We have a 3 T1s that are bonded going into a 3925 with a VWIC3-4MFT that after a BGP flap 2 days ago the circuits have been taking errors very slowly. The errors are only hitting groups, Runts, Input errors, and CRC. The number for each are very close so it seems that one error is possibly incrementing each one most of the time.
Serial0/0/0:1 is up, line protocol is up
Hardware is DSX1
Description: T1 #1
MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 08:01:26
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 7000 bits/sec, 7 packets/sec
5 minute output rate 17000 bits/sec, 13 packets/sec
538137 packets input, 298406346 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
52 runts, 0 giants, 0 throttles
66 input errors, 66 CRC, 0 frame, 0 overrun, 0 ignored, 2 abort
813272 packets output, 124510162 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Serial0/0/1:1 is up, line protocol is up
Hardware is DSX1
Description: T1 #2
MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 08:01:26
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6000 bits/sec, 7 packets/sec
5 minute output rate 19000 bits/sec, 13 packets/sec
537649 packets input, 298666906 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
54 runts, 0 giants, 0 throttles
66 input errors, 66 CRC, 0 frame, 0 overrun, 0 ignored, 1 abort
812353 packets output, 123977448 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Serial0/0/2:1 is up, line protocol is up
Hardware is DSX1
Description: T1 #3
MTU 1500 bytes, BW 1536 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 2/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, crc 16, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 08:01:36
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 6000 bits/sec, 7 packets/sec
5 minute output rate 16000 bits/sec, 13 packets/sec
538272 packets input, 298721911 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
63 runts, 0 giants, 0 throttles
78 input errors, 78 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
812051 packets output, 123939850 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions
Multilink1 is up, line protocol is up
Hardware is multilink group interface
Description: Circuit ID -###
Internet address is #####
MTU 1500 bytes, BW 4608 Kbit/sec, DLY 20000 usec,
reliability 255/255, txload 3/255, rxload 1/255
Encapsulation PPP, LCP Open, multilink Open
Open: IPCP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 2 seconds on reset
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 08:04:21
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/31236 (size/max total/drops)
5 minute input rate 31000 bits/sec, 22 packets/sec
5 minute output rate 60000 bits/sec, 32 packets/sec
1601259 packets input, 883989751 bytes, 0 no buffer
Received 0 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
189 input errors, 0 CRC, 189 frame, 0 overrun, 0 ignored, 0 abort
1742927 packets output, 373824823 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 output buffer failures, 0 output buffers swapped out
0 carrier transition
Router392501#sho network-clocks
Network Clock Configuration
---------------------------
Priority Clock Source Clock State Clock Type
1 T1 0/1/0 GOOD T1
2 T1 0/2/0 GOOD T1
10 Backplane GOOD PLL
Current Primary Clock Source
---------------------------
Priority Clock Source Clock State Clock Type
1 T1 0/1/0 GOOD T1
network-clock-participate wic 0
network-clock-participate wic 1
network-clock-participate wic 2
network-clock-select 1 T1 0/1/0
network-clock-select 2 T1 0/2/0
!
Controllers T1 0/1/0 and 0/2/0 are Voice PRI T1s going into VWIC3-2MFT
The location doesn't really see any issues, but it is something I am trying to troubleshoot. Problem is the site runs pretty much 24/7 and they are in their busy time so trying things is kind of out of the question. The circuit provider and the local end carrier say they have tested all 3 clean and that it is our equipment. 95% of the time it is a carrier issue, but it is like pulling teeth to get full testing done. Upon researching I found that this may be a clocking issue, but not 100% sure how to test this. The T1s ran fine since middle of May and this is the first issue. The site has 3 Bonded T1s going into a 2nd router as well that are running fine. The only difference is Router 2 does not have any Voice PRIs and only the 3 bonded T1s
