2961 - Multilink frame and abort errors on 4 data T1 bundle

[devonmackay]

We have 4 data T1s providing our office with 6Mbs of internet bandwidth.

I have been trying to track down the reason(s) for the steadily increasing frame and abort errors on the Multilink interface of our new router.

We have a new 2961 with 2 2 port T1 interface cards (VWIC2-2MFT-T1/E1).

At first it looked like the T1s were completely clean, but after diving down a bit the last of the 4 T1s does appear to have a decent amount of slip and error seconds.

Is that something that would cause the Multilink interface to show input, frame, and abort errors?

Any config or debug commands I should start with to narrow down what might be causing this problem?

Thanks!

Config snippets:

card type t1 0 0
card type t1 0 1
!
controller T1 0/0/0
clock source internal
cablelength short 440
channel-group 0 timeslots 1-24
description HCFD-XXXXXX
!
controller T1 0/0/1
clock source internal
cablelength short 440
channel-group 0 timeslots 1-24
description HCFD-XXXXXX
!
controller T1 0/1/0
clock source internal
cablelength short 440
channel-group 0 timeslots 1-24
description HCFD-XXXXXX
!
controller T1 0/1/1
clock source internal
cablelength short 440
channel-group 0 timeslots 1-24
description HCFD-XXXXXX
!
interface Multilink1
ip address X.X.X.X 255.255.255.252
ip nat outside
ip virtual-reassembly
ppp multilink
ppp multilink group 1
ppp multilink fragment disable
!
interface Serial0/0/0:0
description T1 : HCFD-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/0/1:0
description T1 : HCFD-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/0:0
description T1 : HCFD-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/1:0
description T1 : HCFD-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1

Output of various show commands:

hq1#sh int serial 0/0/0:0 | include input

  Last input 00:00:00, output 00:00:00, output hang never

  5 minute input rate 225000 bits/sec, 239 packets/sec

     1171775 packets input, 146136108 bytes, 0 no buffer

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

hq1#sh int serial 0/0/0:0 | include input

  Last input 00:00:00, output 00:00:00, output hang never

  5 minute input rate 225000 bits/sec, 239 packets/sec

     1171776 packets input, 146136168 bytes, 0 no buffer

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

hq1#sh int serial 0/0/0:0 | include input

  Last input 00:00:00, output 00:00:00, output hang never

  5 minute input rate 225000 bits/sec, 239 packets/sec

     1171777 packets input, 146136228 bytes, 0 no buffer

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

hq1#sh int serial 0/0/0:0 | include input

  Last input 00:00:00, output 00:00:00, output hang never

  5 minute input rate 225000 bits/sec, 239 packets/sec

     1171778 packets input, 146136288 bytes, 0 no buffer

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort

hq1#sh int multilink 1 | include input

  Last input 00:00:00, output never, output hang never

  5 minute input rate 862000 bits/sec, 258 packets/sec

     1241153 packets input, 546518598 bytes, 0 no buffer

     2220 input errors, 0 CRC, 1678 frame, 0 overrun, 0 ignored, 542 abort

hq1#sh controller t1 0/0/0

T1 0/0/0 is up.

  Applique type is Channelized T1

  Cablelength is short 440

  Description: Bandwidth.com : HCFD-577813

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info Firmware: 20100222, FPGA: 13, spm_count = 0

  Framing is ESF, Line Code is B8ZS, Clock Source is Line.

  CRC Threshold is 320. Reported from firmware  is 320.

  Data in current interval (257 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 1:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 2:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 3:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 4:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 5:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 5 15 minute intervals):

     0 Line Code Violations, 0 Path Code Violations,

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

hq1#sh controller t1 0/0/1

T1 0/0/1 is up.

  Applique type is Channelized T1

  Cablelength is short 440

  Description: Bandwidth.com : HCFD-577814

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info Firmware: 20100222, FPGA: 13, spm_count = 0

  Framing is ESF, Line Code is B8ZS, Clock Source is Line.

  CRC Threshold is 320. Reported from firmware  is 320.

  Data in current interval (276 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     1 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     1 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 1:

     0 Line Code Violations, 0 Path Code Violations

     3 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     3 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 2:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 3:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 4:

     0 Line Code Violations, 0 Path Code Violations

     1 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     1 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 5:

     0 Line Code Violations, 0 Path Code Violations

     1 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     1 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 5 15 minute intervals):

     0 Line Code Violations, 0 Path Code Violations,

     5 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

     5 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

hq1#sh controller t1 0/1/0

T1 0/1/0 is up.

  Applique type is Channelized T1

  Cablelength is short 440

  Description: Bandwidth.com : HCFD-577815

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info Firmware: 20100222, FPGA: 13, spm_count = 0

  Framing is ESF, Line Code is B8ZS, Clock Source is Line.

  CRC Threshold is 320. Reported from firmware  is 320.

  Data in current interval (285 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 1:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 2:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 3:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 4:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 5:

     0 Line Code Violations, 0 Path Code Violations

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 5 15 minute intervals):

     0 Line Code Violations, 0 Path Code Violations,

     0 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

     0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

hq1#sh controller t1 0/1/1

T1 0/1/1 is up.

  Applique type is Channelized T1

  Cablelength is short 440

  Description: Bandwidth.com : HCFD-577816

  No alarms detected.

  alarm-trigger is not set

  Soaking time: 3, Clearance time: 10

  AIS State:Clear  LOS State:Clear  LOF State:Clear

  Version info Firmware: 20100222, FPGA: 13, spm_count = 0

  Framing is ESF, Line Code is B8ZS, Clock Source is Line.

  CRC Threshold is 320. Reported from firmware  is 320.

  Data in current interval (286 seconds elapsed):

     0 Line Code Violations, 0 Path Code Violations

     116 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     116 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 1:

     0 Line Code Violations, 0 Path Code Violations

     366 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     366 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 2:

     0 Line Code Violations, 852 Path Code Violations

     368 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 4 Degraded Mins

     399 Errored Secs, 46 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 3:

     0 Line Code Violations, 5 Path Code Violations

     368 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 2 Degraded Mins

     368 Errored Secs, 1 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 4:

     0 Line Code Violations, 0 Path Code Violations

     370 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     370 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Data in Interval 5:

     0 Line Code Violations, 0 Path Code Violations

     370 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

     370 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

  Total Data (last 5 15 minute intervals):

     0 Line Code Violations, 857 Path Code Violations,

     1842 Slip Secs, 0 Fr Loss Secs, 0 Line Err Secs, 6 Degraded Mins,

     1873 Errored Secs, 47 Bursty Err Secs, 0 Severely Err Secs, 0 Unavail Secs

hq1# show version

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.0(1)M4, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Thu 28-Oct-10 18:32 by prod_rel_team

ROM: System Bootstrap, Version 15.0(1r)M9, RELEASE SOFTWARE (fc1)

hq1 uptime is 2 days, 5 hours, 56 minutes

System returned to ROM by power-on

System image file is "flash0:c2900-universalk9-mz.SPA.150-1.M4.bin"

Last reload type: Normal Reload

<SNIP>

Cisco CISCO2921/K9 (revision 1.0) with 483328K/40960K bytes of memory.

Processor board ID FTX1447AJ6Y

3 Gigabit Ethernet interfaces

4 Serial interfaces

4 Channelized T1/PRI ports

DRAM configuration is 64 bits wide with parity enabled.

255K bytes of non-volatile configuration memory.

254464K bytes of ATA System CompactFlash 0 (Read/Write)

[vmiller]

Your symptoms could be a failure on the telco side, or timing.

Is there a reason your clock source is set to internal /

[devonmackay]

Nuts... I posted the wrong snippet, sorry about that.

That is what I get for lazily copying out of gVim instead of running show run again.

The other SHOW commands were current though, after clearing counters and letting it run a bit.

Here is a FRESH config snippet:

card type t1 0 0
card type t1 0 1
!
!
controller T1 0/0/0
cablelength short 440
channel-group 0 timeslots 1-24
description Bandwidth.com : XXX-XXXXXX
!
controller T1 0/0/1
cablelength short 440
channel-group 0 timeslots 1-24
description Bandwidth.com : XXX-XXXXXX
!
controller T1 0/1/0
cablelength short 440
channel-group 0 timeslots 1-24
description Bandwidth.com : XXX-XXXXXX
!
controller T1 0/1/1
cablelength short 440
channel-group 0 timeslots 1-24
description Bandwidth.com : XXX-XXXXXX
!
!
interface Multilink1
ip address X.X.X.X 255.255.255.252
ip nat outside
ip virtual-reassembly
ppp multilink
ppp multilink group 1
ppp multilink fragment disable
!
interface Serial0/0/0:0
description Bandwidth.com T1 : XXX-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/0/1:0
description Bandwidth.com T1 : XXX-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/0:0
description Bandwidth.com T1 : XXX-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1
!
interface Serial0/1/1:0
description Bandwidth.com T1 : XXX-XXXXXX
no ip address
encapsulation ppp
ppp multilink
ppp multilink group 1