Re: WAN congestion causing interface resets?

Colin Higgins · ‎03-09-2011

I have a router with a multilinked (2) T1 bundle that connects to a MPLS network.

I am using BGP as my routing protocol.

I am having problems in this site and others with carrier transitions, crc errors, input errors, frame errors, and output drops. It is not uncommon for link utilization to go up to 99% during backups, and my provider (AT&T) is claiming that over-utilization is causing the interface to go down (reset).

Here is what the physical interface looks like (note the errors):

Serial0/0/0 is up, line protocol is up
Hardware is GT96K with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 8/255, rxload 15/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 94000 bits/sec, 44 packets/sec
5 minute output rate 50000 bits/sec, 44 packets/sec
     106372481 packets input, 1579483919 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     145 input errors, 145 CRC, 66 frame, 37 overrun, 0 ignored, 102 abort
     141062298 packets output, 2551614815 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up DSR=up DTR=up RTS=up CTS=up

and the multilink is:

Multilink1 is up, line protocol is up
Hardware is multilink group interface
Internet address is XXXX
MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec,
     reliability 255/255, txload 8/255, rxload 18/255
Encapsulation PPP, LCP Open, multilink Open
Listen: CDPCP
Open: IPCP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 2 seconds on reset
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 2w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 40935
Queueing strategy: Class-based queueing
Output queue: 0/1000/64/40935 (size/max total/threshold/drops)
     Conversations 0/77/256 (active/max active/max total)
     Reserved Conversations 2/2 (allocated/max allocated)
     Available Bandwidth 194 kilobits/sec
5 minute input rate 222000 bits/sec, 86 packets/sec
5 minute output rate 99000 bits/sec, 77 packets/sec
     198903171 packets input, 3903303225 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     83 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     267648768 packets output, 2917984847 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transition

I do have WAN QoS enabled. The policies are:

policy-map wanqos
class voice
priority 2000
set dscp ef
class cos2
bandwidth 54
set dscp af31
class stuntraffic
bandwidth 56
set dscp af41
class class-default
set ip dscp default
fair-queue
policy-map qos_outbound
class enhanced_qos
bandwidth 56
set ip dscp af31
class class-default
set ip dscp default

class-map match-all enhanced_qos
match access-group name enhanced_qos_map
class-map match-all cos3
match dscp af21
class-map match-any cos2
match access-group name voice-control
match dscp af31
match precedence 3
class-map match-all stuntraffic
match access-group 100
class-map match-all voice
match access-group name voice

Is it possible for congestion on this interface to actually cause the entire thing to go down (flapping basically)?

rassoul.ghaznavi · ‎03-09-2011

Hi,

In some circumstances, yes it is possible,

What I would do is enable ip nbar protocol-discovery and find out what is happening on the interface and what traffic it that.

Network Group

Kishore Chennupati · ‎03-09-2011

Hi,

When you say resets, are you talking about the physical resets because I dont see any from your output. All the errors you are talking about can be due to physical issues like bad cabling between the NTU and the router and also due to the max-utilisation and your router is not able to keep up.

Did you try shaping?

Here is what the physical interface looks like (note the errors):

Serial0/0/0 is up, line protocol is up
Hardware is GT96K with integrated T1 CSU/DSU
MTU 1500 bytes, BW 1536 Kbit, DLY 20000 usec,
     reliability 255/255, txload 8/255, rxload 15/255
Encapsulation PPP, LCP Open, multilink Open
Link is a member of Multilink bundle Multilink1, loopback not set
Keepalive set (10 sec)
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 2w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 94000 bits/sec, 44 packets/sec
5 minute output rate 50000 bits/sec, 44 packets/sec
     106372481 packets input, 1579483919 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     145 input errors, 145 CRC, 66 frame, 37 overrun, 0 ignored, 102 abort
     141062298 packets output, 2551614815 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transitions
     DCD=up DSR=up DTR=up RTS=up CTS=up

and the multilink is:

Multilink1 is up, line protocol is up
Hardware is multilink group interface
Internet address is XXXX
MTU 1500 bytes, BW 3072 Kbit, DLY 100000 usec,
     reliability 255/255, txload 8/255, rxload 18/255
Encapsulation PPP, LCP Open, multilink Open
Listen: CDPCP
Open: IPCP, loopback not set
Keepalive set (10 sec)
DTR is pulsed for 2 seconds on reset
Last input 00:00:00, output never, output hang never
Last clearing of "show interface" counters 2w6d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 40935    <<< not good
Queueing strategy: Class-based queueing
Output queue: 0/1000/64/40935 (size/max total/threshold/drops)
     Conversations 0/77/256 (active/max active/max total)
     Reserved Conversations 2/2 (allocated/max allocated)
     Available Bandwidth 194 kilobits/sec
5 minute input rate 222000 bits/sec, 86 packets/sec
5 minute output rate 99000 bits/sec, 77 packets/sec
     198903171 packets input, 3903303225 bytes, 0 no buffer
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     83 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     267648768 packets output, 2917984847 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 output buffer failures, 0 output buffers swapped out
     0 carrier transition

HTH,

Regards

Kishore

Please rate if helpful

Colin Higgins · ‎03-10-2011

I cleared the counters after getting the interface resets and carrier transitions: that's why they don't appear here.

But I have been getting them in several locations on this MPLS network.

Congestion occurs because of large data transfers between databases mostly. The QoS policies obviously prioritize voice over the WAN (which is necessary).

I'm not sure how traffic shaping would fit in with the QoS policies I have enabled

meperuma · ‎03-10-2011

Hi interface reset may happen due to missing l2 keepalives also, looking at the output, the physical layer may not be good.

Worth to look at mlppp statistics also.

Sent from Cisco Technical Support iPhone App