Overruns Detected on C2900 GigabitEthernet0/2 Interface

Robert Hefner · ‎01-30-2017

Hey folks,

I have been experiencing some issue with bandwidth through our C2900 ISR. I have a 500 Mbps circuit that is symmetrical. I am getting 392.06DL/286.96UL. I have been working with our service provider and we have confirmed that their equipment is set properly and is achieving the prescribed speeds. I have cleared the counters in interface GigabitEthernet0/2 and then performed a speedtest from speedtest.net. The overruns in the "sh int" below are from the speedtest.net as it takes some time for the C2900 to show overruns under normal network traffic loads. The interface on the C2900 is negotiating at 1 Gbps but it appears to have a hard time with 500 Mbps.

The RAM and CPU on the C2900 are operating well within their limits. CPU: 30% RAM: 18%

Can anyone shed any light on what might be causing this bottleneck on the C2900? I am starting to wonder if the QoS policy could be causing issues with throughput.

Below the interface stats I have included the config of the C2900.

Thanks in advance for any input.

Robert

GigabitEthernet0/2 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is 588d.098b.8f4a (bia 588d.098b.8f4a)
Description: *** Uplink to 0Switch GigE1/0/46 ***
Internet address is 10.0.0.1/16
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 255/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:07:50
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
5 minute input rate 8141000 bits/sec, 1313 packets/sec
5 minute output rate 8155000 bits/sec, 1362 packets/sec
1066759 packets input, 858035342 bytes, 0 no buffer
Received 138 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 68 overrun, 0 ignored
0 watchdog, 23 multicast, 0 pause input
1154883 packets output, 930232868 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
16 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Building configuration...

Current configuration : 6616 bytes
!
! No configuration change since last restart
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname 0RouterMid
!
boot-start-marker
boot-end-marker
!
!
logging buffered 51200 warnings
!
no aaa new-model
!
!
no ipv6 cef
ip auth-proxy max-login-attempts 5
ip admission max-login-attempts 5
!
!
!
!
!
ip domain name thepartnership.local
ip cef
!
multilink bundle-name authenticated
!
!
crypto pki token default removal timeout 0
!
crypto pki trustpoint TP-self-signed-587546248
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-587546248
revocation-check none
rsakeypair TP-self-signed-587546248
!
!

license udi pid CISCO2911/K9 sn FTX1449A1WR
license boot module c2900 technology-package securityk9
!

!
redundancy
!
!
!
!
!
!
class-map match-all VoIP
match access-group name VoIP
!
!
policy-map QoS
class VoIP
priority percent 33
class class-default
fair-queue
!
!
!
!
!
!
!
!
!
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
description *** Uplink to 0RouterEdge GigE0/1 ***
ip address 172.16.253.1 255.255.255.252
no ip proxy-arp
ip flow ingress
ip virtual-reassembly in
duplex full
speed 1000
!
interface GigabitEthernet0/1
description *** Uplink to 0Switch GigE1/0/46 VLAN 500 ***
ip address 172.16.1.254 255.255.255.0
ip access-group MAN-in in
no ip proxy-arp
ip flow ingress
ip virtual-reassembly in
duplex full
speed 1000
service-policy output QoS
!
interface GigabitEthernet0/2
description *** Uplink to 0Switch GigE1/0/46 ***
ip address 10.0.0.1 255.255.0.0
no ip proxy-arp
ip flow ingress
ip virtual-reassembly in
duplex full
speed 1000
service-policy output QoS
!
ip forward-protocol nd
!
ip http server
ip http authentication local
ip http secure-server
ip http timeout-policy idle 60 life 86400 requests 10000
ip flow-export destination 10.0.1.2 2055
!
ip route 0.0.0.0 0.0.0.0 172.16.253.2
ip route 10.1.0.0 255.255.0.0 172.16.1.1
ip route 10.2.0.0 255.255.0.0 172.16.1.2
ip route 10.3.0.0 255.255.0.0 172.16.1.3
ip route 10.4.0.0 255.255.0.0 172.16.1.4
ip route 10.5.0.0 255.255.0.0 172.16.1.5
ip route 10.6.0.0 255.255.0.0 172.16.1.6
ip route 10.7.0.0 255.255.0.0 172.16.1.2
ip route 10.252.2.0 255.255.255.0 172.16.1.2
ip route 10.253.2.0 255.255.255.0 172.16.1.2
ip route 10.253.7.0 255.255.255.0 172.16.1.2
ip route 10.254.1.0 255.255.255.0 172.16.1.1
ip route 10.254.2.0 255.255.255.0 172.16.1.2
ip route 10.254.3.0 255.255.255.0 172.16.1.3
ip route 10.254.4.0 255.255.255.0 172.16.1.4
ip route 10.254.5.0 255.255.255.0 172.16.1.5
ip route 10.254.6.0 255.255.255.0 172.16.1.6
ip route 172.16.254.0 255.255.255.0 172.16.253.2
ip route 192.168.253.0 255.255.255.0 172.16.1.2
!
ip access-list extended MAN-in
permit ip 10.0.0.0 0.0.255.255 any
permit ip any 10.0.0.0 0.0.255.255
permit ip 10.1.0.0 0.0.255.255 any
permit ip any 10.1.0.0 0.0.255.255
permit ip 10.2.0.0 0.0.255.255 any
permit ip any 10.2.0.0 0.0.255.255
permit ip 10.3.0.0 0.0.255.255 any
permit ip any 10.3.0.0 0.0.255.255
permit ip 10.4.0.0 0.0.255.255 any
permit ip any 10.4.0.0 0.0.255.255
permit ip 10.5.0.0 0.0.255.255 any
permit ip any 10.5.0.0 0.0.255.255
permit ip 10.6.0.0 0.0.255.255 any
permit ip any 10.6.0.0 0.0.255.255
permit ip 10.254.1.0 0.0.0.255 any
permit ip any 10.254.1.0 0.0.0.255
permit ip 10.254.2.0 0.0.0.255 any
permit ip any 10.254.2.0 0.0.0.255
permit ip 10.254.3.0 0.0.0.255 any
permit ip any 10.254.3.0 0.0.0.255
permit ip 10.254.4.0 0.0.0.255 any
permit ip any 10.254.4.0 0.0.0.255
permit ip 10.254.5.0 0.0.0.255 any
permit ip any 10.254.5.0 0.0.0.255
permit ip 10.254.6.0 0.0.0.255 any
permit ip any 10.254.6.0 0.0.0.255
permit ip 10.253.2.0 0.0.0.255 any
permit ip any 10.253.2.0 0.0.0.255
permit ip 192.168.253.0 0.0.0.255 any
permit ip any 192.168.253.0 0.0.0.255
permit ip 10.252.2.0 0.0.0.255 any
permit ip any 10.252.2.0 0.0.0.255
permit ip 10.7.0.0 0.0.255.255 any
permit ip any 10.7.0.0 0.0.255.255
permit ip 10.254.7.0 0.0.0.255 any
permit ip any 10.254.7.0 0.0.0.255
permit ip 10.253.7.0 0.0.0.255 any
permit ip any 10.253.7.0 0.0.0.255
deny ip 10.253.2.0 0.0.0.255 10.0.0.0 0.0.255.255
deny ip 10.0.0.0 0.0.255.255 10.253.2.0 0.0.0.255
deny ip 10.253.2.0 0.0.0.255 172.16.254.0 0.0.0.255
permit ip 172.16.1.0 0.0.0.255 any
permit ip any 172.16.1.0 0.0.0.255
ip access-list extended VoIP
permit ip host 10.0.1.11 any
permit ip any host 10.0.1.11
permit ip host 10.0.1.13 any
permit ip any host 10.0.1.13
permit ip host 10.0.1.19 any
permit ip any host 10.0.1.19
!
!
!
snmp-server community *** RO
snmp-server enable traps entity-sensor threshold
!
control-plane
!
!
!
line con 0
login local
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
privilege level 15
login local
transport input telnet ssh
!
scheduler allocate 20000 1000
!
end

Reza Sharifi · ‎01-30-2017

Hi,

Do you know if the the overruns accumulate only when the circuit is being highly utilized or it is all the time regardless of much bandwidth is being used? If it is doing this when the circuit is being utilized at 300Mb, 400Mb or more than maybe the router can't handle the amount of traffic. I am not sure what model 2900 you have, but I am including a link here for you to look at the performance of the 2900 series routers.

https://www.cisco.com/web/partners/downloads/765/tools/quickreference/routerperformance.pdf

Also, to eliminate the QOS, you may want to remove it from the interface and see if it makes any difference.

HTH

Robert Hefner · ‎02-01-2017

Thank for the response Reza. the overruns accumulate under normal load as well. When I clear counters I am able to see as substantial increase when pushing the circuit. Under normal network load the counters increment much slower.

The model is a 2911. I am a bit confused as the interfaces are Gigabit interfaces so I am not sure why they are having a hard time handling the 500 Mbps stream.

I saw this thread posted a few years back on the overrun issues and they stated its a limitation of the hardware. They stated in the thread, "However, in the majority of cases, it indicates that the receiving capability of the interface was exceeded." As you can see we are not exceeding the speed of the interface. So I am still at a loss on this one.

Thread:

https://supportforums.cisco.com/document/13796/overruns-counter-show-interfaces-command-output-increasing

As for the QoS I tried removing this and this had no effect.

Anyone have any thoughts or ideas on this?

Thanks - Robert

Joseph W. Doherty · ‎02-01-2017

An interface can handle a single frame w/o a problem. (Or a least it should.)

What your referenced link was trying to explain, when there's a burst of frames, at line rate, the interface only has so much buffer space to save them. The device needs to drain the interface receive buffer. If it cannot keep up with interface (and many small software based routers cannot), the interface will run out of space to save the frame. I.e. the interface has been overrun.

No 2900 is really suitable for 500 Mbps; also no 3900 is either, if you want to avoid bumping into device performance limitations. For 500 Mbps, you need a 4K ISR. However, as except when speed testing, you overrun rate is very low, you might accept that. Or, you might try increasing the rx-ring limit (if possible) and/or making your configuration as simple/clean as possible.

For the latter, even small changes might help. For example:

ip access-list extended MAN-in
permit ip 10.0.0.0 0.0.255.255 any
permit ip any 10.0.0.0 0.0.255.255
permit ip 10.1.0.0 0.0.255.255 any
permit ip any 10.1.0.0 0.0.255.255
permit ip 10.2.0.0 0.0.255.255 any
permit ip any 10.2.0.0 0.0.255.255
permit ip 10.3.0.0 0.0.255.255 any
permit ip any 10.3.0.0 0.0.255.255
permit ip 10.4.0.0 0.0.255.255 any
permit ip any 10.4.0.0 0.0.255.255
permit ip 10.5.0.0 0.0.255.255 any
permit ip any 10.5.0.0 0.0.255.255
permit ip 10.6.0.0 0.0.255.255 any
permit ip any 10.6.0.0 0.0.255.255
.
.

could become

ip access-list extended MAN-in
permit ip 10.0.0.0 0.3.255.255 any
permit ip any 10.0.0.0 0.3.255.255
permit ip 10.4.0.0 0.1.255.255 any
permit ip any 10.4.0.0 0.1.255.255
permit ip 10.6.0.0 0.0.255.255 any
permit ip any 10.6.0.0 0.0.255.255
.
.

Also reordering ACEs in hit frequency can help a bit too.

Robert Hefner · ‎02-06-2017

Joseph,

I have a Catalyst 3750 that can handle L3 you think the Gig interfaces on this device can handle the throughput or do you think I would likely run into the same overrun issues as I am seeing on the ISR?

Robert

Joseph W. Doherty · ‎02-06-2017

An original series should be able to handle ingress gig w/o issues, as it has ASICs. However, 3750s have their own performance bottlenecks. Their fabric can be one, although unlikely you would bump into that, they, depending on model, cannot run all ports concurrently at wire-rate, also unlikely you'll bump into that, they are also infamous for drop packets on egress, that you may bump into.