Cisco 2921 - Interface Overrun/drop/flushes

[Anuar Shahrin]

Hi All,

I have a problem with overrun on the interface cisco 2921. 2 router running HSRP with both have the same symptom.

Need a second opinion of the issue based on the output i captured. I suspecting the router could not handle the traffic coming in from the switch.

+Software Version 15.1(4)M4

+config - multiple sub-if (up to 15)

+Multicast enabled (pim spares mode)

+HSRP

+OSPF

Other Symptom

+ high CPU history

+intermittent packet.

!!!---- show interface ---before clear----

ROUTER#
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
  Description: Connect to Switch
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:12:23
  Input queue: 15/75/36921/2439824 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13770000 bits/sec, 8073 packets/sec
  5 minute output rate 144000 bits/sec, 106 packets/sec
     6031245 packets input, 1280034830 bytes, 0 no buffer
     Received 2083128 broadcasts (1515152 IP multicasts)
     0 runts, 0 giants, 717 throttles
     0 input errors, 0 CRC, 0 frame, 1347844 overrun, 0 ignored
     0 watchdog, 2071721 multicast, 0 pause input
     80058 packets output, 13604287 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     396 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
  Description: Connect to Switch
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:12:38
  Input queue: 76/75/37580/2503252 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13781000 bits/sec, 8064 packets/sec
  5 minute output rate 147000 bits/sec, 109 packets/sec
     6152652 packets input, 1307001153 bytes, 0 no buffer
     Received 2126220 broadcasts (1548513 IP multicasts)
     0 runts, 0 giants, 733 throttles
     0 input errors, 0 CRC, 0 frame, 1380864 overrun, 0 ignored
     0 watchdog, 2113497 multicast, 0 pause input
     81762 packets output, 13897389 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     404 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER#
ROUTER#clear counters gigabitEthernet 0/2
Clear "show interface" counters on this interface [confirm]
ROUTER#

!!!!!---- After clear -----

ROUTER#sh int gi0/2                      
GigabitEthernet0/2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
  Description: Connect to Switch
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:00:27
  Input queue: 2/75/1222/70748 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13764000 bits/sec, 8049 packets/sec
  5 minute output rate 148000 bits/sec, 109 packets/sec
     218222 packets input, 46527193 bytes, 0 no buffer
     Received 75068 broadcasts (57538 IP multicasts)
     0 runts, 0 giants, 25 throttles
     0 input errors, 0 CRC, 0 frame, 45800 overrun, 0 ignored
     0 watchdog, 73419 multicast, 0 pause input
     3018 packets output, 510797 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     14 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
  Description: Connect to Switch
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:00:30
  Input queue: 0/75/1270/77237 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13746000 bits/sec, 8042 packets/sec
  5 minute output rate 147000 bits/sec, 108 packets/sec
     238197 packets input, 50489917 bytes, 0 no buffer
     Received 82628 broadcasts (62305 IP multicasts)
     0 runts, 0 giants, 28 throttles
     0 input errors, 0 CRC, 0 frame, 50655 overrun, 0 ignored
     0 watchdog, 81876 multicast, 0 pause input
     3297 packets output, 556242 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     16 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
  Description: Connect to Switch
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 3/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:00:31
  Input queue: 3/75/1292/79833 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 13746000 bits/sec, 8042 packets/sec
  5 minute output rate 147000 bits/sec, 108 packets/sec
     245531 packets input, 52083889 bytes, 0 no buffer
     Received 85228 broadcasts (64156 IP multicasts)
     0 runts, 0 giants, 29 throttles
     0 input errors, 0 CRC, 0 frame, 52401 overrun, 0 ignored
     0 watchdog, 84209 multicast, 0 pause input
     3412 packets output, 575631 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     16 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ROUTER#

!!!! ------ CPU Process

ROUTER#show proc cpu sort | ex 0.00%
CPU utilization for five seconds: 27%/19%; one minute: 27%; five minutes: 27%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 128  1109823624   202752864       5473  5.75%  5.50%  5.51%   0 IP Input         
 281    31807992  1102871632         28  0.55%  0.55%  0.55%   0 OSPF-1 Hello     
 270    10231128  3726474754          0  0.47%  0.47%  0.47%   0 IP SLAs Responde
 262    41307784  1578028613         26  0.23%  0.23%  0.23%   0 HSRP IPv4        
 257    22417192   619911584         36  0.23%  0.29%  0.30%   0 PIM Process      
 224         816         677       1205  0.23%  0.07%  0.01% 389 Virtual Exec     
 261     3567176   952910715          3  0.15%  0.13%  0.15%   0 HSRP Common      
  99     2862380  1933783714          1  0.15%  0.14%  0.15%   0 Ethernet Msec Ti
  83     3335860    61482597         54  0.15%  0.15%  0.15%   0 Netclock Backgro
 264     2415156   760694228          3  0.07%  0.09%  0.08%   0 MFIB_mrib_write  
 265     1344932   102613045         13  0.07%  0.04%  0.05%   0 MFIB_mrib_read   
 271    30731956   117528380        261  0.07%  0.11%  0.11%   0 IP SNMP          
 258     1889812   153885093         12  0.07%  0.04%  0.05%   0 Mwheel Process   
  31    17904760   203806740         87  0.07%  0.11%  0.10%   0 ARP Input        

!!!! ----- interface switching

ROUTER#sh int gi0/2 switching
GigabitEthernet0/2 Connect to Switch
          Throttle count    7541011
                   Drops         RP  264575116         SP          0
             SPD Flushes       Fast 2466411544        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs 2627347225      Drops          0

    Protocol  IP                  
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process 1246302482  605097162  541603367 3599316687
            Cache misses          0          -          -          -
                    Fast 3087414019 1557498509  354125801 3079658973
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP             
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      25556    1967812
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Spanning Tree       
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          4        942          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process  230990873  975994062    1733446  110940544
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     253013  117398032     284153  107977512
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other               
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    6059021  360207174    1534274   92056440
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
ROUTER#
ROUTER#sh int gi0/2 switching
GigabitEthernet0/2 Connect to Switch
          Throttle count    7541026
                   Drops         RP  264575988         SP          0
             SPD Flushes       Fast 2466466609        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs 2627351535      Drops          0

    Protocol  IP                  
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process 1246371949  613675553  541604258 3599393405
            Cache misses          0          -          -          -
                    Fast 3087532367 1585313305  354127543 3080042278
               Auton/SSE          0          0          0          0

    Protocol  DEC MOP             
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0      25556    1967812
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Spanning Tree       
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          4        942          0          0
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  ARP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process  230991204  976014014    1733462  110941568
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  CDP                 
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process     253014  117398496     284153  107977512
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other               
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    6059035  360208007    1534277   92056620
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    NOTE: all counts are cumulative and reset only after a reload.
ROUTER#

ROUTER#sh ip traffic           
IP statistics:
  Rcvd:  196907298 total, 2554872193 local destination
         3 format errors, 0 checksum errors, 2034224601 bad hop count
         0 unknown protocol, 2 not a gateway
         0 security failures, 0 bad options, 44851110 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 44851110 alert, 0 cipso, 0 ump
         0 other
  Frags: 2184 reassembled, 38 timeouts, 0 couldn't reassemble
         71967695 fragmented, 143953139 fragments, 437567 couldn't fragment
  Bcast: 75860742 received, 10281 sent
  Mcast: 2292797203 received, 697462590 sent
  Sent:  909586917 generated, 2613238730 forwarded
  Drop:  34179 encapsulation failed, 0 unresolved, 0 no adjacency
         25 no route, 0 unicast RPF, 0 forced drop
         0 options denied
  Drop:  0 packets with source IP address zero
  Drop:  0 packets with internal loop back IP address
         20930 physical broadcast
  Reinj: 0 in input feature path, 0 in output feature path

[Sunil Bhadauria]

Hello Anuar ,

port overruns mean that your router is receiving data faster than it can handle. You either need a faster router than a 2900 series or else larger input buffers. 

Options to recover this situation are : 

1) Try increasing input buffer , whether this will help or not depends on amount of burst being received from other side .

2) Apply QoS shaping on other side interface , that will help avoid any burst spikes and yet will certainly help in reducing / avoiding these errors .

Hope this helps 

Sunil Bhadauria

! Kindly do rate and accordingly mark correct answers to help forum !

[LukaszTJB]

Hello,

There might be a MTU mismatch in your Network and your Router is busy dealing with the IP fragments.

The Frags statistic doesn't look good to me.

Kind regards

Lukasz

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I recall what especially can lead to input queuing is process switching, which your stats reveal in a couple of stats, such as the 5+% of the IP Input process, and your interface switching stats.

Also know, although a 2921 has gig interfaces, it's far, far from having gig forwarding capacity.

What you might first try is running your Ethernet interfaces at 100 Mbps.  This to slow how fast traffic can slam into your router.  You can also try increasing your interface input queues (as also noted by Sunil).  (You can even try maxing this parameter.)

Then you can investigate the cause of your IP Input process percentage.  You would also investigate whether dealing with packet fragmentation (as noted by Lukasz), which will cause much higher CPU processing.