10-02-2014 08:17 PM - edited 03-04-2019 11:53 PM
Hi All,
I have a problem with overrun on the interface cisco 2921. 2 router running HSRP with both have the same symptom.
Need a second opinion of the issue based on the output i captured. I suspecting the router could not handle the traffic coming in from the switch.
+Software Version 15.1(4)M4
+config - multiple sub-if (up to 15)
+Multicast enabled (pim spares mode)
+HSRP
+OSPF
Other Symptom
+ high CPU history
+intermittent packet.
!!!---- show interface ---before clear----
ROUTER#
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
Description: Connect to Switch
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:12:23
Input queue: 15/75/36921/2439824 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 13770000 bits/sec, 8073 packets/sec
5 minute output rate 144000 bits/sec, 106 packets/sec
6031245 packets input, 1280034830 bytes, 0 no buffer
Received 2083128 broadcasts (1515152 IP multicasts)
0 runts, 0 giants, 717 throttles
0 input errors, 0 CRC, 0 frame, 1347844 overrun, 0 ignored
0 watchdog, 2071721 multicast, 0 pause input
80058 packets output, 13604287 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
396 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
Description: Connect to Switch
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:12:38
Input queue: 76/75/37580/2503252 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 13781000 bits/sec, 8064 packets/sec
5 minute output rate 147000 bits/sec, 109 packets/sec
6152652 packets input, 1307001153 bytes, 0 no buffer
Received 2126220 broadcasts (1548513 IP multicasts)
0 runts, 0 giants, 733 throttles
0 input errors, 0 CRC, 0 frame, 1380864 overrun, 0 ignored
0 watchdog, 2113497 multicast, 0 pause input
81762 packets output, 13897389 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
404 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER#
ROUTER#clear counters gigabitEthernet 0/2
Clear "show interface" counters on this interface [confirm]
ROUTER#
!!!!!---- After clear -----
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
Description: Connect to Switch
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:00:27
Input queue: 2/75/1222/70748 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 13764000 bits/sec, 8049 packets/sec
5 minute output rate 148000 bits/sec, 109 packets/sec
218222 packets input, 46527193 bytes, 0 no buffer
Received 75068 broadcasts (57538 IP multicasts)
0 runts, 0 giants, 25 throttles
0 input errors, 0 CRC, 0 frame, 45800 overrun, 0 ignored
0 watchdog, 73419 multicast, 0 pause input
3018 packets output, 510797 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
14 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
Description: Connect to Switch
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:00:30
Input queue: 0/75/1270/77237 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 13746000 bits/sec, 8042 packets/sec
5 minute output rate 147000 bits/sec, 108 packets/sec
238197 packets input, 50489917 bytes, 0 no buffer
Received 82628 broadcasts (62305 IP multicasts)
0 runts, 0 giants, 28 throttles
0 input errors, 0 CRC, 0 frame, 50655 overrun, 0 ignored
0 watchdog, 81876 multicast, 0 pause input
3297 packets output, 556242 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
16 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER#sh int gi0/2
GigabitEthernet0/2 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is f872.ea19.c5b2 (bia f872.ea19.c5b2)
Description: Connect to Switch
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 3/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:00:31
Input queue: 3/75/1292/79833 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 13746000 bits/sec, 8042 packets/sec
5 minute output rate 147000 bits/sec, 108 packets/sec
245531 packets input, 52083889 bytes, 0 no buffer
Received 85228 broadcasts (64156 IP multicasts)
0 runts, 0 giants, 29 throttles
0 input errors, 0 CRC, 0 frame, 52401 overrun, 0 ignored
0 watchdog, 84209 multicast, 0 pause input
3412 packets output, 575631 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
16 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER#
!!!! ------ CPU Process
ROUTER#show proc cpu sort | ex 0.00%
CPU utilization for five seconds: 27%/19%; one minute: 27%; five minutes: 27%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
128 1109823624 202752864 5473 5.75% 5.50% 5.51% 0 IP Input
281 31807992 1102871632 28 0.55% 0.55% 0.55% 0 OSPF-1 Hello
270 10231128 3726474754 0 0.47% 0.47% 0.47% 0 IP SLAs Responde
262 41307784 1578028613 26 0.23% 0.23% 0.23% 0 HSRP IPv4
257 22417192 619911584 36 0.23% 0.29% 0.30% 0 PIM Process
224 816 677 1205 0.23% 0.07% 0.01% 389 Virtual Exec
261 3567176 952910715 3 0.15% 0.13% 0.15% 0 HSRP Common
99 2862380 1933783714 1 0.15% 0.14% 0.15% 0 Ethernet Msec Ti
83 3335860 61482597 54 0.15% 0.15% 0.15% 0 Netclock Backgro
264 2415156 760694228 3 0.07% 0.09% 0.08% 0 MFIB_mrib_write
265 1344932 102613045 13 0.07% 0.04% 0.05% 0 MFIB_mrib_read
271 30731956 117528380 261 0.07% 0.11% 0.11% 0 IP SNMP
258 1889812 153885093 12 0.07% 0.04% 0.05% 0 Mwheel Process
31 17904760 203806740 87 0.07% 0.11% 0.10% 0 ARP Input
!!!! ----- interface switching
ROUTER#sh int gi0/2 switching
GigabitEthernet0/2 Connect to Switch
Throttle count 7541011
Drops RP 264575116 SP 0
SPD Flushes Fast 2466411544 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 2627347225 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 1246302482 605097162 541603367 3599316687
Cache misses 0 - - -
Fast 3087414019 1557498509 354125801 3079658973
Auton/SSE 0 0 0 0
Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 25556 1967812
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 4 942 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 230990873 975994062 1733446 110940544
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 253013 117398032 284153 107977512
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 6059021 360207174 1534274 92056440
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
ROUTER#
ROUTER#sh int gi0/2 switching
GigabitEthernet0/2 Connect to Switch
Throttle count 7541026
Drops RP 264575988 SP 0
SPD Flushes Fast 2466466609 SSE 0
SPD Aggress Fast 0
SPD Priority Inputs 2627351535 Drops 0
Protocol IP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 1246371949 613675553 541604258 3599393405
Cache misses 0 - - -
Fast 3087532367 1585313305 354127543 3080042278
Auton/SSE 0 0 0 0
Protocol DEC MOP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 0 0 25556 1967812
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Spanning Tree
Switching path Pkts In Chars In Pkts Out Chars Out
Process 4 942 0 0
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol ARP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 230991204 976014014 1733462 110941568
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol CDP
Switching path Pkts In Chars In Pkts Out Chars Out
Process 253014 117398496 284153 107977512
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
Protocol Other
Switching path Pkts In Chars In Pkts Out Chars Out
Process 6059035 360208007 1534277 92056620
Cache misses 0 - - -
Fast 0 0 0 0
Auton/SSE 0 0 0 0
NOTE: all counts are cumulative and reset only after a reload.
ROUTER#
ROUTER#sh ip traffic
IP statistics:
Rcvd: 196907298 total, 2554872193 local destination
3 format errors, 0 checksum errors, 2034224601 bad hop count
0 unknown protocol, 2 not a gateway
0 security failures, 0 bad options, 44851110 with options
Opts: 0 end, 0 nop, 0 basic security, 0 loose source route
0 timestamp, 0 extended security, 0 record route
0 stream ID, 0 strict source route, 44851110 alert, 0 cipso, 0 ump
0 other
Frags: 2184 reassembled, 38 timeouts, 0 couldn't reassemble
71967695 fragmented, 143953139 fragments, 437567 couldn't fragment
Bcast: 75860742 received, 10281 sent
Mcast: 2292797203 received, 697462590 sent
Sent: 909586917 generated, 2613238730 forwarded
Drop: 34179 encapsulation failed, 0 unresolved, 0 no adjacency
25 no route, 0 unicast RPF, 0 forced drop
0 options denied
Drop: 0 packets with source IP address zero
Drop: 0 packets with internal loop back IP address
20930 physical broadcast
Reinj: 0 in input feature path, 0 in output feature path
10-03-2014 02:38 AM
Hello Anuar ,
port overruns mean that your router is receiving data faster than it can handle. You either need a faster router than a 2900 series or else larger input buffers.
Options to recover this situation are :
1) Try increasing input buffer , whether this will help or not depends on amount of burst being received from other side .
2) Apply QoS shaping on other side interface , that will help avoid any burst spikes and yet will certainly help in reducing / avoiding these errors .
Hope this helps
Sunil Bhadauria
! Kindly do rate and accordingly mark correct answers to help forum !
10-11-2014 10:03 AM
Hello,
There might be a MTU mismatch in your Network and your Router is busy dealing with the IP fragments.
The Frags statistic doesn't look good to me.
Kind regards
Lukasz
10-13-2014 05:27 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
I recall what especially can lead to input queuing is process switching, which your stats reveal in a couple of stats, such as the 5+% of the IP Input process, and your interface switching stats.
Also know, although a 2921 has gig interfaces, it's far, far from having gig forwarding capacity.
What you might first try is running your Ethernet interfaces at 100 Mbps. This to slow how fast traffic can slam into your router. You can also try increasing your interface input queues (as also noted by Sunil). (You can even try maxing this parameter.)
Then you can investigate the cause of your IP Input process percentage. You would also investigate whether dealing with packet fragmentation (as noted by Lukasz), which will cause much higher CPU processing.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide