03-11-2020 04:08 AM
Hi,
I know this is a recurring topic, but I haven't been able to find a solution yet. No operational impact (as i know of) but I dont like to see overrun errors on a gigabit interface with such low utilization.
This is one of two public interfaces (secondary interface), used for internet access at the office and attached to secondary DMVPN tunnel. Its is incrementing even now when they are all asleep over there.
I've checked netflow, and utilization is quite low overall. Any ideas?
(I am aware of the statement:
In a small number of cases, the overrun counter may be incremented because of a software defect. However, in the majority of cases, it indicates that the receiving capability of the interface was exceeded. Nothing can be done on the router that reports overruns. If possible, the rate that frames are coming should be controlled at the remote end of the connection. Otherwise, if the number of overruns is high, the hardware should be upgraded. )
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M3, RELEASE SOFTWARE (fc2) Cisco CISCO2911/K9 (revision 1.0) with 487424K/36864K bytes of memory.
GigabitEthernet0/2 is up, line protocol is up Hardware is CN Gigabit Ethernet, address is 002c.c8e2.8582 (bia 002c.c8e2.8582) Description: ** Second connection toXXX Office - Public **$ETH-LAN$$FW_OUTSIDE$ Internet address is X.X.X.X/29 MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full Duplex, 1Gbps, media type is RJ45 output flow-control is unsupported, input flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 0/75/0/948271 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: Class-based queueing Output queue: 0/1000/0 (size/max total/drops) 5 minute input rate 1058000 bits/sec, 108 packets/sec 5 minute output rate 125000 bits/sec, 51 packets/sec 924191088 packets input, 2462862979 bytes, 1857 no buffer Received 2130658 broadcasts (0 IP multicasts) 0 runts, 0 giants, 69 throttles 2646058 input errors, 0 CRC, 0 frame, 2646058 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 654499061 packets output, 1717785467 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 1 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
interface GigabitEthernet0/2
description ** Second connection to XXX Office - Public **$ETH-LAN$$FW_OUTSIDE$
ip address X.X.X.X.X 255.255.255.248
ip access-group ACLLine2 in
no ip redirects
no ip proxy-arp
ip flow ingress
ip nat outside
ip inspect CBACFilter out
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled
service-policy output QoS
end
Buffer elements: 711 in free list 84430885 hits, 0 misses, 1242 created Public buffer pools: Small buffers, 104 bytes (total 101, permanent 50, peak 323 @ 1w4d): 93 in free list (20 min, 150 max allowed) 87252899 hits, 2457 misses, 6595 trims, 6646 created 17 failures (0 no memory) Middle buffers, 600 bytes (total 67, permanent 25, peak 90 @ 3w1d): 47 in free list (10 min, 150 max allowed) 98667793 hits, 1599 misses, 4233 trims, 4275 created 73 failures (0 no memory) Big buffers, 1536 bytes (total 93, permanent 50, peak 387 @ 1w4d): 92 in free list (5 min, 150 max allowed) 31857196 hits, 13400 misses, 7415 trims, 7458 created 9000 failures (0 no memory) VeryBig buffers, 4520 bytes (total 15, permanent 10, peak 35 @ 1w4d): 15 in free list (0 min, 100 max allowed) 4896 hits, 4466 misses, 377 trims, 382 created 4407 failures (0 no memory) Large buffers, 5024 bytes (total 2, permanent 0, peak 12 @ 1w4d): 2 in free list (0 min, 10 max allowed) 757 hits, 3699 misses, 438 trims, 440 created 3688 failures (0 no memory) Huge buffers, 18024 bytes (total 10, permanent 0, peak 30 @ 2w1d): 10 in free list (4 min, 10 max allowed) 1300 hits, 2947 misses, 1117 trims, 1127 created 2526 failures (0 no memory) Interface buffer pools: CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 3w4d): 101 in free list (100 min, 200 max allowed) 0 hits, 0 misses, 228 trims, 229 created 0 failures (0 no memory) Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 3w4d): 101 in free list (100 min, 100 max allowed) 0 hits, 0 misses CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 3w4d): 101 in free list (100 min, 200 max allowed) 0 hits, 0 misses, 228 trims, 229 created 0 failures (0 no memory) Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 3w4d): 101 in free list (132 min, 132 max allowed) 3952 hits, 0 misses CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 3w4d): 26 in free list (25 min, 50 max allowed) 0 hits, 0 misses, 228 trims, 229 created 0 failures (0 no memory) IPC buffers, 4096 bytes (total 2, permanent 2): 1 in free list (1 min, 8 max allowed) 1 hits, 0 fallbacks, 0 trims, 0 created 0 failures (0 no memory) CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 3w4d): 3 in free list (2 min, 4 max allowed) 0 hits, 0 misses, 228 trims, 229 created 0 failures (0 no memory) CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 3w4d): 2 in free list (1 min, 2 max allowed) 0 hits, 0 misses, 228 trims, 229 created 0 failures (0 no memory) IPC Medium buffers, 16384 bytes (total 2, permanent 2): 2 in free list (1 min, 8 max allowed) 0 hits, 0 fallbacks, 0 trims, 0 created 0 failures (0 no memory) IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 3w4d): 17 in free list (16 min, 16 max allowed) 0 hits, 0 misses, 36893 trims, 36894 created 0 failures (0 no memory) Header pools: Header buffers, 0 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 575 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 2729564 hits in cache, 63 misses in cache Particle Clones: 1024 clones, 2693119 hits, 0 misses Public particle pools: F/S buffers, 1664 bytes (total 768, permanent 768): 256 in free list (128 min, 1024 max allowed) 512 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) 512 max cache size, 512 in cache 0 hits in cache, 0 misses in cache Normal buffers, 1676 bytes (total 3840, permanent 3840): 3840 in free list (128 min, 4096 max allowed) 279788186 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) Private particle pools: HQF Particle buffers, 0 bytes (total 2000, permanent 2000): 2000 in free list (500 min, 2000 max allowed) 28618086 hits, 0 misses, 0 trims, 0 created 0 failures (0 no memory) IDS SM buffers, 240 bytes (total 128, permanent 128): 0 in free list (0 min, 128 max allowed) 128 hits, 0 fallbacks 128 max cache size, 128 in cache 0 hits in cache, 0 misses in cache GigabitEthernet0/0 buffers, 1664 bytes (total 1024, permanent 1024): 0 in free list (0 min, 1024 max allowed) 1024 hits, 279788186 fallbacks 1024 max cache size, 768 in cache 1134675411 hits in cache, 0 misses in cache GigabitEthernet0/1 buffers, 1664 bytes (total 1024, permanent 1024): 0 in free list (0 min, 1024 max allowed) 1024 hits, 0 fallbacks 1024 max cache size, 768 in cache 85599899 hits in cache, 0 misses in cache GigabitEthernet0/2 buffers, 1664 bytes (total 1024, permanent 1024): 0 in free list (0 min, 1024 max allowed) 1024 hits, 0 fallbacks 1024 max cache size, 768 in cache 925161349 hits in cache, 0 misses in cache
GigabitEthernet0/2 Switching path Pkts In Chars In Pkts Out Chars Out Processor 8693361 2707085783 28693587 1978706503 Route cache 915515352 4057718733 625820939 4038810422 Total 924208713 2469837220 654514526 1722549629
Throttle count 69 Drops RP 0 SP 0 SPD Flushes Fast 948271 SSE 0 SPD Aggress Fast 0 SPD Priority Inputs 2132539 Drops 0 Protocol IP Switching path Pkts In Chars In Pkts Out Chars Out Process 7440194 2614383738 28401465 1930953146 Cache misses 0 - - - Fast 915531905 4064439248 625832221 4042719458 Auton/SSE 0 0 0 0 Protocol ARP Switching path Pkts In Chars In Pkts Out Chars Out Process 2132433 127945980 326 19560 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0 Protocol Other Switching path Pkts In Chars In Pkts Out Chars Out Process 0 0 221079 13264740 Cache misses 0 - - - Fast 0 0 0 0 Auton/SSE 0 0 0 0
HOSTNAME#show processes cpu sorted CPU utilization for five seconds: 9%/6%; one minute: 10%; five minutes: 8% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 172 32244368 81181995 397 2.00% 1.68% 1.56% 0 IP Input 417 1337028 544016923 2 0.71% 0.79% 0.78% 0 IP SLAs XOS Even 194 383172 277450531 1 0.23% 0.25% 0.24% 0 Ethernet Msec Ti 3 992 806 1230 0.23% 0.24% 0.10% 388 SSH Process 247 210524 1395484 150 0.07% 0.00% 0.00% 0 ADJ resolve proc 33 1556476 10905841 142 0.07% 0.06% 0.07% 0 ARP Input 92 315488 2215307 142 0.07% 0.07% 0.07% 0 Per-Second Jobs 168 114468 68672241 1 0.07% 0.05% 0.07% 0 IPAM Manager 128 117064 34544561 3 0.07% 0.05% 0.05% 0 VRRS Main thread 9 0 2 0 0.00% 0.00% 0.00% 0 Timers 8 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro
HOSTNAME#show proc cpu history HOSTNAME 11:22:28 AM Wednesday Mar 11 2020 UTC 666666888889999966666999997777788888999999999977777999999999 100 90 80 70 60 50 40 30 20 10 ************************************************************ 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 CPU% per second (last 60 seconds) 11 1 1111 1 2 41 1 2 12 11 11 1 1111 311 111 629990899900088085940908998699998079998009708089012193009010 100 90 80 70 60 50 40 * 30 * * * * * 20 * * * * * * * * 10 ############################################################ 0....5....1....1....2....2....3....3....4....4....5....5....6 0 5 0 5 0 5 0 5 0 5 0 CPU% per minute (last 60 minutes) * = maximum CPU% # = average CPU% 772935339499798959998437622394763687477896674952322222212226522222222112 307442389387360478476246868866801082598807983644554726384145332152133794 100 * ** * * * * * 90 * * ** * * **** * * ** * 80 * * ** *** **** * * * * **** * * 70 ** * * ****** **** ** * * ** ******* * * 60 ** * * *********** ** * ** *** ******* * * 50 ** * * * *********** ** **** *********** ** ** 40 ** * * *************** ** ***** ************** * ** 30 ***********#*****##****************#****#****** ** * * ** * 20 **********###########**************#########***************#************ 10 ###################################################***###########******* 0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.. 0 5 0 5 0 5 0 5 0 5 0 5 0 CPU% per hour (last 72 hours) * = maximum CPU% # = average CPU%
Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL ----------------------------------------------------------------------------------------------------------------- * GigabitEthernet0/2 0 0 0 0 297000 60 175000 54 69
Packet inspection statistics [process switch:fast switch] tcp packets: [4324074:950480967] udp packets: [6180212:145153209] http packets: [2371485:127872735] ftp packets: [317:0] Interfaces configured for inspection 2 Session creations since subsystem startup or last reset 8344635 Current session counts (estab/half-open/terminating) [71:2:0] Maxever session counts (estab/half-open/terminating) [1610:1233:210] Last session created 00:00:00 Last statistic reset never Last session creation rate 65 Maxever session creation rate 2802 Last half-open session total 2 TCP reassembly statistics received 3318562 packets out-of-order; dropped 559731 peak memory usage 1519 KB; current usage: 0 KB peak queue length 512
03-11-2020 05:28 AM
Hi,
Try to use a different port, tell your ISP about the problem as well and upgrade to a more stable version. This could be one thing, a software "feature".
A second thing. Does your ISP have some kind of policing/shaping configured egress/ingress? What could happen is the following: maybe you have multiple remote spokes and you have a big enough number of packets which needs to be sent/received at the same time (like IGP hellos from the HUB and IGP hellos to the hub; IKE DPD, NHRP messages) which means you have microbursts at regular intervals which could cause overruns, due to the Gig controller internals.
If it was not the first problem, speak with your ISP to reconfigure their policer/shaper, to allow for more bursting, basically reconfigure.
Regards,
Cristian Matei.
03-11-2020 05:56 AM
Thanks :)
We do have some plans to update IOS, but waiting for the new contract to be active.
This is a spoke, but connected to 13 peers now. (main office and other branch offices/warehouses)
I will contact the ISP and see where that leads.
There isn't any operational impact, but it annoys me :)
03-11-2020 01:53 PM
Hello,
the input/overrun errors could be caused by microbursts. What exactly does the QoS policy on the interface do ? Make sure it has some sort of shaping, as that usually smoothes out traffic bursts.
Your buffers have a lot of misses. You might want to try and increase them (use the values below to start out with and monitor the impact):
buffers small max-free 300
buffers small min-free 40
buffers small permanent 100
!
buffers middle max-free 300
buffers middle min-free 20
buffers middle permanent 50
!
buffers big max-free 300
buffers big min-free 10
buffers big permanent 100
!
buffers verybig max-free 200
buffers verybig min-free 10
buffers verybig permanent 20
!
buffers large max-free 20
buffers large min-free 10
buffers large permanent 10
!
buffers huge max-free 20
buffers huge min-free 8
buffers huge permanent 10
03-12-2020 01:26 AM
Thanks, I am trying this. Cleared counters and monitoring.
Turns out there was errors on all ports of this router. 0/0 is the lan port, the other two connects to two different ISP's
show interface Gi0/0 | i drops Input queue: 0/75/2491/87646 (size/max/drops/flushes); Total output drops: 0 94901 unknown protocol drops show interface Gi0/0 | i overrun 238401 input errors, 0 CRC, 0 frame, 238401 overrun, 0 ignored show interface Gi0/1 | i overrun 671 input errors, 0 CRC, 0 frame, 671 overrun, 0 ignored show interface Gi0/2 | i overrun 2824790 input errors, 0 CRC, 0 frame, 2824790 overrun, 0 ignored
03-12-2020 02:12 AM
03-11-2020 04:56 PM
Post the complete output to the following commands:
show interface Gi0/2 controller | b Cumulative Statistics
03-12-2020 01:24 AM
I don't have this command available
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide