cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3734
Views
0
Helpful
7
Replies

Input overrun on gigabit interface with low utilization

Tboss99
Level 1
Level 1

Hi, 

I know this is a recurring topic, but I haven't been able to find a solution yet. No operational impact (as i know of) but I dont like to see overrun errors on a gigabit interface with such low utilization. 

This is one of two public interfaces (secondary interface), used for internet access at the office and attached to secondary DMVPN tunnel. Its is incrementing even now when they are all asleep over there. 

I've checked netflow, and utilization is quite low overall. Any ideas? 

(I am aware of the statement:
In a small number of cases, the overrun counter may be incremented because of a software defect. However, in the majority of cases, it indicates that the receiving capability of the interface was exceeded. Nothing can be done on the router that reports overruns. If possible, the rate that frames are coming should be controlled at the remote end of the connection. Otherwise, if the number of overruns is high, the hardware should be upgraded.   ) 

 

Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.4(3)M3, RELEASE SOFTWARE (fc2)
Cisco CISCO2911/K9 (revision 1.0) with 487424K/36864K bytes of memory.



GigabitEthernet0/2 is up, line protocol is up
  Hardware is CN Gigabit Ethernet, address is 002c.c8e2.8582 (bia 002c.c8e2.8582)
  Description: ** Second connection toXXX Office - Public **$ETH-LAN$$FW_OUTSIDE$
  Internet address is X.X.X.X/29
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/948271 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/0 (size/max total/drops)
  5 minute input rate 1058000 bits/sec, 108 packets/sec
  5 minute output rate 125000 bits/sec, 51 packets/sec
     924191088 packets input, 2462862979 bytes, 1857 no buffer
     Received 2130658 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 69 throttles
     2646058 input errors, 0 CRC, 0 frame, 2646058 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     654499061 packets output, 1717785467 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     1 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

interface GigabitEthernet0/2
description ** Second connection to XXX Office - Public **$ETH-LAN$$FW_OUTSIDE$
ip address X.X.X.X.X 255.255.255.248
ip access-group ACLLine2 in
no ip redirects
no ip proxy-arp
ip flow ingress
ip nat outside
ip inspect CBACFilter out
ip virtual-reassembly in
duplex auto
speed auto
no mop enabled
service-policy output QoS
end
Buffer elements:
     711 in free list
     84430885 hits, 0 misses, 1242 created

Public buffer pools:
Small buffers, 104 bytes (total 101, permanent 50, peak 323 @ 1w4d):
     93 in free list (20 min, 150 max allowed)
     87252899 hits, 2457 misses, 6595 trims, 6646 created
     17 failures (0 no memory)
Middle buffers, 600 bytes (total 67, permanent 25, peak 90 @ 3w1d):
     47 in free list (10 min, 150 max allowed)
     98667793 hits, 1599 misses, 4233 trims, 4275 created
     73 failures (0 no memory)
Big buffers, 1536 bytes (total 93, permanent 50, peak 387 @ 1w4d):
     92 in free list (5 min, 150 max allowed)
     31857196 hits, 13400 misses, 7415 trims, 7458 created
     9000 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 15, permanent 10, peak 35 @ 1w4d):
     15 in free list (0 min, 100 max allowed)
     4896 hits, 4466 misses, 377 trims, 382 created
     4407 failures (0 no memory)
Large buffers, 5024 bytes (total 2, permanent 0, peak 12 @ 1w4d):
     2 in free list (0 min, 10 max allowed)
     757 hits, 3699 misses, 438 trims, 440 created
     3688 failures (0 no memory)
Huge buffers, 18024 bytes (total 10, permanent 0, peak 30 @ 2w1d):
     10 in free list (4 min, 10 max allowed)
     1300 hits, 2947 misses, 1117 trims, 1127 created
     2526 failures (0 no memory)

Interface buffer pools:
CF Small buffers, 104 bytes (total 101, permanent 100, peak 101 @ 3w4d):
     101 in free list (100 min, 200 max allowed)
     0 hits, 0 misses, 228 trims, 229 created
     0 failures (0 no memory)
Generic ED Pool buffers, 512 bytes (total 101, permanent 100, peak 101 @ 3w4d):
     101 in free list (100 min, 100 max allowed)
     0 hits, 0 misses
CF Middle buffers, 600 bytes (total 101, permanent 100, peak 101 @ 3w4d):
     101 in free list (100 min, 200 max allowed)
     0 hits, 0 misses, 228 trims, 229 created
     0 failures (0 no memory)
Syslog ED Pool buffers, 600 bytes (total 133, permanent 132, peak 133 @ 3w4d):
     101 in free list (132 min, 132 max allowed)
     3952 hits, 0 misses
CF Big buffers, 1536 bytes (total 26, permanent 25, peak 26 @ 3w4d):
     26 in free list (25 min, 50 max allowed)
     0 hits, 0 misses, 228 trims, 229 created
     0 failures (0 no memory)
IPC buffers, 4096 bytes (total 2, permanent 2):
     1 in free list (1 min, 8 max allowed)
     1 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)
CF VeryBig buffers, 4520 bytes (total 3, permanent 2, peak 3 @ 3w4d):
     3 in free list (2 min, 4 max allowed)
     0 hits, 0 misses, 228 trims, 229 created
     0 failures (0 no memory)
CF Large buffers, 5024 bytes (total 2, permanent 1, peak 2 @ 3w4d):
     2 in free list (1 min, 2 max allowed)
     0 hits, 0 misses, 228 trims, 229 created
     0 failures (0 no memory)
IPC Medium buffers, 16384 bytes (total 2, permanent 2):
     2 in free list (1 min, 8 max allowed)
     0 hits, 0 fallbacks, 0 trims, 0 created
     0 failures (0 no memory)
IPC Large buffers, 65535 bytes (total 17, permanent 16, peak 17 @ 3w4d):
     17 in free list (16 min, 16 max allowed)
     0 hits, 0 misses, 36893 trims, 36894 created
     0 failures (0 no memory)

Header pools:
Header buffers, 0 bytes (total 768, permanent 768):
     256 in free list (128 min, 1024 max allowed)
     575 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     2729564 hits in cache, 63 misses in cache

Particle Clones:
     1024 clones, 2693119 hits, 0 misses

Public particle pools:
F/S buffers, 1664 bytes (total 768, permanent 768):
     256 in free list (128 min, 1024 max allowed)
     512 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
     512 max cache size, 512 in cache
     0 hits in cache, 0 misses in cache
Normal buffers, 1676 bytes (total 3840, permanent 3840):
     3840 in free list (128 min, 4096 max allowed)
     279788186 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)

Private particle pools:
HQF Particle buffers, 0 bytes (total 2000, permanent 2000):
     2000 in free list (500 min, 2000 max allowed)
     28618086 hits, 0 misses, 0 trims, 0 created
     0 failures (0 no memory)
IDS SM buffers, 240 bytes (total 128, permanent 128):
     0 in free list (0 min, 128 max allowed)
     128 hits, 0 fallbacks
     128 max cache size, 128 in cache
     0 hits in cache, 0 misses in cache
GigabitEthernet0/0 buffers, 1664 bytes (total 1024, permanent 1024):
     0 in free list (0 min, 1024 max allowed)
     1024 hits, 279788186 fallbacks
     1024 max cache size, 768 in cache
     1134675411 hits in cache, 0 misses in cache
GigabitEthernet0/1 buffers, 1664 bytes (total 1024, permanent 1024):
     0 in free list (0 min, 1024 max allowed)
     1024 hits, 0 fallbacks
     1024 max cache size, 768 in cache
     85599899 hits in cache, 0 misses in cache
GigabitEthernet0/2 buffers, 1664 bytes (total 1024, permanent 1024):
     0 in free list (0 min, 1024 max allowed)
     1024 hits, 0 fallbacks
     1024 max cache size, 768 in cache
     925161349 hits in cache, 0 misses in cache

 

GigabitEthernet0/2
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
               Processor    8693361 2707085783   28693587 1978706503
             Route cache  915515352 4057718733  625820939 4038810422
                   Total  924208713 2469837220  654514526 1722549629
          Throttle count         69
                   Drops         RP          0         SP          0
             SPD Flushes       Fast     948271        SSE          0
             SPD Aggress       Fast          0
            SPD Priority     Inputs    2132539      Drops          0

    Protocol  IP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    7440194 2614383738   28401465 1930953146
            Cache misses          0          -          -          -
                    Fast  915531905 4064439248  625832221 4042719458
               Auton/SSE          0          0          0          0

    Protocol  ARP
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process    2132433  127945980        326      19560
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0

    Protocol  Other
          Switching path    Pkts In   Chars In   Pkts Out  Chars Out
                 Process          0          0     221079   13264740
            Cache misses          0          -          -          -
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
HOSTNAME#show processes cpu sorted
CPU utilization for five seconds: 9%/6%; one minute: 10%; five minutes: 8%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 172    32244368    81181995        397  2.00%  1.68%  1.56%   0 IP Input
 417     1337028   544016923          2  0.71%  0.79%  0.78%   0 IP SLAs XOS Even
 194      383172   277450531          1  0.23%  0.25%  0.24%   0 Ethernet Msec Ti
   3         992         806       1230  0.23%  0.24%  0.10% 388 SSH Process
 247      210524     1395484        150  0.07%  0.00%  0.00%   0 ADJ resolve proc
  33     1556476    10905841        142  0.07%  0.06%  0.07%   0 ARP Input
  92      315488     2215307        142  0.07%  0.07%  0.07%   0 Per-Second Jobs
 168      114468    68672241          1  0.07%  0.05%  0.07%   0 IPAM Manager
 128      117064    34544561          3  0.07%  0.05%  0.05%   0 VRRS Main thread
   9           0           2          0  0.00%  0.00%  0.00%   0 Timers
   8           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro
HOSTNAME#show proc cpu history

HOSTNAME  11:22:28 AM Wednesday Mar 11 2020 UTC





      666666888889999966666999997777788888999999999977777999999999
  100
   90
   80
   70
   60
   50
   40
   30
   20
   10 ************************************************************
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per second (last 60 seconds)




      11   1    1111 1 2 41 1    2     12    11 11 1  1111 311 111
      629990899900088085940908998699998079998009708089012193009010
  100
   90
   80
   70
   60
   50
   40                    *
   30                  * *       *      *                  *
   20 *            *   * *       *      *       *          *
   10 ############################################################
     0....5....1....1....2....2....3....3....4....4....5....5....6
               0    5    0    5    0    5    0    5    0    5    0
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%




      772935339499798959998437622394763687477896674952322222212226522222222112
      307442389387360478476246868866801082598807983644554726384145332152133794
  100         * ** *   * *        *                *
   90    *    * ** * * ****       *     *    **    *
   80    *    * ** *** ****  *    * *   *  ****  * *
   70 ** *    * ****** ****  **   * *   ** ******* *             *
   60 ** *    * ***********  **   * ** *** ******* *             *
   50 ** * *  * ***********  **   **** *********** **            **
   40 ** * * *************** **  ***** ************** *          **
   30 ***********#*****##****************#****#****** ** * *     **   *
   20 **********###########**************#########***************#************
   10 ###################################################***###########*******
     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
               0    5    0    5    0    5    0    5    0    5    0    5    0
                   CPU% per hour (last 72 hours)
                  * = maximum CPU%   # = average CPU%
  Interface                   IHQ       IQD       OHQ       OQD      RXBS      RXPS      TXBS      TXPS      TRTL
-----------------------------------------------------------------------------------------------------------------
* GigabitEthernet0/2            0         0         0         0    297000        60    175000        54        69
Packet inspection statistics [process switch:fast switch]
  tcp packets: [4324074:950480967]
  udp packets: [6180212:145153209]
  http packets: [2371485:127872735]
  ftp packets: [317:0]
Interfaces configured for inspection 2
Session creations since subsystem startup or last reset 8344635
Current session counts (estab/half-open/terminating) [71:2:0]
Maxever session counts (estab/half-open/terminating) [1610:1233:210]
Last session created 00:00:00
Last statistic reset never
Last session creation rate 65
Maxever session creation rate 2802
Last half-open session total 2
TCP reassembly statistics
  received 3318562 packets out-of-order; dropped 559731
  peak memory usage 1519 KB; current usage: 0 KB
  peak queue length 512

 

7 Replies 7

Cristian Matei
VIP Alumni
VIP Alumni

Hi,

 

    Try to use a different port, tell your ISP about the problem as well and upgrade to a more stable version. This could be one thing, a software "feature".

    A second thing. Does your ISP have some kind of policing/shaping configured egress/ingress? What could happen is the following: maybe you have multiple remote spokes and you have a big enough number of packets which needs to be sent/received at the same time (like IGP hellos from the  HUB and IGP hellos to the hub; IKE DPD, NHRP messages) which means you have microbursts at regular intervals which could cause overruns, due to the Gig controller internals.

    If it was not the first problem, speak with your ISP to reconfigure their policer/shaper, to allow for more bursting, basically reconfigure.

 

Regards,

Cristian Matei.

 

 

Thanks :) 

We do have some plans to update IOS, but waiting for the new contract to be active. 

This is a spoke, but connected to 13 peers now. (main office and other branch offices/warehouses) 

I will contact the ISP and see where that leads. 

There isn't any operational impact, but it annoys me :)  

Hello,

 

the input/overrun errors could be caused by microbursts. What exactly does the QoS policy on the interface do ? Make sure it has some sort of shaping, as that usually smoothes out traffic bursts.

 

Your buffers have a lot of misses. You might want to try and increase them (use the values below to start out with and monitor the impact):

 

buffers small max-free 300
buffers small min-free 40
buffers small permanent 100
!
buffers middle max-free 300
buffers middle min-free 20
buffers middle permanent 50
!
buffers big max-free 300
buffers big min-free 10
buffers big permanent 100
!
buffers verybig max-free 200
buffers verybig min-free 10
buffers verybig permanent 20
!
buffers large max-free 20
buffers large min-free 10
buffers large permanent 10
!
buffers huge max-free 20
buffers huge min-free 8
buffers huge permanent 10

Thanks, I am trying this. Cleared counters and monitoring. 

Turns out there was errors on all ports of this router. 0/0 is the lan port, the other two connects to two different ISP's

 

show interface Gi0/0 | i drops
Input queue: 0/75/2491/87646 (size/max/drops/flushes); Total output drops: 0
94901 unknown protocol drops
show interface Gi0/0 | i overrun
238401 input errors, 0 CRC, 0 frame, 238401 overrun, 0 ignored
show interface Gi0/1 | i overrun
671 input errors, 0 CRC, 0 frame, 671 overrun, 0 ignored
show interface Gi0/2 | i overrun
2824790 input errors, 0 CRC, 0 frame, 2824790 overrun, 0 ignored

Configure QoS.
Overruns are "too much" packets which arrive to the interface (receive) and drop because they could not be processed.
Overruns is the opposite to "total output drops" (output) when the interface sends too much packets downstream and the downstream clients start dropping packets.

Leo Laohoo
Hall of Fame
Hall of Fame

Post the complete output to the following commands: 

show interface Gi0/2 controller | b Cumulative Statistics

 

I don't have this command available 

Review Cisco Networking for a $25 gift card