increasing Input errors and overrun

Adnan Khan · ‎03-04-2020

Hi have CISCO 2951 Router and I am getting increasing input errors and overruns and CPU was around 98% to 99% what could be the issue or how i can resolve this? Do I need to apply any QoS for inbound traffic.

Outgoing interface seems find and no input errors.

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 36/255, rxload 5/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 00:34:22
Input queue: 3/75/66788/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 22450000 bits/sec, 11148 packets/sec
30 second output rate 143684000 bits/sec, 17439 packets/sec
30934502 packets input, 7699669203 bytes, 0 no buffer
Received 864 broadcasts (0 IP multicasts)
0 runts, 0 giants, 1274 throttles
402 input errors, 0 CRC, 0 frame, 402 overrun, 0 ignored
0 watchdog, 863 multicast, 0 pause input
47091432 packets output, 58453733003 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Leo Laohoo · ‎03-04-2020

@Adnan Khan wrote:

Last clearing of "show interface" counters 00:34:22

No offense, but this is not really helpful.

Very difficult to make an assessment when someone has cleaned out the counters.

@Adnan Khan wrote:

Full Duplex, 1Gbps, media type is RJ45

Post the complete output to the command "sh run interface <BLAH>" (minus the IP addressing and keys).

Adnan Khan · ‎03-04-2020

I cleared the counter to check how fast input errors increasing. Here is the output of interface.

GigabitEthernet0/1 is up, line protocol is up
Hardware is PQ3_TSEC, address is ######
Internet address is xx.xx.xx.xx
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 29/255, rxload 7/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 01:04:13
Input queue: 0/75/66788/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 27744000 bits/sec, 8621 packets/sec
30 second output rate 115153000 bits/sec, 13552 packets/sec
50132298 packets input, 13970608825 bytes, 0 no buffer
Received 1622 broadcasts (0 IP multicasts)
0 runts, 0 giants, 1274 throttles
516 input errors, 0 CRC, 0 frame, 516 overrun, 0 ignored
0 watchdog, 1621 multicast, 0 pause input
78288265 packets output, 92486812313 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Georg Pauwen · ‎03-04-2020

Hello,

post the running configuration of the router. Is this the WAN or LAN interface ? You could try and increase the hold-queue on the interface to e.g. 'hold-queue 1200', or use a QoS policy to shape your traffic. Input errors and overruns could be caused by bursty traffic...

Adnan Khan · ‎03-04-2020

IT is LAN interface. Here is the interface config. What is the default hold-queue setting and what will be the impact of setting as 1200. Can you give any Qos config example. This issue is happening because of bursty traffic most of the traffic is for internet video streaming and web browsing.

interface GigabitEthernet0/1
ip address x.x.x.x
ip nat inside
ip virtual-reassembly in
ip tcp adjust-mss 1452
ip policy route-map xxxxx
standby 2 ip x.x.x.x
standby 2 priority 200
standby 2 preempt
load-interval 30
duplex auto
speed auto

Adnan Khan · ‎03-04-2020

I cleared the counter to check how fast input errors increasing. Here is the output of interface.

GigabitEthernet0/1 is up, line protocol is up
Hardware is PQ3_TSEC, address is ######
Internet address is xx.xx.xx.xx
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 29/255, rxload 7/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full Duplex, 1Gbps, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters 01:04:13
Input queue: 0/75/66788/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 27744000 bits/sec, 8621 packets/sec
30 second output rate 115153000 bits/sec, 13552 packets/sec
50132298 packets input, 13970608825 bytes, 0 no buffer
Received 1622 broadcasts (0 IP multicasts)
0 runts, 0 giants, 1274 throttles
516 input errors, 0 CRC, 0 frame, 516 overrun, 0 ignored
0 watchdog, 1621 multicast, 0 pause input
78288265 packets output, 92486812313 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Georg Pauwen · ‎03-04-2020

Hello,

the default for the in and out hold queues is 75 and 40, respectively. I would start with doubling both values and see if the output rate drops decrease:

--> hold-queue in 150
--> hold-queue out 80

Also, double the numbers below gradually and see if that has an effect:

ip virtual-reassembly max-reassemblies number 32 (16 is the default)
ip virtual-reassembly max-fragments number 64 (32 is the default)

Also, remove the line 'ip tcp adjust-mss 1452' from the interface...

Adnan Khan · ‎03-08-2020

Hi Georg,

Many thanks for your reply. I will check and update you also If i need to implement Traffic Policing then how I should do for incoming traffic?

Leo Laohoo · ‎03-08-2020

If the port is a LAN port, what is the "sh interface <PORT>" output of the LAN directly connected to the port?

Adnan Khan · ‎03-08-2020

I can only see hold-queue and there is no option for in or out after that. It is 2951 Router.

Georg Pauwen · ‎03-10-2020

Hello,

since it is the LAN interface, traffic shaping doesn't make much sense.

What about the previous suggestions ? Did you try these ?

If you only see a hold queue value, gradually double the values and check if that makes a difference

--> hold-queue 200

Also, double the numbers below gradually and see if that has an effect:

ip virtual-reassembly max-reassemblies number 32 (16 is the default)
ip virtual-reassembly max-fragments number 64 (32 is the default)

Also, remove the line 'ip tcp adjust-mss 1452' from the interface...

Adnan Khan · ‎03-10-2020

Hi,

Not yet changed. Changing these values cause any service disruption?

Deepak Kumar · ‎03-09-2020

Hi,

As expert persons are advised to check remote end details and increase the queue limit etc. But I want to highlight an additional point as Why CPU is running high? Have you checked the details? There may be some other reason?

We have seen that many times a physical media issue, duplex or speed mismatch will cause this issue. Please verify all the above.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

Adnan Khan · ‎03-10-2020

Spoiler

Hi Deepak,

I did not see any CRC on the interface but still input errors and input overruns were increasing also I can see input queue drops were seen also. Right now CPU is running normally around 25% but errors were still increasing gradually. Any solution you have?