interface input error

[uthayaman]

We have cisoc 2821 at one of branch and created five sub inetrfaces for different vlans.Output of Show interface shows very frequent increase in the input error count.

I have changed the physical cable and switch port on the other side.But still error rate is increasing.

When the traffic is less error rate is low but with high traffic it is increasing drastically.My router process is very less(4%) only.What could be possible reason.

we are using c2800nm-advsecurityk9-mz.124-20.T1.bin version in the router.

show inter gig 0/1
GigabitEthernet0/1 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 0024.9779.83c1 (bia 0024.9779.83c1)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:03:26
  Input queue: 0/75/104/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 634000 bits/sec, 465 packets/sec
  5 minute output rate 268000 bits/sec, 197 packets/sec
     63629 packets input, 7882225 bytes, 0 no buffer
    Received 23970 broadcasts, 0 runts, 0 giants, 2 throttles
     7 input errors, 0 CRC, 0 frame, 0 overrun, 7 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     30443 packets output, 3432884 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     11 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

drops with respect to queue is increasing after a reboot of router. Should it be a queue issue, how shall i troubleshoot.

[Peter Paluch · Accepted solution]

Hello,

According the documentation at

http://www.cisco.com/en/US/docs/ios/interface/command/reference/ir_s4.html#wp1215572

the "ignored" input errors are caused by the insufficient buffer in the interface controller:

Number of received packets ignored by the  interface because the interface hardware ran low on internal buffers.  These buffers are different than the system buffers. Broadcast storms  and bursts of noise can cause the ignored count to be increased.

Also, there are throttles recorded on the interface. The same document says:

Number of times the receiver on the port was disabled, possibly because of buffer or processor overload.

Thus, it is conceivable that there are large bursts of traffic entering the interface that are causing these errors to increase. Nevertheless, it is not usual for these errors to increase. Can you perhaps post the entire interface configuration? Also, do you perform any traffic monitoring to see if there are indeed some traffic spikes?

Best regards,

Peter

[Ganesh Hariharan]

GigabitEthernet0/1 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 0024.9779.83c1 (bia 0024.9779.83c1)
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 00:03:26
  Input queue: 0/75/104/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 634000 bits/sec, 465 packets/sec
  5 minute output rate 268000 bits/sec, 197 packets/sec
     63629 packets input, 7882225 bytes, 0 no buffer
     Received 23970 broadcasts, 0 runts, 0 giants, 2 throttles
     7 input errors, 0 CRC, 0 frame, 0 overrun, 7 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     30443 packets output, 3432884 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     11 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Hi,

Check out the below link on troubleshooting on input and output quee drops

http://www.cisco.com/en/US/customer/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

Hey Peter,

Your explantion about any problem is fantastic

Hope to Help !!

Ganesh.H

[uthayaman]

Hi,

Thanks for the reply.

Peter: I have enabled netflow analyzer and i didnt find any peak. How can i find  suddent traffic burst in the network.

interface GigabitEthernet0/1
no ip address
ip flow ingress
duplex auto
speed auto
!
interface GigabitEthernet0/1.1
encapsulation dot1Q 1 native
!
interface GigabitEthernet0/1.2
  encapsulation dot1Q 2
ip address 10.50.2.3 255.255.255.0
ip flow ingress
standby 1 ip 10.50.2.1
standby 1 priority 90
standby 1 preempt
standby 1 track 1 decrement 20
!
interface GigabitEthernet0/1.3
  encapsulation dot1Q 3
ip address 10.50.3.3 255.255.255.0
ip flow ingress
standby 2 ip 10.50.3.8
standby 2 priority 90
standby 2 preempt
standby 2 track 1 decrement 20
!
interface GigabitEthernet0/1.4
  encapsulation dot1Q 4
ip address 10.50.4.3 255.255.255.0
ip flow ingress
standby 3 ip 10.50.4.1
standby 3 preempt
standby 3 track 1 decrement 20
!
interface GigabitEthernet0/1.5
  encapsulation dot1Q 5
ip address 10.50.5.3 255.255.255.0
ip flow ingress
standby 4 ip 10.50.5.1
standby 4 priority 90
standby 4 preempt
standby 4 track 1 decrement 20
!
interface GigabitEthernet0/1.6
  encapsulation dot1Q 6
ip address 10.50.6.3 255.255.255.192
ip flow ingress
standby 5 ip 10.50.6.1
standby 5 priority 90
standby 5 preempt
standby 5 track 1 decrement 20
!
interface GigabitEthernet0/1.7
  encapsulation dot1Q 7
ip address 10.50.6.67 255.255.255.192
ip flow ingress
standby 6 ip 10.50.6.65
standby 6 priority 90
standby 6 preempt
standby 6 track 1 decrement 20
!
interface GigabitEthernet0/1.8
  encapsulation dot1Q 8
ip address 10.50.6.131 255.255.255.192
ip access-group OMNE in
ip flow ingress
standby 7 ip 10.50.6.129
standby 7 priority 90
standby 7 preempt
standby 7 track 1 decrement 20
!
interface GigabitEthernet0/1.9
  encapsulation dot1Q 9
ip address 10.50.6.195 255.255.255.224
ip access-group CSL in
ip flow ingress
standby 8 ip 10.50.6.193
standby 8 priority 90
standby 8 preempt
standby 8 track 1 decrement 20
!
interface GigabitEthernet0/1.10
encapsulation dot1Q 10
ip address 10.50.6.227 255.255.255.224
ip flow ingress
standby 12 ip 10.50.6.225
standby 12 priority 90
standby 12 preempt
standby 12 track 1 decrement 20
!
interface GigabitEthernet0/1.100
encapsulation dot1Q 100
ip address 172.17.25.3 255.255.255.0
ip flow ingress
standby 9 ip 172.17.25.1
standby 9 priority 90
standby 9 preempt
standby 9 track 1 decrement 20
!
interface GigabitEthernet0/1.101
  encapsulation dot1Q 101
ip address 10.111.64.3 255.255.255.0
ip flow ingress
standby 10 ip 10.111.64.1
standby 10 preempt
standby 10 track 1 decrement 20
!
interface GigabitEthernet0/1.102
  encapsulation dot1Q 102
ip address 10.111.65.3 255.255.255.0
ip flow ingress
standby 11 ip 10.111.65.1
standby 11 preempt
standby 11 track 1 decrement 20
!
interface GigabitEthernet0/1.114
encapsulation dot1Q 114
ip address 10.114.0.3 255.255.255.0
ip flow ingress
standby 13 ip 10.114.0.1
standby 13 priority 90
standby 13 preempt
standby 13 track 1 decrement 20

[Peter Paluch]

Hello,

Detecting spikes can be a problem - they appear momentarily and are not easily detectable without continuous network monitoring. Relying on summed and averaged data as in NetFlow won't show up their presence.

However, after seeing your configuration I do believe that excessive traffic could account for your problem. You have currently around 13 subinterfaces created under Gi0/1. It is conceivable that the router has a lot of work performing inter-VLAN routing, and with some other features also running at the router, the CPU is simply unable to process sudden bursts of traffic as soon as the frames arrive, occassionally resulting in frames discarded on the interface. I have to point out that even these router platforms have Gigabit Ethernet interfaces, their overall throughput may (and probably will) be significantly lower. To perform inter-VLAN routing between 13 VLANs, a multilayer switch would be much more preferable.

My question is: does this cause any visible, perceptible connectivity problems, or are you simply bothered by the show interfaces output?

Best regards,

Peter

[uthayaman]

Thx Peter,

We are facing intermitant packet drop in the network. Facing packet drop  while reaching my gateway itself. Is there any way to prove my managment that this router resource is not suitable for my environment and go for an upgrade(Any were can i get logs for this resource un availablity).

-uthay

[Peter Paluch]

Hello Uthay,

Proving that a device is technically unsuitable may be somewhat difficult. What I am now suggesting is that from the design viewpoint, it is not appropriate to perform inter-VLAN routing by a single router-on-stick. That the router indeed does not have enough processing power has yet to be ascertained.

Can you produce the packet losses on demand, i.e. can you always perform the reachability test and see that the packets are getting lost? What I suggest is temporarily disallowing individual VLANs on the trunk towards your 2821 router to decrease the amount of traffic hitting the interface, and seeing whether the packet losses stop appearing. I suggest proceeding as follows:

• First, shutdown the respective subinterface on the router using the shutdown command. This should gracefully allow the HSRP to migrate towards another active router.
• Then, on the trunk (i.e. on the switch), prune off the VLAN of the deactivated subinterface using the command switchport trunk allowed vlan remove command. The reason for this is that deactivating the subinterface on the router will not prevent the traffic in the particular VLAN from hitting the router's interface. Although there will be no subinterface to process the traffic, the tagged frames will probably occupy some space in the interface buffer before the router discovers that there is no subinterface for them and that they should be dropped.

If, after pruning off individual VLANs, you discover that the network starts behaving normally and the intermittent packet losses disappear, you have at least pinpointed the existing traffic as the cause of your problem. You could then perform more fine-grained testing to see if it is a single VLAN that contains traffic making these problems.

Taking into consideration that you are running HSRP, I suppose this test should run fine without strong connectivity outages but as I am not familiar with your network, caution is called for.

By the way, what is the second device in the HSRP group? Is it the same router type? Does it also report ignored packets on its interfaces?

Do you believe you could perform this experiment?

Best regards,

Peter

[uthayaman]

Hello Peter,

Thx....

We have performed this test, but we have not done the VLAN prune. Will do it tonight and update.

I am getting same issue on the second router also. That is also same make,model and version.

[Peter Paluch]

Hello Uthay,

Thank you - yes, the VLAN pruning is probably necessary. Always make sure using the show interface trunk that the VLANs have indeed been pruned off the trunk.

The fact that your second router exhibits similar symptoms is probably an indication that indeed, we are probably having problems with the amount of traffic to be handled by the routers. At least we can exclude the possibility of a cabling problem or a faulty hardware on one device.

Please keep us informed!

Best regards,

Peter