1941 Router - Input Queue Traffic Drops

[hiltonstroud]

Hopefully someone might be able to shed some light on some networking issues I am having. 

Our routing core is nexus 5ks, however to a remote site we have a 100Mb full duplex private microwave/wireless system that is used for dataprotection/backups.  We can't conenct the microwave/wireless system to the Nexus 5ks or 2k line card cores as it just shuts the port down due to it thinking the microwave system is a swith.  So we have used a 1941 cisco router we have to connect via our 2960 access stack layer which connectes to our nexus 5ks.  To make the microwave system connect correctly 1 interface gig0/0 has been set to 100Mb full duplex as this is what the mw/wireless system requires.  interface gig0/1 is auto negotiate at 1Gb which connects to the 2960.  Traffic seems ok until it hits the 1941 router, interface gig0/1 doesn't show any "Total output drops" on the Input Queue, however interface gig0/0 is showing "Total output drops" on the input queue which to me indicates an issuse why I am seeing poor performance over this link. 

I assume there is some tunning or rate limiting needed or maybe flow control? however I am not sure how to attack this issue to try and resolve.  Any help will be greatly appreiated. 

Below is the ouput of int gig0/0 to see confiruration and drops. 

sh int gig0/0
GigabitEthernet0/0 is up, line protocol is up
Hardware is CN Gigabit Ethernet, address is "blanked"
Internet address is 192.168.200.1/24
MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 42/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:04, output 00:00:00, output hang never
Last clearing of "show interface" counters 17:20:54
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 92498
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 174000 bits/sec, 216 packets/sec
5 minute output rate 16592000 bits/sec, 1599 packets/sec
159059863 packets input, 3829429892 bytes, 92495 no buffer
Received 8567 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 4294744112 multicast, 0 pause input
0 input packets with dribble condition detected
265371302 packets output, 1902884476 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

sh int gig0/0 summary

*: interface is up
IHQ: pkts in input hold queue IQD: pkts dropped from input queue
OHQ: pkts in output hold queue OQD: pkts dropped from output queue
RXBS: rx rate (bits/sec) RXPS: rx rate (pkts/sec)
TXBS: tx rate (bits/sec) TXPS: tx rate (pkts/sec)
TRTL: throttle count

Interface IHQ IQD OHQ OQD RXBS RXPS TXBS TXPS TRTL
------------------------------------------------------------------------
* GigabitEthernet0/0 0 0 0 92498 45005000 3958 17383000 3369 0

[RyanB]

I see "92495 no buffer".

This, from my understanding, means that 92,495 out of the 92,498 packets dropped were due to the fact that your buffer was full.

If there's nothing else connected to the 1941 router than a single link to the 2960 access stack, limit that connection to 100Mbps as well. There's no reason to allow 1Gbps into the 1941 if all it can ever do is leave at 100Mbps.

[hiltonstroud]

Hi Ryan, 

Thank you for the response.  It's starting to make sense for me slowly. I was leaning to that packets dropped due to the interface essentially being flooded, or as you put it bugger was full.   I am not sure what buffer this device and will have to look in to that side. 

There is nothing else connected on the 1941 router other then the single link to the 2960 access stack.  So in this case are you saying to set the speed on the 2960 port to 100Mbps Full duplex, which may require the 1941 gig0/1 port to be set to 100MB Full duplex as well so they talk?
Just wondering as I can't interupt the current transfers atm and hoping to do some of this stuff while it's running.

My other thought is, if I set the speed ont he 2960 port, am I just shifting the dropouts issue to the next device up the line as such from the 1941 router? or does the 2690 have be ability to handle this?  

Many thanks again Ryan, I will also start to try understand the link you had in the next post below as well. 
Cheers,

[RyanB]

In the show interfaces interface-identifier command output:
The no buffer counter increments when the interface fails to obtain a buffer for an inbound packet.
Both the no buffer and drops (input queue) counters increment when the interface fails to obtain a buffer for an inbound packet.
A no buffer counter that increments in the show interfaces output correlates to the misses counter that increments in the show buffers output. The appropriate buffer pool may be tuned.

https://www.cisco.com/c/en/us/support/docs/interfaces-modules/channel-interface-processors/14620-41.html

[Joseph W. Doherty]

Gig in to FE out, output drops are to be expected.

What can you do?

If these are transient bursts, you can increase your egress queue depth. The default of 40 supports a typical 10 Mbps LAN or about a T1 WAN, i.e. likely much smaller than your BDP requires.

If you're dealing with sustained congestion, you might be able to mitigate the overall number of egress drops by implementing "smarter" drop management. For example either FQ or RED might decrease the overall drop percentage.

BTW, Cisco recommends a 1941 for only up to 25 Mbps.

[hiltonstroud]

Hi Joeseph, Thank you for your reply, appreiated. 

I have played with the max size on the output queue and have limited the drops, however I think I have both a traniset burst as well as sustained congestion occuring.  The Increase in max output queue has limited the drops but I think is just masking the issue to a certain point.  I agree Gig into FE isn't the best and is def looking like the core of the issue.  It seems to me I need to move up the stack to the 2960's port to possible deal with it there and have both ports on the 1941 router set to 100Mb, that way can maybe use more power unti to handle the Gig to 100/FE conversion. 

If I change speed on a interface I assume it will need to go down and up to apply so current transferes through this unti would be interuppted? 

Any chance you could point me in the direction of this FQ or RED suggestion? I am not sure what that is. 

I understand the cisco 1941 isn't the fast unit, however it should be able to handle a transfering a 100Mb connection at around the 9-10Mbps speed.  There is only an in and out connection on this unit so this is it's sole job as such so should be fine to handle the job at hand in theory. 

Cheers,

[Joseph W. Doherty]

Yes, a 1941 should be able to handle 9-10 Mbps, but as you reduce drops, the throughput might try to increase, and if it increase enough, the capacity of the 1941 might begin to impact your traffic.

As to changing upstream interface also to 100 Mbps, that should, in theory, just move the bottleneck further upstream. Often that doesn't help unless either the upstream device has "better" QoS feature set to work with (unlikely compared to your 1941) or you have one major host pushing the link, and you slow its interface.

As to FQ (fair-queue) or RED (random early drop), you find much information on them if you search Cisco's main web site. However, RED (or WRED) can be a tricky technology to tune for optimal operation (which Cisco's documentation won't mention).

Oh, and yes, changing interface speeds is going to impact transit traffic.