08-22-2011 11:28 PM - edited 03-07-2019 01:49 AM
Running a 3110X blade chassis and on alot of the etherchannels i'm seeing a lot of output drops.
running IOS version 12.2(55)SE.
here are some show commands:
BS001#show int po6
Port-channel6 is up, line protocol is up (connected)
Hardware is EtherChannel, address is e804.6288.9b06 (bia e804.6288.9b06)
Description: LACP-ServerSlot-6
MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is unknown
Media-type configured as connector
input flow-control is off, output flow-control is unsupported
Members in this channel: Gi1/0/6 Gi2/0/6
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 7240
show int gi1/0/6
GigabitEthernet1/0/6 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is e804.6288.a806 (bia e804.6288.a806)
Description: Server Slot 6
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseX
Media-type configured as connector
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:23, output 00:00:00, output hang never
Last clearing of "show interface" counters 11w4d
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 9
#remote command 2 show int gi2/0/6
Switch : 2 :
------------
GigabitEthernet2/0/6 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is e804.6288.9b06 (bia e804.6288.9b06)
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 2/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseX
Media-type configured as connector
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:12, output 00:00:06, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 7231
BS001#show platform port-asic stats drop gi1/0/6
Interface Gi1/0/6 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 9
Queue 4
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 5
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 6
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 7
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
BS001# remote command 2 show platform port-asic stats drop gi2/0/6
Switch : 2 :
------------
Interface Gi2/0/6 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 7231
Queue 4
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 5
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 6
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 7
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Any idea whats going on?
I have a similar issue with a 3750 running the same version of IOS. Never had this issue with previous versions of IOS.
QoS is disabled on this chassis.
CPU and memory are fine and utilization on the interfaces are low, monitoring shows it has never been above 100Mb.
08-23-2011 09:51 PM
anyone?
08-24-2011 11:10 AM
Hi Luca,
Output drops are caused by lack of egress buffers on interfaces. This is usually caused by bursty traffic, high number of packets in a short period. Although the average rate might not be high, if the port tries to send out a lot of traffic in short time, buffers might be not available and port will drop traffic.
Therefore I would suggest the following to be done:
- Try doing a SPAN capture on the ports to see if you can identify short bursts of packets, either unicast, multicast or broadcast.
- Configure the etherchannel load-balance method on the switch to something more granular, such as src-dst-ip to make sure that links are utilized evenly and more efficiently, rather than one link carrying much more traffic than the other.
Understanding EtherChannel Load Balancing and Redundancy on Catalyst Switches
http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml
On the other hand, the output drops value does not seem very high and I would not be concerned about it if the packet loss is less than 1%.
Best regards,
Andras
08-24-2011 07:52 PM
While i understand the bufferers issue, i think something has changed in recent IOS. I have a simialar blade running an earlier version of IOS 12.2(22) and i don't see this issue. Same goes for some 3750s i have around as well.
Anyway - it does look like one link is being uses way more than the other. Will using the src-dst-ip assist with more even load balancing? What does cisco reccomend?
08-24-2011 08:49 PM
Hi Luca
When qos is disabled, all traffic uses txqueue 3 weight 2. So this is kind of taildrop which indicates congestion on this link.
Whether changing etherchannel load-balance algorithm helps or not depends on the traffic src/dst. By default it uses src mac, so if all traffic that causes congestion is from same host to the same destination host there will be no much difference if you change the load-balance to src-dst-ip. But if the traffic is more sporadic , say, from same source to different destinations, changing the load-balance algorithm will load balance the traffic better.
Cheers
Lixia
08-25-2011 12:50 AM
In older versions, there were some bugs which caused the output drops counter to stay at 0, although it was dropping traffic and the correct values are visible in HW counters with the 'sh platform port-asic stats drop' command.
I would recommend to try different load-balance methods to see if one helps, you might consider trying src-dst-ip first as that one usually proves to be the best choice due to the granularity it provides, your setup might be different though, so choose the one which fits you best.
Andras
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide