Output drops on etherchannel

LucaSalvatore_2 · ‎08-22-2011

Running a 3110X blade chassis and on alot of the etherchannels i'm seeing a lot of output drops.

running IOS version 12.2(55)SE.

here are some show commands:

BS001#show int po6

Port-channel6 is up, line protocol is up (connected)

Hardware is EtherChannel, address is e804.6288.9b06 (bia e804.6288.9b06)

Description: LACP-ServerSlot-6

MTU 1500 bytes, BW 2000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 2/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is unknown

Media-type configured as connector

input flow-control is off, output flow-control is unsupported

Members in this channel: Gi1/0/6 Gi2/0/6

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 7240

show int gi1/0/6

GigabitEthernet1/0/6 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is e804.6288.a806 (bia e804.6288.a806)

Description: Server Slot 6

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 2/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseX

Media-type configured as connector

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:23, output 00:00:00, output hang never

Last clearing of "show interface" counters 11w4d

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 9

#remote command 2 show int gi2/0/6

Switch : 2 :

------------

GigabitEthernet2/0/6 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is e804.6288.9b06 (bia e804.6288.9b06)

MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

reliability 255/255, txload 1/255, rxload 2/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseX

Media-type configured as connector

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:12, output 00:00:06, output hang never

Last clearing of "show interface" counters never

Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 7231

BS001#show platform port-asic stats drop gi1/0/6

Interface Gi1/0/6 TxQueue Drop Statistics

Queue 0

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 1

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 2

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 3

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 9

Queue 4

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 5

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 6

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 7

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

BS001# remote command 2 show platform port-asic stats drop gi2/0/6

Switch : 2 :

------------

Interface Gi2/0/6 TxQueue Drop Statistics

Queue 0

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 1

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 2

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 3

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 7231

Queue 4

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 5

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 6

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Queue 7

Weight 0 Frames 0

Weight 1 Frames 0

Weight 2 Frames 0

Any idea whats going on?

I have a similar issue with a 3750 running the same version of IOS. Never had this issue with previous versions of IOS.

QoS is disabled on this chassis.

CPU and memory are fine and utilization on the interfaces are low, monitoring shows it has never been above 100Mb.

LucaSalvatore_2 · ‎08-23-2011

anyone?

andtoth · ‎08-24-2011

Hi Luca,

Output drops are caused by lack of egress buffers on interfaces. This is usually caused by bursty traffic, high number of packets in a short period. Although the average rate might not be high, if the port tries to send out a lot of traffic in short time, buffers might be not available and port will drop traffic.

Therefore I would suggest the following to be done:

- Try doing a SPAN capture on the ports to see if you can identify short bursts of packets, either unicast, multicast or broadcast.

- Configure the etherchannel load-balance method on the switch to something more granular, such as src-dst-ip to make sure that links are utilized evenly and more efficiently, rather than one link carrying much more traffic than the other.

Understanding EtherChannel Load Balancing and Redundancy on Catalyst Switches

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

On the other hand, the output drops value does not seem very high and I would not be concerned about it if the packet loss is less than 1%.

Best regards,

Andras

LucaSalvatore_2 · ‎08-24-2011

While i understand the bufferers issue, i think something has changed in recent IOS. I have a simialar blade running an earlier version of IOS 12.2(22) and i don't see this issue. Same goes for some 3750s i have around as well.

Anyway - it does look like one link is being uses way more than the other. Will using the src-dst-ip assist with more even load balancing? What does cisco reccomend?

lixwang · ‎08-24-2011

Hi Luca

When qos is disabled, all traffic uses txqueue 3 weight 2. So this is kind of taildrop which indicates congestion on this link.

Whether changing etherchannel load-balance algorithm helps or not depends on the traffic src/dst. By default it uses src mac, so if all traffic that causes congestion is from same host to the same destination host there will be no much difference if you change the load-balance to src-dst-ip. But if the traffic is more sporadic , say, from same source to different destinations, changing the load-balance algorithm will load balance the traffic better.

Cheers

Lixia

andtoth · ‎08-25-2011

In older versions, there were some bugs which caused the output drops counter to stay at 0, although it was dropping traffic and the correct values are visible in HW counters with the 'sh platform port-asic stats drop' command.

I would recommend to try different load-balance methods to see if one helps, you might consider trying src-dst-ip first as that one usually proves to be the best choice due to the granularity it provides, your setup might be different though, so choose the one which fits you best.

Andras