08-03-2010 07:29 AM - edited 03-06-2019 12:17 PM
I've been doing some searching around and can't seem to come up with a good solid answer here. I have quite a few 2960's in a manufacturing environment that are experiencing a large number of Total Output Drops. The gigabit uplinks arn't even close to fully subscribed and if I'm lucky they run at a peak of 4 - 5 Mb on a 1Gb uplink. Something really funky is going on. The individual switchports seem to have the total output drops and the uplinks at 1Gb do not have any output drops and never peak above 20Mb over a 3 month span on a 1Gb link. Quite frustrating. Please see my counter's below, I'm definitly up for suggestions!
********# sh int fa0/10
FastEthernet0/10 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 001b.2a56.9f0a (bia 001b.2a56.9f0a)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, media type is 10/100BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 11119416
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 113000 bits/sec, 43 packets/sec
5 minute output rate 62000 bits/sec, 44 packets/sec
9994159 packets input, 2559801911 bytes, 0 no buffer
Received 13426 broadcasts (698 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 698 multicast, 0 pause input
0 input packets with dribble condition detected
17250074 packets output, 2754594875 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
********#sh int gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 001b.2a56.9f31 (bia 001b.2a56.9f31)
Description: Connection to CSW1 Port G6/14
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 224000 bits/sec, 149 packets/sec
5 minute output rate 243000 bits/sec, 130 packets/sec
329488291 packets input, 98533510592 bytes, 0 no buffer
Received 95771995 broadcasts (73824483 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 73824483 multicast, 0 pause input
0 input packets with dribble condition detected
223115516 packets output, 43537654594 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out
LC_F1_2960_2#show controllers utilization
Port Receive Utilization Transmit Utilization
Fa0/1 0 0
Fa0/2 0 0
Fa0/3 0 0
Fa0/4 0 0
Fa0/5 0 0
Fa0/6 0 0
Fa0/7 0 0
Fa0/8 0 0
Fa0/9 0 0
Fa0/10 0 0
Fa0/11 0 0
Fa0/12 0 0
Fa0/13 0 0
Fa0/14 0 0
Fa0/15 0 0
Fa0/16 0 0
Fa0/17 0 0
Fa0/18 0 0
Fa0/19 0 0
Fa0/20 0 0
Fa0/21 0 0
Fa0/22 0 0
Fa0/23 0 0
Fa0/24 0 0
Fa0/25 0 0
Fa0/26 0 0
Fa0/27 0 0
Fa0/28 0 0
Fa0/29 0 0
Fa0/30 0 0
Fa0/31 0 0
Fa0/32 0 0
Fa0/33 0 0
Fa0/34 0 0
Fa0/35 0 0
Fa0/36 0 0
Fa0/37 0 0
Fa0/38 0 0
Fa0/39 0 0
Fa0/40 0 0
Fa0/41 0 0
Fa0/42 0 0
Fa0/43 0 0
Fa0/44 0 0
Fa0/45 0 0
Fa0/46 0 0
Fa0/47 0 0
Fa0/48 0 0
Gi0/1 0 0
Gi0/2 0 0
Total Ports : 50
Switch Receive Bandwidth Percentage Utilization : 0
Switch Transmit Bandwidth Percentage Utilization : 0
Switch Fabric Percentage Utilization : 0
Port-asic Port Drop Statistics - Summary
========================================
RxQueue 0 Drop Stats: 0
RxQueue 1 Drop Stats: 0
RxQueue 2 Drop Stats: 0
RxQueue 3 Drop Stats: 0
Port 0 TxQueue Drop Stats: 0
Port 1 TxQueue Drop Stats: 0
Port 2 TxQueue Drop Stats: 0
Port 3 TxQueue Drop Stats: 0
Port 4 TxQueue Drop Stats: 5734547
Port 5 TxQueue Drop Stats: 0
Port 6 TxQueue Drop Stats: 5733888
Port 7 TxQueue Drop Stats: 0
Port 8 TxQueue Drop Stats: 5733653
Port 9 TxQueue Drop Stats: 12621701
Port 10 TxQueue Drop Stats: 12621214
Port 11 TxQueue Drop Stats: 12622766
Port 12 TxQueue Drop Stats: 5735188
Port 13 TxQueue Drop Stats: 5734667
Port 14 TxQueue Drop Stats: 11119214
Port 15 TxQueue Drop Stats: 0
Port 16 TxQueue Drop Stats: 0
Port 17 TxQueue Drop Stats: 5733365
Port 18 TxQueue Drop Stats: 0
Port 19 TxQueue Drop Stats: 12782012
Port 20 TxQueue Drop Stats: 0
Port 21 TxQueue Drop Stats: 0
Port 22 TxQueue Drop Stats: 5734815
Port 23 TxQueue Drop Stats: 5733704
Port 24 TxQueue Drop Stats: 0
Port 25 TxQueue Drop Stats: 12621039
Supervisor TxQueue Drop Statistics
Queue 0: 0
Queue 1: 0
Queue 2: 0
Queue 3: 0
Queue 4: 0
Queue 5: 0
Queue 6: 0
Queue 7: 0
Queue 8: 0
Queue 9: 0
Queue 10: 0
Queue 11: 0
Queue 12: 0
Queue 13: 0
Queue 14: 0
Queue 15: 0
Port-asic Port Drop Statistics - Details
========================================
RxQueue Drop Statistics
Queue 0
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Queue 1
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Queue 2
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Queue 3
Weight 0 Frames: 0
Weight 1 Frames: 0
Weight 2 Frames: 0
Port 4 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 5734547
....
LC_F1_2960_2#show platform pm if-numbers
interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa0/1 1 1 1 0/2 1 1 1 local Yes Yes
Fa0/2 2 2 2 0/3 1 2 2 local Yes Yes
Fa0/3 3 3 3 0/4 1 3 3 local Yes Yes
Fa0/4 4 4 4 0/5 1 4 4 local Yes Yes
Fa0/5 5 5 5 0/6 1 5 5 local Yes Yes
Fa0/6 6 6 6 0/7 1 6 6 local Yes Yes
Fa0/7 7 7 7 0/8 1 7 7 local Yes Yes
Fa0/8 8 8 8 0/9 1 8 8 local Yes Yes
Fa0/9 9 9 9 0/10 1 9 9 local Yes Yes
Fa0/10 10 10 10 0/11 1 10 10 local Yes Yes
Fa0/11 11 11 11 0/12 1 11 11 local Yes Yes
Fa0/12 12 12 12 0/13 1 12 12 local Yes Yes
Fa0/13 13 13 13 0/14 1 13 13 local Yes Yes
Fa0/14 14 14 14 0/15 1 14 14 local Yes Yes
Fa0/15 15 15 15 0/16 1 15 15 local Yes Yes
Fa0/16 16 16 16 0/17 1 16 16 local Yes Yes
Fa0/17 17 17 17 0/18 1 17 17 local Yes Yes
Fa0/18 18 18 18 0/19 1 18 18 local Yes Yes
Fa0/19 19 19 19 0/20 1 19 19 local Yes Yes
Fa0/20 20 20 20 0/21 1 20 20 local Yes Yes
Fa0/21 21 21 21 0/22 1 21 21 local Yes Yes
interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa0/22 22 22 22 0/23 1 22 22 local Yes Yes
Fa0/23 23 23 23 0/24 1 23 23 local Yes Yes
Fa0/24 24 24 24 0/25 1 24 24 local Yes Yes
Fa0/25 25 25 25 1/2 1 25 25 local Yes Yes
Fa0/26 26 26 26 1/3 1 26 26 local Yes Yes
Fa0/27 27 27 27 1/4 1 27 27 local Yes Yes
Fa0/28 28 28 28 1/5 1 28 28 local Yes Yes
Fa0/29 29 29 29 1/6 1 29 29 local Yes Yes
Fa0/30 30 30 30 1/7 1 30 30 local Yes Yes
Fa0/31 31 31 31 1/8 1 31 31 local Yes Yes
Fa0/32 32 32 32 1/9 1 32 32 local Yes Yes
Fa0/33 33 33 33 1/10 1 33 33 local Yes Yes
Fa0/34 34 34 34 1/11 1 34 34 local Yes Yes
Fa0/35 35 35 35 1/12 1 35 35 local Yes Yes
Fa0/36 36 36 36 1/13 1 36 36 local Yes Yes
Fa0/37 37 37 37 1/14 1 37 37 local Yes Yes
Fa0/38 38 38 38 1/15 1 38 38 local Yes Yes
Fa0/39 39 39 39 1/16 1 39 39 local Yes Yes
Fa0/40 40 40 40 1/17 1 40 40 local Yes Yes
Fa0/41 41 41 41 1/18 1 41 41 local Yes Yes
Fa0/42 42 42 42 1/19 1 42 42 local Yes Yes
Fa0/43 43 43 43 1/20 1 43 43 local Yes Yes
Fa0/44 44 44 44 1/21 1 44 44 local Yes Yes
Fa0/45 45 45 45 1/22 1 45 45 local Yes Yes
interface gid gpn lpn port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa0/46 46 46 46 1/23 1 46 46 local Yes Yes
Fa0/47 47 47 47 1/24 1 47 47 local Yes Yes
Fa0/48 48 48 48 1/25 1 48 48 local Yes Yes
Gi0/1 49 49 49 1/1 1 1 49 local Yes Yes
Gi0/2 50 50 50 1/0 1 2 50 local Yes Yes
St1 104 104 0 16/0 1 0 0 internal Yes Yes
10-26-2010 05:00 PM
I am very curious to know why a 100% correct post by Ian above has been low-rated as 2 stars.
However I do not think the responsible person will come forward and much less with a rational explaination.
I have rated 5 the post above, as that is what is worth.
10-26-2010 05:59 PM
I am very curious to know why a 100% correct post by Ian above has been low-rated as 2 stars.
Probably because the OP didn't like the answer (even though it's correct).03-31-2011 02:13 PM
Everyone, please excuse my hijacking this thread.
Ian, I manage a largish-network. We have a Zimbra email system, and at times that correlate with a poor user experience (slowness, timeouts), the interfaces to which those servers connect experence bursts of TXOutputQdrops. (I see these both in the IOS command-line stats and in our Zenoss monitoring system.) They are connected to a 3560 running Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(46)SE, RELEASE SOFTWARE (fc2). 5-minute traffic stats show pretty tame traffic - 30-100 Mbps on a Gig interface.
What puzzles me is how QoS could possibly help. The traffic, depending on the server (MTA or user GUI interface) is either all port 25 or all port 443. So, how could I prioritize packets to stop the TXOQDs?
Also, I'm still confused as to if there can be any culpability of the server. If it cannot "keep up", could that cause the TXOQD counter to increment? I would think not, but as I said, I am confused.
A few stats:
ECB-3560G-1A.6#sh platform port-asic stats drop gi0/19
Interface Gi0/19 TxQueue Drop Statistics
Queue 0
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 1
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 2
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 0
Queue 3
Weight 0 Frames 0
Weight 1 Frames 0
Weight 2 Frames 787
ECB-3560G-1A.6#sh controllers ethernet-controller gi0/17
Transmit GigabitEthernet0/17 Receive
2648991964 Bytes 4025987434 Bytes
85242420 Unicast frames 1207153645 Unicast frames
6422004 Multicast frames 66 Multicast frames
10878354 Broadcast frames 14058 Broadcast frames
0 Too old frames 4025074428 Unicast bytes
0 Deferred frames 13294 Multicast bytes
0 MTU exceeded frames 899712 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 2230358639 Minimum size frames
0 8 collision frames 432550698 65 to 127 byte frames
0 9 collision frames 1276924028 128 to 255 byte frames
0 10 collision frames 541007290 256 to 511 byte frames
0 11 collision frames 193595908 512 to 1023 byte frames
0 12 collision frames 827698502 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
2072121016 64 byte frames 0 Valid frames, too small
512618292 127 byte frames
1112788246 255 byte frames 0 Too old frames
330206005 511 byte frames 0 Valid oversize frames
348277996 1023 byte frames 0 System FCS error frames
21498519 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames
ECB-3560G-1A.6#sh controllers ethernet-controller gi0/19
Transmit GigabitEthernet0/19 Receive
19327170 Bytes 2804425779 Bytes
606210102 Unicast frames 887334761 Unicast frames
6422512 Multicast frames 181 Multicast frames
10875562 Broadcast frames 17700 Broadcast frames
0 Too old frames 2803255761 Unicast bytes
0 Deferred frames 37218 Multicast bytes
0 MTU exceeded frames 1132800 Broadcast bytes
0 1 collision frames 0 Alignment errors
0 2 collision frames 0 FCS errors
0 3 collision frames 0 Oversize frames
0 4 collision frames 0 Undersize frames
0 5 collision frames 0 Collision fragments
0 6 collision frames
0 7 collision frames 872201 Minimum size frames
0 8 collision frames 272164604 65 to 127 byte frames
0 9 collision frames 43565501 128 to 255 byte frames
0 10 collision frames 107116223 256 to 511 byte frames
0 11 collision frames 31669377 512 to 1023 byte frames
0 12 collision frames 431964736 1024 to 1518 byte frames
0 13 collision frames 0 Overrun frames
0 14 collision frames 0 Pause frames
0 15 collision frames
0 Excessive collisions 0 Symbol error frames
0 Late collisions 0 Invalid frames, too large
0 VLAN discard frames 0 Valid frames, too large
0 Excess defer frames 0 Invalid frames, too small
11558448 64 byte frames 0 Valid frames, too small
174269841 127 byte frames
172657955 255 byte frames 0 Too old frames
24708751 511 byte frames 0 Valid oversize frames
22087049 1023 byte frames 0 System FCS error frames
218226132 1518 byte frames 0 RxPortFifoFull drop frame
0 Too large frames
0 Good (1 coll) frames
0 Good (>1 coll) frames
And a snip of thesh controllers ethernet-controller port-asic statistics outpit
...
Switch 1, PortASIC 9 Statistics
---------------------------------------------------------------------------
0 RxQ-0, wt-0 enqueue frames 0 RxQ-0, wt-0 drop frames
52 RxQ-0, wt-1 enqueue frames 0 RxQ-0, wt-1 drop frames
0 RxQ-0, wt-2 enqueue frames 0 RxQ-0, wt-2 drop frames
0 RxQ-1, wt-0 enqueue frames 0 RxQ-1, wt-0 drop frames
0 RxQ-1, wt-1 enqueue frames 0 RxQ-1, wt-1 drop frames
0 RxQ-1, wt-2 enqueue frames 0 RxQ-1, wt-2 drop frames
0 RxQ-2, wt-0 enqueue frames 0 RxQ-2, wt-0 drop frames
0 RxQ-2, wt-1 enqueue frames 0 RxQ-2, wt-1 drop frames
3009663797 RxQ-2, wt-2 enqueue frames 0 RxQ-2, wt-2 drop frames
0 RxQ-3, wt-0 enqueue frames 0 RxQ-3, wt-0 drop frames
0 RxQ-3, wt-1 enqueue frames 0 RxQ-3, wt-1 drop frames
0 RxQ-3, wt-2 enqueue frames 0 RxQ-3, wt-2 drop frames
0 TxBufferFull Drop Count 0 Rx Fcs Error Frames
0 TxBufferFrameDesc BadCrc16 0 Rx Invalid Oversize Frames
0 TxBuffer Bandwidth Drop Cou 0 Rx Invalid Too Large Frames
0 TxQueue Bandwidth Drop Coun 0 Rx Invalid Too Large Frames
0 TxQueue Missed Drop Statist 0 Rx Invalid Too Small Frames
122 RxBuffer Drop DestIndex Cou 0 Rx Too Old Frames
0 SneakQueue Drop Count 0 Tx Too Old Frames
0 Learning Queue Overflow Fra 0 System Fcs Error Frames
0 Learning Cam Skip Count
0 Sup Queue 0 Drop Frames 0 Sup Queue 8 Drop Frames
0 Sup Queue 1 Drop Frames 0 Sup Queue 9 Drop Frames
0 Sup Queue 2 Drop Frames 0 Sup Queue 10 Drop Frames
0 Sup Queue 3 Drop Frames 0 Sup Queue 11 Drop Frames
0 Sup Queue 4 Drop Frames 0 Sup Queue 12 Drop Frames
0 Sup Queue 5 Drop Frames 0 Sup Queue 13 Drop Frames
0 Sup Queue 6 Drop Frames 0 Sup Queue 14 Drop Frames
0 Sup Queue 7 Drop Frames 0 Sup Queue 15 Drop Frames
===========================================================================
I'm in over my head here, and help greatly appreciated.
Thanks!
Steve
05-25-2012 08:48 AM
Richard,
Did you ever resolve this issue or find a solution? We are seeing the exact same behvaior described here. We are running multiple WS-C2960S-48TD-L switches with the latest maintenance release, 12.2(58)SE2.
11-29-2012 01:10 AM
Hello!
I just confirm, that we have similar problem on stacked switch platform WS-C3750G-24TS-1U & WS-C3750E-24TD with release 12.2(58)SE2. We did update IOS version recently. We didn't saw this problem on previouse version,
Do you have any solution for this issue? Is it bug?
11-29-2012 01:19 AM
We also had this issue.
If you have a test-bed; switch of mls qos in global mode.
Then check if your performance improves drastically. If so, you wil be in the same issue as we had.
We later finetuned mls to have mls without high drops.
11-29-2012 04:37 AM
We use the same mls qos config as on previous version 12.2(55)SE.
Moreover we do not see such statistic in output of command "sh mls qos interface statistics".
Command "sh interface" shows 3-4 Mega packets per second, when traffic on interface less then 1Gbps and rate no change after update.
I think this bug!
11-29-2012 05:40 AM
We also had great performance on 1000BT interfaces , but lousy performance on 100BT connections.
This is happening due to the much bigger buffers on 1Gig interfaces (when using mls qos).
So try : no mls qos (on a testbed)
11-29-2012 10:49 AM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
2960 uses the same QoS architecture as 3560 and 3750, hence the following excellent guide is very relevant and applicable:http://cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a0080883f9e.shtml
Another excellent reference: https://supportforums.cisco.com/docs/DOC-8093
02-03-2013 07:55 AM
Hi,
maybe the main problem is due to ingress vs. egress speed difference.
Your ingress is 1Gbps, egress ist 100 Mbps ?
Have a look at CSCte64832.
I found it on release notes
I think it is a overall problem, when traffic bursts are on the network with ingress/egress speed mismatch.
I also have the problem with citrix traffic and software distribution like SCCM.
We lowered the drops with queue threshold increases, but we used higher thresholds than 2000. It's seems like experimenting in the life network.
The other way is to use 1Gbps Client ports, wie tried it (Cat2960S + Gigabit Endsystems) but could not role out in the field because of usage of 100 Mbps IP-phone.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide