cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
59194
Views
65
Helpful
24
Replies

2960 High Output Drops

rhornberger
Level 1
Level 1

I've been doing some searching around and can't seem to come up with a good solid answer here.  I have quite a few 2960's in a manufacturing environment that are experiencing a large number of Total Output Drops.  The gigabit uplinks arn't even close to fully subscribed and if I'm lucky they run at a peak of 4 - 5 Mb on a 1Gb uplink.  Something really funky is going on.  The individual switchports seem to have the total output drops and the uplinks at 1Gb do not have any output drops and never peak above 20Mb over a 3 month span on a 1Gb link.  Quite frustrating.  Please see my counter's below, I'm definitly up for suggestions!

********#  sh int fa0/10
FastEthernet0/10 is up, line protocol is up (connected)
  Hardware is Fast Ethernet, address is 001b.2a56.9f0a (bia 001b.2a56.9f0a)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100BaseTX
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 11119416
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 113000 bits/sec, 43 packets/sec
  5 minute output rate 62000 bits/sec, 44 packets/sec
     9994159 packets input, 2559801911 bytes, 0 no buffer
     Received 13426 broadcasts (698 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 698 multicast, 0 pause input
     0 input packets with dribble condition detected
     17250074 packets output, 2754594875 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

********#sh int gi0/1
GigabitEthernet0/1 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 001b.2a56.9f31 (bia 001b.2a56.9f31)
  Description: Connection to CSW1 Port G6/14
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
  input flow-control is off, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:01, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 224000 bits/sec, 149 packets/sec
  5 minute output rate 243000 bits/sec, 130 packets/sec
     329488291 packets input, 98533510592 bytes, 0 no buffer
     Received 95771995 broadcasts (73824483 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 73824483 multicast, 0 pause input
     0 input packets with dribble condition detected
     223115516 packets output, 43537654594 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 PAUSE output
     0 output buffer failures, 0 output buffers swapped out

LC_F1_2960_2#show controllers utilization
Port       Receive Utilization  Transmit Utilization
Fa0/1              0                    0
Fa0/2              0                    0
Fa0/3              0                    0
Fa0/4              0                    0
Fa0/5              0                    0
Fa0/6              0                    0
Fa0/7              0                    0
Fa0/8              0                    0
Fa0/9              0                    0
Fa0/10             0                    0
Fa0/11             0                    0
Fa0/12             0                    0
Fa0/13             0                    0
Fa0/14             0                    0
Fa0/15             0                    0
Fa0/16             0                    0
Fa0/17             0                    0
Fa0/18             0                    0
Fa0/19             0                    0
Fa0/20             0                    0
Fa0/21             0                    0
Fa0/22             0                    0
Fa0/23             0                    0
Fa0/24             0                    0
Fa0/25             0                    0
Fa0/26             0                    0
Fa0/27             0                    0
Fa0/28             0                    0
Fa0/29             0                    0
Fa0/30             0                    0
Fa0/31             0                    0
Fa0/32             0                    0
Fa0/33             0                    0
Fa0/34             0                    0
Fa0/35             0                    0
Fa0/36             0                    0
Fa0/37             0                    0
Fa0/38             0                    0
Fa0/39             0                    0
Fa0/40             0                    0
Fa0/41             0                    0
Fa0/42             0                    0
Fa0/43             0                    0
Fa0/44             0                    0
Fa0/45             0                    0
Fa0/46             0                    0
Fa0/47             0                    0
Fa0/48             0                    0
Gi0/1              0                    0
Gi0/2              0                    0

Total Ports : 50
Switch Receive Bandwidth Percentage Utilization  : 0
Switch Transmit Bandwidth Percentage Utilization : 0

Switch Fabric Percentage Utilization : 0

Port-asic Port Drop Statistics - Summary
========================================
  RxQueue 0 Drop Stats: 0
  RxQueue 1 Drop Stats: 0
  RxQueue 2 Drop Stats: 0
  RxQueue 3 Drop Stats: 0

  Port  0 TxQueue Drop Stats: 0
  Port  1 TxQueue Drop Stats: 0
  Port  2 TxQueue Drop Stats: 0
  Port  3 TxQueue Drop Stats: 0
  Port  4 TxQueue Drop Stats: 5734547
  Port  5 TxQueue Drop Stats: 0
  Port  6 TxQueue Drop Stats: 5733888
  Port  7 TxQueue Drop Stats: 0
  Port  8 TxQueue Drop Stats: 5733653
  Port  9 TxQueue Drop Stats: 12621701
  Port 10 TxQueue Drop Stats: 12621214
  Port 11 TxQueue Drop Stats: 12622766
  Port 12 TxQueue Drop Stats: 5735188
  Port 13 TxQueue Drop Stats: 5734667
  Port 14 TxQueue Drop Stats: 11119214
  Port 15 TxQueue Drop Stats: 0

  Port 16 TxQueue Drop Stats: 0
  Port 17 TxQueue Drop Stats: 5733365
  Port 18 TxQueue Drop Stats: 0
  Port 19 TxQueue Drop Stats: 12782012
  Port 20 TxQueue Drop Stats: 0
  Port 21 TxQueue Drop Stats: 0
  Port 22 TxQueue Drop Stats: 5734815
  Port 23 TxQueue Drop Stats: 5733704
  Port 24 TxQueue Drop Stats: 0
  Port 25 TxQueue Drop Stats: 12621039

  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 0
    Queue  9: 0
    Queue 10: 0
    Queue 11: 0

    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

Port-asic Port Drop Statistics - Details
========================================
  RxQueue Drop Statistics
    Queue 0
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 1
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 2
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0
    Queue 3
    Weight 0 Frames: 0
    Weight 1 Frames: 0
    Weight 2 Frames: 0

  Port 4 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 5734547
....

LC_F1_2960_2#show platform pm if-numbers

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa0/1     1    1    1    0/2  1    1    1    local     Yes     Yes
Fa0/2     2    2    2    0/3  1    2    2    local     Yes     Yes
Fa0/3     3    3    3    0/4  1    3    3    local     Yes     Yes
Fa0/4     4    4    4    0/5  1    4    4    local     Yes     Yes
Fa0/5     5    5    5    0/6  1    5    5    local     Yes     Yes
Fa0/6     6    6    6    0/7  1    6    6    local     Yes     Yes
Fa0/7     7    7    7    0/8  1    7    7    local     Yes     Yes
Fa0/8     8    8    8    0/9  1    8    8    local     Yes     Yes
Fa0/9     9    9    9    0/10 1    9    9    local     Yes     Yes
Fa0/10    10   10   10   0/11 1    10   10   local     Yes     Yes
Fa0/11    11   11   11   0/12 1    11   11   local     Yes     Yes
Fa0/12    12   12   12   0/13 1    12   12   local     Yes     Yes
Fa0/13    13   13   13   0/14 1    13   13   local     Yes     Yes
Fa0/14    14   14   14   0/15 1    14   14   local     Yes     Yes
Fa0/15    15   15   15   0/16 1    15   15   local     Yes     Yes
Fa0/16    16   16   16   0/17 1    16   16   local     Yes     Yes
Fa0/17    17   17   17   0/18 1    17   17   local     Yes     Yes
Fa0/18    18   18   18   0/19 1    18   18   local     Yes     Yes
Fa0/19    19   19   19   0/20 1    19   19   local     Yes     Yes
Fa0/20    20   20   20   0/21 1    20   20   local     Yes     Yes
Fa0/21    21   21   21   0/22 1    21   21   local     Yes     Yes

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa0/22    22   22   22   0/23 1    22   22   local     Yes     Yes
Fa0/23    23   23   23   0/24 1    23   23   local     Yes     Yes
Fa0/24    24   24   24   0/25 1    24   24   local     Yes     Yes
Fa0/25    25   25   25   1/2  1    25   25   local     Yes     Yes
Fa0/26    26   26   26   1/3  1    26   26   local     Yes     Yes
Fa0/27    27   27   27   1/4  1    27   27   local     Yes     Yes
Fa0/28    28   28   28   1/5  1    28   28   local     Yes     Yes
Fa0/29    29   29   29   1/6  1    29   29   local     Yes     Yes
Fa0/30    30   30   30   1/7  1    30   30   local     Yes     Yes
Fa0/31    31   31   31   1/8  1    31   31   local     Yes     Yes
Fa0/32    32   32   32   1/9  1    32   32   local     Yes     Yes
Fa0/33    33   33   33   1/10 1    33   33   local     Yes     Yes
Fa0/34    34   34   34   1/11 1    34   34   local     Yes     Yes
Fa0/35    35   35   35   1/12 1    35   35   local     Yes     Yes
Fa0/36    36   36   36   1/13 1    36   36   local     Yes     Yes
Fa0/37    37   37   37   1/14 1    37   37   local     Yes     Yes
Fa0/38    38   38   38   1/15 1    38   38   local     Yes     Yes
Fa0/39    39   39   39   1/16 1    39   39   local     Yes     Yes
Fa0/40    40   40   40   1/17 1    40   40   local     Yes     Yes
Fa0/41    41   41   41   1/18 1    41   41   local     Yes     Yes
Fa0/42    42   42   42   1/19 1    42   42   local     Yes     Yes
Fa0/43    43   43   43   1/20 1    43   43   local     Yes     Yes
Fa0/44    44   44   44   1/21 1    44   44   local     Yes     Yes
Fa0/45    45   45   45   1/22 1    45   45   local     Yes     Yes

interface gid  gpn  lpn  port slot unit slun port-type lpn-idb gpn-idb
----------------------------------------------------------------------
Fa0/46    46   46   46   1/23 1    46   46   local     Yes     Yes
Fa0/47    47   47   47   1/24 1    47   47   local     Yes     Yes
Fa0/48    48   48   48   1/25 1    48   48   local     Yes     Yes
Gi0/1     49   49   49   1/1  1    1    49   local     Yes     Yes
Gi0/2     50   50   50   1/0  1    2    50   local     Yes     Yes
St1       104  104  0    16/0  1    0    0    internal  Yes     Yes

24 Replies 24

I am very curious to know why a 100% correct post by Ian above has been low-rated as 2 stars.

However I do not think the responsible person will come forward and much less with a rational explaination.

I have rated 5 the post above, as that is what is worth.

I am very curious to know why a 100% correct post by Ian above has been low-rated as 2 stars.

Probably because the OP didn't like the answer (even though it's correct).

Everyone, please excuse my hijacking this thread.

Ian, I manage a largish-network. We have a Zimbra email system, and at times that correlate with a poor user experience (slowness, timeouts), the interfaces to which those servers connect experence bursts of TXOutputQdrops. (I see these both in the IOS command-line stats and in our Zenoss monitoring system.) They are connected to a 3560 running Cisco IOS Software, C3560 Software (C3560-IPBASEK9-M), Version 12.2(46)SE, RELEASE SOFTWARE (fc2). 5-minute traffic stats show pretty tame traffic - 30-100 Mbps on a Gig interface.

What puzzles me is how QoS could possibly help. The traffic, depending on the server (MTA or user GUI interface) is either all port 25 or all port 443. So, how could I prioritize packets to stop the TXOQDs?

Also, I'm still confused as to if there can be any culpability of the server. If it cannot "keep up", could that cause the TXOQD counter to increment? I would think not, but as I said, I am confused.

A few stats:

ECB-3560G-1A.6#sh platform port-asic stats drop gi0/19

  Interface Gi0/19 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 787


ECB-3560G-1A.6#sh controllers ethernet-controller gi0/17

     Transmit GigabitEthernet0/17             Receive
   2648991964 Bytes                       4025987434 Bytes
     85242420 Unicast frames              1207153645 Unicast frames
      6422004 Multicast frames                    66 Multicast frames
     10878354 Broadcast frames                 14058 Broadcast frames
            0 Too old frames              4025074428 Unicast bytes
            0 Deferred frames                  13294 Multicast bytes
            0 MTU exceeded frames             899712 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames          2230358639 Minimum size frames
            0 8 collision frames           432550698 65 to 127 byte frames
            0 9 collision frames          1276924028 128 to 255 byte frames
            0 10 collision frames          541007290 256 to 511 byte frames
            0 11 collision frames          193595908 512 to 1023 byte frames
            0 12 collision frames          827698502 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
   2072121016 64 byte frames                       0 Valid frames, too small
    512618292 127 byte frames
   1112788246 255 byte frames                      0 Too old frames
    330206005 511 byte frames                      0 Valid oversize frames
    348277996 1023 byte frames                     0 System FCS error frames
     21498519 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

ECB-3560G-1A.6#sh controllers ethernet-controller gi0/19

     Transmit GigabitEthernet0/19             Receive
     19327170 Bytes                       2804425779 Bytes
    606210102 Unicast frames               887334761 Unicast frames
      6422512 Multicast frames                   181 Multicast frames
     10875562 Broadcast frames                 17700 Broadcast frames
            0 Too old frames              2803255761 Unicast bytes
            0 Deferred frames                  37218 Multicast bytes
            0 MTU exceeded frames            1132800 Broadcast bytes
            0 1 collision frames                   0 Alignment errors
            0 2 collision frames                   0 FCS errors
            0 3 collision frames                   0 Oversize frames
            0 4 collision frames                   0 Undersize frames
            0 5 collision frames                   0 Collision fragments
            0 6 collision frames
            0 7 collision frames              872201 Minimum size frames
            0 8 collision frames           272164604 65 to 127 byte frames
            0 9 collision frames            43565501 128 to 255 byte frames
            0 10 collision frames          107116223 256 to 511 byte frames
            0 11 collision frames           31669377 512 to 1023 byte frames
            0 12 collision frames          431964736 1024 to 1518 byte frames
            0 13 collision frames                  0 Overrun frames
            0 14 collision frames                  0 Pause frames
            0 15 collision frames
            0 Excessive collisions                 0 Symbol error frames
            0 Late collisions                      0 Invalid frames, too large
            0 VLAN discard frames                  0 Valid frames, too large
            0 Excess defer frames                  0 Invalid frames, too small
     11558448 64 byte frames                       0 Valid frames, too small
    174269841 127 byte frames
    172657955 255 byte frames                      0 Too old frames
     24708751 511 byte frames                      0 Valid oversize frames
     22087049 1023 byte frames                     0 System FCS error frames
    218226132 1518 byte frames                     0 RxPortFifoFull drop frame
            0 Too large frames
            0 Good (1 coll) frames
            0 Good (>1 coll) frames

And a snip of thesh controllers ethernet-controller port-asic statistics outpit

...

Switch 1, PortASIC 9 Statistics
---------------------------------------------------------------------------
         0 RxQ-0, wt-0 enqueue frames            0 RxQ-0, wt-0 drop frames
        52 RxQ-0, wt-1 enqueue frames            0 RxQ-0, wt-1 drop frames
         0 RxQ-0, wt-2 enqueue frames            0 RxQ-0, wt-2 drop frames

         0 RxQ-1, wt-0 enqueue frames            0 RxQ-1, wt-0 drop frames
         0 RxQ-1, wt-1 enqueue frames            0 RxQ-1, wt-1 drop frames
         0 RxQ-1, wt-2 enqueue frames            0 RxQ-1, wt-2 drop frames

         0 RxQ-2, wt-0 enqueue frames            0 RxQ-2, wt-0 drop frames
         0 RxQ-2, wt-1 enqueue frames            0 RxQ-2, wt-1 drop frames
3009663797 RxQ-2, wt-2 enqueue frames            0 RxQ-2, wt-2 drop frames

         0 RxQ-3, wt-0 enqueue frames            0 RxQ-3, wt-0 drop frames
         0 RxQ-3, wt-1 enqueue frames            0 RxQ-3, wt-1 drop frames
         0 RxQ-3, wt-2 enqueue frames            0 RxQ-3, wt-2 drop frames


         0 TxBufferFull Drop Count               0 Rx Fcs Error Frames
         0 TxBufferFrameDesc BadCrc16            0 Rx Invalid Oversize Frames
         0 TxBuffer Bandwidth Drop Cou           0 Rx Invalid Too Large Frames
         0 TxQueue Bandwidth Drop Coun           0 Rx Invalid Too Large Frames
         0 TxQueue Missed Drop Statist           0 Rx Invalid Too Small Frames
       122 RxBuffer Drop DestIndex Cou           0 Rx Too Old Frames
         0 SneakQueue Drop Count                 0 Tx Too Old Frames
         0 Learning Queue Overflow Fra           0 System Fcs Error Frames
         0 Learning Cam Skip Count

         0 Sup Queue 0 Drop Frames               0 Sup Queue 8 Drop Frames
         0 Sup Queue 1 Drop Frames               0 Sup Queue 9 Drop Frames
         0 Sup Queue 2 Drop Frames               0 Sup Queue 10 Drop Frames
         0 Sup Queue 3 Drop Frames               0 Sup Queue 11 Drop Frames
         0 Sup Queue 4 Drop Frames               0 Sup Queue 12 Drop Frames
         0 Sup Queue 5 Drop Frames               0 Sup Queue 13 Drop Frames
         0 Sup Queue 6 Drop Frames               0 Sup Queue 14 Drop Frames
         0 Sup Queue 7 Drop Frames               0 Sup Queue 15 Drop Frames
===========================================================================

I'm in over my head here, and help greatly appreciated.

Thanks!

Steve

Richard,

Did you ever resolve this issue or find a solution?  We are seeing the exact same behvaior described here.  We are running multiple WS-C2960S-48TD-L switches with the latest maintenance release, 12.2(58)SE2.

Hello!

I just confirm, that we have similar problem on stacked switch platform WS-C3750G-24TS-1U & WS-C3750E-24TD with release 12.2(58)SE2. We did update IOS version recently. We didn't saw this problem on previouse version,

Do you have any solution for this issue? Is it bug?

We also had this issue.

If you have a test-bed; switch of mls qos in global mode.

Then check if your performance improves drastically. If so, you wil be in the same issue as we had.

We later finetuned mls to have mls without high drops.

We use the same mls qos config as on previous version 12.2(55)SE.

Moreover we do not see such statistic in output of command "sh mls qos interface statistics".

Command "sh interface" shows 3-4 Mega packets per second, when traffic on interface less then 1Gbps and rate no change after update.

I think this bug!

We also had great performance on 1000BT interfaces , but lousy performance on 100BT connections.

This is happening due to the much bigger buffers on 1Gig interfaces (when using mls qos).

So try  : no mls qos (on a testbed) 

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

2960 uses the same QoS architecture as 3560 and 3750, hence the following excellent guide is very relevant and applicable:

http://cisco.com/en/US/products/hw/switches/ps5023/products_tech_note09186a0080883f9e.shtml

Another excellent reference: https://supportforums.cisco.com/docs/DOC-8093

alois.heilmaier
Level 1
Level 1

Hi,

maybe the main problem is due to ingress vs. egress speed difference.

Your ingress is 1Gbps, egress ist 100 Mbps ?

Have a look at CSCte64832.

I found it on release notes

http://www.cisco.com/en/US/docs/switches/lan/catalyst2960/software/release/12.2_53_se/release/notes/OL22233.html

I think it is a overall problem, when traffic bursts are on the network with ingress/egress speed mismatch.

I also have the problem with citrix traffic and software distribution like SCCM.

We lowered the drops with queue threshold increases, but we used higher thresholds than 2000. It's seems like experimenting in the life network.

The other way is to use 1Gbps Client ports, wie tried it (Cat2960S + Gigabit Endsystems) but could not role out in the field because of usage of 100 Mbps IP-phone.

Review Cisco Networking for a $25 gift card