cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2876
Views
0
Helpful
12
Replies

WS-C2960S-24TS-L - Output Drops when Low Utilization

coredialeng
Level 1
Level 1

Hi,

I'm trying to troubleshoot output drops on an uplink interface. Here is the situation:

  • Model: WS-C2960S-24TS-L
  • 1 GE uplink, trunking ( gi1/0/23)
  • QoS is not enabled ( no mls qos )
  • Link utilization for the uplink is never more than 20%
  • The output of "show buffers" does not show incrementing "missed" counters
  • The "show platform port-asic stats misc gi1/0/23" does counter TxBufferFullDropCount is nonzero

Some output below:

as2-1-1#show platform port-asic stats miscellaneous gi1/0/23

Port-asic Misc Statistics

=====================================

    TxBufferFullDropCount               676470

    TxBufferFrameDescBadCrc16DropCountA 0

    TxBufferFrameDescBadCrc16DropCountB 0

    TxBufferBandwidthDropCount          0

    TxQueueBandwidthDropCount           0

    TxQueueMissedDropStatistics         0

    SneakQueueDropCount                 0

    RxBufferDropDestIndexCountA         0

    RxBufferDropDestIndexCountB         0

    LearningQueueOverflowFrames         1

    RxNetworkProcessorPacketCount0      0

    RxNetworkProcessorPacketCount1      0

as2-1-1#

as2-1-1#show platform port-asic stats drop gi1/0/23

  Interface Gi1/0/23 TxQueue Drop Statistics

!!! Output omitted

      Weight 0 Frames 648845

      Weight 1 Frames 0

      Weight 2 Frames 4

    Queue 2

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 3

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 26293

!!! Output omitted

as2-1-1#

Has anyone overcome this issue without QoS or replacing hardware?

12 Replies 12

Leo Laohoo
Hall of Fame
Hall of Fame

Post the following output:

1.  sh version;

2.  sh interface Gi 1/0/23; and

3.  sh controll e Gi 1/0/23

Hi Leo,

Thanks for your reply. I've included a zip file containing the outputs you requested and several graphs. There are discontinuities in the graphs because of a memory issue we have with our monitoring server ( unrelated ). I've also include the output of "show platform port-asic stat drop gi1/0/23" in the file "stat_drop.txt"

One thing I notice is that there are drops in multiple queues ( stat_drop.txt) when QoS is disabled. My understanding is that when QoS is disabled globally, there is a single queue ( i.e.e Queue 1 ).

I've done usual checks while output drops were incrementing:

  • show proc cpu sorted ==> No individual process with high utilization
  • show mem failures alloc ==> No output
  • show buffer failures ==> None listed
  • show proc cpu history ==> No spikes, nomial utilziation is ~30 - 40%
  • show buffers ==> No incrementing "missed counters"
  • Link speed and duplex match on either end

As you'll note from the graphs, the utilization on the link is really low. I've searched through the 111 published bugs on the Cisco bugtracker tool as well and found none relevant to our environment.

This seems to be a widespread issue on the 2960S platform at and before the code level I'm runing ( 12.2(55)SE2 ). Multiple forums suggested upgrading to at least 12.2(58) but others have shown this not to have solved the issue. Other forum post show implementing QoS ( or disabling) it to be a fix, however these things are not currently implemented in our environment and so do not apply.

T.I.A,

Joe

H0ly crap!  I can't read your file.

Cut and paste the outputs I've requested in the thread.

I would upgrade your IOS to 12.2(55)SE8.

Hi Leo,

Sorry for the late reply. Was off today. See below:

#### Show Ver ####

Cisco IOS Software, C2960S Software (C2960S-UNIVERSALK9-M), Version 12.2(55)SE2, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2011 by Cisco Systems, Inc.

Compiled Tue 11-Jan-11 02:23 by prod_rel_team

Image text-base: 0x00003000, data-base: 0x01B00000

ROM: Bootstrap program is Alpha board boot loader

BOOTLDR: C2960S Boot Loader (C2960S-HBOOT-M) Version 12.2(53r)SE, RELEASE SOFTWARE (fc3)

as2-1-1 uptime is 1 week, 3 days, 8 hours, 11 minutes

System returned to ROM by power-on

System restarted at 13:19:33 EDT Mon Jul 22 2013

System image file is "flash:/c2960s-universalk9-mz.122-55.SE2/c2960s-universalk9-mz.122-55.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C2960S-24TS-L (PowerPC) processor (revision C0) with 131072K bytes of memory.

Processor board ID FOC1508X3U0

Last reset from power-on

1 Virtual Ethernet interface

1 FastEthernet interface

28 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C2960S-24TS-L (PowerPC) processor (revision C0) with 131072K bytes of memory.

Processor board ID FOC1508X3U0

Last reset from power-on

1 Virtual Ethernet interface

1 FastEthernet interface

28 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : B8:BE:BF:30:1C:80

Motherboard assembly number     : 73-11910-06

Power supply part number        : 341-0328-02

Motherboard serial number       : FOC15077RQ0

Power supply serial number      : DCA1432M6W8

Model revision number           : C0

Motherboard revision number     : A0

Model number                    : WS-C2960S-24TS-L

Daughterboard assembly number   : 73-11933-04

Daughterboard serial number     : FOC150643E7

System serial number            : FOC1508X3U0

Top Assembly Part Number        : 800-30954-02

Top Assembly Revision Number    : B0

Version ID                      : V02

CLEI Code Number                : COMGG00ARB

Daughterboard revision number   : A0

Hardware Board Revision Number  : 0x01

Switch Ports Model              SW Version            SW Image                

------ ----- -----              ----------            ----------              

*    1 28    WS-C2960S-24TS-L   12.2(55)SE2           C2960S-UNIVERSALK9-M    

Configuration register is 0xF

### Show int ###

as2-1-1#show int gi1/0/23

GigabitEthernet1/0/23 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is b8be.bf30.1c97 (bia b8be.bf30.1c97)

  Description: PHY|L2|rhost:cs2-1-1|rport:gi1/1

  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,

     reliability 255/255, txload 1/255, rxload 1/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:01, output 00:00:03, output hang never

  Last clearing of "show interface" counters 06:52:22

  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 2647

  Queueing strategy: fifo

  Output queue: 0/40 (size/max)

  5 minute input rate 1659000 bits/sec, 1287 packets/sec

  5 minute output rate 2558000 bits/sec, 1198 packets/sec

     76224138 packets input, 11348240054 bytes, 0 no buffer

     Received 468086 broadcasts (114552 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 114552 multicast, 0 pause input

     0 input packets with dribble condition detected

     79685680 packets output, 36998897195 bytes, 0 underruns

     0 output errors, 0 collisions, 0 interface resets

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 PAUSE output

     0 output buffer failures, 0 output buffers swapped out

as2-1-1#

### Show controllers ###

as2-1-1#show controllers ethernet-controller gi1/0/23       

     Transmit GigabitEthernet1/0/23           Receive

   1360350908 Bytes                       4218320450 Bytes                   

   1594259627 Unicast frames              2733236679 Unicast frames          

       371647 Multicast frames               4347119 Multicast frames        

       564581 Broadcast frames              12786960 Broadcast frames        

            0 Too old frames              2854689159 Unicast bytes           

            0 Deferred frames              545250746 Multicast bytes         

            0 MTU exceeded frames          818380545 Broadcast bytes         

            0 1 collision frames                   0 Alignment errors        

            0 2 collision frames                   0 FCS errors              

            0 3 collision frames                   0 Oversize frames         

            0 4 collision frames                   0 Undersize frames        

            0 5 collision frames                   0 Collision fragments     

            0 6 collision frames      

            0 7 collision frames           201288077 Minimum size frames     

            0 8 collision frames           867680707 65 to 127 byte frames   

            0 9 collision frames           107819549 128 to 255 byte frames  

            0 10 collision frames           60471382 256 to 511 byte frames  

            0 11 collision frames           79485747 512 to 1023 byte frames 

            0 12 collision frames         1433625296 1024 to 1518 byte frames

            0 13 collision frames                  0 Overrun frames          

            0 14 collision frames                  0 Pause frames            

            0 15 collision frames     

            0 Excessive collisions                 0 Symbol error frames     

            0 Late collisions                      0 Invalid frames, too large

            0 VLAN discard frames                  0 Valid frames, too large 

            0 Excess defer frames                  0 Invalid frames, too small

     96727731 64 byte frames                       0 Valid frames, too small 

    414347781 127 byte frames         

    292401594 255 byte frames                      0 Too old frames          

    247938903 511 byte frames                      0 Valid oversize frames   

    201842506 1023 byte frames                     0 System FCS error frames 

    341937340 1518 byte frames                     0 RxPortFifoFull drop frame

            0 Too large frames        

            0 Good (1 coll) frames    

            0 Good (>1 coll) frames   

as2-1-1#

### Show platform ... ###

as2-1-1#show platform port-asic stats drop gi1/0/23

  Interface Gi1/0/23 TxQueue Drop Statistics

    Queue 0

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 1

      Weight 0 Frames 651608

      Weight 1 Frames 0

      Weight 2 Frames 4

    Queue 2

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 3

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 27867

    Queue 4

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 5

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 6

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

    Queue 7

      Weight 0 Frames 0

      Weight 1 Frames 0

      Weight 2 Frames 0

as2-1-1#

As mentioned, there is no QoS enabled globally. This is a simple layer 2 switch configuration.

Also attached the graphs.

Thanks again.

Joe

As mentioned, there is no QoS enabled globally

Ok, thanks for the output.  I see no line errors.  I see, however, "output drops".

Disable interface-level QoS and see if output drops still increments.


Can you tell me what's connected to Gi 1/0/23?

Leo,

Interface QoS is also disabled:

as2-1-1#show mls qos inter gi1/0/23

GigabitEthernet1/0/23

QoS is disabled. When QoS is enabled, following settings will be applied <<<<<<<<<<< QoS disabled

trust state: not trusted

trust mode: not trusted

trust enabled flag: ena

COS override: dis

default COS: 0

DSCP Mutation Map: Default DSCP Mutation Map

Trust device: none

qos mode: port-based

as2-1-1#

A Cisco 4948 switch is connected on the gi1/0/23 interface (in trunk mode ).

Did you disable QoS on the 4948-side too?

Good catch,

Inteface QoS is enabled on the other side of the trunk, something I hadn't known about/looked at before:

cs-2-1-1#show qos inter gi1/1

QoS is disabled globally

Port QoS is enabled

Administrative Port Trust State: 'untrusted'

Operational Port Trust State: 'dscp'

Trust device: none

Default DSCP: 0 Default CoS: 0

Appliance trust: none

Tx-Queue   Bandwidth   ShapeRate   Priority   QueueSize

             (bps)       (bps)                (packets)

  1        250000000   disabled    N/A        2336    

  2        250000000   disabled    N/A        2336    

  3        250000000   disabled    normal     2336    

  4        250000000   disabled    N/A        2336    

cs-2-1-1#

However, how would enabling QoS on the far end affect Output drops on the near end?

However, how would enabling QoS on the far end affect Output drops on the near end?

It's the opposite.  If you enable QoS in a LAN, there's a significant chance of getting output drops.

Leo Laohoo wrote:

However, how would enabling QoS on the far end affect Output drops on the near end?

It's the opposite.  If you enable QoS in a LAN, there's a significant chance of getting output drops.

That make sense if QoS is enabled on every switch in the LAN, in this case, only the upstream switch has port QoS enabled. On the downlink switch, there is no QoS and all traffic is treated equally.

I guess my question is, how would having QoS enabled on the upstream switch cause output drops on the downstream switch (e.g. for traffic egressing to the upstream switch )?

Hello,

 

Did the issue finally resolved ?
I am asking because we are facing a similar issue.

 

Regards,

Without knowing the exact specifics of your particulat problem, one of the first thinsg you could usually try is to configure 'qos queue-softmax-multiplier 1200' on your switch...

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: