2960 and 3560 high output drops ios version 12.2(53)se2

[bob brennan]

i have been experiencing a large amount of output queue drops on the 2960 and 3560 switch ports.

here are some of the things i have tried and the conclusions i have reached.

1. I changed the queuing on one of the 2960 switches numerous times to the following settings

srr-queue bandwidth share 1 75 1 25 (original setting – the 75% is for queue 1 call signaling and 25% for queue 3 best effort )

srr-queue bandwidth share 1 50 1 50

srr-queue bandwidth share 1 25 1 75

            after changing the queue setting I still experienced drops. In most cases it was less drops as the % increased for queue 3. in some cases it actually stayed the same or increased.

2. I removed qos from the switch I was testing and 2 - 3560 switches. The 3560 switches were experiencing a lot of drops as well. On some ports over a 1 hour period the drop rate was as high as 5%

After removing qos I saw no drops on the 3560 switches. But on the 2960 switch I still saw a small amount of drops on a few ports. With qos all 11 10/100 ports had drops as high as 1000 over the same time period. With qos removed only 2 of the ports experienced drops and they were a small amount 1 and 91 drops.

Once I enabled qos on the switches the drops started occurring again on all 3 switches at a rapid rate.

3. this leads me to conclude that we may have multiple issues
   1. almost all the ports experiencing drops are 100 mb ports (about 95%). Even when qos was totally removed I saw drops on some ports. I believe at this point the 100mb ports are experiencing bursts in traffic that they can’t handle and packets get dropped. We need to install 1gb cards in the clients.
   2. The current ios we are running on the switches has numerous issues, such as, high cpu utilization and output packet drop issues that have been seen by others. When removing qos completely the amount of drops were none or decreased significantly. I believe they are issues with the ios that need to be addressed by cisco. Current upgrades according to other users have not resolved the issue and a fix is not scheduled until 3/11.
   3. We will need to fine tune the qos configuration on our switches once we implement voip

i have read numerous posts on this issue. so i know i am not the only one seeing it. i contacted cisco tac about the issue and they continue to say that bursty traffic is causing all the drops. but what they can't account for is when removing qos almost all the drop go away.

has anyone else experienced this and if so were you able to fine a resolution. thank you.

i have also attached a show tech support file.

[Lei Tian]

Hi Bob,

You are welcome! I am glad that I can help.

Regards,

Lei Tian

[danny.carroll]

Upgrade your ios to 12.2(55) and let me know if that helps. It's helped others.

If it does help let me know. I have some switches with the same issue and that is what i've suggested to my boss.

https://cisco-support2.uat3.hosted.jivesoftware.com/thread/154488

[bob brennan]

hey danny,

according to the other posts i have read on this issue, the upgrade to 12.2(55) doesn't resolve the issue. according to one post i read a fix is scheduled for march 2011 in 12.2(58), according to the post. but i implemented the following qos changes recommended on this post and i am no longer seeing drops.

global changes

mls qos queue-set output 1 threshold 1 150 300 100 400
mls qos queue-set output 1 threshold 2 150 300 50 400
mls qos queue-set output 1 threshold 3 150 300 50 400
mls qos queue-set output 1 threshold 4 200 300 100 400
mls qos queue-set output 1 buffers 10 20 10 60

interface changes

srr-queue bandwidth share 10 20 10 60

hope that helps.

[danny.carroll]

Unfortunately i'm seeing drops on my supervisor.

Supervisor TxQueue Drop Statistics
    Queue  0: 37076
    Queue  1: 0
    Queue  2: 0
    Queue  3: 1071053
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 769441
    Queue  9: 0
    Queue 10: 1027404
    Queue 11: 0
    Queue 12: 0
    Queue 13: 0
    Queue 14: 38879547
    Queue 15: 0

[paolo bevilacqua]

Drops are most likely due to applications trying to transmit faster than interface speed, thing than in networking has been proven impossible so far.

[danny.carroll]

The main queue that i'm having issues is the dstats queue, which is the drop stats queue. That's all i know about it.

I'm hoping on the ios upgrade.

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc               105040525  0          0          0          0
stp               9346673    0          9343664    0          0
ipc               11447788   0          0          0          0
routing protocol  26482989   0          0          0          0
L2 protocol       547553     0          43         0          0
remote console    0          0          0          0          0
sw forwarding     0          0          0          0          0
host              2231757    0          0          0          0
broadcast         10729955   0          0          0          0
cbt-to-spt        0          0          0          0          0
igmp snooping     19845858   0          0          0          0
icmp              0          0          0          0          0
logging           0          0          0          0          0
rpf-fail          0          0          0          0          0
dstats            78486304   0          0          0          0
cpu heartbeat     46374282   0          0          0          0

Supervisor TxQueue Drop Statistics
  Queue  0: 37160
  Queue  1: 0
  Queue  2: 0
  Queue  3: 1071053
  Queue  4: 0
  Queue  5: 0
  Queue  6: 0
  Queue  7: 0
  Queue  8: 769528
  Queue  9: 0
  Queue 10: 1027404
  Queue 11: 0
  Queue 12: 0
  Queue 13: 0
  Queue 14: 38908191
  Queue 15: 0

[urbinir]

Hi

I have the same problem of packet dropped on a Cisco 3560E (10/100/1000 ports and uplink at 10GE) with IOS 12.2.(53)S2

In my scenario the 3560 has a portchannel with two GE ports connected to a server and this server does the backup of all documents,e-mail ecc ecc of the customers. I have no qos enabled so all the switch works with default value.

I notice the dropped output an interface only (the traffic was very high near 800Mb/s) while the other interace had no traffic in output so I changed the load-balance method for source-ip instead of source-mac

The situation seems to work fine because the switch started to use both the interfece but , after a while , the packet dropping started again even if the traffic was at 300/400Mb/s

It seems that I have a problem with bursty traffic

Must I enable the mls qos and srr to solve this problem?

The other server connect to the same switch haven't any packet dropped

If I configure the mls qos could I have problem with these servers?

thanks

Riccardo

[paolo bevilacqua]

As mentioned above. Drops are due to burst of traffic, are normal  and there is nothing you can do about it.

[bob brennan]

i have been working w/ cisco tac on this issue and their response has been that bursty traffic is causing the issue.

my response to that is below

1. I changed the queuing on one of the 2960 switches numerous times to the following settings

srr-queue bandwidth share 1 75 1 25 (original setting – the 75% is for queue 1 call signaling and 25% for queue 3 best effort )

srr-queue bandwidth share 1 50 1 50

srr-queue bandwidth share 1 25 1 75

            after changing the queue setting I still experienced drops. In most cases it was less drops as the % increased for queue 3. in some cases it actually stayed the same or increased.

2. I removed qos from the switch I was testing and 2 switches on the 6^th floor of hw. The switches on the 6^th floor were 3560 switches that were experiencing a lot of drops as well. On some ports over a 1 hour period the drop rate was as high as 5%

After removing qos I saw no drops on the 3560 switches in hw. But on the 2960 switch in admin I still saw a small amount of drops on a few ports. With qos all 11 10/100 ports had drops. With qos removed only 2 of the ports experienced drops and they were a small amount 1 and 91 drops.

Once I enabled qos on the switches the drops started occurring again on all 3 switches at a rapid rate.

3. this leads me to conclude that we may have multiple issues
   1. almost all the ports experiencing drops are 100 mb ports (about 99%). Even when qos was totally removed I saw drops. I believe at this point the 100mb ports are experiencing bursts in traffic that they can’t handle and packets get dropped. We need to install 1gb cards in the clients.
   2. The current ios we are running on the switches has numerous issues, such as, high cpu utilization and output packet drop issues that have been seen by others. When removing qos completely the amount of drops were none or decreased significantly. I believe they are issues with the ios that need to be addressed by cisco. Current upgrades according to other users have not resolved the issue and a fix is not scheduled until 3/11.
   3. We will need to fine tune the qos configuration on our switches once we start deploying voip.

after tuning the qos with commands i mentioned earlier and letting it run for a few days, i am seeing no drops on the 3560 switches that support only 10/100. i am still seeing a few drops on the 2960s switches that support 10/100/1000. only ports running at 10/100 are getting drops and now it's only a few like 20 or 30 as opposed to thousands of drops int the same time period.

i am going to test the new qos commands on some other 2960 switches i have that support only 10/100 and users connecting to these switches have complained about slowness and see if that helps. i will postr the results in a few days.

[bob brennan]

this is cisco tac's response to this issue

Hi Bob,

Thanks for the update!

I would like to tell you that the drops are due to traffic bursts on the interfaces.

Tweaking of threshold on the queues is a good option as it will not drop the packets early and would rather allot more buffers to the queues.

If this also does not work than you can wait till March 2011 for the new IOS release and upgrade the switch to that IOS.

Thank you for your time and understanding.

Since I am familiar with your service request, I would like to retain ownership to avoid any delays in resolving your issue.

Thanks & Regards

Sumit Sharma

Customer Support Engineer - LAN Switching

Cisco Systems, Inc. 

since i have tuned the qos settings with recommendations provided by lei on this post, i have significantly reduced the amount of drops. in some cases none at all. please see the attached file for a 24 hour time frame sample.  lei, the last recommendation for increasing the max threshold for queue 4 to 1000 worked like a charm. thank you.

at this point the qos recommendations have resolved most of my drop issues. i also spoke to users on these switches and they are also seeing a performance increase. i am still seeing some drops but i believe most of these are related to bursty traffic and possible issues with the ios (high cpu utilization). when the new ios is released i am definitely going to take a look at upgrading.

thank again for everyone's help and insight.

[urbinir]

hey danny,

according to the other posts i have read on this issue, the upgrade to 12.2(55) doesn't resolve the issue. according to one post i read a fix is scheduled for march 2011 in 12.2(58), according to the post. but i implemented the following qos changes recommended on this post and i am no longer seeing drops.

global changes

mls qos queue-set output 1 threshold 1 150 300 100 400
..................................................................................

Hi

does anyone tried the 12.2(58)SE1 to solve the problem of this post?

Please let me know a feedback..

Regards Riccardo

[Leo Laohoo]

I must be the first person to use the new 12.2(58)SE1.  So far so good.  Testing is good.  I'm now rolling it into the "wild". 

[pyandle]

Hi there, I have some 2960 switches running the LAN lite image that are also experiancing drops on queue 4 for best effort traffic when QoS is enabled.

We have upgraded the IOS to 12.2(58)SE1 but it hasn't resolved the issue.

sh mls qos int fa0/1 buffers
FastEthernet0/1
The port is mapped to qset : 1
The allocations between the queues are : 25 25 25 25

sh platform port-asic stats drop fa0/1

  Interface Fa0/1 TxQueue Drop Statistics
    Queue 0
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 1
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 2
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 0
    Queue 3
      Weight 0 Frames 0
      Weight 1 Frames 0
      Weight 2 Frames 74007

the show mls qos queue-set command does not exist.

Unfortunately the LAN Lite image doesn't look to provide the following commands in order to manually modify the buffers etc.

mls qos queue-set output 1 threshold

mls qos queue-set output 1 buffers

srr-queue bandwidth share

Are there any other options avilable in a LAN Lite image ?

[Leo Laohoo]

LAN Lite won't do QoS.

[pyandle]

As far as I'm aware basic QoS is provided in terms of egress queueing and priority.  It doesn't have auto QoS.

I need to amend the buffers defined for egress as per previous recomendations.

This is a PoE switch so is expected to carry VoIP traffic, which i wouldn't expect to be at the expense of the majority of the traffic. 

Can anyone shed some light (excuse the pun).

sh mls qos int fa0/1
FastEthernet0/1
trust state: trust cos
trust mode: trust cos
trust enabled flag: ena
COS override: dis
default COS: 0

sh mls qos int fa0/1 statistics
FastEthernet0/1 (All statistics are in packets)

  dscp: incoming
-------------------------------

  0 -  4 :      113392            0            0            0            0
  5 -  9 :           0            0            0            0            0
10 - 14 :           0            0            0            0            0
15 - 19 :           0            0            0            0            0
20 - 24 :           0            0            0            0         1418
25 - 29 :           0            0            0            0            0
30 - 34 :           0            0            0            0            0
35 - 39 :           0            0            0            0            0
40 - 44 :           0            0            0            0            0
45 - 49 :           0        16866            0            0            0
50 - 54 :           0            0            0            0            0
55 - 59 :           0            0            0            0            0
60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

  0 -  4 :      160378            0            0            0            0
  5 -  9 :           0            0            0            0            0
10 - 14 :           0            0            0            0            0
15 - 19 :           0            0            0            0            0
20 - 24 :           0            0            0            0         1105
25 - 29 :           0           74            0            0            0
30 - 34 :           0            0            0            0            0
35 - 39 :           0            0            0            0            0
40 - 44 :           0            0            0            0            0
45 - 49 :           0        16942            0         2124            0
50 - 54 :           0            0            0            0            0
55 - 59 :           0            0            0            0            0
60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------

  0 -  4 :      113901            0            0         1418            0
  5 -  7 :       16866            0            0
  cos: outgoing
-------------------------------

  0 -  4 :      228734            0            0            0            0
  5 -  7 :           0          575            7
  output queues enqueued:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           2           0           0
queue 1:           0       10433       12542
queue 2:           0           0           0
queue 3:           0           0      218801

  output queues dropped:
queue:    threshold1   threshold2   threshold3
-----------------------------------------------
queue 0:           0           0           0
queue 1:           0           0           0
queue 2:           0           0           0
queue 3:           0           0        2657

sh ver
Cisco IOS Software, C2960 Software (C2960-LANLITEK9-M), Version 12.2(58)SE1, RELEASE SOFTWARE (fc1)