Buy or Renew
Buy or Renew
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
2354
Views
5
Helpful
5
Replies

Troubleshooting output queue drops/qos drops

Go to solution
billmatthews
Level 1
Level 1

‎05-02-2013 10:11 AM - edited ‎03-07-2019 01:09 PM

We're trying to troubleshoot some performance problems on a stack of 3750.  We initially noticed lots of output queue drops (no input drops, no errors). 

I've followed this technote: http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

Eventually I also looked at qos drops, and I see lots of drops

  dscp: incoming
-------------------------------
  0 -  4 :  1457734311            0      2154834            0      2649209
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0   3196936291
 25 - 29 :           0      5003169            0            0            0
 30 - 34 :           0            0        22316            0            0
 35 - 39 :           0            1            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0   2512040185            0     74672627            0
 50 - 54 :          40            0            0            0            3
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------
  0 -  4 :   360986293           15        16189            0        83333
  5 -  9 :           0            0            0         8519            0
 10 - 14 :   195960893            0            3            8            0
 15 - 19 :           0          122            0     22750877            0
 20 - 24 :           4            0            2            2   3642595093
 25 - 29 :           0      7574890            0           22            0
 30 - 34 :           0            0    502252620            0            7
 35 - 39 :           0        47816            0            6            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0      5266457            0    181700371            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------
  0 -  4 :    54270256            0            0   3201939457        22317
  5 -  7 :  2512040177     75355489     14772028
  cos: outgoing
-------------------------------
  0 -  4 :   711773765    195969423     27760830   3650170005    502300449
  5 -  7 :  4244506266    181700371    191977167
  output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------
 queue 0:           2           0  4244506267
 queue 1:    43692439    45670782  4023848561
 queue 2:           0           0   530061259
 queue 3:         164   195969259   651039679
  output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------
 queue 0:            0            0            0
 queue 1:         2747         2794       162044
 queue 2:            0            0            0
 queue 3:           43     20903265   2849834164
Policer: Inprofile:            0 OutofProfile:            0

So that has me thinking we have a qos problem.  I believe I need to assign more bandwidth to queue 4? (read this article https://supportforums.cisco.com/community/netpro/network-infrastructure/switching/blog/2011/04/11/output-drops-due-to-qos-on-296035603750-switches)

show mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      10      10      26      54
threshold1:     138     138      36      20
threshold2:     138     138      77      50
reserved  :      92      92     100      67
maximum   :     138     400     318     400
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      16       6      17      61
threshold1:     149     118      41      42
threshold2:     149     118      68      72
reserved  :     100     100     100     100
maximum   :     149     235     272     242

But as a side note, I'm also seeing some asic drops, I'm not sure if this is related?

Port-asic Port Drop Statistics - Summary
========================================
  RxQueue Drop Statistics Slice0
   RxQueue 0 Drop Stats Slice0: 0
   RxQueue 1 Drop Stats Slice0: 0
   RxQueue 2 Drop Stats Slice0: 0
   RxQueue 3 Drop Stats Slice0: 0
  RxQueue Drop Statistics Slice1
   RxQueue 0 Drop Stats Slice1: 0
   RxQueue 1 Drop Stats Slice1: 0
   RxQueue 2 Drop Stats Slice1: 1
   RxQueue 3 Drop Stats Slice1: 0
  Port  0 TxQueue Drop Stats: 61335
  Port  1 TxQueue Drop Stats: 0
  Port  2 TxQueue Drop Stats: 0
  Port  3 TxQueue Drop Stats: 0
  Port  4 TxQueue Drop Stats: 0
  Port  5 TxQueue Drop Stats: 0
  Port  6 TxQueue Drop Stats: 0
  Port  7 TxQueue Drop Stats: 0
  Port  8 TxQueue Drop Stats: 0
  Port  9 TxQueue Drop Stats: 0
  Port 10 TxQueue Drop Stats: 0
  Port 11 TxQueue Drop Stats: 0
  Port 12 TxQueue Drop Stats: 0
  Port 13 TxQueue Drop Stats: 0
  Port 14 TxQueue Drop Stats: 69127
  Port 15 TxQueue Drop Stats: 0
  Port 16 TxQueue Drop Stats: 0
  Port 17 TxQueue Drop Stats: 0
  Port 18 TxQueue Drop Stats: 0
  Port 19 TxQueue Drop Stats: 0
  Port 20 TxQueue Drop Stats: 0
  Port 21 TxQueue Drop Stats: 0
  Port 22 TxQueue Drop Stats: 0
  Port 23 TxQueue Drop Stats: 0
  Port 24 TxQueue Drop Stats: 0
  Port 25 TxQueue Drop Stats: 0
  Port 26 TxQueue Drop Stats: 0
  Port 27 TxQueue Drop Stats: 0
  Supervisor TxQueue Drop Statistics
    Queue  0: 0
    Queue  1: 0
    Queue  2: 0
    Queue  3: 0
    Queue  4: 0
    Queue  5: 0
    Queue  6: 0
    Queue  7: 0
    Queue  8: 0
    Queue  9: 0
    Queue 10: 0
    Queue 11: 0
    Queue 12: 0
    Queue 13: 0
    Queue 14: 0
    Queue 15: 0

Are those relatd to my QoS problems?  And any suggestions on how to redistribute my queue-set?

Thanks

Bill

Labels:
0 Helpful
3 Accepted Solutions

Accepted Solutions

Go to solution
paolo bevilacqua
Hall of Fame
Hall of Fame

‎05-02-2013 12:59 PM

Remove all QoS configuration, that is totally unnecessary on LAN, and all your problems will be solved.

View solution in original post

0 Helpful

Go to solution
InayathUlla Sharieff
Cisco Employee
Cisco Employee

‎05-02-2013 05:44 PM

Hi Bill,

As Paolo suggested kindly remove the QOS and update us the result.

If you are not supposed to remove then check the following things;

Output qos drops in many networks are expected and the best solution is

to increase the line speed, which is not the feasible solution here.

Here we need to apply some solution to control output drops, based on

the trail and run method by implementing various bandwidth ratios and

threshold values for the traffic that gets affected. From the queueing

interface output all drops occurred in Queue 1. Queue 1 handles traffic

mapped with COS values 0 1. No drops are reported for other queues,

queues 1 and 3. Lets not consider Queue 4 which is a priority queue and

is always dedicated to drop sensitive traffic such voice and video.

  Packets dropped on Transmit:

    BPDU packets:  0

    queue              dropped  [cos-map]

    ---------------------------------------------

    1                        0  [0 1 ]

    2                     1477  [2 3 4 ]

    3                        0  [6 7 ]

    4                        0  [5 ]

-          Identify what type of traffics are classified for COS 2, 3,

4.

-          Suppose lets say, if more traffic are expected with a cos

value of 2, then its a best practice to dedicate one queue for that

traffic. In this case, other traffic cos 3 and 4, depending upon the

needs, they can be put to be handled by different queues.

-          Accordingly MAP the cos values to the appropriate queued ID,

to minimize drops.

-          Use "wrr-queue cos-map" command to map the appropriate cos to

a particular queue.

-          You may also try increasing the buffer ratios and increase

its wrr threshold values for a particular queue to see if that would

help.

-          Use "wrr-queue bandwidth" command to increase the bandwidth

percentage of a particular queue.

-          Use "wrr-queue queue-limit" command to increase the transmit

queue ratio.

-          Use "wrr-queue random-detect min-threshold" & w"wrr-queue

random-detect max-threshold" commands to configure the drop minimum and

maximum thresholds of a particular queue.

-          Refer the link below to know the usage of all the above

commands in detail.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note

09186a008015bf98.shtml

HTH

Regards

Inayath

Please rate all usefull posts.

View solution in original post

0 Helpful

Go to solution
Joseph W. Doherty
Hall of Fame
Hall of Fame

‎05-02-2013 05:52 PM

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Indeed what you've posted shows drops.  Your queue set parameters are (cough) "interesting".

The problem with the 2960/3560/3750 egress queuing architecture, is when you enable QoS it, by default, evenly provisions 4 egress queues with the resources that were devoted to just 1 egress queue.  This commonly results in drops, not seen before QoS was enabled.  (This is one reason Paolo is advising disabling QoS.  However, you can still have congestion on a port with the non-QoS single egress queue, and if you do, and if you have "special" traffic performance requirements, then disabling QoS might not be a good option.)

If you really need QoS, then you likely will need to also tune it for your requirements.  As your existing queue set configurations are non-default, I would presume someone has been trying to tune the QoS configuration.

Unfortunately, QoS tuning often is time consuming, as you need to monitor to determine if your are getting the performance from it your traffic requires.

Also unfortunately, understanding the queue set parameters, and how then interrelate, can be confusing (at least I had found them so).

There's another QoS document on these support forums, that has lots of good info about 3750 QoS: https://supportforums.cisco.com/docs/DOC-8093

View solution in original post

0 Helpful
5 Replies 5

Go to solution
paolo bevilacqua
Hall of Fame
Hall of Fame

‎05-02-2013 12:59 PM

Remove all QoS configuration, that is totally unnecessary on LAN, and all your problems will be solved.

0 Helpful

Go to solution
InayathUlla Sharieff
Cisco Employee
Cisco Employee

‎05-02-2013 05:44 PM

Hi Bill,

As Paolo suggested kindly remove the QOS and update us the result.

If you are not supposed to remove then check the following things;

Output qos drops in many networks are expected and the best solution is

to increase the line speed, which is not the feasible solution here.

Here we need to apply some solution to control output drops, based on

the trail and run method by implementing various bandwidth ratios and

threshold values for the traffic that gets affected. From the queueing

interface output all drops occurred in Queue 1. Queue 1 handles traffic

mapped with COS values 0 1. No drops are reported for other queues,

queues 1 and 3. Lets not consider Queue 4 which is a priority queue and

is always dedicated to drop sensitive traffic such voice and video.

  Packets dropped on Transmit:

    BPDU packets:  0

    queue              dropped  [cos-map]

    ---------------------------------------------

    1                        0  [0 1 ]

    2                     1477  [2 3 4 ]

    3                        0  [6 7 ]

    4                        0  [5 ]

-          Identify what type of traffics are classified for COS 2, 3,

4.

-          Suppose lets say, if more traffic are expected with a cos

value of 2, then its a best practice to dedicate one queue for that

traffic. In this case, other traffic cos 3 and 4, depending upon the

needs, they can be put to be handled by different queues.

-          Accordingly MAP the cos values to the appropriate queued ID,

to minimize drops.

-          Use "wrr-queue cos-map" command to map the appropriate cos to

a particular queue.

-          You may also try increasing the buffer ratios and increase

its wrr threshold values for a particular queue to see if that would

help.

-          Use "wrr-queue bandwidth" command to increase the bandwidth

percentage of a particular queue.

-          Use "wrr-queue queue-limit" command to increase the transmit

queue ratio.

-          Use "wrr-queue random-detect min-threshold" & w"wrr-queue

random-detect max-threshold" commands to configure the drop minimum and

maximum thresholds of a particular queue.

-          Refer the link below to know the usage of all the above

commands in detail.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note

09186a008015bf98.shtml

HTH

Regards

Inayath

Please rate all usefull posts.

0 Helpful

Go to solution
Joseph W. Doherty
Hall of Fame
Hall of Fame

‎05-02-2013 05:52 PM

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Indeed what you've posted shows drops.  Your queue set parameters are (cough) "interesting".

The problem with the 2960/3560/3750 egress queuing architecture, is when you enable QoS it, by default, evenly provisions 4 egress queues with the resources that were devoted to just 1 egress queue.  This commonly results in drops, not seen before QoS was enabled.  (This is one reason Paolo is advising disabling QoS.  However, you can still have congestion on a port with the non-QoS single egress queue, and if you do, and if you have "special" traffic performance requirements, then disabling QoS might not be a good option.)

If you really need QoS, then you likely will need to also tune it for your requirements.  As your existing queue set configurations are non-default, I would presume someone has been trying to tune the QoS configuration.

Unfortunately, QoS tuning often is time consuming, as you need to monitor to determine if your are getting the performance from it your traffic requires.

Also unfortunately, understanding the queue set parameters, and how then interrelate, can be confusing (at least I had found them so).

There's another QoS document on these support forums, that has lots of good info about 3750 QoS: https://supportforums.cisco.com/docs/DOC-8093

0 Helpful

Go to solution
billmatthews
Level 1
Level 1
In response to Joseph W. Doherty

‎05-03-2013 07:12 AM

Thank you all for the excellent advice, +5

My first step will be to remove QoS and monitor.  To remove QoS do I just "no" all the mls qos commands individually?  Or is there one global command to rip it all out (no mls qos)?

Sorry for asking a question I should be able to test myself -- but unfortunately I don't currently have a lab box.

0 Helpful

Go to solution
InayathUlla Sharieff
Cisco Employee
Cisco Employee
In response to billmatthews

‎05-03-2013 07:19 AM

Hi Bill,

Command to remove the QOS is:

Config t

no mls qos

Ref:

mls qos (global configuration mode)

To enable the quality of service (QoS) functionality globally, use the mls qos command in global configuration mode. To disable the QoS functionality globally, use the no form of this command.

mls qos

no mls qos

HTH

Regards

Inayath

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Customers Also Viewed These Support Documents
Review Cisco Networking for a $25 gift card