cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1955
Views
5
Helpful
5
Replies

Troubleshooting output queue drops/qos drops

billmatthews
Level 1
Level 1

We're trying to troubleshoot some performance problems on a stack of 3750.  We initially noticed lots of output queue drops (no input drops, no errors). 

I've followed this technote: http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml

Eventually I also looked at qos drops, and I see lots of drops

  dscp: incoming

-------------------------------

  0 -  4 :  1457734311            0      2154834            0      2649209

  5 -  9 :           0            0            0            0            0

10 - 14 :           0            0            0            0            0

15 - 19 :           0            0            0            0            0

20 - 24 :           0            0            0            0   3196936291

25 - 29 :           0      5003169            0            0            0

30 - 34 :           0            0        22316            0            0

35 - 39 :           0            1            0            0            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0   2512040185            0     74672627            0

50 - 54 :          40            0            0            0            3

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  dscp: outgoing

-------------------------------

  0 -  4 :   360986293           15        16189            0        83333

  5 -  9 :           0            0            0         8519            0

10 - 14 :   195960893            0            3            8            0

15 - 19 :           0          122            0     22750877            0

20 - 24 :           4            0            2            2   3642595093

25 - 29 :           0      7574890            0           22            0

30 - 34 :           0            0    502252620            0            7

35 - 39 :           0        47816            0            6            0

40 - 44 :           0            0            0            0            0

45 - 49 :           0      5266457            0    181700371            0

50 - 54 :           0            0            0            0            0

55 - 59 :           0            0            0            0            0

60 - 64 :           0            0            0            0

  cos: incoming

-------------------------------

  0 -  4 :    54270256            0            0   3201939457        22317

  5 -  7 :  2512040177     75355489     14772028

  cos: outgoing

-------------------------------

  0 -  4 :   711773765    195969423     27760830   3650170005    502300449

  5 -  7 :  4244506266    181700371    191977167

  output queues enqueued:

queue: threshold1 threshold2 threshold3

-----------------------------------------

queue 0:           2           0  4244506267

queue 1:    43692439    45670782  4023848561

queue 2:           0           0   530061259

queue 3:         164   195969259   651039679

  output queues dropped:

queue: threshold1 threshold2 threshold3

-----------------------------------------

queue 0:            0            0            0

queue 1:         2747         2794       162044

queue 2:            0            0            0

queue 3:           43     20903265   2849834164

Policer: Inprofile:            0 OutofProfile:            0


So that has me thinking we have a qos problem.  I believe I need to assign more bandwidth to queue 4? (read this article https://supportforums.cisco.com/community/netpro/network-infrastructure/switching/blog/2011/04/11/output-drops-due-to-qos-on-296035603750-switches)

show mls qos queue-set

Queueset: 1

Queue     :       1       2       3       4

----------------------------------------------

buffers   :      10      10      26      54

threshold1:     138     138      36      20

threshold2:     138     138      77      50

reserved  :      92      92     100      67

maximum   :     138     400     318     400

Queueset: 2

Queue     :       1       2       3       4

----------------------------------------------

buffers   :      16       6      17      61

threshold1:     149     118      41      42

threshold2:     149     118      68      72

reserved  :     100     100     100     100

maximum   :     149     235     272     242


But as a side note, I'm also seeing some asic drops, I'm not sure if this is related?

Port-asic Port Drop Statistics - Summary

========================================

  RxQueue Drop Statistics Slice0

   RxQueue 0 Drop Stats Slice0: 0

   RxQueue 1 Drop Stats Slice0: 0

   RxQueue 2 Drop Stats Slice0: 0

   RxQueue 3 Drop Stats Slice0: 0

  RxQueue Drop Statistics Slice1

   RxQueue 0 Drop Stats Slice1: 0

   RxQueue 1 Drop Stats Slice1: 0

   RxQueue 2 Drop Stats Slice1: 1

   RxQueue 3 Drop Stats Slice1: 0

  Port  0 TxQueue Drop Stats: 61335

  Port  1 TxQueue Drop Stats: 0

  Port  2 TxQueue Drop Stats: 0

  Port  3 TxQueue Drop Stats: 0

  Port  4 TxQueue Drop Stats: 0

  Port  5 TxQueue Drop Stats: 0

  Port  6 TxQueue Drop Stats: 0

  Port  7 TxQueue Drop Stats: 0

  Port  8 TxQueue Drop Stats: 0

  Port  9 TxQueue Drop Stats: 0

  Port 10 TxQueue Drop Stats: 0

  Port 11 TxQueue Drop Stats: 0

  Port 12 TxQueue Drop Stats: 0

  Port 13 TxQueue Drop Stats: 0

  Port 14 TxQueue Drop Stats: 69127

  Port 15 TxQueue Drop Stats: 0

  Port 16 TxQueue Drop Stats: 0

  Port 17 TxQueue Drop Stats: 0

  Port 18 TxQueue Drop Stats: 0

  Port 19 TxQueue Drop Stats: 0

  Port 20 TxQueue Drop Stats: 0

  Port 21 TxQueue Drop Stats: 0

  Port 22 TxQueue Drop Stats: 0

  Port 23 TxQueue Drop Stats: 0

  Port 24 TxQueue Drop Stats: 0

  Port 25 TxQueue Drop Stats: 0

  Port 26 TxQueue Drop Stats: 0

  Port 27 TxQueue Drop Stats: 0

  Supervisor TxQueue Drop Statistics

    Queue  0: 0

    Queue  1: 0

    Queue  2: 0

    Queue  3: 0

    Queue  4: 0

    Queue  5: 0

    Queue  6: 0

    Queue  7: 0

    Queue  8: 0

    Queue  9: 0

    Queue 10: 0

    Queue 11: 0

    Queue 12: 0

    Queue 13: 0

    Queue 14: 0

    Queue 15: 0


Are those relatd to my QoS problems?  And any suggestions on how to redistribute my queue-set?

Thanks

Bill

3 Accepted Solutions

Accepted Solutions

paolo bevilacqua
Hall of Fame
Hall of Fame

Remove all QoS configuration, that is totally unnecessary on LAN, and all your problems will be solved.

View solution in original post

InayathUlla Sharieff
Cisco Employee
Cisco Employee

Hi Bill,

As Paolo suggested kindly remove the QOS and update us the result.

If you are not supposed to remove then check the following things;

Output qos drops in many networks are expected and the best solution is

to increase the line speed, which is not the feasible solution here.

Here we need to apply some solution to control output drops, based on

the trail and run method by implementing various bandwidth ratios and

threshold values for the traffic that gets affected. From the queueing

interface output all drops occurred in Queue 1. Queue 1 handles traffic

mapped with COS values 0 1. No drops are reported for other queues,

queues 1 and 3. Lets not consider Queue 4 which is a priority queue and

is always dedicated to drop sensitive traffic such voice and video.

  Packets dropped on Transmit:

    BPDU packets:  0

    queue              dropped  [cos-map]

    ---------------------------------------------

    1                        0  [0 1 ]

    2                     1477  [2 3 4 ]

    3                        0  [6 7 ]

    4                        0  [5 ]

-          Identify what type of traffics are classified for COS 2, 3,

4.

-          Suppose lets say, if more traffic are expected with a cos

value of 2, then its a best practice to dedicate one queue for that

traffic. In this case, other traffic cos 3 and 4, depending upon the

needs, they can be put to be handled by different queues.

-          Accordingly MAP the cos values to the appropriate queued ID,

to minimize drops.

-          Use "wrr-queue cos-map" command to map the appropriate cos to

a particular queue.

-          You may also try increasing the buffer ratios and increase

its wrr threshold values for a particular queue to see if that would

help.

-          Use "wrr-queue bandwidth" command to increase the bandwidth

percentage of a particular queue.

-          Use "wrr-queue queue-limit" command to increase the transmit

queue ratio.

-          Use "wrr-queue random-detect min-threshold" & w"wrr-queue

random-detect max-threshold" commands to configure the drop minimum and

maximum thresholds of a particular queue.

-          Refer the link below to know the usage of all the above

commands in detail.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note

09186a008015bf98.shtml

HTH

Regards

Inayath

Please rate all usefull posts.

View solution in original post

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Indeed what you've posted shows drops.  Your queue set parameters are (cough) "interesting".

The problem with the 2960/3560/3750 egress queuing architecture, is when you enable QoS it, by default, evenly provisions 4 egress queues with the resources that were devoted to just 1 egress queue.  This commonly results in drops, not seen before QoS was enabled.  (This is one reason Paolo is advising disabling QoS.  However, you can still have congestion on a port with the non-QoS single egress queue, and if you do, and if you have "special" traffic performance requirements, then disabling QoS might not be a good option.)

If you really need QoS, then you likely will need to also tune it for your requirements.  As your existing queue set configurations are non-default, I would presume someone has been trying to tune the QoS configuration.

Unfortunately, QoS tuning often is time consuming, as you need to monitor to determine if your are getting the performance from it your traffic requires.

Also unfortunately, understanding the queue set parameters, and how then interrelate, can be confusing (at least I had found them so).

There's another QoS document on these support forums, that has lots of good info about 3750 QoS: https://supportforums.cisco.com/docs/DOC-8093

View solution in original post

5 Replies 5

paolo bevilacqua
Hall of Fame
Hall of Fame

Remove all QoS configuration, that is totally unnecessary on LAN, and all your problems will be solved.

InayathUlla Sharieff
Cisco Employee
Cisco Employee

Hi Bill,

As Paolo suggested kindly remove the QOS and update us the result.

If you are not supposed to remove then check the following things;

Output qos drops in many networks are expected and the best solution is

to increase the line speed, which is not the feasible solution here.

Here we need to apply some solution to control output drops, based on

the trail and run method by implementing various bandwidth ratios and

threshold values for the traffic that gets affected. From the queueing

interface output all drops occurred in Queue 1. Queue 1 handles traffic

mapped with COS values 0 1. No drops are reported for other queues,

queues 1 and 3. Lets not consider Queue 4 which is a priority queue and

is always dedicated to drop sensitive traffic such voice and video.

  Packets dropped on Transmit:

    BPDU packets:  0

    queue              dropped  [cos-map]

    ---------------------------------------------

    1                        0  [0 1 ]

    2                     1477  [2 3 4 ]

    3                        0  [6 7 ]

    4                        0  [5 ]

-          Identify what type of traffics are classified for COS 2, 3,

4.

-          Suppose lets say, if more traffic are expected with a cos

value of 2, then its a best practice to dedicate one queue for that

traffic. In this case, other traffic cos 3 and 4, depending upon the

needs, they can be put to be handled by different queues.

-          Accordingly MAP the cos values to the appropriate queued ID,

to minimize drops.

-          Use "wrr-queue cos-map" command to map the appropriate cos to

a particular queue.

-          You may also try increasing the buffer ratios and increase

its wrr threshold values for a particular queue to see if that would

help.

-          Use "wrr-queue bandwidth" command to increase the bandwidth

percentage of a particular queue.

-          Use "wrr-queue queue-limit" command to increase the transmit

queue ratio.

-          Use "wrr-queue random-detect min-threshold" & w"wrr-queue

random-detect max-threshold" commands to configure the drop minimum and

maximum thresholds of a particular queue.

-          Refer the link below to know the usage of all the above

commands in detail.

http://www.cisco.com/en/US/products/hw/switches/ps700/products_tech_note

09186a008015bf98.shtml

HTH

Regards

Inayath

Please rate all usefull posts.

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Indeed what you've posted shows drops.  Your queue set parameters are (cough) "interesting".

The problem with the 2960/3560/3750 egress queuing architecture, is when you enable QoS it, by default, evenly provisions 4 egress queues with the resources that were devoted to just 1 egress queue.  This commonly results in drops, not seen before QoS was enabled.  (This is one reason Paolo is advising disabling QoS.  However, you can still have congestion on a port with the non-QoS single egress queue, and if you do, and if you have "special" traffic performance requirements, then disabling QoS might not be a good option.)

If you really need QoS, then you likely will need to also tune it for your requirements.  As your existing queue set configurations are non-default, I would presume someone has been trying to tune the QoS configuration.

Unfortunately, QoS tuning often is time consuming, as you need to monitor to determine if your are getting the performance from it your traffic requires.

Also unfortunately, understanding the queue set parameters, and how then interrelate, can be confusing (at least I had found them so).

There's another QoS document on these support forums, that has lots of good info about 3750 QoS: https://supportforums.cisco.com/docs/DOC-8093

Thank you all for the excellent advice, +5

My first step will be to remove QoS and monitor.  To remove QoS do I just "no" all the mls qos commands individually?  Or is there one global command to rip it all out (no mls qos)?

Sorry for asking a question I should be able to test myself -- but unfortunately I don't currently have a lab box.

Hi Bill,

Command to remove the QOS is:

Config t

no mls qos

Ref:

mls qos (global configuration mode)

To enable the quality of service (QoS) functionality globally, use the mls qos command in global configuration mode. To disable the QoS functionality globally, use the no form of this command.

mls qos

no mls qos

HTH

Regards

Inayath

Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco