cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1948
Views
20
Helpful
6
Replies

Output Drops Due to QoS on Access Layer Switch 3750

TM13
Level 1
Level 1

All,

  We having this issue and i just went through the doc and having some question on it

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/116089-technote-switches-output-drops-qos-00.html

 

  1. How to find DSCP number from many packets and drop numbers like:

(All statistics are in packets)

  dscp: incoming
-------------------------------

  0 -  4 :   306582988           12     66874383            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :   540065157            0         1563            0            0
 15 - 19 :           0      8297379            0    456182577            0
 20 - 24 :           0            0         9648            0     94207135
 25 - 29 :        5960     21899074            0    975626032            0
 30 - 34 :           0            0            0            0     17541371
 35 - 39 :           0   3124937463            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0    310563890            0     49126249            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

  0 -  4 :  2351172567          870       814120            0            5
  5 -  9 :           0            5            0       664882            0
 10 - 14 :  1040333219            0          127            0        92576
 15 - 19 :           0    139107378            0    550290000            0
 20 - 24 :           2            0            0            0      3982696
 25 - 29 :           0     15411473            4      9028496            0
 30 - 34 :           0            0     12243780            0     45251290
 35 - 39 :           2    756337522            0            0            0
 40 - 44 :     4968689            0           34            0            3
 45 - 49 :           0    186543342            0     16486553            0
 50 - 54 :           1            0            1            0          900
 55 - 59 :           0            0            0            0            2
 60 - 64 :           0            0            0      2502542
  cos: incoming
-------------------------------

  0 -  4 :   915591884    540066720    464490990   1091738204   3142478834
  5 -  7 :   310563890     49126249      1280370
  cos: outgoing
-------------------------------

  0 -  4 :  2354911404   1041200878    690815892     28422669    813832594
  5 -  7 :   191512068     16487455      6457284
  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0   191512419
 queue 1:     3921076      594772    52868524
 queue 2:           0           0  1504645786
 queue 3:      664882  1040426034  2353605945

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0           0           0
 queue 2:           0           0     7772375
 queue 3:        1523     1023494      132058

Policer: Inprofile:            0 OutofProfile:            0

#sh mls qos maps dscp-output-q
   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9
     ------------------------------------------------------------
      0 :    04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-01 04-02
      1 :    04-02 04-02 04-02 04-02 04-02 04-02 03-03 03-03 03-03 03-03
      2 :    03-03 03-03 03-03 03-03 02-03 02-03 02-03 02-03 02-03 02-03
      3 :    02-03 02-03 03-03 03-03 03-03 03-03 03-03 03-03 03-03 03-03
      4 :    01-03 01-03 01-03 01-03 01-03 01-03 01-03 01-03 02-03 02-03
      5 :    02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03 02-03
      6 :    02-03 02-03 02-03 02-03

 

i am trying to find the packets from queues, what marking they have ... 

2. On the Document they increased threshold 2 map to 3100, how they calculated that it should be 3100? also what is the maximum? do i change it to fit in my scenario? 

 

  1. In this example, queue-set 2 and queue 1 thresholds are changed. Both threshold 1 and threshold 2 are mapped to 3100 so that they can pull buffer from the reserved pool if required.

    Switch(config)#mls qos queue-set output 2 threshold 1 3100 3100 100 3200
  2. Verify that the changes reflect under correct queue and queue-set.
    Switch#show mls qos queue-set
    Queueset: 1
    Queue     :       1       2       3       4
    ----------------------------------------------
    buffers   :      25      25      25      25
    threshold1:     100     200     100     100
    threshold2:     100     200     100     100
    reserved  :      50      50      50      50
    maximum   :     400     400     400     400
    Queueset: 2
    Queue     :       1       2       3       4
    ----------------------------------------------
    buffers   :      70      10      10      10
    threshold1:    3100     100     100     100
    threshold2:    3100     100     100     100
    reserved  :     100      50      50      50
    maximum   :    3200     400     400     400

3. Also how to change or best practical qos maps dscp-output-q ?

We have 2 different 3750 version, 1 is 12.2E and 15.2E both having different maps ...

#sh mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     138     138      36      20
threshold2:     138     138      77      50
reserved  :      92      92     100      67
maximum   :     138     400     318     400
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     149     200     100     100
threshold2:     149     200     100     100
reserved  :     100      50      50      50
maximum   :     149     400     400     400
 

 

and 

 

sh mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      15      30      35      20
threshold1:     100      80     100      60
threshold2:     100      90     100     100
reserved  :     100     100     100     100
maximum   :     100     400     400     400
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      16       6      17      61
threshold1:     149     118      41      42
threshold2:     149     118      68      72
reserved  :     100     100     100     100
maximum   :     149     235     272     242

 

6 Replies 6

daniel.dib
Level 7
Level 7

Worth noting from the output is that the queues under drops are listed as 0, 1, 2, 3 but they are referred to as 1, 2, 3, 4 under the DSCP to output queue mapping.

You have drops in queue 3, threshold 3 and queue 4, all thresholds. Have you modified the default mappings?

The Catalyst 3750 is well known for not handling bursts gracefully, do you actually need QoS as disabling usually gives better performance than untuned QoS.

This document also gives some theory about the thresholds and buffers:

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/116102-qanda-egress-00.html

I don't know how they calculated the value 3100 but basically by putting a higher value (max is 3200) the queue can borrow more buffers from the common pool for Weighted Tail Drop (WTD) before dropping packets.

You may have to assign more buffers to an egress queue depending on how much traffic is ending up in each queue.

What kind of applications is transiting the switch and what is their expected marking?

Here is a post I did on Catalyst QoS in the past:

http://lostintransit.se/2012/03/07/catalyst-qos/

This document is the most granular I've seen on Catalyst QoS:

https://supportforums.cisco.com/document/31581/egress-qos

You can use some of the show platform commands to get more information on your drops such as:

show platform port-asic stats enqueue gi1/0/25

show platform port-asic stats drop gi1/0/25

There are potential solutions under chapter 3 of that post.

Daniel Dib
CCIE #37149
CCDE #20160011

Please rate helpful posts.

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

First, be aware of, on 3560/3750, some stats refer to the four egress queues as 0..3, others 1..4, which is great for making it even more confusing.

q1: The dscp egress table shows you the count of packets with the different ToS markings.  So, for example, your:

  dscp: outgoing
-------------------------------   0 -  4 :  2351172567          870       814120            0            5

Provides the count of egress packets with ToS markings 0..4, zero being DSCP BE.  (A lot of those, which is often the case.)

If you want to know what queue and WTD a ToS marking goes to, you look at your  Dscp-outputq-threshold map.  If we want to see where the BE packets go, we look for a DSCP value of zero, e.g.:

#sh mls qos maps dscp-output-q
   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9
     ------------------------------------------------------------
      0 :    04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-03 04-01 04-02

In the above, they go to queue 4, WTD 3.

Your egress stats, shows whether packets have been enqueued or dropped.  Remembering my first note, the stats that apply to those BE packets are:

  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0   191512419
 queue 1:     3921076      594772    52868524
 queue 2:           0           0  1504645786
 queue 3:      664882  1040426034  2353605945

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:           0           0           0
 queue 2:           0           0     7772375
 queue 3:        1523     1023494      132058

Remember, though, other packets may map to the same queue and WTD, so these stats may, or may not, just apply to BE tagged packets.

q2: I don't know how they decided on using 3100; probably they just wanted something a little under max.  What they chose may be just to make an example.

Maximum value varies per IOS version.  Older versions only supported a max of 400, later versions support a max of 3200 (if I recall correctly).

I'm unsure what your "scenario" is.

q3: It depends on what you're trying to accomplish and the nature of the traffic flowing thru your device.

As Daniel has already noted, 3750s are well known for egress interfaces dropping packets.  This because they don't have large buffer resources (2 MB per 24 copper ports or per uplink ports), and their default buffer settings, especially when QoS in enabled, might be far from optimal for your traffic.

Personally, I have had great success enabling QoS and setting the buffers to used a shared buffer approach (more like the earlier 35xx switches).  However, I cannot say whether such an approach would work for you.

Hi Joseph,

   Thanks for the great explanation, really helpful, can you share to me the template of your QoS buffers settings?

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I would probably need management's approval, so I should probably avoid posting our actual templates.  However, what you can experiment with is . . .

given:

mls qos queue-set output <queue-set> threshold <queue> <WTD1> <WTD2> <reserved> <MAX>

Unless you want early tail drops for some traffic, set WTD1=WTD2=MAX

Set reserved, smaller, much smaller or to minimum possible.  (This increases the RAM to the common pool rather than reserved to port pools.

Lastly, set MAX larger, much larger or to maximum possible.  (This increases how many frames can be logically queued; buffers will be taken from common pool.  You still have drops if you exhaust pool resources or hit logical queue limit.)

BTW, as logical limits are percentages, if maximums for above isn't sufficient, you can vary ratios, per queue, using the buffer statement.

Some experimentation, with tuning, is often required.  Often you see a huge difference, and get to diminishing results.  For example, I had a couple of ISCSI devices, connected to a 3750 port having multiple drops every second.  Tuning the above settings, I got down to just a couple of drops a day.

Hi Joseph,

  Thanks for reply, we also having ISCSI also connected to 3750 and having some Video surveillance and file sharing which are reflecting, because we giving high priority to file sharing but video surveillance giving unexpected results when users watching cameras in different places on their laptop ...  can i get your contact for details?

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You're welcome to send me private messages, via the support forums, but I'm very limited by my employer in outside job related activities, so, unfortunately, I wouldn't be able to assist much more than what I've already done.