cisco 3750 QoS - OutDiscard Errors

[John Dufour]

I have a couple stacks of 3750x 24 port switches, I am seeing a decent amount of Outdiscards on my interfaces. The research I have done indicates this could be a QoS Problem, looked like the buffer ques were set to low... After tweeking them I still see the errors, I removed QoS from the switch and I am still seeing the OutDiscard Errors.  What eles could this be if not a QoS Problem?

QoS Created by Auto QoS Policy............   Here is the out put..

QoS is disabled
QoS ip packet dscp rewrite is enabled

mls qos map policed-dscp  24 26 46 to 0

mls qos map cos-dscp 0 8 16 24 32 46 48 56

mls qos srr-queue input bandwidth 90 10

mls qos srr-queue input threshold 1 8 16

mls qos srr-queue input threshold 2 34 66

mls qos srr-queue input buffers 67 33

mls qos srr-queue input cos-map queue 1 threshold 2 1

mls qos srr-queue input cos-map queue 1 threshold 3 0

mls qos srr-queue input cos-map queue 2 threshold 1 2

mls qos srr-queue input cos-map queue 2 threshold 2 4 6 7

mls qos srr-queue input cos-map queue 2 threshold 3 3 5

mls qos srr-queue input dscp-map queue 1 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue input dscp-map queue 1 threshold 3 0 1 2 3 4 5 6 7

mls qos srr-queue input dscp-map queue 1 threshold 3 32

mls qos srr-queue input dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23

mls qos srr-queue input dscp-map queue 2 threshold 2 33 34 35 36 37 38 39 48

mls qos srr-queue input dscp-map queue 2 threshold 2 49 50 51 52 53 54 55 56

mls qos srr-queue input dscp-map queue 2 threshold 2 57 58 59 60 61 62 63

mls qos srr-queue input dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue input dscp-map queue 2 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output cos-map queue 1 threshold 3 5

mls qos srr-queue output cos-map queue 2 threshold 3 3 6 7

mls qos srr-queue output cos-map queue 3 threshold 3 2 4

mls qos srr-queue output cos-map queue 4 threshold 2 1

mls qos srr-queue output cos-map queue 4 threshold 3 0

mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55

mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63

mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23

mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39

mls qos srr-queue output dscp-map queue 4 threshold 1 8

mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7

mls qos queue-set output 1 threshold 1 138 138 92 138

mls qos queue-set output 1 threshold 2 138 138 92 400

mls qos queue-set output 1 threshold 3 36 77 100 318

mls qos queue-set output 1 threshold 4 20 50 67 400

mls qos queue-set output 2 threshold 1 149 149 100 149

mls qos queue-set output 2 threshold 2 118 118 100 235

mls qos queue-set output 2 threshold 3 41 68 100 272

mls qos queue-set output 2 threshold 4 42 72 100 242

mls qos queue-set output 1 buffers 10 10 26 54

mls qos queue-set output 2 buffers 16 6 17 61

class-map match-all AutoQoS-VoIP-RTP-Trust

match ip dscp ef

class-map match-all AutoQoS-VoIP-Control-Trust

match ip dscp cs3  af31

!

!

policy-map AutoQoS-Police-CiscoPhone

class AutoQoS-VoIP-RTP-Trust

  set dscp ef

  police 320000 8000 exceed-action policed-dscp-transmit

class AutoQoS-VoIP-Control-Trust

  set dscp cs3

  police 32000 8000 exceed-action policed-dscp-transmit

interface GigabitEthernet1/0/3

switchport access vlan 50

switchport mode access

switchport voice vlan 90

srr-queue bandwidth share 10 10 60 20

queue-set 2

priority-queue out

mls qos trust device cisco-phone

mls qos trust cos

auto qos voip cisco-phone

spanning-tree portfast

service-policy input AutoQoS-Police-CiscoPhone

mls qos map policed-dscp  24 26 46 to 0

mls qos map cos-dscp 0 8 16 24 32 46 48 56

mls qos srr-queue input bandwidth 90 10

mls qos srr-queue input threshold 1 8 16

mls qos srr-queue input threshold 2 34 66

mls qos srr-queue input buffers 67 33

mls qos srr-queue input cos-map queue 1 threshold 2 1

mls qos srr-queue input cos-map queue 1 threshold 3 0

mls qos srr-queue input cos-map queue 2 threshold 1 2

mls qos srr-queue input cos-map queue 2 threshold 2 4 6 7

mls qos srr-queue input cos-map queue 2 threshold 3 3 5

mls qos srr-queue input dscp-map queue 1 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue input dscp-map queue 1 threshold 3 0 1 2 3 4 5 6 7

mls qos srr-queue input dscp-map queue 1 threshold 3 32

mls qos srr-queue input dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23

mls qos srr-queue input dscp-map queue 2 threshold 2 33 34 35 36 37 38 39 48

mls qos srr-queue input dscp-map queue 2 threshold 2 49 50 51 52 53 54 55 56

mls qos srr-queue input dscp-map queue 2 threshold 2 57 58 59 60 61 62 63

mls qos srr-queue input dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue input dscp-map queue 2 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output cos-map queue 1 threshold 3 5

mls qos srr-queue output cos-map queue 2 threshold 3 3 6 7

mls qos srr-queue output cos-map queue 3 threshold 3 2 4

mls qos srr-queue output cos-map queue 4 threshold 2 1

mls qos srr-queue output cos-map queue 4 threshold 3 0

mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47

mls qos srr-queue output dscp-map queue 2 threshold 3 24 25 26 27 28 29 30 31

mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55

mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63

mls qos srr-queue output dscp-map queue 3 threshold 3 16 17 18 19 20 21 22 23

mls qos srr-queue output dscp-map queue 3 threshold 3 32 33 34 35 36 37 38 39

mls qos srr-queue output dscp-map queue 4 threshold 1 8

mls qos srr-queue output dscp-map queue 4 threshold 2 9 10 11 12 13 14 15

mls qos srr-queue output dscp-map queue 4 threshold 3 0 1 2 3 4 5 6 7

mls qos queue-set output 1 threshold 1 138 138 92 138

mls qos queue-set output 1 threshold 2 138 138 92 400

mls qos queue-set output 1 threshold 3 36 77 100 318

mls qos queue-set output 1 threshold 4 20 50 67 400

mls qos queue-set output 2 threshold 1 149 149 100 149

mls qos queue-set output 2 threshold 2 118 118 100 235

mls qos queue-set output 2 threshold 3 41 68 100 272

mls qos queue-set output 2 threshold 4 42 72 100 242

mls qos queue-set output 1 buffers 10 10 26 54

mls qos queue-set output 2 buffers 16 6 17 61

class-map match-all AutoQoS-VoIP-RTP-Trust

match ip dscp ef

class-map match-all AutoQoS-VoIP-Control-Trust

match ip dscp cs3  af31

!

!

policy-map AutoQoS-Police-CiscoPhone

class AutoQoS-VoIP-RTP-Trust

  set dscp ef

  police 320000 8000 exceed-action policed-dscp-transmit

class AutoQoS-VoIP-Control-Trust

  set dscp cs3

  police 32000 8000 exceed-action policed-dscp-transmit

interface GigabitEthernet1/0/3

switchport access vlan 50

switchport mode access

switchport voice vlan 90

srr-queue bandwidth share 10 10 60 20

queue-set 2

priority-queue out

mls qos trust device cisco-phone

mls qos trust cos

auto qos voip cisco-phone

spanning-tree portfast

service-policy input AutoQoS-Police-CiscoPhone

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Might be simply an oversubscribed interface.

If buffers are too shallow for your BDP, packets are dropped prematurely.  If buffers are correct for BDP, transient or sustained over subscription will still drop packets (as it should - also too deep queues can be counter productive except perhaps in cases of very brief transient over subscription).

[John Dufour]

Can you point me into any specific direction on where to look in order to determin if this os in deed the case?

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

Shashank provides good information.  Beyond that, it can be difficult to tell unless you can correlate ingress load to egress load.  If your egress interfaces shows consistent high outbound load, that can be a good indicator, but transient bursts are difficult to actually see without additional instrumentation.

NB: on some router IOSs there's QoS Bandwidth Estimation (which uses Corvil technology) that can provide some insights into microbursts, but not a feature available on switches (to my knowledge).

[Shashank Singh]

Hi John,

There are several bugs prior to 12.2(53)SE that can show spurious output queue discards (OQD) on 3750 stack. If you are running anything older that 12.2(53)SE, there is a chance that you are hitting a software bug and the OQD counters do not actually mean that the packets are getting dropped. Please upgrade to 12.2(53)SE or later and recheck OQD counters.

If you are running a newer code, and wish to look into oversubscription, check out the following output for the interface dropping packets: "sh platform port-asic stat drop gix/y". This will tell you about  the asic level drops due to oversubscription.

Please rate this post if this helped answer your query.

Regards,

Shashank

[John Dufour]

Running version 12.2.(53)SE2 

Reading a few other posts I modified the thresholds

Orginal AutoQoS Config

mls qos queue-set output 2 buffers 16 6 17 61

New modified config

mls qos queue-set output 2 buffers 10 10 10 70 

(This was because the drops were happening only in queue 4)

Also I changed the buffers

Orginal config   mls qos queue-set output 2 threshold 4 32 72 100 242

new config     mls qos queue-set output 2 threshold 4 3100 3100 100 3200

[michael.hinz]

Hi John,

I see the exact same behavior on almost all my 3750 (none G, G, X, ... types) type switches.  I worked with TAC on it and got the answer that this is a purely "display bug only" and can "be ignored".  I was also told to apply the latest code (15.X) which addresses the issue in the 3750X series.  I tried that with no success.

So I kept digging.  Cisco actually has a guideline / recommended setup for 3750X switches on their website which I find very interesting.  I learned about it during Cisco Live this year and would never have found it otherwise.

http://www.cisco.com/en/US/docs/solutions/SBA/February2013/Cisco_SBA_BN_LANDeploymentGuide-Feb2013.pdf (see page 21, step 9)

I compared this guide to my configuration and noticed the following difference. The default queue settings 3750X generated via Auto QOS look like this:

     mls qos queue-set output 1 threshold 3 100 100 100 400

Here is the same queue setting from the recommendations document (see link above):

     mls qos queue-set output 1 threshold 3 100 100 100 3200

This is similar to your queue change setting from above.  I have implemented this Cisco recommded setting and have seen almost no OutDiscards since.

So thanks for pointing in the right direction. TAC never confirmed this to be the right step to me and did not want me to change the buffers.  I feel safe however after I read this to be recommended by a Cisco configuration guide.

Mike

[Joseph W. Doherty]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

Michael, you realize you're responding to post over two years ago?

In any case, whether the two different threshold configuration you note actually operate differently depends on how the other queue buffers are defined.

If all the buffers reserve all their buffer allocations, none of the thresholds should be albe to acquire more than 100%.  I.e. the 400 or 3200 doesn't matter.

Even if other queues leave actual buffer space to the common pool, unless you carefully configure, unlikely you'll even be able to acquire 400% let alone 3200%.

Cisco's basic documentation on 3560/3750 buffers is somewhat unclear, but if you haven't seen it, there an excellent document in this support forums:

https://supportforums.cisco.com/docs/DOC-8093

[michael.hinz]

Hello Joseph,

Yes, I did realize that the post was original from 2 years ago.  I still wanted to respond to it rather than opening a new case since it describes almost identically what I am experiencing in my environment.  During my internet search for a solution I have come across other posts that started in 2011 but still receive up-to-date posts that describe the same problem like in this post (i.e. http://thwack.solarwinds.com/thread/41782).  The only difference I found is that this post provided a direction with the change of buffers that seems to lead to a solution to the issue.  At least in my case I saw the problem seems to have vanished from all my 3750X switches after I implemented the Cisco recommended setting.

Let me quote directly from the Cisco LAN deployment guide:

Step 9: If the access switch is a Cisco Catalyst 3750-X, 3560-X, or 2960-S, increase the buffers for the default queue. This modification of the global QoS settings improves the ability to handle high bandwidth bursty traffic. In global configuration mode, add the following command:

mls qos queue-set output 1 threshold 3 100 100 100 3200

http://www.cisco.com/en/US/docs/solutions/SBA/February2013/Cisco_SBA_BN_LANDeploymentGuide-Feb2013.pdf, page 21>

Again, this might not work for everybody but in my case I have deployed this now to various independent 3750X deployments across my WAN and it solved the issue with the high amount of OutDiscards I was seeing before.  So I am glad for having found this Cisco LAN deployment guide step.

[John Dufour]

reading it now, good find.

[John Dufour]

How can I tell if I am starving out some other traffic, or if I am causing a delay?  Since I created this configuration change the out-discards have been  clean,