Total output drops and traffic shaping

[Fabri_Fabri]

(TL;DR - Traffic shaping is the solution to avoid total output drops ?)

Hello,

we have 2 offices connected via a layer 1 SDH link of the provider. Link speed is 100MB.

From each side, it connects with our 2960 Cisco.

QoS is disabled on both 2960.

Each trunk interface of the 2 switches is configured as follow:

interface GigabitEthernet1/0/43
 description OFFICE_A
 switchport mode trunk
 switchport nonegotiate
 speed 100
 duplex full

interface GigabitEthernet1/0/48
 description OFFICE_B
 switchport mode trunk
 switchport nonegotiate
 speed 100
 duplex full

All PC clients are connected to 1000MB FULL.

Colleagues told me about network issues and find out that total output drops continue to raise either when moving a file or doing voip calls(every network activity actually).

So I tried ---> sh mls qos int gi1/0/43 statistics:

GigabitEthernet1/0/43 (All statistics are in packets)

  dscp: incoming
-------------------------------

  0 -  4 :   113503347            0            0            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0            0
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0      9220010            0         1379            0
 50 - 54 :           0            0            0            0         1915
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

  0 -  4 :   299820870          585        62116          210         4768
  5 -  9 :        1377        33660         2438            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0            0
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0      9238097            0           78            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------

  0 -  4 :   125157334           47          301           49          198
  5 -  7 :          49         1320          142
  cos: outgoing
-------------------------------

  0 -  4 :   316790643            0            0            0            0
  5 -  7 :           0            0      2800400
  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:     1644620     7125839     2800400
 queue 2:           0           0           0
 queue 3:           0           0   308020185

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:        2350           0           0
 queue 2:           0           0           0
 queue 3:           0           0    10473885

Policer: Inprofile:            0 OutofProfile:            0

and

sh mls qos int gi1/0/48 statistics
GigabitEthernet1/0/48 (All statistics are in packets)

  dscp: incoming
-------------------------------

  0 -  4 :    12442901        91894     13070488         1479       883459
  5 -  9 :      134835      6368811        95991         3448            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0        36992            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0            0
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0    454690986            0         5375            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

  0 -  4 :   814048023            0            0            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0            0
 35 - 39 :           0            0            0            0            0
 40 - 44 :      216227            0            0            0            0
 45 - 49 :           0    452982384            0        97733            0
 50 - 54 :           0            0            0            0        95502
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------

  0 -  4 :  1964488839            0            0            1            0
  5 -  7 :           0            0            0
  cos: outgoing
-------------------------------

  0 -  4 :  1431395147         1803        10204         3011         8372
  5 -  7 :        1414        84277    149768027
  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:    25097116   164208919   149852304
 queue 2:           0           0           0
 queue 3:           0           0  1272774324

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:         134           1           0
 queue 2:           0           0           0
 queue 3:           0           0      187257

Policer: Inprofile:            0 OutofProfile:            0

Why I have such high numbers ?

A friend of mine told me to enable traffic shaping. Do you agree ?

Can you please suggest me a method to solve this issue using srr-queue bandwidth share / shape ?

Any help much appreciated.

Best regads.

[Fabri_Fabri]

Anyone  ?Thanks

[Mark Malone]

Hi qos must be enabled to run that command , do a show mls qos see if its globally enabled , the high numbers 452982384   is your voice traffic tagging packets with EF DSCP 46 and being placed in its queue , without specific ports set for qos everything becomes untrusted when this is enabled , couple of options either remove it globally or set auto qos voip trust and mls qos trust dscp on your access ports and mls qos trust dscp on your uplinks to mark the traffic properly        

[Fabri_Fabri]

Hi Mark,

switch7#show mls qos
QoS is disabled
QoS ip packet dscp rewrite is enabled

set auto qos voip trust --> is it at access port level ?
mls qos trust dscp --> is it at access port level ?

mls qos trust dscp on your uplinks to mark the traffic properly     --> you mean to execute that command only on port that connect two offices (trunk mode) ? Is this command enough to mark the traffic properly ?

Last but not least: executing above commands, disrupts connectivity ?

Thanks, your suggestions are really helpful.

[Mark Malone]

yes it will have an effect,  traffic will start to be prioritized in queues so do it in a maintenance window just in case , i have rolled it out before without any user impact though but better to be safe.

QOS needs to be marked from source - destination to have full effect but you can prioritize the traffic locally so your voice traffic is not getting hammered and will be pushed through first , thought mls would be enabled to be able to see statistics regarding an interface but obviously not , this is the quickest way to do it at layer 2 let the switch automatically do the shaping , you can be more granular with your layer 3 qos where you break out your wan links

The way our global net is set for layer 2 qos is access ports with pc/phone are set with auto qos trust voip this will automatically put in mls qos trust dscp or trust cos does not really matter which 1 as the switch maps the cos-dscp , why we use these specific commands it covers all vendor phones and softphones other commands like cisco-phone are specific

Your uplink trunks or routed links to your next switch can use mls qos trust dscp, basically your phone sends a packet with a marking of EF 46 DSCP or COS 5 which is the same the access port will trust the marking and place it in the correct queue prioritizing it as its voice the uplink will also trust the marking and carry it on , with QOS its best to mark the packets closest to the source as possible so allowing the phone to do it is the way most people would set it , phones originally sends the marked packet , switch just trusts and carrys it until it gets to your layer 3 and then you would use your service policy with class-maps to prioritize specific traffic for your wan

There is multiple ways to do this but this is a quick way if your not over familiar with what way the queues works at hardware level best to let the switch setup all the queuing shaping automatically then you can adjust if required

These may help

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst6500/ios/12-2SX/configuration/guide/book/auto_qos.html

http://www.davidsudjiman.info/2012/02/14/cisco-3560-mls-qos-part-1/

[Fabri_Fabri]

Ok, I will try to do in a maintenance window.

To summarize:
 

On both 2960:

set auto qos voip trust

On uplink trunks of both switches:

mls qos trust dscp

On ALL access ports of the switch where VOIP phones are connected:

mls qos trust dscp

Is it correct ? Do I need anything else ?

I really appreciate, thanks.

mls qos trust dscp

[Mark Malone]

If your switch is not capable of doing auto qos your voice will still be prioritized just using mls qos trust dscp on its access ports as the phone will mark the packet at the source with its priority marking

You can check if your image supports it , just go to any interface in config mode and type---auto ?

this will tell you if its allowed in that image as something or nothing will come up in terms of options  , and you would need a contract to get it off the Cisco website

yes you have the above correct just get rid of the set in the auto qos trust voip if its available on your ios version , also turn it on globally -- mls qos

[Fabri_Fabri]

switch7(config-if)#auto qos voip ?
  cisco-phone      Trust the QoS marking of Cisco IP Phone
  cisco-softphone  Trust the QoS marking of Cisco IP SoftPhone
  trust            Trust the DSCP/CoS marking

So it seems supported, doesn't it ?

Summarizing again:

1) Globally on both switches:

mls qos

2) Per interface (VOIP ones)

auto qos voip trust
mls qos trust dscp

3) On uplink trunks of both switches:

mls qos trust dscp

Is it correct or am I missing something ?

Thanks !!!!!

[Mark Malone]

Yes thats correct its supported trust covers you for all devices so use that rather than specific softphone or cisco-phone , yes thats the way we have it setup and it works and you can confirm using wireshark as 1 method make a call from a source access port with phone and span the port you will be able to open up the packet and see thats its marked with EF 46 DSCP for voice confirming ports are carrying marking.

when you enable auto qos on the ports you will see queues will appear in the config under the interfaces  , these can be altered but most of the time they work fine as default we have never had to change them at access layer switches 

[Fabri_Fabri]

Hi Mark I will try your hints as soon as I can :-)

A last question do you suggest to align also firmware versions ? Now one switch has a firmware and the other one has another version.

Thanks !

[Mark Malone]

I would not worry about the IOS too much just keep them up to date not at the oldest versions but never at the newest either as if you hit a bug there may be no fix on the latest image and you will have to wait for Ciscos DE team to write 1 so always best to stay back a couple of releases from the latest

Usually we don't upgrade unless we hit a bug or we require a new feature that the newer IOS might have or its getting near deferred release stage

[Fabri_Fabri]

Thanks ! I owe you a bier (or 2,3 ... ) :-)

[Mark Malone]

no problem just rate the post if its useful thanks

[Fabri_Fabri]

Ops .. my switches have this image:

C2960S-UNIVERSALK9-M

auto qos needs LAN based image ? Is it true ? Can I just download it on my 2960 from the website ?

Thanks.