Re: Total output drops increasing on WS-C2960C when QoS is enabled

UweKiewel0453 · ‎11-08-2020

Setup:

WS-C3560CX, firewall internal port connected to gi0/2

WS-C2960C-8TC-S, IPTV box connected to port fa0/1, speed 100m auto

WS-C2960GC-8TC-S, IPTV boxes connectet to port gi0/3 and gi0/5, speed 100m auto

The inter-switch connections are 1G trunks. IPTV traffic is separate vlan502

If I enable QoS (mls qos) I am facing a rapid increasing of "Total output drops" at these interfaces, where the IPTV boxes are connected. The effect are freezes in IPTV. That has not a good waf...

My provider send the musticast IPTV packets with dscp 34. The switch port with connection to the internal firewall port is configured to trust dscp

interface GigabitEthernet0/2
 switchport trunk allowed vlan 1,101,501-504,901
 switchport mode trunk
 mls qos trust dscp
end

some stats at the interface where IPTV box is connected:

WS2960-02#show interface gi0/3
GigabitEthernet0/3 is up, line protocol is up (connected)
  Hardware is Gigabit Ethernet, address is 1833.9d8b.f683 (bia 1833.9d8b.f683)
  MTU 9100 bytes, BW 100000 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 20/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX
  input flow-control is on, output flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input never, output 00:00:00, output hang never
  Last clearing of "show interface" counters 5w3d
  Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 174765
  Queueing strategy: fifo
  Output queue: 0/4096 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 7994000 bits/sec, 735 packets/sec
     3986501 packets input, 426971434 bytes, 0 no buffer
     Received 714402 broadcasts (714311 multicasts)
     0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 714311 multicast, 0 pause input
     0 input packets with dribble condition detected
     38008178 packets output, 48472986367 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

WS2960-02#show mls qos interface gi0/3 statistics
GigabitEthernet0/3 (All statistics are in packets)

  dscp: incoming
-------------------------------

  0 -  4 :           0            0            0            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0       251586
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0            0            0            0            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

  0 -  4 :       52405            0            0            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0 
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0         1015            0      1797321
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0       172348            0        22941            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------

  0 -  4 :      256227            0            0            0            0
  5 -  7 :           0            0            0
  cos: outgoing
-------------------------------

  0 -  4 :       63304            0            0         0      1792084
  5 -  7 :      172348        22941       143542
  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:      172348           0           0
 queue 1:       60518      119769      143609
 queue 2:    12549334           0           0
 queue 3:     1814957           0           0

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:      173370           0           0
 queue 1:         664           0           0
 queue 2:           1           0           0
 queue 3:         652           0           0

Policer: Inprofile:            0 OutofProfile:            0

WS2960-02#

The trunk ports at the C2960er's are also configured to trust dscp.

So, the question is now: Why is the switch dropping the IPTV packets if QoS is enabled and how to avoid this?

thanks

Uwe

Georg Pauwen · ‎11-09-2020

Hello,

when you enable QoS with 'mls qos', the switch automatically generates queues and thresholds (that you can configure). There is a way to finetune these values. For that, I need the output of:

show mls qos maps dscp-output-q

show mls qos queue-set

In the meantime, what happens when you disable QoS altogether ?

UweKiewel0453 · ‎11-09-2020

Hi @Georg Pauwen

If I disable QoS, there are no drops and no freezes.

Your request at the 3560:

WS3560-01#show mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400

WS3560-01#show mls qos maps dscp-output-q
   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9
     ------------------------------------------------------------
      0 :    02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01
      1 :    02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01
      2 :    03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01
      3 :    03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      4 :    01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01
      5 :    04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      6 :    04-01 04-01 04-01 04-01

... and the 2960:

WS2960-02#show mls qos maps dscp-output-q
   Dscp-outputq-threshold map:
     d1 :d2    0     1     2     3     4     5     6     7     8     9
     ------------------------------------------------------------
      0 :    02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01
      1 :    02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01
      2 :    03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01
      3 :    03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      4 :    01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01
      5 :    04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01
      6 :    04-01 04-01 04-01 04-01

WS2960-02#show mls qos queue-set
Queueset: 1
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:    1000    1000    1000    1000
threshold2:    1000    1000    1000    1000
reserved  :      50      50      50      50
maximum   :    1000    1000    1000    1000
Queueset: 2
Queue     :       1       2       3       4
----------------------------------------------
buffers   :      25      25      25      25
threshold1:     100     200     100     100
threshold2:     100     200     100     100
reserved  :      50      50      50      50
maximum   :     400     400     400     400

Thanks,

Uwe

Georg Pauwen · ‎11-09-2020

Hello,

try and configure the WS-C2960-2 with these global values:

WS-C2960-02(config)#mls qos queue-set output 2 buffers 10 10 10 70
WS-C2960-02(config)#mls qos queue-set output 2 threshold 4 3100 3100 100 3200

and the interface GigabitEthernet0/3 with:

interface GigabitEthernet0/3
queue-set 2

and see if the output drops disappear on that interface. If it works, in order to calculate the values for the other interfaces, post the output of:

show mls qos interface x statistics

for all interfaces that show the drops.

marce1000 · ‎11-09-2020

- Check if this document can be helpful :

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/116089-technote-switches-output-drops-qos-00.html

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

Joseph W. Doherty · ‎11-09-2020

"So, the question is now: Why is the switch dropping the IPTV packets if QoS is enabled and how to avoid this?"

The usual reason for seeing drops is because those switches, and the 3560/3750 switches, when QoS is enabled, define four egress queues per port, and "reserve" buffer space per port. Effectively, by default, you have less buffers per QoS egress queue so often output drops increase. (Also, the 2960s have less buffer resources than the aforementioned 3Ks.)

How to avoid this is to change resource allocation defaults. In your stats, all the drops are happening at threshold 1. You might just change that, but then your drops might start to show at threshold 2 or the queue limit. So, suggest you set all 3 values to their maximum values. (NB: The maximum values vary based on IOS version.)

First, increasing logical queue limits often helps as also does placing more buffers back into the shared buffer pool:

Try:
mls qos queue-set output 1 threshold 1 3200 3200 25 3200
mls qos queue-set output 1 threshold 2 3200 3200 25 3200
mls qos queue-set output 1 threshold 3 3200 3200 25 3200
mls qos queue-set output 1 threshold 4 3200 3200 25 3200

See what happens before making any additional changes.

UweKiewel0453 · ‎11-09-2020

Thanks to @Georg Pauwen, tanks to @Joseph W. Doherty

With @Joseph W. Doherty suggestion I have the best result. There are no drops for multicast IPTV.

The new issue is, to have drops with netflix

I cleared the mls qos interface stat. So, the following output contain counters from netflix stream only:

  dscp: incoming
-------------------------------

  0 -  4 :           0            0            0            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0        39170
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0            0            0            0            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  dscp: outgoing
-------------------------------

  0 -  4 :      222829            0            0            0            0
  5 -  9 :           0            0            0            0            0
 10 - 14 :           0            0            0            0            0
 15 - 19 :           0            0            0            0            0
 20 - 24 :           0            0            0            0            0
 25 - 29 :           0            0            0            0            0
 30 - 34 :           0            0            0            0           18
 35 - 39 :           0            0            0            0            0
 40 - 44 :           0            0            0            0            0
 45 - 49 :           0            0            0           34            0
 50 - 54 :           0            0            0            0            0
 55 - 59 :           0            0            0            0            0
 60 - 64 :           0            0            0            0
  cos: incoming
-------------------------------

  0 -  4 :       39171            0            0            0            0
  5 -  7 :           0            0            0
  cos: outgoing
-------------------------------

  0 -  4 :      230674            0            0            0            0
  5 -  7 :           0           34          208
  output queues enqueued:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:      230642         132         208
 queue 2:           0           0           0
 queue 3:          34           0           0

  output queues dropped:
 queue:    threshold1   threshold2   threshold3
-----------------------------------------------
 queue 0:           0           0           0
 queue 1:        1392           0           0
 queue 2:           0           0           0
 queue 3:           0           0           0

Policer: Inprofile:            0 OutofProfile:            0

The drop rate is much less than the droprate at IPTV before adjusting the theshold size, but the drops are present.

Georg Pauwen · ‎11-09-2020

Hello,

I wonder if it might not just be easier to turn off 'mls qos' altogether...? By manipulating the queues, you might in the end just shift the drops to other queues. As you stated earlier by yourself, disabling mls qos results in no drops.

Joseph W. Doherty · ‎11-09-2020

Hmm, if you used the values I suggested, I'm unsure why you seeing enqueued stats for thresholds 2 and 3 as it appears all those packets are marked BE.

We need to watch carefully what happens as we make additional changes, as it's often easy enough to make things worse.

BTW, what Georg suggests about disabling QoS is a worthwhile suggestion, unless you really want to protect one class of traffic against another. I.e. sometimes when you oversubscribe bandwidth, the best you can do is target which traffic is going to suffer first from lack of resources.

Anyway, what I suggest you might try next is decreasing reserved queue buffers releasing them back to the common pool.

For example:
mls qos queue-set output 1 threshold 1 3200 3200 10 3200
mls qos queue-set output 1 threshold 2 3200 3200 10 3200
mls qos queue-set output 1 threshold 3 3200 3200 10 3200

mls qos queue-set output 1 threshold 4 3200 3200 10 3200

Also, please do another "show mls qos queue-set" and confirm which queue-set you're using on the interface.

michelfares · ‎05-19-2021

I have similar issue where my cisco 2960 is connection to ASA firewall

I am facing an increasing packet log

can you help?

Joseph W. Doherty · ‎05-19-2021

"can you help?"

Maybe (but possibly maybe not).

Anyway, is QoS active on your 2960 and if so, what are the QoS parameter settings?

michelfares · ‎05-19-2021

mls qos queue-set output 1 threshold 1 3200 3200 10 3200
mls qos queue-set output 1 threshold 2 3200 3200 10 3200
mls qos queue-set output 1 threshold 3 3200 3200 10 3200
mls qos queue-set output 1 threshold 4 3200 3200 10 3200
mls qos queue-set output 1 buffers 90 5 4 1

Joseph W. Doherty · ‎05-19-2021

So, what queue(s) are you seeing drops?

BTW, in cases of sustained over subscription (which may, or may not, be your case), you'll generally always have drops. If that's your situation, you'll either need more bandwidth or, if the bandwidth consuming flows slow when they detect congestion, you can sometimes "manage" source transmission rates (decreasing the drop percentage, but possibly not eliminating it).