11-08-2020 11:00 PM
Setup:
WS-C3560CX, firewall internal port connected to gi0/2
WS-C2960C-8TC-S, IPTV box connected to port fa0/1, speed 100m auto
WS-C2960GC-8TC-S, IPTV boxes connectet to port gi0/3 and gi0/5, speed 100m auto
The inter-switch connections are 1G trunks. IPTV traffic is separate vlan502
If I enable QoS (mls qos) I am facing a rapid increasing of "Total output drops" at these interfaces, where the IPTV boxes are connected. The effect are freezes in IPTV. That has not a good waf...
My provider send the musticast IPTV packets with dscp 34. The switch port with connection to the internal firewall port is configured to trust dscp
interface GigabitEthernet0/2 switchport trunk allowed vlan 1,101,501-504,901 switchport mode trunk mls qos trust dscp end
some stats at the interface where IPTV box is connected:
WS2960-02#show interface gi0/3 GigabitEthernet0/3 is up, line protocol is up (connected) Hardware is Gigabit Ethernet, address is 1833.9d8b.f683 (bia 1833.9d8b.f683) MTU 9100 bytes, BW 100000 Kbit/sec, DLY 100 usec, reliability 255/255, txload 20/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive set (10 sec) Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX input flow-control is on, output flow-control is unsupported ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output 00:00:00, output hang never Last clearing of "show interface" counters 5w3d Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 174765 Queueing strategy: fifo Output queue: 0/4096 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 7994000 bits/sec, 735 packets/sec 3986501 packets input, 426971434 bytes, 0 no buffer Received 714402 broadcasts (714311 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 714311 multicast, 0 pause input 0 input packets with dribble condition detected 38008178 packets output, 48472986367 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 unknown protocol drops 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 pause output 0 output buffer failures, 0 output buffers swapped out
WS2960-02#show mls qos interface gi0/3 statistics GigabitEthernet0/3 (All statistics are in packets) dscp: incoming ------------------------------- 0 - 4 : 0 0 0 0 0 5 - 9 : 0 0 0 0 0 10 - 14 : 0 0 0 0 0 15 - 19 : 0 0 0 0 0 20 - 24 : 0 0 0 0 0 25 - 29 : 0 0 0 0 0 30 - 34 : 0 0 0 0 251586 35 - 39 : 0 0 0 0 0 40 - 44 : 0 0 0 0 0 45 - 49 : 0 0 0 0 0 50 - 54 : 0 0 0 0 0 55 - 59 : 0 0 0 0 0 60 - 64 : 0 0 0 0 dscp: outgoing ------------------------------- 0 - 4 : 52405 0 0 0 0 5 - 9 : 0 0 0 0 0 10 - 14 : 0 0 0 0 0 15 - 19 : 0 0 0 0 0 20 - 24 : 0 0 0 0 0 25 - 29 : 0 0 0 0 0 30 - 34 : 0 0 1015 0 1797321 35 - 39 : 0 0 0 0 0 40 - 44 : 0 0 0 0 0 45 - 49 : 0 172348 0 22941 0 50 - 54 : 0 0 0 0 0 55 - 59 : 0 0 0 0 0 60 - 64 : 0 0 0 0 cos: incoming ------------------------------- 0 - 4 : 256227 0 0 0 0 5 - 7 : 0 0 0 cos: outgoing ------------------------------- 0 - 4 : 63304 0 0 0 1792084 5 - 7 : 172348 22941 143542 output queues enqueued: queue: threshold1 threshold2 threshold3 ----------------------------------------------- queue 0: 172348 0 0 queue 1: 60518 119769 143609 queue 2: 12549334 0 0 queue 3: 1814957 0 0 output queues dropped: queue: threshold1 threshold2 threshold3 ----------------------------------------------- queue 0: 173370 0 0 queue 1: 664 0 0 queue 2: 1 0 0 queue 3: 652 0 0 Policer: Inprofile: 0 OutofProfile: 0 WS2960-02#
The trunk ports at the C2960er's are also configured to trust dscp.
So, the question is now: Why is the switch dropping the IPTV packets if QoS is enabled and how to avoid this?
thanks
Uwe
11-09-2020 12:03 AM - edited 11-09-2020 12:45 AM
Hello,
when you enable QoS with 'mls qos', the switch automatically generates queues and thresholds (that you can configure). There is a way to finetune these values. For that, I need the output of:
show mls qos maps dscp-output-q
show mls qos queue-set
In the meantime, what happens when you disable QoS altogether ?
11-09-2020 01:16 AM
If I disable QoS, there are no drops and no freezes.
Your request at the 3560:
WS3560-01#show mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 100 200 100 100 threshold2: 100 200 100 100 reserved : 50 50 50 50 maximum : 400 400 400 400 Queueset: 2 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 100 200 100 100 threshold2: 100 200 100 100 reserved : 50 50 50 50 maximum : 400 400 400 400
WS3560-01#show mls qos maps dscp-output-q Dscp-outputq-threshold map: d1 :d2 0 1 2 3 4 5 6 7 8 9 ------------------------------------------------------------ 0 : 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 1 : 02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01 2 : 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 3 : 03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 4 : 01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01 5 : 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 6 : 04-01 04-01 04-01 04-01
... and the 2960:
WS2960-02#show mls qos maps dscp-output-q Dscp-outputq-threshold map: d1 :d2 0 1 2 3 4 5 6 7 8 9 ------------------------------------------------------------ 0 : 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 02-01 1 : 02-01 02-01 02-01 02-01 02-01 02-01 03-01 03-01 03-01 03-01 2 : 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 03-01 3 : 03-01 03-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 4 : 01-01 01-01 01-01 01-01 01-01 01-01 01-01 01-01 04-01 04-01 5 : 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 04-01 6 : 04-01 04-01 04-01 04-01
WS2960-02#show mls qos queue-set Queueset: 1 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 1000 1000 1000 1000 threshold2: 1000 1000 1000 1000 reserved : 50 50 50 50 maximum : 1000 1000 1000 1000 Queueset: 2 Queue : 1 2 3 4 ---------------------------------------------- buffers : 25 25 25 25 threshold1: 100 200 100 100 threshold2: 100 200 100 100 reserved : 50 50 50 50 maximum : 400 400 400 400
Thanks,
Uwe
11-09-2020 02:35 AM
Hello,
try and configure the WS-C2960-2 with these global values:
WS-C2960-02(config)#mls qos queue-set output 2 buffers 10 10 10 70
WS-C2960-02(config)#mls qos queue-set output 2 threshold 4 3100 3100 100 3200
and the interface GigabitEthernet0/3 with:
interface GigabitEthernet0/3
queue-set 2
and see if the output drops disappear on that interface. If it works, in order to calculate the values for the other interfaces, post the output of:
show mls qos interface x statistics
for all interfaces that show the drops.
11-09-2020 12:53 AM
- Check if this document can be helpful :
M.
11-09-2020 09:15 AM - edited 11-09-2020 09:15 AM
"So, the question is now: Why is the switch dropping the IPTV packets if QoS is enabled and how to avoid this?"
The usual reason for seeing drops is because those switches, and the 3560/3750 switches, when QoS is enabled, define four egress queues per port, and "reserve" buffer space per port. Effectively, by default, you have less buffers per QoS egress queue so often output drops increase. (Also, the 2960s have less buffer resources than the aforementioned 3Ks.)
How to avoid this is to change resource allocation defaults. In your stats, all the drops are happening at threshold 1. You might just change that, but then your drops might start to show at threshold 2 or the queue limit. So, suggest you set all 3 values to their maximum values. (NB: The maximum values vary based on IOS version.)
First, increasing logical queue limits often helps as also does placing more buffers back into the shared buffer pool:
Try:
mls qos queue-set output 1 threshold 1 3200 3200 25 3200
mls qos queue-set output 1 threshold 2 3200 3200 25 3200
mls qos queue-set output 1 threshold 3 3200 3200 25 3200
mls qos queue-set output 1 threshold 4 3200 3200 25 3200
See what happens before making any additional changes.
11-09-2020 12:34 PM
Thanks to @Georg Pauwen, tanks to @Joseph W. Doherty
With @Joseph W. Doherty suggestion I have the best result. There are no drops for multicast IPTV.
The new issue is, to have drops with netflix
I cleared the mls qos interface stat. So, the following output contain counters from netflix stream only:
dscp: incoming ------------------------------- 0 - 4 : 0 0 0 0 0 5 - 9 : 0 0 0 0 0 10 - 14 : 0 0 0 0 0 15 - 19 : 0 0 0 0 0 20 - 24 : 0 0 0 0 0 25 - 29 : 0 0 0 0 0 30 - 34 : 0 0 0 0 39170 35 - 39 : 0 0 0 0 0 40 - 44 : 0 0 0 0 0 45 - 49 : 0 0 0 0 0 50 - 54 : 0 0 0 0 0 55 - 59 : 0 0 0 0 0 60 - 64 : 0 0 0 0 dscp: outgoing ------------------------------- 0 - 4 : 222829 0 0 0 0 5 - 9 : 0 0 0 0 0 10 - 14 : 0 0 0 0 0 15 - 19 : 0 0 0 0 0 20 - 24 : 0 0 0 0 0 25 - 29 : 0 0 0 0 0 30 - 34 : 0 0 0 0 18 35 - 39 : 0 0 0 0 0 40 - 44 : 0 0 0 0 0 45 - 49 : 0 0 0 34 0 50 - 54 : 0 0 0 0 0 55 - 59 : 0 0 0 0 0 60 - 64 : 0 0 0 0 cos: incoming ------------------------------- 0 - 4 : 39171 0 0 0 0 5 - 7 : 0 0 0 cos: outgoing ------------------------------- 0 - 4 : 230674 0 0 0 0 5 - 7 : 0 34 208 output queues enqueued: queue: threshold1 threshold2 threshold3 ----------------------------------------------- queue 0: 0 0 0 queue 1: 230642 132 208 queue 2: 0 0 0 queue 3: 34 0 0 output queues dropped: queue: threshold1 threshold2 threshold3 ----------------------------------------------- queue 0: 0 0 0 queue 1: 1392 0 0 queue 2: 0 0 0 queue 3: 0 0 0 Policer: Inprofile: 0 OutofProfile: 0
The drop rate is much less than the droprate at IPTV before adjusting the theshold size, but the drops are present.
11-09-2020 12:44 PM
Hello,
I wonder if it might not just be easier to turn off 'mls qos' altogether...? By manipulating the queues, you might in the end just shift the drops to other queues. As you stated earlier by yourself, disabling mls qos results in no drops.
11-09-2020 02:09 PM
Hmm, if you used the values I suggested, I'm unsure why you seeing enqueued stats for thresholds 2 and 3 as it appears all those packets are marked BE.
We need to watch carefully what happens as we make additional changes, as it's often easy enough to make things worse.
BTW, what Georg suggests about disabling QoS is a worthwhile suggestion, unless you really want to protect one class of traffic against another. I.e. sometimes when you oversubscribe bandwidth, the best you can do is target which traffic is going to suffer first from lack of resources.
Anyway, what I suggest you might try next is decreasing reserved queue buffers releasing them back to the common pool.
For example:
mls qos queue-set output 1 threshold 1 3200 3200 10 3200
mls qos queue-set output 1 threshold 2 3200 3200 10 3200
mls qos queue-set output 1 threshold 3 3200 3200 10 3200
mls qos queue-set output 1 threshold 4 3200 3200 10 3200
Also, please do another "show mls qos queue-set" and confirm which queue-set you're using on the interface.
05-19-2021 05:39 AM
I have similar issue where my cisco 2960 is connection to ASA firewall
I am facing an increasing packet log
can you help?
05-19-2021 08:03 AM
"can you help?"
Maybe (but possibly maybe not).
Anyway, is QoS active on your 2960 and if so, what are the QoS parameter settings?
05-19-2021 10:26 AM
05-19-2021 12:57 PM
So, what queue(s) are you seeing drops?
BTW, in cases of sustained over subscription (which may, or may not, be your case), you'll generally always have drops. If that's your situation, you'll either need more bandwidth or, if the bandwidth consuming flows slow when they detect congestion, you can sometimes "manage" source transmission rates (decreasing the drop percentage, but possibly not eliminating it).
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide