3850 priority queue drops
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2021 04:20 PM - edited 01-11-2021 06:13 PM
I have a 3850 that I’m trying to figure out what is going on with queue drops on threshold 1 on the priority queue. I'm using 1P7Q3T with auto QOS on a ten gig interface with a CIR of 2 Gig setup using the bandwidth command.
Below is how the queue buffers are setup, (default)
DATA Port:5 GPN:3 AFD:Disabled FlatAFD:Disabled QoSMap:3 HW Queues: 40 - 47
DrainFast:Disabled PortSoftStart:4 - 1872
----------------------------------------------------------
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 7 187 9 187 5 0 0 0 5 2496
1 1 4 0 10 320 8 320 5 125 5 2496
2 1 4 0 11 500 8 320 5 125 5 2496
3 1 4 0 11 500 8 320 5 125 5 2496
4 1 4 0 11 500 8 320 5 125 5 2496
5 1 4 0 11 500 8 320 5 125 5 2496
6 1 4 0 11 500 8 320 5 125 5 2496
7 1 4 0 12 1248 9 798 6 312 5 2496
Priority Shaped/shared weight shaping_step sharpedWeight
-------- ------------- ------ ------------ -------------
0 1 Shaped 6375 255 0
1 7 Shared 125 0 0
2 7 Shared 125 0 0
3 7 Shared 125 0 0
4 7 Shared 250 0 0
5 7 Shared 250 0 0
6 7 Shared 125 0 0
7 7 Shared 50 0 0
Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2
------- ------- ------- ------- ------- ------- ------- ------- ------
0 0 298 0 0 333 0 0 374 0
1 0 255 0 0 285 0 0 320 0
2 0 398 0 0 445 0 0 500 0
3 0 398 0 0 445 0 0 500 0
4 0 398 0 0 445 0 0 500 0
5 0 398 0 0 445 0 0 500 0
6 0 398 0 0 445 0 0 500 0
7 0 994 0 0 1111 0 0 1248 0
The problem is I'm getting discards on that interface and looking at the queue stats it appears to be on the priority queue on threshold 1 and I'm not sure why that is. I'm having a hard time figuring out why I'm dropping priority packets on queue0 at threshold 1,
DATA Port:5 Enqueue Counters --------------------------------------------------------------------------------------------- Q Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2 Qpolicer (Count) (Bytes) (Bytes) (Bytes) (Bytes) - ------- -------------------- -------------------- -------------------- -------------------- 0 0 0 3040662254 1033265 0 1 0 1805733 25876676 357248 0 2 0 0 0 0 0 3 0 0 0 727548867 0 4 0 0 0 1341365282 0 5 0 0 0 35966063126 0 6 0 0 0 2468421 0 7 0 0 0 11016902233 0 DATA Port:5 Drop Counters ------------------------------------------------------------------------------------------------------------------------------- Q Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop QpolicerDrop (Bytes) (Bytes) (Bytes) (Bytes) (Bytes) (Bytes) - -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- 0 0 4983541 0 0 0 0 1 0 0 0 0 0 0 2 0 0 0 0 0 0 3 0 0 0 0 0 0 4 0 0 0 0 0 0 5 0 0 0 0 0 0 6 0 0 0 0 0 0 7 0 0 0 0 0 0 AGGR-CAT3K-FRV-MA-AGGR-SW#
---------------------------------------------------------- DTS Hardmax Softmax PortSMin GlblSMin PortStEnd ----- -------- -------- -------- -------- --------- 0 1 7 187 9 187 5 0 0 0 5 2496 1 1 4 0 10 320 8 320 5 125 5 2496 2 1 4 0 11 500 8 320 5 125 5 2496 3 1 4 0 11 500 8 320 5 125 5 2496 4 1 4 0 11 500 8 320 5 125 5 2496 5 1 4 0 11 500 8 320 5 125 5 2496 6 1 4 0 11 500 8 320 5 125 5 2496 7 1 4 0 12 1248 9 798 6 312 5 2496 Priority Shaped/shared weight shaping_step sharpedWeight -------- ------------- ------ ------------ ------------- 0 1 Shaped 6375 255 0 1 7 Shared 125 0 0 2 7 Shared 125 0 0 3 7 Shared 125 0 0 4 7 Shared 250 0 0 5 7 Shared 250 0 0 6 7 Shared 125 0 0 7 7 Shared 50 0 0 Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2 ------- ------- ------- ------- ------- ------- ------- ------- ------ 0 0 298 0 0 333 0 0 374 0 1 0 255 0 0 285 0 0 320 0 2 0 398 0 0 445 0 0 500 0 3 0 398 0 0 445 0 0 500 0 4 0 398 0 0 445 0 0 500 0 5 0 398 0 0 445 0 0 500 0 6 0 398 0 0 445 0 0 500 0 7 0 994 0 0 1111 0 0 1248 0
Threshold 0 is set to 298 TH 1 has 333 and TH 2 374 but it looks like when a packet is over threshold one it gets dropped. Not sure how a packet will be determined to go to TH1 or TH2 and how I can fix this issue. Right now I see discards on the output interface and I'm afraid I'm dropping priority packets like voice.
TenGigabitEthernet1/0/3 is up, line protocol is up (connected)
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 4702711
Could someone explain to me how the thresholds work on the 3850, why I don’t seem to be using threshold 0 or 2 and what I can do to prevent these drops.
TIA, Paul
- Labels:
-
Catalyst 3000
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-11-2021 04:39 PM
worth adding the command and clear the counter and observe,
qos queue-softmax-multiplier 1200
you can look at the troubleshooting document for more information.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 08:36 AM
This will only increase the queue for not priority queues, from above the issue im having is drops on the priority queue on TH1, which i read is the default. Using your suggested command will not help with the priority queue drops
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 04:59 PM
noted that the suggested one not worked. let me re-visit the full Q and update my comments, meantime try other suggestions in the post.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 08:58 AM
I'm no expert on the 3850, but I do wonder about "shared" in regards to Q0.
I also wonder if PQ on a 3850 is more like a router's LLQ, with an implicit policer rather than the earlier 3750s whose PQ was "pure" PQ.
What's the actual auto-QoS that's being used?
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 01:10 PM
Hi, Joseph, Q0 says shaped not shared. Having a hard time trying to figure out how Cisco sets up these Queues, best info I go was from https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/200594-Catalyst-3850-Troubleshooting-Output-dr.html but its not for IOS 16.x
sh policy-map AutoQos-4.0-Output-Policy Policy Map AutoQos-4.0-Output-Policy Class AutoQos-4.0-Output-Priority-Queue priority level 1 20 (%) Class AutoQos-4.0-Output-Control-Mgmt-Queue bandwidth remaining 10 (%) queue-limit dscp cs2 percent 80 queue-limit dscp cs3 percent 90 queue-limit dscp cs6 percent 100 queue-limit dscp cs7 percent 100 queue-buffers ratio 10 Class AutoQos-4.0-Output-Multimedia-Conf-Queue bandwidth remaining 10 (%) queue-buffers ratio 10 Class AutoQos-4.0-Output-Trans-Data-Queue bandwidth remaining 10 (%) queue-buffers ratio 10 Class AutoQos-4.0-Output-Bulk-Data-Queue bandwidth remaining 5 (%) queue-buffers ratio 10 Class AutoQos-4.0-Output-Scavenger-Queue bandwidth remaining 5 (%) queue-buffers ratio 10 Class AutoQos-4.0-Output-Multimedia-Strm-Queue bandwidth remaining 10 (%) queue-buffers ratio 10 Class class-default bandwidth remaining 25 (%) queue-buffers ratio 25
I changed some of the bandwidth remaining as I found the default 30% for the priority queue too high. I also noticed that the shared queues have by default low default buffers which I can adjust with the queue-softmax-multiplier command and the queue ration command inside the class map, just not sure why I'm getting dropped packets on TH1 for the priority Q which is the default TH from what I read. Also not sure if there a way to adjust the priority TH1 and if its even worth it as i will introduce more delay but i dont feel comfortable with the drops. Could the auto qos Policy priority queue just be matching too many packets? for the TH1 level, usually on priority queues I have just EF when I create a policy manually.
DATA Port:5 Drop Counters ------------------------------------------------------------------------------------------------------------------------------- Q Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop QpolicerDrop (Bytes) (Bytes) (Bytes) (Bytes) (Bytes) (Bytes) - -------------------- -------------------- -------------------- -------------------- -------------------- -------------------- 0 0 18216 0 0 0 0 Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 18216
TIA, Paul
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 01:18 PM
I actually was able to increase the priority Thresholds by dropping the queue-buffers ratio on the scavenger class to 5.
DATA Port:5 GPN:3 AFD:Disabled FlatAFD:Disabled QoSMap:3 HW Queues: 40 - 47 DrainFast:Disabled PortSoftStart:4 - 1872 ---------------------------------------------------------- DTS Hardmax Softmax PortSMin GlblSMin PortStEnd ----- -------- -------- -------- -------- --------- 0 1 7 250 8 250 5 0 0 0 5 2496 1 1 4 0 10 320 8 320 5 125 5 2496 2 1 4 0 11 500 8 320 5 125 5 2496 3 1 4 0 11 500 8 320 5 125 5 2496 4 1 4 0 11 500 8 320 5 125 5 2496 5 1 4 0 12 248 9 158 6 62 5 2496 6 1 4 0 11 500 8 320 5 125 5 2496 7 1 4 0 13 1248 10 798 7 312 5 2496 Priority Shaped/shared weight shaping_step sharpedWeight -------- ------------- ------ ------------ ------------- 0 1 Shaped 6375 255 0 1 7 Shared 125 0 0 2 7 Shared 125 0 0 3 7 Shared 125 0 0 4 7 Shared 250 0 0 5 7 Shared 250 0 0 6 7 Shared 125 0 0 7 7 Shared 50 0 0 Weight0 Max_Th0 Min_Th0 Weigth1 Max_Th1 Min_Th1 Weight2 Max_Th2 Min_Th2 ------- ------- ------- ------- ------- ------- ------- ------- ------ 0 0 398 0 0 445 0 0 500 0 1 0 255 0 0 285 0 0 320 0 2 0 398 0 0 445 0 0 500 0 3 0 398 0 0 445 0 0 500 0 4 0 398 0 0 445 0 0 500 0 5 0 197 0 0 220 0 0 248 0 6 0 398 0 0 445 0 0 500 0 7 0 994 0 0 1111 0 0 1248 0 AGGR-CAT3K-FRV-MA-AGGR-SW#
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 02:43 PM
"Hi, Joseph, Q0 says shaped not shared."
Oops, indeed. I meant to write shaped (as all the other queues are shared).
Looking at your auto-QoS policy-map, the "shaping", likely comes from the bandwidth allocation. Unclear (to me) whether excess is actually shaped or policed. (I would expect the latter to avoid PQ adding latency.)
Anyway, recently there was another 3850 poster having PQ drops issues, and I thought the queue limits were too small. Had trouble finding a way to increase them, because as you noted, qos queue-softmax-multiplier didn't seem to make a difference on PQ.
That person was using both level 1 and level 2 PQ, which also has unclear (as in, IMO, not being well documented) implications. From reading the documentation using just one PQ and/or one PQ w/o a specified level, might also impact how thresholds are assigned.
As to how packets are assigned to queues, the auto-QoS class-maps, would, hopefully, reveal that.
BTW, my impression was that:
Hardmax Softmax PortSMin GlblSMin PortStEnd
values might all be adjustable, but dependent on what "knobs" a particular IOS version supports.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
01-12-2021 02:51 PM
You can definitely influence, increase the PQ limits by adjusting the queue-buffers ratio in other non PQ queues.
Thanks again for your input, I appreciate it as usual.
Paul
