Output Drops on Priority Queue 1 on Cisco 9300 Switch

anil.kumar2508 · ‎12-16-2020

Hello Community members,

I am struggling with Output drops on PQ1 (QoS) on my several cisco 9300 series switches. Ports(Te1/1/1 & Te1/1/2) configured as port-channel. Packets are only dropping on interface Te1/1/1. qos queue-softmax-multiplier 1200 already applied. Please help in resolving this issue.

Please find my config

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 62 C9300-48U 16.6.8 CAT9K_IOSXE INSTALL

TenGigabitEthernet1/1/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is xxxxx
Description: "Link to "
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:02, output hang never
Last clearing of "show interface" counters 2d05h
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 2632463712
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 134000 bits/sec, 160 packets/sec
5 minute output rate 1909000 bits/sec, 231 packets/sec
42106757 packets input, 17561926962 bytes, 0 no buffer
Received 3887697 broadcasts (3665080 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 3665080 multicast, 0 pause input
0 input packets with dribble condition detected
49175423 packets output, 48495711222 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

My Output policy

Policy Map EGRESS_policy
Class Voice_EGRESS
priority level 1 10 (%)
Class VIDEO_EGRESS
priority level 2 20 (%)
Class SGL_EGRESS
bandwidth remaining 5 (%)
queue-buffers ratio 10
Class BUSINESS_EGRESS
bandwidth remaining 30 (%)
queue-buffers ratio 30
Class HTD_EGRESS
bandwidth remaining 10 (%)
queue-buffers ratio 10
Class class-default
bandwidth remaining 25 (%)
queue-buffers ratio 25
queue-limit dscp cs1 percent 80

Switch#$fed switch active qos queue config interface te1/1/1
DATA Port:52 GPN:1344 AFD:Enabled QoSMap:0 HW Queues: 416 - 423
DrainFast:Disabled PortSoftStart:2 - 10800
----------------------------------------------------------
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 6 65 6 65 0 0 0 0 6 14400
1 1 7 60 7 2880 2 180 0 0 6 14400
2 1 4 0 11 2400 2 150 1 75 6 14400
3 1 4 0 12 7200 2 450 1 225 6 14400
4 1 4 0 11 2400 2 150 1 75 6 14400
5 1 4 0 13 6000 2 375 1 187 6 14400
6 1 4 0 5 0 0 0 0 0 6 14400
7 1 4 0 5 0 0 0 0 0 6 14400

Switch#$fed switch active qos queue stats interface te1/1/1

DATA Port:52 Enqueue Counters
-------------------------------
Queue Buffers Enqueue-TH0 Enqueue-TH1 Enqueue-TH2
----- ------- ----------- ----------- -----------
0 0 0 48172083 21562065282
1 0 0 0 1404210692
2 0 0 0 665075768
3 0 0 0 28934704
4 0 0 0 5574111
5 0 20081050 0 56583672096
6 0 0 0 0
7 0 0 0 0

DATA Port:52 Drop Counters
-------------------------------
Queue Drop-TH0 Drop-TH1 Drop-TH2 SBufDrop QebDrop
----- ----------- ----------- ----------- ----------- -----------
0 0 67878 34555331980 0 0
1 0 0 0 0 0
2 0 0 0 0 0
3 0 0 0 0 0
4 0 0 0 0 0
5 0 0 0 0 0
6 0 0 0 0 0
7 0 0 0 0 0

See Queue drops in attached pic.

Thanks a lot.

Georg Pauwen · ‎12-16-2020

Hello,

looks like the drops occur in the first class (Voice_EGRESS) in your policy. Change the queue buffer ratios as below:

Policy Map EGRESS_policy
Class Voice_EGRESS
priority level 1 10 (%)
queue-buffers ratio 35
Class VIDEO_EGRESS
priority level 2 20 (%)
Class SGL_EGRESS
bandwidth remaining 5 (%)
queue-buffers ratio 10
Class BUSINESS_EGRESS
bandwidth remaining 30 (%)
queue-buffers ratio 10
Class HTD_EGRESS
bandwidth remaining 10 (%)
queue-buffers ratio 10
Class class-default
bandwidth remaining 25 (%)
queue-buffers ratio 10
queue-limit dscp cs1 percent 80

anil.kumar2508 · ‎12-16-2020

Hi George,

Thanks for the reply but unfortunately drops are still increasing.

Switch#sh int te1/1/1
TenGigabitEthernet1/1/1 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is xxxxxxxx
Description: "Link "
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseSX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:02, output hang never
Last clearing of "show interface" counters 00:05:48
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 18751648
Queueing strategy: Class-based queueing
Output queue: 0/40 (size/max)
5 minute input rate 129000 bits/sec, 157 packets/sec
5 minute output rate 1940000 bits/sec, 238 packets/sec
54419 packets input, 6098718 bytes, 0 no buffer
Received 6607 broadcasts (6295 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 6295 multicast, 0 pause input
0 input packets with dribble condition detected
84137 packets output, 86081217 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

Policy Map EGRESS_policy
Class Voice
priority level 1 10 (%)
queue-buffers ratio 35
Class VIDEO_EGRESS
priority level 2 20 (%)
Class SGL_EGRESS
bandwidth remaining 5 (%)
queue-buffers ratio 10
Class BUSINESS_EGRESS
bandwidth remaining 30 (%)
queue-buffers ratio 10
Class HTD_EGRESS
bandwidth remaining 10 (%)
queue-buffers ratio 10
Class class-default
bandwidth remaining 25 (%)
queue-buffers ratio 10
queue-limit dscp cs1 percent 80

Can we remove percent from priority Queue 1? what would be the effect? Also can we somehow increase the burst size?

Georg Pauwen · ‎12-16-2020

Hello,

the problem is possible with the other side. What do you have connected on the other side of the port channel ? Check the corresponding interface for input errors or input queue drops...

anil.kumar2508 · ‎12-16-2020

Hi,

Other side it is connected to cisco WS-C6509-E. No errors or output drops on port..

interface GigabitEthernet1/6/4
description Critical
switchport
switchport trunk allowed vlan 1,72,168,253,313,431,432,610,741-744,835
switchport mode trunk
channel-group 103 mode active
service-policy type lan-queuing input QUEUING-2Q4T-IN
service-policy type lan-queuing output QUEUING-1P3Q4T-OUT
end

GigabitEthernet1/6/4 is up, line protocol is up (connected)
Hardware is C6k 1000Mb 802.3, address is xx.xx.xx.
Description: Critical
MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 1000BaseSX
input flow-control is off, output flow-control is off
Clock mode is auto
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of "show interface" counters 00:25:29
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 3060000 bits/sec, 459 packets/sec
5 minute output rate 466000 bits/sec, 190 packets/sec
664585 packets input, 557439144 bytes, 0 no buffer
Received 839 broadcasts (733 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
0 input packets with dribble condition detected
716725 packets output, 350182572 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 PAUSE output
0 output buffer failures, 0 output buffers swapped out

Georg Pauwen · ‎12-16-2020

Hello,

on interface GigabitEthernet1/6/4, try and turn flow control on:

flowcontrol receive on

anil.kumar2508 · ‎12-16-2020

Still No change after the flowcontrol receive on

Joseph W. Doherty · ‎12-16-2020

Switch#$fed switch active qos queue config interface te1/1/1
DATA Port:52 GPN:1344 AFD:Enabled QoSMap:0 HW Queues: 416 - 423
DrainFast:Disabled PortSoftStart:2 - 10800
----------------------------------------------------------
DTS Hardmax Softmax PortSMin GlblSMin PortStEnd
----- -------- -------- -------- -------- ---------
0 1 6 65 6 65 0 0 0 0 6 14400
1 1 7 60 7 2880 2 180 0 0 6 14400
2 1 4 0 11 2400 2 150 1 75 6 14400
3 1 4 0 12 7200 2 450 1 225 6 14400
4 1 4 0 11 2400 2 150 1 75 6 14400
5 1 4 0 13 6000 2 375 1 187 6 14400
6 1 4 0 5 0 0 0 0 0 6 14400
7 1 4 0 5 0 0 0 0 0 6 14400

I believe, what I've highlighted, above, is the cause of your drops.

From my reading, with "qos queue-softmax-multiplier 1200" already applied, I would expect a larger value. Maybe a bug?

Anyway what can you try, beyond trying moving to a later IOS version?

Your 9300-48U should be a dual ASIC model, and buffer resources are per ASIC. So, using a different port, if possible, might relieve the problem (also perhaps why you don't see this issue on the other Etherchannel link - unless that's because of hashing choice).

You also might try a variant of what Georg already suggested, using the "queue-buffer ratio" command, but just on Voice class (alone).

You might also try increasing the bandwidth allocation to this class (or removing it, which I think just disables the policer). BTW, the priority command also offers a burst parameter, but I don't think you're hitting the implicit police limit. (So why change bandwidth allocation? It might increase buffer allocations.)

You might also try the class queue-limit command, on this class.

Further, unless you know you have a lot of Voice traffic, as a level 1 PQ, seeing queuing for this traffic would seem unusual, especially as microbursts (and as your 5 minute load average is low). You might confirm some other traffic isn't incorrectly/"illegally" using this class.

anil.kumar2508 · ‎12-16-2020

Hi Joseph,

i did couple of tests on another switch facing same issue.

1) Removed output policy from the port----> No output drops at all.

2) I swapped the P1 Queue with P2 Queue i.e. my voice traffic now hitting P2 Queue----> But still i saw drops on P1 Queue only. No drops on P2 Queue . So i think we can rule out issue with my voice traffic.

In this switch 5 security cameras are connected and having very less traffic.

Even if i increase PQ1 percent to 30% still packet drops are there. priority level 1 percent 30

So why only PQ1 is dropping traffic.

Why need to add buffers-ratio even if traffic volume is low. Same Traffic is not dropping on PQ2 and no queue-buffers allocated to this queue.

Georg Pauwen · ‎12-16-2020

Hello,

actually, one of Joseph's suggestions was what I was thinking of next. No queue buffers ratios on anything but the voice class:

Policy Map EGRESS_policy
Class Voice_EGRESS
priority level 1 10 (%)
queue-buffers ratio 75
Class VIDEO_EGRESS
priority level 2 20 (%)
Class SGL_EGRESS
bandwidth remaining 5 (%)
Class BUSINESS_EGRESS
bandwidth remaining 30 (%)
Class HTD_EGRESS
bandwidth remaining 10 (%)
Class class-default
bandwidth remaining 25 (%)
queue-limit dscp cs1 percent 80

Joseph W. Doherty · ‎12-17-2020

Yea, what I would like to know/see, is with such a change, whether Q0's Softmax value changes.

Joseph W. Doherty · ‎12-17-2020

"So why only PQ1 is dropping traffic."

Possibly for the same reason I've already noted, i.e. Q0's Softmax value is too small. (I'm still wondering if it's a bug.)

When you removed the policy-map you also changed over to the default queuing policy, which uses different buffer values.

Other possible solutions, don't configure two level PQ. Or just use the level 2 PQ for all your real-time traffic.