AF22 drops in ISR4451 while the link is not congested

vaibhav-shukla22 · ‎05-09-2024

Hi I have a problem and I am finding it really hard to get thee right answer from documentations:

The customer has 100Mb link and below are the bandwidth allocation to different classes:
100m port = EF = 17.5m / AF1 = 5.5m / AF2 = 16.5m / AF3 = 22m / AF4 = 11m

The customer is experiencing packet drops in the AF22 class. Overall, link utilization is not going beyond 30% of 100Mb. Now, their concern is why AF22 is dropping packets if the bandwidth is available. Also, if the AF22 class is full, will it do a tail drop even if the bandwidth is available on the link and other classes? QOS should only kick in when the link is congested but in this case, bandwidth on the link is not going beyond 30%.

Cust-Router#sh policy-map int gi0/0/1 output
GigabitEthernet0/0/1

Service-policy output: Policy-OUT-G0/0/1

Class-map: class-default (match-any)
1525832486 packets, 445519751019 bytes
5 minute offered rate 4135000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 409 packets
(queue depth/total drops/no-buffer drops) 0/931123/0
(pkts output/bytes output) 1521130809/444997955872
shape (average) cir 100000000, bc 400000, be 400000
target shape rate 100000000
Overhead Accounting Enabled

Service-policy : Cust-OUT-G0/0/1

queue stats for all priority classes:
Queueing
queue limit 512 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 69514603/12760486459

Class-map: ce_mgmt_bundled_output (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: access-group 199
Queueing
queue limit 4688 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 0/0
police:
cir 8000 bps, bc 8000 bytes, be 16000 bytes
conformed 0 packets, 0 bytes; actions:
set-prec-transmit 6
exceeded 0 packets, 0 bytes; actions:
set-prec-transmit 6
violated 0 packets, 0 bytes; actions:
set-prec-transmit 6
conformed 0000 bps, exceeded 0000 bps, violated 0000 bps
bandwidth remaining ratio 14
Overhead Accounting enabled

Exp-weight-constant: 6 (1/64)
Mean queue depth: 0 packets
class Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 0/0 0/0 0/0 1172 2344 1/10
1 0/0 0/0 0/0 1318 2344 1/10
2 0/0 0/0 0/0 1465 2344 1/10
3 0/0 0/0 0/0 1611 2344 1/10
4 0/0 0/0 0/0 1758 2344 1/10
5 0/0 0/0 0/0 1904 2344 1/10
6 0/0 0/0 0/0 176 352 1/10
7 0/0 0/0 0/0 2197 2344 1/10

Class-map: ce_ef_output (match-any)
69514603 packets, 12760486459 bytes
5 minute offered rate 224000 bps, drop rate 0000 bps
Match: dscp cs5 (40) ef (46)
police:
cir 21584000 bps, bc 30000 bytes, be 30000 bytes
conformed 69514603 packets, 12760486459 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
drop
violated 0 packets, 0 bytes; actions:
drop
conformed 224000 bps, exceeded 0000 bps, violated 0000 bps
Priority: Strict, b/w exceed drops: 0

Class-map: ce_af4_output (match-any)
7026867 packets, 6166093876 bytes
5 minute offered rate 88000 bps, drop rate 0000 bps
Match: dscp cs4 (32) af41 (34) af42 (36) af43 (38)
Queueing
queue limit 4688 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 7026867/6166093876
bandwidth remaining ratio 174
Overhead Accounting enabled

Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

cs4 20/2360 0/0 0/0 1758 2344 1/10
af41 0/0 0/0 0/0 611 1222 1/10
af42 6716655/5954167244 0/0 0/0 183 489 1/10
af43 310192/211924272 0/0 0/0 183 489 1/10

Class-map: ce_af3_output (match-any)
2627325 packets, 873212147 bytes
5 minute offered rate 7000 bps, drop rate 0000 bps
Match: dscp cs3 (24) af31 (26) af32 (28) af33 (30)
Queueing
queue limit 4688 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 2627325/873212147
bandwidth remaining ratio 349
Overhead Accounting enabled

Exp-weight-constant: 10 (1/1024)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

cs3 2423080/864605841 0/0 0/0 1611 2344 1/10
af31 202494/8504748 0/0 0/0 863 1726 1/10
af32 1751/101558 0/0 0/0 259 690 1/10
af33 0/0 0/0 0/0 259 690 1/10

Class-map: ce_af2_output (match-any)
786936776 packets, 142616579534 bytes
5 minute offered rate 862000 bps, drop rate 0000 bps
Match: dscp cs2 (16) af21 (18) af22 (20) af23 (22)
Queueing
queue limit 4688 packets
(queue depth/total drops/no-buffer drops) 0/842684/0
(pkts output/bytes output) 786094092/142465158947
bandwidth remaining ratio 262
Overhead Accounting enabled

Exp-weight-constant: 10 (1/1024)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

cs2 20/2360 0/0 0/0 1465 2344 1/10
af21 847541/62180630 0/0 0/0 750 1500 1/10
af22 785246531/142402975957 245351/42443765 597333/108976822 225 600 1/10
af23 0/0 0/0 0/0 225 600 1/10

Class-map: ce_af1_output (match-any)
10229 packets, 880699 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp cs1 (8) af11 (10) af12 (12) af13 (14)
Queueing
queue limit 4688 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 10229/880699
bandwidth remaining ratio 87
Overhead Accounting enabled

Exp-weight-constant: 8 (1/256)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

cs1 144/15952 0/0 0/0 1318 2344 1/10
af11 2834/289388 0/0 0/0 431 862 1/10
af12 7231/572999 0/0 0/0 129 345 1/10
af13 20/2360 0/0 0/0 129 345 1/10

Class-map: class-default (match-any)
659716684 packets, 283102498123 bytes
5 minute offered rate 2932000 bps, drop rate 1000 bps
Match: any
Queueing
queue limit 4688 packets
(queue depth/total drops/no-buffer drops) 0/88439/0
(pkts output/bytes output) 655857693/282732123744
bandwidth remaining ratio 98
Overhead Accounting enabled

Exp-weight-constant: 8 (1/256)
Mean queue depth: 0 packets
class Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 641804972/279764959667 43862/51708675 44577/54727064 458 824 1/5
1 0/0 0/0 0/0 458 824 1/5
2 0/0 0/0 0/0 458 824 1/5
3 0/0 0/0 0/0 458 824 1/5
4 0/0 0/0 0/0 458 824 1/5
5 0/0 0/0 0/0 458 824 1/5
6 14052721/2967164077 0/0 0/0 915 1830 1/10
7 0/0 0/0 0/0 458 824 1/5

chrihussey · ‎05-14-2024

Is the traffic coming into the router on a Gig interface and then existing on the 100Mbps interface? If this is the case it would be easy for the 100Mbps queue to easily get overwhelmed especially with bursty traffic. I've seen that on both switch and router platforms with and without QoS applied. If you're not exceeding 30% of the link, perhaps remove the policy map and see if the results are the same.

Joseph W. Doherty · ‎05-14-2024

"Overall, link utilization is not going beyond 30% of 100Mb."

Laugh, if I only had a nickel for every time utilization is ONLY ##%! (BTW, generally understanding QoS for bandwidth management is often not well understood, but then, IMO, most QoS training is poor too.)

Your drop stats say otherwise (about link loading), also there are drops in the class-default class too.

Insufficient information to say, but too many drops can actually cause utilization stats to decrease.

One of your possible problems might be your queues are too large. Yes, that might seem counter intuitive, but too large queues can make TCP flows believe there's more bandwidth than there actually is, and when too large queues overflow, besides possibly massive drop events, flow rates don't behave optimally.

I see WRED is being used, which I generally recommend NOT be used unless you're a QoS expert, but what's interesting about the AF22 WRED stats is the number of WRED tail drops, both that there are any at all and relative to the random drops.

Anyway to answer your questions. . .

Now, their concern is why AF22 is dropping packets if the bandwidth is available.

It's unclear whether bandwidth is actually available, but queue is filling enough to have WRED random drops or tail drops.

Also, if the AF22 class is full, will it do a tail drop even if the bandwidth is available on the link and other classes?

Again, unclear bandwidth is available. However, a bandwidth class can usually exceed and use other otherwise unused bandwidth.

QOS should only kick in when the link is congested but in this case, bandwidth on the link is not going beyond 30%.

Hmm, and how exactly are you measuring the 30% utilization? Whether QoS "kicks in" with congestion, depends on the nature of the QoS and what you consider to be congestion.

Personally, I consider an interface is congested anytime a packet cannot be immediately transmitted. Cisco's QoS isn't quite as sensitive. It usually first requires the interface hardware queue to overflow, or in this case (?) for a shaper to queue traffic and with shapers (or policers), you need to understand Bc/Be and Tc.