Solved: QOS. Total output drops without congestion.

Igor Patsuk · ‎01-19-2021

I am seeing behavior I don't understand. I hope someone can help.

Total output drops without congestion.

c1100-universalk9_ias.16.09.06.SPA.bin

RT#sh domain iwan1 master exits

RT#sh int gi0/0/0
GigabitEthernet0/0/0 is up, line protocol is up
Hardware is C1111-2x1GE, address is 2c4f.526d.4c00 (bia 2c4f.526d.4c00)
Description: -=C=-
Internet address is 10.254.19.28/20
MTU 1500 bytes, BW 5000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 6/255, rxload 8/255
Encapsulation ARPA, loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is off, input flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:26, output hang never
Last clearing of "show interface" counters never
Input queue: 0/1024/0/0 (size/max/drops/flushes); Total output drops: 1942787
Queueing strategy: Class-based queueing
Output queue: 0/1024 (size/max)
5 minute input rate 163000 bits/sec, 73 packets/sec
5 minute output rate 122000 bits/sec, 71 packets/sec
114369553 packets input, 28152144430 bytes, 0 no buffer
Received 917047 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 56266 multicast, 0 pause input
121874616 packets output, 25368860800 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
389107 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

RT#sh policy-map int gi0/0/0
GigabitEthernet0/0/0

Service-policy output: POLICY-TRANSPORT-13

Class-map: class-default (match-any)
123819380 packets, 26639571390 bytes
5 minute offered rate 125000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/1942763/0
(pkts output/bytes output) 121472744/24815338169
shape (average) cir 5000000, bc 20000, be 20000
target shape rate 5000000

Service-policy : WAN

queue stats for all priority classes:
Queueing
priority level 1
queue limit 512 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 5289123/1106953958

Class-map: INTERACTIVE-VIDEO (match-any)
663 packets, 123318 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp cs4 (32) af41 (34) af42 (36) af43 (38)
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 663/123318
bandwidth remaining 30%
Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

af41 663/123318 0/0 0/0 28 32 1/10
QoS Set
dscp af41
Marker statistics: Disabled

Class-map: STREAMING-VIDEO (match-any)
4319103 packets, 1057699850 bytes
5 minute offered rate 10000 bps, drop rate 0000 bps
Match: dscp af31 (26) af32 (28) af33 (30) cs5 (40)
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 4319103/1057699850
bandwidth remaining 10%
Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

af31 4319103/1057699850 0/0 0/0 28 32 1/10
QoS Set
dscp af31
Marker statistics: Disabled

Class-map: NET-CTRL (match-any)
1319800 packets, 494700635 bytes
5 minute offered rate 4000 bps, drop rate 0000 bps
Match: dscp cs6 (48)
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 915951/428231182
bandwidth remaining 5%
QoS Set
dscp cs6
Marker statistics: Disabled

Class-map: CALL-SIGNALING (match-any)
1546923 packets, 287727678 bytes
5 minute offered rate 1000 bps, drop rate 0000 bps
Match: dscp cs3 (24)
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 1546923/287727678
bandwidth remaining 4%
QoS Set
dscp cs6
Marker statistics: Disabled

Class-map: CRITICAL-DATA (match-any)
37466715 packets, 6584313862 bytes
5 minute offered rate 25000 bps, drop rate 0000 bps
Match: dscp af11 (10) af12 (12) af13 (14) cs2 (16) af21 (18) af22 (20) af23 (22)
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/1683/0
(pkts output/bytes output) 37465032/6582259168
bandwidth remaining 25%
Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

af21 37465032/6582259168 0/0 498/640620 28 32 1/10
QoS Set
dscp af21
Marker statistics: Disabled

Class-map: SCAVENGER (match-any)
291309 packets, 61347722 bytes
5 minute offered rate 1000 bps, drop rate 0000 bps
Match: dscp cs1 (8)
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/4/0
(pkts output/bytes output) 291305/61344074
bandwidth remaining 1%
QoS Set
dscp af11
Marker statistics: Disabled

Class-map: VOICE (match-any)
5289147 packets, 1106960918 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp ef (46)
Priority: Strict, b/w exceed drops: 0

Priority Level: 1
police:
cir 10 %
cir 500000 bps, bc 15625 bytes
conformed 5289123 packets, 1106953958 bytes; actions:
transmit
exceeded 24 packets, 6960 bytes; actions:
drop
conformed 0000 bps, exceeded 0000 bps
QoS Set
dscp ef
Marker statistics: Disabled

Class-map: class-default (match-any)
73585720 packets, 17046697407 bytes
5 minute offered rate 77000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/1941076/0
(pkts output/bytes output) 71644644/15290998941
bandwidth remaining 25%
Exp-weight-constant: 9 (1/512)

Mean queue depth: 0 packets
class Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 69893739/14965300407 20867/19540892 528197/479991494 16 32 1/10
1 0/0 0/0 0/0 18 32 1/10
2 0/0 0/0 0/0 20 32 1/10
3 0/0 0/0 0/0 22 32 1/10
4 0/0 0/0 0/0 24 32 1/10
5 0/0 0/0 0/0 26 32 1/10
6 60278/11290564 0/0 9/1674 28 32 1/10
7 1690627/314407970 1/186 419/77934 30 32 1/10

Giuseppe Larosa · ‎01-19-2021

Hello @Igor Patsuk ,

the external policy creates a logical pipe at 5 Mpbs and this is inherited by child policy WAN

5000000

This is why you have drops in default class even if the physical GE is not congested.

bandwidth remaining percent 25
random-detect

It might be the case to test it without WRED enabled or to increase the BW assigned to default class.

In a case like yours WRED may be causing early drops befofe the queue is filled in an attempt to avoid congestion.

Hope to help

Giuseppe

View solution in original post

Georg Pauwen · ‎01-19-2021

Hello,

it looks like most drops occur in the default queues:

Service-policy output: POLICY-TRANSPORT-13

Class-map: class-default (match-any)
(queue depth/total drops/no-buffer drops) 0/1942763/0

Service-policy : WAN

Class-map: CRITICAL-DATA (match-any)
(queue depth/total drops/no-buffer drops) 0/1683/0

Class-map: class-default (match-any)
(queue depth/total drops/no-buffer drops) 0/1941076/0

Can you post the running config ? Not sure if I read your output properly, but do you have two service policies applied ?

Igor Patsuk · ‎01-19-2021

I don't understand, why? Not congestion, chanlд not utilisation 100%

Joseph W. Doherty · ‎01-19-2021

You don't need 100% utilization to have congestion. (Actually you can also have 100% utilization w/o congestion too.)

Anytime you have any packets, enqueued, i.e. waiting to be transmitted, you have congestion. This can happen even with low average, overtime, utilization.

An extreme case of this would be "microbursts", which you might want to research. E.g.: https://en.wikipedia.org/wiki/Micro-bursting_(networking)

Igor Patsuk · ‎01-19-2021

I thought about it but I don't know to track down.

Igor Patsuk · ‎01-19-2021

Yes, I can

policy-map WAN
class INTERACTIVE-VIDEO
bandwidth remaining percent 30
random-detect dscp-based
set dscp af41
class STREAMING-VIDEO
bandwidth remaining percent 10
random-detect dscp-based
set dscp af31
class NET-CTRL
bandwidth remaining percent 5
set dscp cs6
class CALL-SIGNALING
bandwidth remaining percent 4
set dscp cs6
class CRITICAL-DATA
bandwidth remaining percent 25
random-detect dscp-based
set dscp af21
class SCAVENGER
bandwidth remaining percent 1
set dscp af11
class VOICE
priority level 1
police cir percent 10
set dscp ef
class class-default
bandwidth remaining percent 25
random-detect
policy-map POLICY-TRANSPORT-12
class class-default
shape average 4000000
service-policy WAN
policy-map POLICY-TRANSPORT-13
class class-default
shape average 5000000
service-policy WAN
policy-map Input_DSCP
class NET-CTRL-in
set dscp cs6
class CALL-SIGNALING-in
set dscp af21
class VOICE-in
set dscp ef
class SCAVENGER-in
set dscp cs1
class INTERACTIVE-VIDEO-in
set dscp af41
class STREAMING-VIDEO-in
set dscp af31
class CRITICAL-DATA-in
set dscp af21

interface GigabitEthernet0/0/0
bandwidth 5000
description -=Connect to VOLZ=-
vrf forwarding IWAN-TRANSPORT-VOLZ
ip address
ip access-group ACL-IWAN-PUBLIC in
no ip redirects
no ip unreachables
no ip proxy-arp
negotiation auto
no cdp enable
no mop enabled
no lldp transmit
no lldp receive
service-policy output POLICY-TRANSPORT-13
hold-queue 1024 in
hold-queue 1024 out

Giuseppe Larosa · ‎01-19-2021

Hello @Igor Patsuk ,

the external policy creates a logical pipe at 5 Mpbs and this is inherited by child policy WAN

5000000

This is why you have drops in default class even if the physical GE is not congested.

bandwidth remaining percent 25
random-detect

It might be the case to test it without WRED enabled or to increase the BW assigned to default class.

In a case like yours WRED may be causing early drops befofe the queue is filled in an attempt to avoid congestion.

Hope to help

Giuseppe

Igor Patsuk · ‎01-19-2021

I was do the case to test it without WEED in result increase tail drop.

Georg Pauwen · ‎01-19-2021

Hello,

in addition to Giuseppe's remarks, I have looked at the traffc that actually hits the various classes, and come up with the changes below (marked in bold). I would delete all the random-detects, as these can lead to unpredicatable results. I would also change the voice policer into a voice shaper, if your IOS supports that.

That said, 5Mbps is very low, by any standard. What kind of link/service is that ?

policy-map WAN
class INTERACTIVE-VIDEO
--> bandwidth remaining percent 15
--> no random-detect dscp-based
set dscp af41
class STREAMING-VIDEO
bandwidth remaining percent 10
--> no random-detect dscp-based
set dscp af31
class NET-CTRL
bandwidth remaining percent 5
set dscp cs6
class CALL-SIGNALING
bandwidth remaining percent 4
set dscp cs6
class CRITICAL-DATA
--> bandwidth remaining percent 25
--> no random-detect dscp-based
set dscp af21
class SCAVENGER
bandwidth remaining percent 1
set dscp af11
class VOICE
priority level 1
--> shape average 500000
set dscp ef
class class-default
--> bandwidth remaining percent 40
--> no random-detect
policy-map POLICY-TRANSPORT-12
class class-default
shape average 4000000
service-policy WAN
policy-map POLICY-TRANSPORT-13
class class-default
shape average 5000000
service-policy WAN
policy-map Input_DSCP
class NET-CTRL-in
set dscp cs6
class CALL-SIGNALING-in
set dscp af21
class VOICE-in
set dscp ef
class SCAVENGER-in
set dscp cs1
class INTERACTIVE-VIDEO-in
set dscp af41
class STREAMING-VIDEO-in
set dscp af31
class CRITICAL-DATA-in
set dscp af21