drops in queue on subinterface with out congestion (QOS) - Page 2

Vyacheslav Vlasov · ‎07-24-2014

Hello colleagues

The problem is, that i have ~ 7-8 Mbps on 65Mbps link, but my service policy seems to drop packets according to queue size.

In my understanding, policy map start to drop traffic only when congestion appear, is it so?

same situation on ASR 1002 and 7206

ios Version 15.2(4)S4 and Version 15.1(4)M7

Here is config for policy map on WAN link

class-map CRITICAL

match precedence 5

class-map match-any IMPORTANT

match precedence 3

match precedence 6

class-map BUSSINESS

match precedence 1

policy-map SHAPING

class class-default

shape average 65000000

service-policy SP-QOS

policy-map SP-QOS

Class CRITICAL

priority percent 70

Class IMPORTANT

bandwidth percent 12

Class BUSSINESS

bandwidth percent 12

random-detect dscp-based

Class class-default

bandwidth percent 6

random-detect

interface GigabitEthernet0/0/1.9

encapsulation dot1Q 9

ip address xxxxxxxxx

ip nat outside

ip nbar protocol-discovery

service-policy output SHAPING

and here, what I see in statistic

Service-policy output: SHAPING

Class-map: class-default (match-any)

129798770 packets, 40181696005 bytes

5 minute offered rate 7304000 bps, drop rate 0000 bps

Match: any

Queueing

queue limit 270 packets

(queue depth/total drops/no-buffer drops) 0/11680/0

(pkts output/bytes output) 129779213/40165064207

shape (average) cir 65000000, bc 260000, be 260000

target shape rate 65000000

Service-policy : SP-QOS

queue stats for all priority classes:

Queueing

queue limit 512 packets

(queue depth/total drops/no-buffer drops) 0/0/0

(pkts output/bytes output) 13484092/3912327041

Class-map: CRITICAL (match-all)

13480980 packets, 3911647909 bytes

5 minute offered rate 422000 bps, drop rate 0000 bps

Match: precedence 5

Priority: 70% (45500 kbps), burst bytes 1137500, b/w exceed drops: 0

Class-map: IMPORTANT (match-any)

7721261 packets, 1021340017 bytes

5 minute offered rate 108000 bps, drop rate 0000 bps

Match: precedence 3

Match: precedence 6

Queueing

queue limit 64 packets

(queue depth/total drops/no-buffer drops) 0/0/0

(pkts output/bytes output) 7715467/1020879486

bandwidth 12% (7800 kbps)

Class-map: BUSSINESS (match-all)

47724414 packets, 15630803933 bytes

5 minute offered rate 3393000 bps, drop rate 0000 bps

Match: precedence 1

Queueing

queue limit 64 packets

(queue depth/total drops/no-buffer drops) 0/7388/0

(pkts output/bytes output) 47741826/15628877447

bandwidth 12% (7800 kbps)

Exp-weight-constant: 4 (1/16)

Mean queue depth: 1 packets

dscp Transmitted Random drop Tail drop Minimum Maximum Mark

pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

af11 2990415/861605540 25/32285 645/884994 28 32 1/10

af12 677310/367870926 0/0 10/5466 24 32 1/10

af13 154681/48657019 3/1634 4/307 20 32 1/10

Class-map: class-default (match-any)

60817716 packets, 19603759092 bytes

5 minute offered rate 3363000 bps, drop rate 0000 bps

Match: any

Queueing

queue limit 64 packets

(queue depth/total drops/no-buffer drops) 0/4292/0

(pkts output/bytes output) 60837828/19602980233

bandwidth 6% (3900 kbps)

Exp-weight-constant: 4 (1/16)

Mean queue depth: 1 packets

class Transmitted Random drop Tail drop Minimum Maximum Mark

pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 3474437/1110326227 269/276936 742/851882 16 32 1/10

1 0/0 0/0 0/0 18 32 1/10

2 0/0 0/0 0/0 20 32 1/10

3 0/0 0/0 0/0 22 32 1/10

4 0/0 0/0 0/0 24 32 1/10

5 0/0 0/0 0/0 26 32 1/10

6 0/0 0/0 0/0 28 32 1/10

7 0/0 0/0 0/0 30 32 1/10

Vyacheslav Vlasov · ‎10-23-2014

Joseph, thanks a lot for your help!

I have only "QoS-FE:Implemeniting..." course as a background for QoS methods, and it was clear for me. But after your comments, i`ve understand, that sky is not so clean as i thought :) May be you can advice some helpfull material for deeper understanding (i`m now reading "IP Quality of Service [Srinivas Raju Vegesna]", but may be you can advice some thing more)?

Joseph W. Doherty · ‎10-23-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Unfortunately, I've never come across any really, really good source material that explained how to effectively use QoS in a production environment with real traffic.

Even simple "gotchas" like Cisco interface FIFO queue depth (tx-ring-limit) may need to be adjusted to meet QoS goals, but that's often overlooked.

I'll admit, when I first encountered WRED, I thought it was such a great idea (and in someways it still is), but nothing I initially read mentioned all the issues with it. (Oh, and again, don't misunderstand, it can be useful, but it's a tricky technology to really obtain its best.)

Or, on WRED, besides the most obvious adjustment of min and max thresholds, when should the moving average parameter be adjusted. Or, is 10% really the ideal value for the random drop percentage?

Did your course discuss those? (If not, that's not a knock against your course, it's just determining the correct values for all RED parameters is difficult.)

Anyway, all I can suggest, if you really want to understand QoS, is read a lot. Take any one source with a grain of salt. Read research about "improved" versions of QoS approaches, especially taking note of what's the problem or issue trying to be solved (sometimes more interesting then their proposed gee-wiz solution).

Monitor you results and don't be afraid to adjust, but also be prepared for making things worse. ;)

Vyacheslav Vlasov · ‎10-23-2014

yep, we discuss it :) actually - a lot of interesting information about qos tuning was on this course. But not so much practice on real WAN in my case. But, as for me - it`s very interesting to work on network improvement!

One more time - thank you for your help, time and good advices!

Have a nice day! :)

Akash Agrawal · ‎07-24-2014

Hi,

In this case i see two issues and few config tweaking can help us here. In case of shaping, packet gets queued only when traffic is crossing conform rate(packets (bits) arrived in Tc interval is more than Bc defined). Shaping rate of 65 mb for 1 sec is too high and we have very less traffic on the link but to check whether traffic conforming traffic contract or not, shaping tool uses Token bucket mechanism and in every Tc time interval it replenishes Bc amount of token. Here Bc is set very low which is just 260000 bits and if you calculate Tc it is 4 ms.

Bc = 260000 bits
CIR = 65000000 bits/sec
Tc = Bc/CIR = 260000/65000000
Tc = 4 ms

So every 4 ms, bucket will be filled with 260000 tokens. If within 4 ms router gets 21 packets (1500 bytes) so all tokens will be used and next packet will be queued in the respective queue. So for bursty traffic we may see queue getting filled for short duration and WRED is getting triggered. Probably it is happening for class BUSINESS and class-default.

Even for latency/jitter sensitive traffic Tc is recommended as 10ms or by default it gets calculated as 25ms. You can try below values of Bc and Be for respective Tc.

Tc = 25 ms ---> Bc = 1625000 , Be = 1625000

Tc = 10 ms ---> Bc = 650000 , Be = 650000

Second thing queue-limit for each class is 64 packets, but WRED minimum threshold is set as 16 and maximum as 32. If changing Bc,Be does not help then WRED threshold needs to be increased.

Please do not forget to rate helpful post.

-Akash

Vyacheslav Vlasov · ‎07-27-2014

Thank you Akash!

for your first suggestion - i belive this could happen, if we have a lot of burst traffic. But on monitoring systems, i didn`t see any splashes (30s check rate). May be this burst traffic lasts not too long and i didn`t see it on monitoring...

I`ll try to set 10ms Tc, and will see what happened.

second suggestion - i think you right. That was my confusion, couse i forget the way how WRED works.

Thank you for your help!

Vyacheslav Vlasov · ‎07-27-2014

still see some tail drops in class-default (current Bc is 1625000)

Service-policy output: SHAPING

Class-map: class-default (match-any)
621651 packets, 242393853 bytes
5 minute offered rate 4862000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 270 packets
(queue depth/total drops/no-buffer drops) 0/159/0
(pkts output/bytes output) 621463/242224044
shape (average) cir 65000000, bc 1625000, be 1625000
target shape rate 65000000

Service-policy : SP-QOS

queue stats for all priority classes:
Queueing
queue limit 512 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 26105/5613542

Class-map: CRITICAL (match-all)
26105 packets, 5613542 bytes
5 minute offered rate 116000 bps, drop rate 0000 bps
Match: precedence 5
Priority: 70% (45500 kbps), burst bytes 1137500, b/w exceed drops: 0

Class-map: IMPORTANT (match-any)
17897 packets, 1842729 bytes
5 minute offered rate 41000 bps, drop rate 0000 bps
Match: precedence 3
Match: precedence 6
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 17881/1841225
bandwidth 12% (7800 kbps)

Class-map: BUSSINESS (match-all)
204242 packets, 55251009 bytes
5 minute offered rate 1112000 bps, drop rate 0000 bps
Match: precedence 1
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 204242/55251009
bandwidth 12% (7800 kbps)
Exp-weight-constant: 4 (1/16)
Mean queue depth: 1 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

af11 171891/43885889 0/0 0/0 28 32 1/10
af12 24896/6077339 0/0 0/0 24 32 1/10
af13 7455/5287781 0/0 0/0 20 32 1/10

Class-map: class-default (match-any)
373394 packets, 179683059 bytes
5 minute offered rate 3607000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/159/0
(pkts output/bytes output) 373235/179518268
bandwidth 6% (3900 kbps)
Exp-weight-constant: 4 (1/16)
Mean queue depth: 1 packets
class Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 373227/179516812 47/51277 112/113514 16 32 1/10
1 0/0 0/0 0/0 18 32 1/10
2 0/0 0/0 0/0 20 32 1/10
3 0/0 0/0 0/0 22 32 1/10
4 8/1456 0/0 0/0 24 32 1/10
5 0/0 0/0 0/0 26 32 1/10
6 0/0 0/0 0/0 28 32 1/10
7 0/0 0/0 0/0 30 32 1/10

Akash Agrawal · ‎07-28-2014

Hi,

As I mentioned before, please try to increase WRED minimum,maximum threshold. right now it is 16 32, you may increase it to 32 48.

Vyacheslav Vlasov · ‎07-28-2014

Done, cleared statistic - get TailDrop hit in a first few seconds

Class-map: class-default (match-any)
60351 packets, 22005334 bytes
5 minute offered rate 559000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/4/0
(pkts output/bytes output) 60347/22003490
bandwidth 6% (3900 kbps)
Exp-weight-constant: 4 (1/16)
Mean queue depth: 1 packets
class Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 60345/22003126 3/1774 1/70 32 48 1/10
1 0/0 0/0 0/0 18 32 1/10
2 0/0 0/0 0/0 20 32 1/10
3 0/0 0/0 0/0 22 32 1/10
4 2/364 0/0 0/0 24 32 1/10
5 0/0 0/0 0/0 26 32 1/10
6 0/0 0/0 0/0 28 32 1/10
7 0/0 0/0 0/0 30 32 1/10

Akash Agrawal · ‎07-28-2014

If you see number of drops have been reduced significantly. Earlier there were 159 drops (random + tail) in total 373227 transmitted packets which means 1 drop out of 2347 packets. Now it is 1 drop out of 15086 packets. So drop rate has been reduced by 6 times. If you want to reduce drop further, need to increase WRED threshold more. If you have bursty traffic in network, you need to have larger queue to hold the bursty traffic.

Vyacheslav Vlasov · ‎07-28-2014

i`ve checked my first post statistic for Tail Drop -

class Transmitted Random drop Tail drop Minimum Maximum Mark

pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 3474437/1110326227 269/276936 742/851882 16 32 1/10

So it about 1 of 12916 packet drop rate

today statistic

class Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 32108690/9746181825 895/1117314 3940/4774446 32 48 1/10

So it about 1 of 8149 packet is Tail droped.

I`ll try to set thresh a little bit larger, but don`t look like it helpes a lot.

Maybe set Bc larger?

Akash Agrawal · ‎07-30-2014

Hi,

Definitely you can try with increasing Bc. However would like to share below info related to WRED max threshold in HQF IOS.

The IOS you are using, i guess HQF is implemented and in HQF queue-limit of each class takes precedence over max-threshold of the WRED unlike in pre-HQF images. By default class queue-limit is 64 so WRED threshold can not be exceed more than the queue-limit(64). If you want to increase WRED max threshold more than 64 then queue-limit also needs to be increased. You can try to increase queue-limit to 1000 and WRED also as min-600 ,max-1000. Just for testing.

Vyacheslav Vlasov · ‎08-01-2014

thanks, i`ll try!