Traffic shaping queue limit and packet drops

MARTIN LAUB · ‎01-16-2014

Hi, I have an ASR1001 with the below nested policy map applied to a GE WAN interface on the outbound direction. The parent policy-map is just a shaper and the child policy-map does the actual class prioritization.

policy-map QoS_Out_to_rest

 class Mgmt_Out

  bandwidth 16

  set ip precedence 6

 class Voice_Out

  police 150000000 20480000 40960000 conform-action set-prec-transmit 5 exceed-action drop  violate-action drop

  priority

 class Platinum_Out

  bandwidth 198000

  random-detect

  random-detect precedence 3 1045 1065

  police 198000000 50181000 10036200 conform-action set-prec-transmit 3 exceed-action set-prec-transmit 3 violate-action set-prec-transmit 3

 class Silver_Out

  bandwidth 12000

  random-detect

  police 12000000 2457000 737200 conform-action set-prec-transmit 1 exceed-action set-prec-transmit 1 violate-action set-prec-transmit 1

 class class-default

  bandwidth 239840

  random-detect

  police 239840000 44997000 90165000 conform-action set-prec-transmit 0 exceed-action set-prec-transmit 0 violate-action set-prec-transmit 0


policy-map Shaping_pmap

 class class-default

  shape average 600000000 60000000 60000000

  service-policy QoS_Out_to_rest

The problem I'm having is a high packet drop rate seen in the class-default on the child policy, which reflects in the parent policy as shown below in red:

WAN-CPE#sh policy-map interface gigabitEthernet 0/0/1 output

GigabitEthernet0/0/1

Service-policy output: Shaping_pmap

Class-map: class-default (match-any)

145971879 packets, 46014416510 bytes

30 second offered rate 174778000 bps, drop rate 114000 bps

Match: any

Queueing

queue limit 2499 packets

(queue depth/total drops/no-buffer drops) 0/34125/0

(pkts output/bytes output) 145937765/45977616422

shape (average) cir 600000000, bc 60000000, be 60000000

target shape rate 600000000

Service-policy : QoS_Out_to_rest

queue stats for all priority classes:

Queueing

queue limit 512 packets

(queue depth/total drops/no-buffer drops) 0/0/0

(pkts output/bytes output) 86981436/18979811341

'.............................................

...............................................

Class-map: class-default (match-any)

57822243 packets, 26207121807 bytes

30 second offered rate 101754000 bps, drop rate 153000 bps

Match: any

Queueing

queue limit 998 packets

(queue depth/total drops/no-buffer drops) 0/34125/0

(pkts output/bytes output) 57788118/26170310963

bandwidth 239840 kbps

Exp-weight-constant: 4 (1/16)

Mean queue depth: 1 packets

class Transmitted Random drop Tail drop Minimum Maximum Mark

pkts/bytes pkts/bytes pkts/bytes thresh thresh prob

0 57788118/26170310963 10331/11180773 23794/25630071 24 49 1/10

1 0/0 0/0 0/0 27 49 1/10

2 0/0 0/0 0/0 30 49 1/10

3 0/0 0/0 0/0 33 49 1/10

4 0/0 0/0 0/0 36 49 1/10

5 0/0 0/0 0/0 39 49 1/10

6 0/0 0/0 0/0 42 49 1/10

7 0/0 0/0 0/0 45 49 1/10

police:

I tried increasing the queue-limit on the child pmap default class up to 3-fold but the traffic drops continued at virtually the smae rate. Then I tried increasing the queue limit on the parent pmap but the IOS apparently dioes not support manually setting the queue-limit on the parent pmap.

Is there any other option I could try? The shaper average rate is well above the actual average traffic so more bandwidth doesn't seem to be the solution.

Thanks in advance for any suggestions on the matter.

Martin

Vasilii Mikhailovskii · ‎01-16-2014

Hello.

I think you need to tune min/max threshold as it's done under Platinum_Out class.

I would guess that 500000 bytes (1000-1400 packets) for max threshold (and twice less for min threshold) could be enough, but try different values.

Please aware that ARS is doing QoS in hardware, so there is a slight delay after configuration before it takes effect. Also examine the log to be aware of any error!

paul driver · ‎01-16-2014

Hello

Your qos config doesn't seem correct me, your policing outbound for one and applying congestion avoidance to the child default class where I think you should be applying some degree of fairness to this catch all class.

For voice traffic I would suggest using LLQ instead of policing and for all the other policed traffic I would personally police this as it enters the router ( lan facing) and then apply some congestion management relating to this traffic on the wan links outbound using hierarichical qos (HQF)

res

Paul

Please don't forget to rate any posts that have been helpful.

Thanks.

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Ashish Arora · ‎01-16-2014

Hi,

It seems like a queuing issue , before providing any feedback here , i would like to know the following things:-

1. Show interface gig (repeat output for 3-4 consecutive attempts).

2. Make sure to put load-interval 30 command under the gig interface before providing any outputs.

3. Are you we sure we are getting CIR 600 megs for this cirtuit, Can we test it by applying legacy QOS (rate-limiting) or Policing.

4. Aren't we geeting any burty traffic ?, This can be checked by a packet capture for 10-15 minutes and see the same under wireshark (go to statastics >> IO graph). Please share this output also.

5.since this is 600 megs link , we should need to check the output hold queue as paskets are getting tail dropped. so need to increase the size slowly and see the behaviour.(Note if we will increase the size at a higher number , we may expect voice delay),so please check the production impact befor applying it.

6. What is the actual issue are we experiance , Do you feel slowness or voice issue or any other issue.

7. I will provide you an action plan after seeing the output asked above and :-

Show interface gig stats

show interface gig switching.

-THank you

Ashish Arora

MARTIN LAUB · ‎01-16-2014

Hi Ashish, please see the command outputs below. Keep in mind that the traffic volume dropped significantly since I opened the discussion, and so did packet drops.

Here are the answers to your questions:

1 and 2: please see below

3 the WAN circuit is a 1 GigE cirtuit. The provider is rate-limiting to 650 M so I'm shaping at 600M just in case

4 traffic IS bursty.There are some applications sending out 1000-UDP packet bursts or so

5 How can I check this please?

6 slow file transfers (topped at 2M). nuttcp shows a lot of TCP retransmissions due to packet drops. Voice is fine. NO end user complaints

7. thanks !

WAN-CPEF#sh int gigabitEthernet 0/0/1

GigabitEthernet0/0/1 is up, line protocol is up

Hardware is ASR1001, address is f0f7.5559.5601 (bia f0f7.5559.5601)

Description: WAN

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 16/255, rxload 35/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive not supported

Full Duplex, 1000Mbps, link type is force-up, media type is LX

output flow-control is on, input flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:05:43, output 00:05:43, output hang never

Last clearing of "show interface" counters 01:31:11

Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 415

Queueing strategy: Class-based queueing

Output queue: 0/40 (size/max)

30 second input rate 137456000 bits/sec, 28430 packets/sec

30 second output rate 66031000 bits/sec, 23256 packets/sec

190300536 packets input, 120170194379 bytes, 0 no buffer

Received 2015 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

152849701 packets output, 55506977247 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

WAN-CPE#sh int gigabitEthernet 0/0/1

GigabitEthernet0/0/1 is up, line protocol is up

Hardware is ASR1001, address is f0f7.5559.5601 (bia f0f7.5559.5601)

Description: WAN

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 16/255, rxload 35/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive not supported

Full Duplex, 1000Mbps, link type is force-up, media type is LX

output flow-control is on, input flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:05:45, output 00:05:45, output hang never

Last clearing of "show interface" counters 01:31:12

Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 415

Queueing strategy: Class-based queueing

Output queue: 0/40 (size/max)

30 second input rate 137456000 bits/sec, 28430 packets/sec

30 second output rate 66031000 bits/sec, 23256 packets/sec

190300536 packets input, 120170194379 bytes, 0 no buffer

Received 2015 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

152849701 packets output, 55506977247 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

WAN-CPE#sh int gigabitEthernet 0/0/1

GigabitEthernet0/0/1 is up, line protocol is up

Hardware is ASR1001, address is f0f7.5559.5601 (bia f0f7.5559.5601)

Description: WAN

MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,

reliability 255/255, txload 19/255, rxload 34/255

Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set

Keepalive not supported

Full Duplex, 1000Mbps, link type is force-up, media type is LX

output flow-control is on, input flow-control is on

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:05:46, output 00:05:46, output hang never

Last clearing of "show interface" counters 01:31:14

Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 415

Queueing strategy: Class-based queueing

Output queue: 0/40 (size/max)

30 second input rate 136538000 bits/sec, 28446 packets/sec

30 second output rate 76238000 bits/sec, 24406 packets/sec

190585301 packets input, 120338570846 bytes, 0 no buffer

Received 2021 broadcasts (0 IP multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 0 multicast, 0 pause input

153116779 packets output, 55627828181 bytes, 0 underruns

0 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

WAN-CPE#sh int gigabitEthernet 0/0/1 switching

GigabitEthernet0/0/1 WAN

Throttle count 0

Drops RP 2453 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 336612027 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 365 34994 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 848669476307 608144422524240 702953328190 280895435014649

Protocol ARP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 41295 2312520 381134 24392576

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 41295 2477700 381104 24390656

NOTE: all counts are cumulative and reset only after a reload.

WAN-CPE#sh int gigabitEthernet 0/0/1 switching

GigabitEthernet0/0/1 WAN

Throttle count 0

Drops RP 2453 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 336612075 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 365 34994 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 848669785041 608144625699596 702953598567 280895542993229

Protocol ARP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 41295 2312520 381134 24392576

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 41295 2477700 381104 24390656

WAN-CPEF#sh int gigabitEthernet 0/0/1 switching

GigabitEthernet0/0/1 WAN

Throttle count 0

Drops RP 2453 SP 0

SPD Flushes Fast 0 SSE 0

SPD Aggress Fast 0

SPD Priority Inputs 336612115 Drops 0

Protocol IP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 365 34994 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 848670068158 608144792696556 702953853921 280895645436152

Protocol ARP

Switching path Pkts In Chars In Pkts Out Chars Out

Process 41295 2312520 381134 24392576

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 0 0 0 0

Protocol Other

Switching path Pkts In Chars In Pkts Out Chars Out

Process 0 0 0 0

Cache misses 0 - - -

Fast 0 0 0 0

Auton/SSE 41295 2477700 381104 24390656

Ashish Arora · ‎01-16-2014

hi ,

THank you for pasting the outputs.

Since traffic rate is somethere close to 136 down and 67 Up. It seems to be ok for the link as Saturated bandwidth allowed is 650 Mbps ;however only 40 Packets can be buffered into the output queue , which is quite less when compared to 1 Gig link. I should say try to change it to 100 and keep on incrementing till 200.You can perform as follow:-

Interface gig 0/0/1

hold-queue 70 out

Check the behaviour

hold-queue 100 out

Check the behaviour now for 5 minutes atleast.

hold-queue 150 out ( monitor it for some time)

We shoudn't increase this a lot beause it may create issues for the voice traffic and experience a little delay in RTP traffic.

Please also try to remove the policy and put it back in.

In order to check the CIR, We should follow the policy :-

Policy-map TEST

class class-default

police rate 600000000

conform-action transmit

exceed-action drop

Now we need to send traffic till it reached the max limit and once it would start dropping , CIR can be roughly observed.

-Thank you.

Ashish

Vasilii Mikhailovskii · ‎01-16-2014

Hello, m.laub.

Have you tried the recommendation to add min/max thresholds for class-default?

policy-map QoS_Out_to_rest

class class-default

random-detect precedence 0 500 900

To adjust policy the good (or even best) practice is to: 1) remove it from interface, 2) adjust policy, 3) apply back to the interface.

Joseph W. Doherty · ‎01-17-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I haven't thoroughly read all the other posts, but I think I noticed someone else noticed your class-default drops are in your WRED settings, 24/49. Two third of your drops are WRED tail drops. Those (default?) values are very likely way too small for 240 Mbps.

BTW, I don't know why you need to rate limit your non-LLQ classes, but if the ASR1001 supports child policy class shaping, I would generally recommend it over policing. I would also recommend analysis of whether you really need to rate limit any of your non-LLQ classes. (I'm guessing you're working with some WAN vendor that "sets aside" a certain amount of bandwidth for the non-LLQ classes, but often that bandwidth is proportionally prioritized. If it is, shaping or policing is not taking advantage of otherwise available bandwidth.)