cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
930
Views
25
Helpful
11
Replies

CBWQF QoS

jacky.cheuk
Level 1
Level 1

Hi,

I have two WAN routers, Router A and Router B. But when i tried to use extended ping with tos 184(ef) or 104(af31) to ping from A to B.

We can always see drops when the traffic is only around 70-80% of the link bandwidth. (every ping test,, although only 4-5 packets out of 1500)

We are using priority queue for the EF class. Doesn't it has the guaranteed bandwidth when congestion is seen? Like here is 30% of 10Mb

Please kindly help, really have no clue to this problem.

 

Router A#ping
Protocol [ip]:
Target IP address: 172.15 .10.2
Repeat count [5]: 1500
Datagram size [100]: 100
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 184
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1500, 100-byte ICMP Echos to 172.15.10.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (1498/1500), round-trip min/avg/max = 148/153/684 ms

 

 

RouterA#ping
Protocol [ip]:
Target IP address: 172.15.10.2
Repeat count [5]: 1500   
Datagram size [100]:
Timeout in seconds [2]:
Extended commands [n]: y
Source address or interface:
Type of service [0]: 104
Set DF bit in IP header? [no]:
Validate reply data? [no]:
Data pattern [0xABCD]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Sweep range of sizes [n]:
Type escape sequence to abort.
Sending 1500, 100-byte ICMP Echos to 172.15.10.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!.!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Success rate is 99 percent (1496/1500), round-trip min/avg/max = 144/215/752 ms


 

11 Replies 11

Hello.

Could you please post "show policy-map int ..." during the issue (from both sides: this router and ICMP responder)? Could you provide QoS configuration from another router?

Some notes about your configuration:

  • sum of queue limit is 4096, are you sure you have them?
  • you are using "set ip precedence" under classes that is not a garanteed way to mark traffic (police should be used instead);
  • you are shaping at 10240000 - is this contracted bandwidth and what interface are you using?

>We are using priority queue for the EF class. Doesn't it has the guaranteed bandwidth when congestion is seen?

Yes it has on the interface, that is why I asked for "show policy-map int", but your provider could drop traffic, if it exceeds CIR.

"you are using "set ip precedence" under classes that is not a garanteed way to mark traffic (police should be used instead);"

 

Could you elaborate and/or provide references?

Hi Vasilii,

First of all ,thanks for the information.

sum of queue limit is 4096, are you sure you have them?

I am not sure actually, how to check if I really having 4096 for the queue depth?

Also, I dont see any buffer issue so far. Before without "hold-queue 4096 in", the total output drop keeps increasing all the time. And with that command, not seeing that behaviour again.

Input queue: 0/4096/0/0 (size/max/drops/flushes); Total output drops: 51972 ( before more than 100K)

 

--------------------------------------------------------------------------

 

GigabitEthernet0/0 buffers, 1664 bytes (total 768, permanent 768):
     0 in free list (0 min, 768 max allowed)
     768 hits, 0 fallbacks
     768 max cache size, 512 in cache
     3029333726 hits in cache, 0 misses in cache

=========================================

you are using "set ip precedence" under classes that is not a garanteed way to mark traffic (police should be used instead);

I always not seeing "Critical-Out" having traffic, is it related?

 Class-map: Critical-Out (match-any)
          182250712 packets, 53367352107 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: access-group name Critical-Out
            182241215 packets, 53366402507 bytes
            30 second rate 0 bps
          Match:  dscp af31 (26) af32 (28) af33 (30)
            9496 packets, 949600 bytes
            30 second rate 0 bps
          Match: ip precedence 3
            0 packets, 0 bytes
            30 second rate 0 bps
          Queueing
          queue limit 1792 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 182250709/53372798539
          bandwidth 50% (5120 kbps)
          QoS Set
            precedence 3
              Packets marked 175064110

==============================================

Here is the "sh policy-map int"

 

sh policy-map int gi0/0 out
 GigabitEthernet0/0

  Service-policy output: QoS-Out

    Class-map: class-default (match-any)
      1964240416 packets, 425034944682 bytes
      30 second offered rate 2119000 bps, drop rate 0 bps
      Match: any
      Queueing
      queue limit 64 packets
      (queue depth/total drops/no-buffer drops) 0/14952/0
      (pkts output/bytes output) 1980395760/457982960237
      shape (average) cir 10240000, bc 40960, be 40960
      target shape rate 10240000

      Service-policy : Sub-QoS-Out

        queue stats for all priority classes:

          queue limit 512 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 135409659/31028231864

        Class-map: Voice-Out (match-any)
          135424571 packets, 30333395174 bytes
          30 second offered rate 2119000 bps, drop rate 0 bps
          Match: access-group name Voice-Out
            135423045 packets, 30331284996 bytes
            30 second rate 2119000 bps
          Match:  dscp ef (46)
            1500 packets, 2100000 bytes
            30 second rate 0 bps
          Match: ip precedence 5
            0 packets, 0 bytes
            30 second rate 0 bps
          Priority: 30% (3072 kbps), burst bytes 76800, b/w exceed drops: 14952

          QoS Set
            precedence 5
              Packets marked 134717720

        Class-map: Critical-Out (match-any)
          182250712 packets, 53367352107 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: access-group name Critical-Out
            182241215 packets, 53366402507 bytes
            30 second rate 0 bps
          Match:  dscp af31 (26) af32 (28) af33 (30)
            9496 packets, 949600 bytes
            30 second rate 0 bps
          Match: ip precedence 3
            0 packets, 0 bytes
            30 second rate 0 bps
          Queueing
          queue limit 1792 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 182250709/53372798539
          bandwidth 50% (5120 kbps)
          QoS Set
            precedence 3
              Packets marked 175064110

        Class-map: class-default (match-any)
          1646565278 packets, 341334268400 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: any

          queue limit 1792 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 1662735392/373581929834

 

Hello.

You wrote: I am not sure actually, how to check if I really having 4096 for the queue depth?

The issue is not about buffers you have, but with queue length and queue delay, that the router adds.

If you have 50% of 10M for critical class and queue length of 1792 packets, then queuing delay may be up to 1000 ms! I can't believe it's acceptable!

The same issue is for Voice class, as it may exceed CIR and all the exceeded traffic will be queued.

PS: you'ld better to use default values (of 64/128 packets) and WRED + FQ in class-default.

Joseph W. Doherty
Hall of Fame
Hall of Fame

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Yes CBWFQ's LLQ is dequeued before any non-LLQ queues.  However, there are many possible reasons you're seeing drops.  Vasilii mentioned one, what's happening within your WAN (if using any kind of WAN cloud).

 

(BTW, besides drops, you shouldn't be seeing such variable ping times for LLQ traffic.  Is your WAN multipoint?)

 

If your WAN provider is providing a logical bandwidth cap, lower than the physical bandwidth, you often need to shape to not exceed that value.  I see you are shaping, but I believe many device shapers don't account for L2 overhead, but WAN providers often do.  I.e. you need to shape slower than the nomimal bandwidth.

 

For 10 Mbps, if possible, running an Ethernet interface at 10 Mbps works better for regulating traffic than a shaper.

 

It's possible the implicit policer, for the LLQ class, is dropping overrate LLQ.

 

PS:

BTW, your CBWFQ configuration statements, shown, is a bit odd in many respects.

 

For example, your Critical-In matches IPPrec 3, but Critical-Out does not.

 

Voice-In matches IPPrec 5 and 7, but Voice-Out does not.

 

Your Voice-Out and Critical-Out reset ToS, for example, losing DSCP EF.

 

Your Critical-In matches IPPrec 3, so there's no need to match the DSCP 3x values.  Likewise for Voice-In matching IPPrec 5, negates the need to match DSCP EF.

 

Your queue depths are rather deep.

Hi Joseph,

Thanks for the information.

 

(BTW, besides drops, you shouldn't be seeing such variable ping times for LLQ traffic. Is your WAN multipoint?)

Yes, this is a MPLS network. The ping is from India ping to Europe.

===============================================

For 10 Mbps, if possible, running an Ethernet interface at 10 Mbps works better for regulating traffic than a shaper

you mean the shaping command is not necessary? Without the shaping command , will the QoS still function properly?

=====================================

Your queue depths are rather deep

I see below article and it stated that the max. queue depth should be 4096, isn't it?

if not, what should be the reasonable queue depth?

====================================

 

BTW, When i try to ping with tos 104, i dont see any traffic on the Critical-Out class. Any idea

 

Class-map: Critical-Out (match-any)
          182250712 packets, 53367352107 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: access-group name Critical-Out
            182241215 packets, 53366402507 bytes
            30 second rate 0 bps
          Match:  dscp af31 (26) af32 (28) af33 (30)
            9496 packets, 949600 bytes
            30 second rate 0 bps
          Match: ip precedence 3
            0 packets, 0 bytes
            30 second rate 0 bps
          Queueing
          queue limit 1792 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 182250709/53372798539
          bandwidth 50% (5120 kbps)
          QoS Set
            precedence 3
              Packets marked 175064110
 

 

Thanks in advance.

 

Hello.

you mean the shaping command is not necessary? Without the shaping command , will the QoS still function properly?

If you configure interface with speed 10/duplex full, you won't need shaper command and nested policy.

BTW, When i try to ping with tos 104, i dont see any traffic on the Critical-Out class. Any idea ?

It might be consumed by Voice class; but you'ld better to wait at least 10 seconds after ping and compare packet number per class.

if not, what should be the reasonable queue depth?

On 10M you may keep queue length default and enable WRED on non-voice classes.

Hi ,

The is a 25Mb ethernet link, we are using QoS Voice/Critical/default -- 30%/50%/20%.

However, we can see sth on the "drop rate" all the time when the utilization of that class is around 5Mb. Any idea?

For 25Mb, we should have 12800 kbps for Critical class. And the Critical class traffic can even overflow to other classes as well, isn't it?

Thanks in advance.

 

 

Class-map: Critical-Out (match-any)
          1564791025 packets, 943221073082 bytes
          30 second offered rate 4874000 bps, drop rate 5000 bps
          Match: access-group name Critical-Out
            1564689716 packets, 943190830806 bytes
            30 second rate 4869000 bps
          Match:  dscp af31 (26) af32 (28) af33 (30)
            5192 packets, 518994 bytes
            30 second rate 4000 bps
          Match: ip precedence 3
            96116 packets, 29727270 bytes
            30 second rate 3000 bps
          Queueing
          queue limit 256 packets
          (queue depth/total drops/no-buffer drops) 0/27288/0
          (pkts output/bytes output) 1564763731/943190504244
          bandwidth 50% (12800 kbps)
          QoS Set
            precedence 3
              Packets marked 1564791026
 

Hello.

It's a different configuration than you paste 5 hours ago:

h policy-map int gi0/0 out
 GigabitEthernet0/0

  Service-policy output: QoS-Out

    Class-map: class-default (match-any)
      1964240416 packets, 425034944682 bytes
      30 second offered rate 2119000 bps, drop rate 0 bps
      Match: any
      Queueing
      queue limit 64 packets
      (queue depth/total drops/no-buffer drops) 0/14952/0
      (pkts output/bytes output) 1980395760/457982960237
      shape (average) cir 10240000, bc 40960, be 40960
      target shape rate 10240000

As it had 10M shaping.

Reagrding new config (of 25M): could you please "show policy-map int G0/0 out" and "show int G0/0"?

Hi Vasilii,

 

Yes, this is another configuration. I changed the queue-limit as per yours and Joseph's suggestion. And this link is a 25Mb Ethernet link.

 

RouterA#sh policy-map int gi0/0 out
 GigabitEthernet0/0

  Service-policy output: QoS-Out

    Class-map: class-default (match-any)
      6281284662 packets, 2256027825911 bytes
      30 second offered rate 15189000 bps, drop rate 7000 bps
      Match: any
      Queueing
      queue limit 64 packets
      (queue depth/total drops/no-buffer drops) 295/120757/0
      (pkts output/bytes output) 2050287858/2375458720506
      shape (average) cir 25600000, bc 102400, be 102400
      target shape rate 25600000

      Service-policy : Sub-QoS-Out

        queue stats for all priority classes:

          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 36016630/3801195186

        Class-map: Voice-Out (match-any)
          36016628 packets, 3662337109 bytes
          30 second offered rate 25000 bps, drop rate 0 bps
          Match: access-group name Voice-Out
            30318868 packets, 3379642769 bytes
            30 second rate 0 bps
          Match:  dscp ef (46)
            3214 packets, 238267 bytes
            30 second rate 0 bps
          Match: ip precedence 5
            5694547 packets, 282456242 bytes
            30 second rate 24000 bps
          Priority: 30% (7680 kbps), burst bytes 192000, b/w exceed drops: 0

          QoS Set
            precedence 5
              Packets marked 36016630

        Class-map: Critical-Out (match-any)
          1594244859 packets, 965878563879 bytes
          30 second offered rate 12603000 bps, drop rate 7000 bps
          Match: access-group name Critical-Out
            1593848191 packets, 965758202728 bytes
            30 second rate 12585000 bps
          Match:  dscp af31 (26) af32 (28) af33 (30)
            6506 packets, 650394 bytes
            30 second rate 0 bps
          Match: ip precedence 3
            390161 packets, 119715792 bytes
            30 second rate 21000 bps
          Queueing
          queue limit 512 packets
          (queue depth/total drops/no-buffer drops) 294/74886/0
          (pkts output/bytes output) 1594169970/965788603988
          bandwidth 50% (12800 kbps)
          QoS Set
            precedence 3
              Packets marked 1594244863

        Class-map: class-default (match-any)
          4651036724 packets, 1286493970070 bytes
          30 second offered rate 2555000 bps, drop rate 0 bps
          Match: any

          queue limit 512 packets
          (queue depth/total drops/no-buffer drops) 0/45871/0
          (pkts output/bytes output) 420101265/1405868924498
          QoS Set
            precedence 0
              Packets marked 4650675067

 

 

RouterA#   sh int gi0/0
GigabitEthernet0/0 is up, line protocol is up  
 
  MTU 1500 bytes, BW 25600 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 152/255, rxload 55/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full Duplex, 100Mbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 5w0d
  Input queue: 0/1152/30/0 (size/max/drops/flushes); Total output drops: 120757
  Queueing strategy: Class-based queueing
  Output queue: 0/1152/0 (size/max total/drops)
  30 second input rate 5583000 bits/sec, 2937 packets/sec
  30 second output rate 15279000 bits/sec, 3286 packets/sec
     5853694046 packets input, 1288961515893 bytes, 0 no buffer
     Received 2582 broadcasts (0 IP multicasts)
     0 runts, 0 giants, 18 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog, 0 multicast, 0 pause input
     6345326283 packets output, 2375500204094 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

 

 

 

Hello.

You are right - Critical class can borrow unused bandwidth of other classes if needed.

I think you observe drops here due to bursts in you Critical class. Notice you see 294 queue depth in sh policy-map...

As far as you are not using WRED nor FQ, Critical class can overflow pretty quickly. I would suggest to configure WRED on Critical class and min/max thresholds about 120 and 180 respectively. This would allow early drops, that helps to manage traffic bursts and TCP synchronization.

PS: you will never get rid of drops, unless you configure infinite queue size (with potentially infinite queuing delay); the only thing you could try is to make drops manageable.

PS2: 7000 bps is not too much, if you run a lot of flows on the link and most of them are bursty.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: