QoS - Dropping traffic with BW Available

[tinochelli]

Hi All,

I am trying to understand why i am getting a drop rate on this class whilst it is <50% utilized.  I have not tried changing the hold-queue or tx-ring-limit as of yet as there is a lot of traffic that utilizes this port so would prefer to get to the bottom of it before committing to changes.

class-map match-any xoxoxo
 description ## XO 5Mb ##
 match access-group name test-acl
 match access-group name test-acl-1

Class-map: xoxoxo (match-any)
              669636947 packets, 617341481032 bytes
              30 second offered rate 1099000 bps, drop rate 13000 bps
              Match: access-group name test-acl
                669636948 packets, 617341481122 bytes
                30 second rate 1099000 bps
              Queueing
              queue limit 64 packets
              (queue depth/total drops/no-buffer drops) 0/6324936/0
              (pkts output/bytes output) 663312012/609484292958
              bandwidth 2400 kbps

    Class-map: class-default (match-any)
              0 packets, 0 bytes
              30 second offered rate 0 bps, drop rate 0 bps
              Match: any
              Queueing
              queue limit 64 packets
              (queue depth/total drops/no-buffer drops) 0/0/0
              (pkts output/bytes output) 0/0
              bandwidth 640 kbps

POLICY APPLIED TO Gi0/2

GigabitEthernet0/2 is up, line protocol is up
  Hardware is iGbE, address is c464.1392.3f82 (bia c464.1392.3f82)
  Description: XXXXXXXXX
  MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
     reliability 255/255, txload 17/255, rxload 8/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full Duplex, 1Gbps, media type is RJ45
  output flow-control is unsupported, input flow-control is unsupported
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 5w2d
  Input queue: 0/75/7484/726531 (size/max/drops/flushes); Total output drops: 132382908
  Queueing strategy: Class-based queueing
  Output queue: 75/1000/0 (size/max total/drops)
  30 second input rate 32795000 bits/sec, 16333 packets/sec
  30 second output rate 69177000 bits/sec, 15921 packets/sec
     3271649900 packets input, 1764478082 bytes, 0 no buffer
     Received 35053136 broadcasts (13965604 IP multicasts)
     0 runts, 21522 giants, 0 throttles
     49324 input errors, 0 CRC, 0 frame, 49324 overrun, 0 ignored
     0 watchdog, 28380386 multicast, 0 pause input
     2946365844 packets output, 1971336889 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     2843283 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output

Thanks

[Madhukrishnan Gopinathan Nair]

Which platform is this?

Madhu

[tinochelli]

It's a CISCO2901/K9 running Version 15.0(1r)M16

[Martin Hruby · Accepted solution]

Hello

This is quite normal if you have bursty traffic, e.g. applications generate a lot of packets in short time interval. Even though you don't actually exceed your bandwidth, you still see drops because there's a large number of bytes (Bc) to be sent in short time (Tc). You can increase the queue-limit so you'll be able to store more exceeding packets during these bursts, but don't go too far (i.e. more than 512).

I see you also have drops in the input queue which is pretty bad. In this case I would recommend having a look at the device sending traffic to this interface and see if you can apply shaping in the outbound direction on that interface (unless you're running delay-sensitive applications like VoIP) to smooth out the bursts.

Best regards,
Martin

[tinochelli]

Sunil, Martin & Joseph,

Thanks for the feedback.  This circuit is a bit different as it is running over satellite so there are incurred latencies that you would not normally expect to see.  Ultimately you have confirmed my initial thought of needing to increase the queue limit.  I will confirm if voice is sent over that as I cannot introduce and jitter into their service. 

I will also look into the input drops but this is not necessarily coming from this service as there are several services utilizing this port.

Thanks!

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Satellite latencies do cause their own problems, but not much for queuing issues on the router.

For VoIP, especially, you may need to shape slow than your downstream rate because I believe many shapers don't count L2 overhead.  Without such counting, you can transmit faster than the media can process.  Also, without such counting, L2 overhead per packet varies, so you can either allow for worse case or average case.  Normally, average case is "good enough", but for VoIP, you might need to favor worse case.

If using a shaper, you also want to check the Tc interval.  Older shaper implementations defaulted to 25 ms, newer often default to 4 ms.  The revised defaults, I suspect, are to better serve VoIP.

Lastly, you sometimes need to minimize the interface tx queue, which is FIFO.  Normally, only when it "overflows" does your QoS engage, but a shaper should engage before that.

[Sunil Bhadauria]

Hello ,

I will try to answer your query .

This may be expected scenario as well .

I do not see your QoS configuration in post , but seems like you are referring output drop rate on class xoxoxo + total output drops on Gi0/2 . You are also pointing that concerned class is given 5 Mb bandwidth .

But notice the offered rate and drop rate we only see average rate of traffic over 30 seconds, which could average out any possible burst caused at sub-millisecond level of flow :

Class-map: xoxoxo (match-any)
              669636947 packets, 617341481032 bytes
              30 second offered rate 1099000 bps, drop rate 13000 bps>>>>>>>>>>>>>>>>>>>>
              Match: access-group name test-acl
                669636948 packets, 617341481122 bytes
                30 second rate 1099000 bps >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
              Queueing
              queue limit 64 packets
              (queue depth/total drops/no-buffer drops) 0/6324936/0
              (pkts output/bytes output) 663312012/609484292958
              bandwidth 2400 kbps

- i do not see more information about CIR ( commited interface rate )/ Bc ( commited Burst ) / Be ( Excess Burst ) on the post for particular class .

But in general any policy ( shaper / policer ) will not allow traffic more then Bc ( burst ) during time Tc ( Tc=Bc/CIR ) .

I believe you must have understood what I mean . It all depends on traffic pattern .

You may try to increase queue-limit for particular class to see if drop rate decreases or lower down . But that will add little delay to packet forwarding , so you need to check for any time sensitive application traffic .

HTH

Sunil Bhadauria

Kindly rate all helpful posts

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

As Sunil has already noted, most likely issue is bursty traffic that's not visible within even a 30 second traffic rate average, but might easily overlow a queue down at the microsecond level.  If the ingress port, to your router is gig, a lot of packets can be queued, during a gig burst, trying to maintain a transmission rate at the megabit level.