Total output drops on router and not QoS drops

[Heino Human]

Hi guys,

I have a few routers in a customers network that continuesly having output drops. The QoS policy applied shows no drops at all. I have cleared the counters and monitored it, it just keeps creeping up. Its not over utlization as I have netflow enabled and don't see it reaching full capacity, so QoS dont even come into the picture. Below in the output you can see the drop count after a week and no drops in QoS classes what so ever....

ORADMORT01#sh inter g0/0
GigabitEthernet0/0 is up, line protocol is up
  Hardware is MV96340 Ethernet, address is 001b.5444.50f8 (bia 001b.5444.50f8)
  Description: Telstra GWIP FNN:N6010599R VLAN:N6046175R
  Internet address is 10.230.14.158/30
  MTU 1500 bytes, BW 4096 Kbit/sec, DLY 100 usec,
     reliability 255/255, txload 12/255, rxload 91/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, media type is T
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters 1w1d
  Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 77998
  Queueing strategy: Class-based queueing
  Output queue: 0/1000/0 (size/max total/drops)
  30 second input rate 1463000 bits/sec, 292 packets/sec
  30 second output rate 193000 bits/sec, 150 packets/sec
     87137921 packets input, 1932383342 bytes, 0 no buffer
     Received 196128 broadcasts, 2 runts, 0 giants, 0 throttles
     4 input errors, 1 CRC, 0 frame, 0 overrun, 1 ignored
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     37810913 packets output, 1780878043 bytes, 0 underruns
     0 output errors, 0 collisions, 0 interface resets
     0 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out
ORADMORT01#sh polic
ORADMORT01#sh policy-map inte
ORADMORT01#sh policy-map interface g0/0
 GigabitEthernet0/0

  Service-policy output: WAN_EGRESS

    Class-map: class-default (match-any)
      37657378 packets, 10198904105 bytes
      30 second offered rate 239000 bps, drop rate 0 bps
      Match: any
      Queueing
      queue limit 64 packets
      (queue depth/total drops/no-buffer drops) 0/0/0
      (pkts output/bytes output) 37855428/10411003452
      shape (average) cir 4096000, bc 16384, be 16384
      target shape rate 4096000

      Service-policy : WAN_CONGESTION_MNGT

        queue stats for all priority classes:
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 4257255/910981902

        Class-map: VOICE_BEARER (match-any)
          4257255 packets, 910981902 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: ip dscp ef (46)
            4257255 packets, 910981902 bytes
            30 second rate 0 bps
          Priority: 900 kbps, burst bytes 22500, b/w exceed drops: 0

          police:
              rate 900000 bps, burst 28125 bytes, peak-burst 1500 bytes
            conformed 4257255 packets, 910981902 bytes; actions:
              transmit
            exceeded 0 packets, 0 bytes; actions:
              drop
            violated 0 packets, 0 bytes; actions:
              drop
            conformed 0 bps, exceed 0 bps, violate 0 bps

        Class-map: VOICE_SIGNALLING (match-any)
          2003973 packets, 132821144 bytes
          30 second offered rate 1000 bps, drop rate 0 bps
          Match: ip dscp cs3 (24)
            2003973 packets, 132821144 bytes
            30 second rate 1000 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 2003973/127387965
          bandwidth 16 kbps

        Class-map: DRIVES (match-any)
          0 packets, 0 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: ip dscp af32 (28)
            0 packets, 0 bytes
            30 second rate 0 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 0/0
          bandwidth 100 kbps

        Class-map: CITRIX (match-any)
          0 packets, 0 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: ip dscp af31 (26)
            0 packets, 0 bytes
            30 second rate 0 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 0/0
          bandwidth 150 kbps

        Class-map: NETWORK_MANAGEMENT (match-any)
          363851 packets, 255045339 bytes
          30 second offered rate 2000 bps, drop rate 0 bps
          Match: access-group name NETWORK_MANAGEMENT_TRAFFIC
            363849 packets, 255044626 bytes
            30 second rate 2000 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 285761/78583417
          bandwidth 64 kbps
          QoS Set
            dscp cs2
              Packets marked 363851
          police:
              rate 64000 bps, burst 2000 bytes, peak-burst 1500 bytes
            conformed 285761 packets, 78583417 bytes; actions:
              transmit
            exceeded 21402 packets, 9652670 bytes; actions:
              drop
            violated 56688 packets, 30309197 bytes; actions:
              drop
            conformed 2000 bps, exceed 0 bps, violate 0 bps

        Class-map: ROUTING_PROTOCOLS (match-any)
          76848 packets, 18608605 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: ip dscp cs6 (48)
            76848 packets, 18608605 bytes
            30 second rate 0 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 76848/4967660
          bandwidth 24 kbps

        Class-map: VIDEO_BEARER (match-any)
          0 packets, 0 bytes
          30 second offered rate 0 bps, drop rate 0 bps
          Match: ip dscp af41 (34)
            0 packets, 0 bytes
            30 second rate 0 bps
          police:
              rate 1024000 bps, burst 102400 bytes, peak-burst 1500 bytes
            conformed 0 packets, 0 bytes; actions:
              transmit
            exceeded 0 packets, 0 bytes; actions:
              drop
            violated 0 packets, 0 bytes; actions:
              drop
            conformed 0 bps, exceed 0 bps, violate 0 bps
          Priority: 1024 kbps, burst bytes 25600, b/w exceed drops: 0

        Class-map: class-default (match-any)
          30955453 packets, 8881447608 bytes
          30 second offered rate 237000 bps, drop rate 0 bps
          Match: any

          queue limit 1024 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 31231591/9289082508
ORADMORT01#

Software version running is 112.4(24)T3 and its a 2821 router. I thought it might be a bug, but cant find anything there either.

I reviewed a fair number of posts online, they all point to QOS, but this is not QoS related.

Thanx for your input, hopefully I'm not the only one with this problem....

H

[Vinit Jain]

Hello,

One thing you can try is disabling flow control on the interface and see if that makes any difference. Disabling flow control will help us understand if there is any kind of congestion happening or not.

Would like to know if the drops are incrementing even after removing the QoS policy. If you haven't verified that, please do.

Another action plan i would suggest is, capturing the below output:

- show ip traffic | in fragment

- show interface gi 0/0 | in rate|drop

Capture the above output multiple times to see if the drops are incrementing at the rate at which the router is unable to fragment some packets. If a large packet is received on a different interface and tries to egress here the packet may have to be fragmented. If the DF bit is set on the same packet then we cannot fragment it and will have to drop the packet. This would also increment the couldn't fragment counter of the show ip traffic command as well.

Hope this helps.

Vinit

[Heino Human]

Thank you guys for your help, especially Joseph W. Doherty. I don't know how i missed that! my goodness, thank you!

I have been scratching my head and we have a few (around 80) routers with this same issue, I was thinking I would have had to upgrade the software on all of them.....

Thank you again!

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

". . . no drops in QoS classes what so ever....":

Class-map: NETWORK_MANAGEMENT (match-any)
          363851 packets, 255045339 bytes
          30 second offered rate 2000 bps, drop rate 0 bps
          Match: access-group name NETWORK_MANAGEMENT_TRAFFIC
            363849 packets, 255044626 bytes
            30 second rate 2000 bps
          Queueing
          queue limit 64 packets
          (queue depth/total drops/no-buffer drops) 0/0/0
          (pkts output/bytes output) 285761/78583417
          bandwidth 64 kbps
          QoS Set
            dscp cs2
              Packets marked 363851
          police:
              rate 64000 bps, burst 2000 bytes, peak-burst 1500 bytes
            conformed 285761 packets, 78583417 bytes; actions:
              transmit
            exceeded 21402 packets, 9652670 bytes; actions:
              drop
            violated 56688 packets, 30309197 bytes; actions:
              drop
            conformed 2000 bps, exceed 0 bps, violate 0 bps

Compare the sum of this policer's drops to the total drops listed for the interface.