cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
677
Views
0
Helpful
8
Replies
Beginner

QoS Dropping Packets

I wonder if you could help with a QoS issue that we have, we are dropping packets at an alarming rate with very little traffic on the wire, I can only assume the QoS is not set-up quite right.

 

These are the interface and QoS stats for a time period between 8am and 10am, the packets are being dropped from Site A to B

 

GigabitEthernet1/0/3 is up, line protocol is up (connected)

  Hardware is Gigabit Ethernet, address is 3c5e.c303.4983 (bia 3c5e.c303.4983)

  Description: 100mb P2P Circuit

  MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

     reliability 255/255, txload 14/255, rxload 89/255

  Encapsulation ARPA, loopback not set

  Keepalive set (10 sec)

  Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

  input flow-control is off, output flow-control is unsupported

  ARP type: ARPA, ARP Timeout 04:00:00

  Last input 00:00:00, output never, output hang never

  Last clearing of "show interface" counters 02:12:31

  Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 5587991

  Queueing strategy: Class-based queueing

  Output queue: 0/4096 (size/max)

  5 minute input rate 35163000 bits/sec, 3707 packets/sec

  5 minute output rate 5801000 bits/sec, 2550 packets/sec

     27403731 packets input, 33666560879 bytes, 0 no buffer

     Received 21889 broadcasts (21845 multicasts)

     0 runts, 0 giants, 0 throttles

     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

     0 watchdog, 21845 multicast, 0 pause input

     0 input packets with dribble condition detected

     19347219 packets output, 7391882600 bytes, 0 underruns

     5587991 output errors, 0 collisions, 0 interface resets

     0 unknown protocol drops

     0 babbles, 0 late collision, 0 deferred

     0 lost carrier, 0 no carrier, 0 pause output

     0 output buffer failures, 0 output buffers swapped out

 

 

  Service-policy output: NCI-QUEUE-SHAPE-OUTPUT-POLICY-PARENT-Inc

 

    Class-map: class-default (match-any) 

      0 packets

      Match: any

      Queueing

     

      (total drops) 5587991

      (bytes output) 7427241975

      shape (average) cir 100000000, bc 400000, be 400000

      target shape rate 100000000

 

      Service-policy : NCI-QUEUE-OUTPUT-POLICY-CHILD-IncreasedD

 

        queue stats for all priority classes:

          Queueing

          priority level 1

          

          (total drops) 0

          (bytes output) 23832294

 

        Class-map: NCI-VOIP-QUEUE (match-any) 

          0 packets

          Match: ip dscp ef (46)

            0 packets, 0 bytes

            5 minute rate 0 bps

          Priority: 10% (10000 kbps), burst bytes 250000,

         

          Priority Level: 1

          queue-buffers ratio 10

 

        Class-map: NCI-VIDEO-QUEUE (match-any) 

          0 packets

          Match: ip dscp af41 (34) af42 (36) af43 (38)

            0 packets, 0 bytes

            5 minute rate 0 bps

          Queueing

         

          queue-limit dscp 34 percent 100

          queue-limit dscp 36 percent 90

          queue-limit dscp 38 percent 80

          (total drops) 0

          (bytes output) 120

          bandwidth remaining 15%

          queue-buffers ratio 5

         

        Class-map: NCI-NET-MGMT-QUEUE (match-any) 

          0 packets

          Match: ip dscp cs2 (16) cs6 (48) cs7 (56)

            0 packets, 0 bytes

            5 minute rate 0 bps

          Queueing

         

          queue-limit dscp 16 percent 80

          queue-limit dscp 48 percent 90

          queue-limit dscp 56 percent 100

          (total drops) 1294807

          (bytes output) 41662725

          bandwidth remaining 13%

          queue-buffers ratio 15

         

 

        Class-map: NCI-DATA-QUEUE (match-any) 

          0 packets

          Match: ip dscp af21 (18) af22 (20) af23 (22)

            0 packets, 0 bytes

            5 minute rate 0 bps

          Queueing

         

          queue-limit dscp 18 percent 100

          queue-limit dscp 20 percent 90

          queue-limit dscp 22 percent 80

          (total drops) 4293184

          (bytes output) 5471263194

          bandwidth remaining 30%

          queue-buffers ratio 35

         

        Class-map: NCI-VoIP-SIG-QUEUE (match-any) 

          0 packets

          Match: ip dscp cs3 (24)

            0 packets, 0 bytes

            5 minute rate 0 bps

          Queueing

         

          (total drops) 0

          (bytes output) 26003918

          bandwidth remaining 5%

          queue-buffers ratio 5

 

        Class-map: NCI-SCAVENGER-QUEUE (match-any) 

          0 packets

          Match:  dscp cs1 (8)

            0 packets, 0 bytes

            5 minute rate 0 bps

          Queueing

         

          (total drops) 0

          (bytes output) 0

          bandwidth remaining 1%

          queue-buffers ratio 5

 

        Class-map: NCI-PUBLIC-QUEUE (match-any) 

          0 packets

          Match:  dscp 6

            0 packets, 0 bytes

            5 minute rate 0 bps

          Queueing

         

          (total drops) 0

          (bytes output) 0

          bandwidth remaining 5%

          queue-buffers ratio 5

 

        Class-map: class-default (match-any) 

          0 packets

          Match: any

          Queueing

         

          (total drops) 0

          (bytes output) 1864479724

          bandwidth remaining 20%

          queue-buffers ratio 20

 

We have a 100mb P2P fibre circuit between the sites

Site A:  Cisco WS-C3850-48T - 03.06.06.E cat3k_caa-universalk9

Site B: Cisco WS-C3560-24PS - 12.2(55)SE11

 

The biggest complaint we have is assessing an SQL Database from site B to A, end users are getting a screen popup saying it’s taking to long to connect to the DB

 

I'm need to find what traffic is being dropped and why

 

QoS

policy-map NCI-QUEUE-OUTPUT-POLICY-CHILD-IncreasedD
 class NCI-VOIP-QUEUE
  priority level 1 percent 10
  queue-buffers ratio 10
 class NCI-VIDEO-QUEUE
  bandwidth remaining percent 15
  queue-buffers ratio 5
  queue-limit dscp af43 percent 80
  queue-limit dscp af42 percent 90
  queue-limit dscp af41 percent 100

 class NCI-NET-MGMT-QUEUE
  bandwidth remaining percent 13
  queue-buffers ratio 15
  queue-limit dscp cs2 percent 80
  queue-limit dscp cs6 percent 90
  queue-limit dscp cs7 percent 100

 class NCI-DATA-QUEUE
  bandwidth remaining percent 30
  queue-buffers ratio 35
  queue-limit dscp af23 percent 80
  queue-limit dscp af22 percent 90
  queue-limit dscp af21 percent 100

 class NCI-VoIP-SIG-QUEUE
  bandwidth remaining percent 5
  queue-buffers ratio 5

 class NCI-SCAVENGER-QUEUE
  bandwidth remaining percent 1
  queue-buffers ratio 5

 class NCI-PUBLIC-QUEUE
  bandwidth remaining percent 5
  queue-buffers ratio 5

 class class-default
  bandwidth remaining percent 20
  queue-buffers ratio 20

policy-map NCI-QUEUE-SHAPE-OUTPUT-POLICY-PARENT-Inc
 class class-default
  shape average percent 100
   service-policy NCI-QUEUE-OUTPUT-POLICY-CHILD-IncreasedD
policy-map NCI-MARK-POLICY
 class NCI-VOIP-CLASS
  set dscp ef
 class NCI-VOIP-SIG-CLASS
  set dscp cs3
 class NCI-VIDEO-CLASS
  set dscp af41
 class NCI-DATA-CLASS
  set dscp af21
 class NCI-NET-MGMT-CLASS
  set dscp cs2
 class NCI-PUBLIC-CLASS
  set dscp 6
 class NCI-SCAVENGER-CLASS
  set dscp cs1
 class class-default
  set dscp default

class-map match-any NCI-VOIP-CLASS
  description VoIP traffic
 match dscp ef

class-map match-any NCI-VOIP-SIG-CLASS
  description Signalling traffic
 match dscp cs3

class-map match-any NCI-VIDEO-CLASS
  description Video Conferencing traffic
 match access-group name NCI-VIDEO-ACL

class-map match-any NCI-DATA-CLASS
  description Business Critical traffic
 match access-group name NCI-DATA-ACL

class-map match-any NCI-NET-MGMT-CLASS
 match access-group name NCI-NET-MGMT-ACL

class-map match-any NCI-PUBLIC-CLASS
  description Public Traffic
 match access-group name NCI-PUBLIC-ACL

class-map match-any NCI-SCAVENGER-CLASS
 match access-group name NCI-SCAVENGER-ACL

ip access-list extended NCI-VIDEO-ACL
 permit udp any any eq 554
 permit udp any any eq 537
 permit udp any any eq 1755
 permit udp any any eq 1736

ip access-list extended NCI-DATA-ACL
 permit tcp any any eq ftp-data
 permit ip host 10.0.20.118 any
 permit ip host 172.22.8.25 any
 permit ip host 172.22.7.6 any
 permit ip host 10.0.17.190 any
 permit tcp any eq 443 any
 permit tcp any eq 389 any
 permit tcp any eq 8443 any
 permit tcp any eq www any
 permit tcp any eq ftp-data any
 permit tcp any eq 995 any
 permit tcp any eq 1521 any
 permit udp any eq 1521 any
 permit tcp any eq 1526 any
 permit udp any eq 1526 any
 permit tcp any eq 1575 any
 permit udp any eq 1575 any
 permit tcp any eq 1630 any
 permit udp any eq 14002 any
 permit udp any eq 14006 any
 permit tcp any eq ftp any
 permit tcp any eq 143 any
 permit tcp any any eq 993
 permit tcp any eq 993 any
 permit tcp any eq smtp any
 permit tcp any eq 465 any
 permit tcp any eq 8080 any
 permit tcp any eq 1433 any
 permit tcp any eq lpd any
 permit tcp any eq 9388 any
 permit tcp any eq 1523 any
 permit udp any eq 1523 any
 permit udp any eq 9388 any
 permit tcp any eq 5343 any
 permit tcp any eq 445 any

ip access-list extended NCI-NET-MGMT-ACL
 permit udp any eq ntp any
 permit tcp any eq 22 any
 permit udp any eq snmp any
 permit udp any eq snmptrap any
 permit tcp any eq 161 any
 permit tcp any eq 162 any
 permit icmp any any
 permit udp any eq syslog any
 permit tcp any eq tacacs any
 permit udp any eq tacacs any
 permit udp any eq bootps any
 permit udp any eq domain any
 permit tcp any eq domain any
 permit udp any eq netbios-dgm any
 permit udp any eq netbios-ns any
 permit tcp any eq 135 any
 permit tcp any eq 3389 any
 permit tcp any eq 9996 any
 permit tcp any eq 7777 any
 permit udp any eq 9996 any
 permit tcp any eq 9100 any
 permit udp any eq 4172 any
 permit tcp any eq 1494 any
 permit tcp any eq 1090 any
 permit tcp any eq 5985 any
 permit tcp any eq 3268 any
 permit tcp any eq 5246 any
 permit tcp any eq 88 any

ip access-list extended NCI-PUBLIC-ACL
 permit ip any 10.9.0.0 0.0.255.255

ip access-list extended NCI-SCAVENGER-ACL
 permit tcp any any eq 3689
 permit udp any any eq 3689
 permit tcp any any range 6881 6999

Everyone's tags (1)
8 REPLIES 8
Hall of Fame Community Legend

Re: QoS Dropping Packets

Is this a 3650/3850 switch?
Beginner

Re: QoS Dropping Packets

Cisco WS-C3850-48T - 03.06.06.E cat3k_caa-universalk9

 

Thanks

Highlighted
Participant

Re: QoS Dropping Packets

Hi,

 

Try issuing this command :

show platform qos queue stats gi 1/0/3    ==> this command reveals in which queue the packet are enqued/dropped

 

show platform qos dscp-cos counters gi 1/0/3  ==> this command reveal the DSCP values seen on ingress and egress packets

OBS: please remember the space between "gi" and "1/0/3" !!

 

But back to your output :

You're dropping packets in

NCI-NET-MGMT-QUEUE   ... (total drops) 1294807    ... cs2, cs6, cs7

NCI-DATA-QUEUE            ... (total drops) 4293184    ... af21, af22, af23

the sum of these gives (Total output drops: 5587991) the same number (you're not always that lucky when looking at Cisco counters ;-)

I wonder why you drop packets in the MGMT queue, but can see that you included Citrix VDI (TCP port 1494) and RAW printing (TCP port 9100) in the MGMT class (assume that you mark the packet ingress on other ports on the switch by the ...MARK policy), and these protocols can be a little BW hungry

 

But if you're sending from 1G ports towards a 100 Mbit shaped port, bursts will cause packet drops.

Only way to remedy this, is to supply more buffers to this queue, but that must be taken from somewhere else.

 

Best Regards

Finn

Beginner

Re: QoS Dropping Packets

Thanks for the reply, I’ve stepped out of the office so will take a look when I get back.

 

Here’s the strange thing though, although it maybe not so strange to the trained eye.

I’ve removed QoS from the interface because there is not traffic on the link and I’m getting the same amount of drops, even when the site are just browsing the internet.

 

We have other interfaces showing the same sort of problems so I will run that command on their interface.

 

Thanks

Participant

Re: QoS Dropping Packets

Hi again,

 

This lead to the suspicion that you actually is sending more than 1Gbit of traffic out the interface, since you do not have an traffic shaper anymore (assume you've removed the service policy).

 

Best Regards

Finn

 

 

Beginner

Re: QoS Dropping Packets

I've checked that with OpManager, results below just confuse it more. No congestion and not sending more than

27.9 Mbps

  •  
    IN
  •  
    OUT
 
 
00:00 - 00:00
 
Total
 
Max
 
Min
 
Average
 
Standard Deviation
 
95th Percentile
 
Category
 
             
221.01 GB 38.25 Mbps 6.6 Mbps 27.9 Mbps 2.94 Mbps 30.48 Mbps Traffic IN
41.63 GB 105.22 Mbps 678.39 Kbps 5.25 Mbps 12.01 Mbps 19.95 Mbps Traffic OUT
Beginner

Re: QoS Dropping Packets

show platform qos dscp-cos counters gi 1/0/3
Ingress DSCP0 52619399124   0            
Ingress DSCP1 0             0            
Ingress DSCP2 25112738      0            
Ingress DSCP3 0             0            
Ingress DSCP4 0             0            
Ingress DSCP5 0             0            
Ingress DSCP6 2047          0            
Ingress DSCP7 0             0            
Ingress DSCP8 13701671193   0            
Ingress DSCP9 0             0            
Ingress DSCP10 104397482     0            
Ingress DSCP11 0             0            
Ingress DSCP12 0             0            
Ingress DSCP13 0             0            
Ingress DSCP14 0             0            
Ingress DSCP15 0             0            
Ingress DSCP16 61634549      0            
Ingress DSCP17 0             0            
Ingress DSCP18 3080342803    0            
Ingress DSCP19 0             0            
Ingress DSCP20 0             0            
Ingress DSCP21 0             0            
Ingress DSCP22 0             0            
Ingress DSCP23 0             0            
Ingress DSCP24 418262395     0            
Ingress DSCP25 0             0            
Ingress DSCP26 3004424       0            
Ingress DSCP27 0             0            
Ingress DSCP28 0             0            
Ingress DSCP29 0             0            
Ingress DSCP30 0             0            
Ingress DSCP31 0             0            
Ingress DSCP32 0             0            
Ingress DSCP33 0             0            
Ingress DSCP34 108686        0            
Ingress DSCP35 0             0            
Ingress DSCP36 0             0            
Ingress DSCP37 0             0            
Ingress DSCP38 0             0            
Ingress DSCP39 0             0            
Ingress DSCP40 0             0            
Ingress DSCP41 0             0            
Ingress DSCP42 0             0            
Ingress DSCP43 0             0            
Ingress DSCP44 0             0            
Ingress DSCP45 0             0            
Ingress DSCP46 1550533909    0            
Ingress DSCP47 0             0            
Ingress DSCP48 302513870     0            
Ingress DSCP49 0             0            
Ingress DSCP50 0             0            
Ingress DSCP51 0             0            
Ingress DSCP52 0             0            
Ingress DSCP53 0             0            
Ingress DSCP54 2             0            
Ingress DSCP55 0             0            
Ingress DSCP56 0             0            
Ingress DSCP57 0             0            
Ingress DSCP58 0             0            
Ingress DSCP59 0             0            
Ingress DSCP60 0             0            
Ingress DSCP61 0             0            
Ingress DSCP62 0             0            
Ingress DSCP63 0             0            
Ingress COS0 60324229464   0            
Ingress COS1 11572964454   0            
Ingress COS2 3263465       0            
Ingress COS3 0             0            
Ingress COS4 0             0            
Ingress COS5 0             0            
Ingress COS6 0             0            
Ingress COS7 111835556     0            
Egress DSCP0 45720354616   0            
Egress DSCP1 0             0            
Egress DSCP2 0             0            
Egress DSCP3 0             0            
Egress DSCP4 168           0            
Egress DSCP5 0             0            
Egress DSCP6 4933          0            
Egress DSCP7 0             0            
Egress DSCP8 9659414       0            
Egress DSCP9 0             0            
Egress DSCP10 4600740       0            
Egress DSCP11 0             0            
Egress DSCP12 0             0            
Egress DSCP13 0             0            
Egress DSCP14 0             0            
Egress DSCP15 0             0            
Egress DSCP16 813172776     0            
Egress DSCP17 0             0            
Egress DSCP18 2791702990    0            
Egress DSCP19 0             0            
Egress DSCP20 0             0            
Egress DSCP21 0             0            
Egress DSCP22 0             0            
Egress DSCP23 0             0            
Egress DSCP24 509168402     0            
Egress DSCP25 0             0            
Egress DSCP26 711251        0            
Egress DSCP27 0             0            
Egress DSCP28 0             0            
Egress DSCP29 0             0            
Egress DSCP30 0             0            
Egress DSCP31 0             0            
Egress DSCP32 417763        0            
Egress DSCP33 0             0            
Egress DSCP34 62350         0            
Egress DSCP35 0             0            
Egress DSCP36 0             0            
Egress DSCP37 0             0            
Egress DSCP38 82288         0            
Egress DSCP39 0             0            
Egress DSCP40 0             0            
Egress DSCP41 0             0            
Egress DSCP42 0             0            
Egress DSCP43 0             0            
Egress DSCP44 0             0            
Egress DSCP45 0             0            
Egress DSCP46 560167476     0            
Egress DSCP47 0             0            
Egress DSCP48 41119286503   0            
Egress DSCP49 0             0            
Egress DSCP50 8             0            
Egress DSCP51 0             0            
Egress DSCP52 0             0            
Egress DSCP53 0             0            
Egress DSCP54 68            0            
Egress DSCP55 0             0            
Egress DSCP56 22250         0            
Egress DSCP57 0             0            
Egress DSCP58 0             0            
Egress DSCP59 0             0            
Egress DSCP60 0             0            
Egress DSCP61 0             0            
Egress DSCP62 0             0            
Egress DSCP63 269           0            
Egress COS0 45776301142   0            
Egress COS1 15476033      0            
Egress COS2 3604874412    0            
Egress COS3 505191634     0            
Egress COS4 562358        0            
Egress COS5 555846854     0            
Egress COS6 41079634368   0            
Egress COS7 22250         0

Beginner

Re: QoS Dropping Packets

Could a mis-configured QoS affect ports that do not have QoS policies attached to them?

 

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err  UnderSize  OutDiscards
Gi1/0/1             0           0    25452737           0          0     25452737
Gi1/0/2             0           0    33403581           0          0     33403581
Gi1/0/3             0           0      279166           0          0       279166
Gi1/0/4             0           0    19374337           0          0     19374337
Gi1/0/5             0           0    82146339           0          0     82146339
Gi1/0/6             0           0       59278           0          0        59278
Gi1/0/7             0           0           0           0          0            0
Gi1/0/8             0           0    18373082           0          0     18373082
Gi1/0/9             0           0    11634632           0          0     11634632
Gi1/0/10            0           0   198306645           0          0    198306645
Gi1/0/11            0           0     8007183           0          0      8007183
Gi1/0/12            0           0   720977758           0          0    720977758
Gi1/0/13            0           0           0           0          0            0
Gi1/0/14            0           0     8308167           0          0      8308167
Gi1/0/15            0           0      129370           0          0       129370
Gi1/0/16            0           0     3987453           0          0      3987453
Gi1/0/17            0           0    12564574           0          0     12564574
Gi1/0/18            0           0    22917583           0          0     22917583
Gi1/0/19            0           0     3900292           0          0      3900292
Gi1/0/20            0           0     1726647           0          0      1726647
Gi1/0/21            0           0    34887365           0          0     34887365
Gi1/0/22            0           0    13796343           0          0     13796343
Gi1/0/23            0           0     5894413           0          0      5894413
Gi1/0/24            0           0           0           0          0            0
Gi1/0/25            0           0           0           0          0            0
Gi1/0/26            0           0    23166434           0          0     23166434
Gi1/0/27            0           0           0           0          0            0
Gi1/0/28            0           0           0           0          0            0
Gi1/0/29            0           0    17422847           0          0     17422847
Gi1/0/30            0           0    29595513           0          0     29595513
Gi1/0/31            0           0           0           0          0            0
Gi1/0/32            0           0           0           0          0            0
Gi1/0/33            0           0           0           0          0            0
Gi1/0/34            0           0     5007989           0          0      5007989
Gi1/0/35            0           0           0           0          0            0
Gi1/0/36            0           0    19793040           0          0     19793040
Gi1/0/37            0           0    12853882           0          0     12853882
Gi1/0/38            0           0           0           0          0            0
Gi1/0/39            0           0    13975161           0          0     13975161
Gi1/0/40            0           0        9036           0          0         9036
Gi1/0/41            0           0           0           0          0            0
Gi1/0/42            0           0           0           0          0            0
Gi1/0/43            0           0           0           0          0            0
Gi1/0/44            0           0           0           0          0            0
Gi1/0/45            0           0      452156           0          0       452156
Gi1/0/46            0           0           0           0          0            0
Gi1/0/47            0           0           0           0          0            0
Gi1/0/48            0           0           0           0          0            0

CreatePlease to create content
Content for Community-Ad
August's Community Spotlight Awards