topic Re: QoS Dropping Packets in Application Networking

QoS Dropping Packets

martynch1 — Wed, 24 Oct 2018 09:23:49 GMT

I wonder if you could help with a QoS issue that we have, we are dropping packets at an alarming rate with very little traffic on the wire, I can only assume the QoS is not set-up quite right.

These are the interface and QoS stats for a time period between 8am and 10am, the packets are being dropped from Site A to B

GigabitEthernet1/0/3 is up, line protocol is up (connected)

Hardware is Gigabit Ethernet, address is 3c5e.c303.4983 (bia 3c5e.c303.4983)

Description: 100mb P2P Circuit

MTU 1500 bytes, BW 100000 Kbit/sec, DLY 100 usec,

reliability 255/255, txload 14/255, rxload 89/255

Encapsulation ARPA, loopback not set

Keepalive set (10 sec)

Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX

input flow-control is off, output flow-control is unsupported

ARP type: ARPA, ARP Timeout 04:00:00

Last input 00:00:00, output never, output hang never

Last clearing of "show interface" counters 02:12:31

Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 5587991

Queueing strategy: Class-based queueing

Output queue: 0/4096 (size/max)

5 minute input rate 35163000 bits/sec, 3707 packets/sec

5 minute output rate 5801000 bits/sec, 2550 packets/sec

27403731 packets input, 33666560879 bytes, 0 no buffer

Received 21889 broadcasts (21845 multicasts)

0 runts, 0 giants, 0 throttles

0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored

0 watchdog, 21845 multicast, 0 pause input

0 input packets with dribble condition detected

19347219 packets output, 7391882600 bytes, 0 underruns

5587991 output errors, 0 collisions, 0 interface resets

0 unknown protocol drops

0 babbles, 0 late collision, 0 deferred

0 lost carrier, 0 no carrier, 0 pause output

0 output buffer failures, 0 output buffers swapped out

Service-policy output: NCI-QUEUE-SHAPE-OUTPUT-POLICY-PARENT-Inc

Class-map: class-default (match-any)

0 packets

Match: any

Queueing

(total drops) 5587991

(bytes output) 7427241975

shape (average) cir 100000000, bc 400000, be 400000

target shape rate 100000000

Service-policy : NCI-QUEUE-OUTPUT-POLICY-CHILD-IncreasedD

queue stats for all priority classes:

Queueing

priority level 1

(total drops) 0

(bytes output) 23832294

Class-map: NCI-VOIP-QUEUE (match-any)

0 packets

Match: ip dscp ef (46)

0 packets, 0 bytes

5 minute rate 0 bps

Priority: 10% (10000 kbps), burst bytes 250000,

Priority Level: 1

queue-buffers ratio 10

Class-map: NCI-VIDEO-QUEUE (match-any)

0 packets

Match: ip dscp af41 (34) af42 (36) af43 (38)

0 packets, 0 bytes

5 minute rate 0 bps

Queueing

queue-limit dscp 34 percent 100

queue-limit dscp 36 percent 90

queue-limit dscp 38 percent 80

(total drops) 0

(bytes output) 120

bandwidth remaining 15%

queue-buffers ratio 5

Class-map: NCI-NET-MGMT-QUEUE (match-any)

0 packets

Match: ip dscp cs2 (16) cs6 (48) cs7 (56)

0 packets, 0 bytes

5 minute rate 0 bps

Queueing

queue-limit dscp 16 percent 80

queue-limit dscp 48 percent 90

queue-limit dscp 56 percent 100

(total drops) 1294807

(bytes output) 41662725

bandwidth remaining 13%

queue-buffers ratio 15

Class-map: NCI-DATA-QUEUE (match-any)

0 packets

Match: ip dscp af21 (18) af22 (20) af23 (22)

0 packets, 0 bytes

5 minute rate 0 bps

Queueing

queue-limit dscp 18 percent 100

queue-limit dscp 20 percent 90

queue-limit dscp 22 percent 80

(total drops) 4293184

(bytes output) 5471263194

bandwidth remaining 30%

queue-buffers ratio 35

Class-map: NCI-VoIP-SIG-QUEUE (match-any)

0 packets

Match: ip dscp cs3 (24)

0 packets, 0 bytes

5 minute rate 0 bps

Queueing

(total drops) 0

(bytes output) 26003918

bandwidth remaining 5%

queue-buffers ratio 5

Class-map: NCI-SCAVENGER-QUEUE (match-any)

0 packets

Match: dscp cs1 (8)

0 packets, 0 bytes

5 minute rate 0 bps

Queueing

(total drops) 0

(bytes output) 0

bandwidth remaining 1%

queue-buffers ratio 5

Class-map: NCI-PUBLIC-QUEUE (match-any)

0 packets

Match: dscp 6

0 packets, 0 bytes

5 minute rate 0 bps

Queueing

(total drops) 0

(bytes output) 0

bandwidth remaining 5%

queue-buffers ratio 5

Class-map: class-default (match-any)

0 packets

Match: any

Queueing

(total drops) 0

(bytes output) 1864479724

bandwidth remaining 20%

queue-buffers ratio 20

We have a 100mb P2P fibre circuit between the sites

Site A: Cisco WS-C3850-48T - 03.06.06.E cat3k_caa-universalk9

Site B: Cisco WS-C3560-24PS - 12.2(55)SE11

The biggest complaint we have is assessing an SQL Database from site B to A, end users are getting a screen popup saying it’s taking to long to connect to the DB

I'm need to find what traffic is being dropped and why

QoS

policy-map NCI-QUEUE-OUTPUT-POLICY-CHILD-IncreasedD
class NCI-VOIP-QUEUE
priority level 1 percent 10
queue-buffers ratio 10
class NCI-VIDEO-QUEUE
bandwidth remaining percent 15
queue-buffers ratio 5
queue-limit dscp af43 percent 80
queue-limit dscp af42 percent 90
queue-limit dscp af41 percent 100

class NCI-NET-MGMT-QUEUE
bandwidth remaining percent 13
queue-buffers ratio 15
queue-limit dscp cs2 percent 80
queue-limit dscp cs6 percent 90
queue-limit dscp cs7 percent 100

class NCI-DATA-QUEUE
bandwidth remaining percent 30
queue-buffers ratio 35
queue-limit dscp af23 percent 80
queue-limit dscp af22 percent 90
queue-limit dscp af21 percent 100

class NCI-VoIP-SIG-QUEUE
bandwidth remaining percent 5
queue-buffers ratio 5

class NCI-SCAVENGER-QUEUE
bandwidth remaining percent 1
queue-buffers ratio 5

class NCI-PUBLIC-QUEUE
bandwidth remaining percent 5
queue-buffers ratio 5

class class-default
bandwidth remaining percent 20
queue-buffers ratio 20

policy-map NCI-QUEUE-SHAPE-OUTPUT-POLICY-PARENT-Inc
class class-default
shape average percent 100
service-policy NCI-QUEUE-OUTPUT-POLICY-CHILD-IncreasedD
policy-map NCI-MARK-POLICY
class NCI-VOIP-CLASS
set dscp ef
class NCI-VOIP-SIG-CLASS
set dscp cs3
class NCI-VIDEO-CLASS
set dscp af41
class NCI-DATA-CLASS
set dscp af21
class NCI-NET-MGMT-CLASS
set dscp cs2
class NCI-PUBLIC-CLASS
set dscp 6
class NCI-SCAVENGER-CLASS
set dscp cs1
class class-default
set dscp default

class-map match-any NCI-VOIP-CLASS
description VoIP traffic
match dscp ef

class-map match-any NCI-VOIP-SIG-CLASS
description Signalling traffic
match dscp cs3

class-map match-any NCI-VIDEO-CLASS
description Video Conferencing traffic
match access-group name NCI-VIDEO-ACL

class-map match-any NCI-DATA-CLASS
description Business Critical traffic
match access-group name NCI-DATA-ACL

class-map match-any NCI-NET-MGMT-CLASS
match access-group name NCI-NET-MGMT-ACL

class-map match-any NCI-PUBLIC-CLASS
description Public Traffic
match access-group name NCI-PUBLIC-ACL

class-map match-any NCI-SCAVENGER-CLASS
match access-group name NCI-SCAVENGER-ACL

ip access-list extended NCI-VIDEO-ACL
permit udp any any eq 554
permit udp any any eq 537
permit udp any any eq 1755
permit udp any any eq 1736

ip access-list extended NCI-DATA-ACL
permit tcp any any eq ftp-data
permit ip host 10.0.20.118 any
permit ip host 172.22.8.25 any
permit ip host 172.22.7.6 any
permit ip host 10.0.17.190 any
permit tcp any eq 443 any
permit tcp any eq 389 any
permit tcp any eq 8443 any
permit tcp any eq www any
permit tcp any eq ftp-data any
permit tcp any eq 995 any
permit tcp any eq 1521 any
permit udp any eq 1521 any
permit tcp any eq 1526 any
permit udp any eq 1526 any
permit tcp any eq 1575 any
permit udp any eq 1575 any
permit tcp any eq 1630 any
permit udp any eq 14002 any
permit udp any eq 14006 any
permit tcp any eq ftp any
permit tcp any eq 143 any
permit tcp any any eq 993
permit tcp any eq 993 any
permit tcp any eq smtp any
permit tcp any eq 465 any
permit tcp any eq 8080 any
permit tcp any eq 1433 any
permit tcp any eq lpd any
permit tcp any eq 9388 any
permit tcp any eq 1523 any
permit udp any eq 1523 any
permit udp any eq 9388 any
permit tcp any eq 5343 any
permit tcp any eq 445 any

ip access-list extended NCI-NET-MGMT-ACL
permit udp any eq ntp any
permit tcp any eq 22 any
permit udp any eq snmp any
permit udp any eq snmptrap any
permit tcp any eq 161 any
permit tcp any eq 162 any
permit icmp any any
permit udp any eq syslog any
permit tcp any eq tacacs any
permit udp any eq tacacs any
permit udp any eq bootps any
permit udp any eq domain any
permit tcp any eq domain any
permit udp any eq netbios-dgm any
permit udp any eq netbios-ns any
permit tcp any eq 135 any
permit tcp any eq 3389 any
permit tcp any eq 9996 any
permit tcp any eq 7777 any
permit udp any eq 9996 any
permit tcp any eq 9100 any
permit udp any eq 4172 any
permit tcp any eq 1494 any
permit tcp any eq 1090 any
permit tcp any eq 5985 any
permit tcp any eq 3268 any
permit tcp any eq 5246 any
permit tcp any eq 88 any

ip access-list extended NCI-PUBLIC-ACL
permit ip any 10.9.0.0 0.0.255.255

ip access-list extended NCI-SCAVENGER-ACL
permit tcp any any eq 3689
permit udp any any eq 3689
permit tcp any any range 6881 6999

Re: QoS Dropping Packets

Leo Laohoo — Wed, 24 Oct 2018 21:34:38 GMT

Is this a 3650/3850 switch?

Re: QoS Dropping Packets

martynch1 — Thu, 25 Oct 2018 06:38:23 GMT

Cisco WS-C3850-48T - 03.06.06.E cat3k_caa-universalk9

Thanks

Re: QoS Dropping Packets

finn.poulsen — Thu, 25 Oct 2018 13:25:40 GMT

Hi,

Try issuing this command :

show platform qos queue stats gi 1/0/3 ==> this command reveals in which queue the packet are enqued/dropped

show platform qos dscp-cos counters gi 1/0/3 ==> this command reveal the DSCP values seen on ingress and egress packets

OBS: please remember the space between "gi" and "1/0/3" !!

But back to your output :

You're dropping packets in

NCI-NET-MGMT-QUEUE ... (total drops) 1294807 ... cs2, cs6, cs7

NCI-DATA-QUEUE ... (total drops) 4293184 ... af21, af22, af23

the sum of these gives (Total output drops: 5587991) the same number (you're not always that lucky when looking at Cisco counters 😉

I wonder why you drop packets in the MGMT queue, but can see that you included Citrix VDI (TCP port 1494) and RAW printing (TCP port 9100) in the MGMT class (assume that you mark the packet ingress on other ports on the switch by the ...MARK policy), and these protocols can be a little BW hungry

But if you're sending from 1G ports towards a 100 Mbit shaped port, bursts will cause packet drops.

Only way to remedy this, is to supply more buffers to this queue, but that must be taken from somewhere else.

Best Regards

Finn

Re: QoS Dropping Packets

martynch1 — Thu, 25 Oct 2018 14:11:38 GMT

Thanks for the reply, I’ve stepped out of the office so will take a look when I get back.

Here’s the strange thing though, although it maybe not so strange to the trained eye.

I’ve removed QoS from the interface because there is not traffic on the link and I’m getting the same amount of drops, even when the site are just browsing the internet.

We have other interfaces showing the same sort of problems so I will run that command on their interface.

Thanks

Re: QoS Dropping Packets

finn.poulsen — Thu, 25 Oct 2018 14:25:21 GMT

Hi again,

This lead to the suspicion that you actually is sending more than 1Gbit of traffic out the interface, since you do not have an traffic shaper anymore (assume you've removed the service policy).

Best Regards

Finn

Re: QoS Dropping Packets

martynch1 — Thu, 25 Oct 2018 16:40:46 GMT

I've checked that with OpManager, results below just confuse it more. No congestion and not sending more than

27.9 Mbps

00:00 - 00:00

Total	Max	Min	Average	Standard Deviation	95th Percentile	Category


221.01 GB	38.25 Mbps	6.6 Mbps	27.9 Mbps	2.94 Mbps	30.48 Mbps	Traffic IN
41.63 GB	105.22 Mbps	678.39 Kbps	5.25 Mbps	12.01 Mbps	19.95 Mbps	Traffic OUT

Re: QoS Dropping Packets

martynch1 — Thu, 25 Oct 2018 17:06:20 GMT

show platform qos dscp-cos counters gi 1/0/3
Ingress DSCP0 52619399124   0
Ingress DSCP1 0             0
Ingress DSCP2 25112738      0
Ingress DSCP3 0             0
Ingress DSCP4 0             0
Ingress DSCP5 0             0
Ingress DSCP6 2047          0
Ingress DSCP7 0             0
Ingress DSCP8 13701671193   0
Ingress DSCP9 0             0
Ingress DSCP10 104397482     0
Ingress DSCP11 0             0
Ingress DSCP12 0             0
Ingress DSCP13 0             0
Ingress DSCP14 0             0
Ingress DSCP15 0             0
Ingress DSCP16 61634549      0
Ingress DSCP17 0             0
Ingress DSCP18 3080342803    0
Ingress DSCP19 0             0
Ingress DSCP20 0             0
Ingress DSCP21 0             0
Ingress DSCP22 0             0
Ingress DSCP23 0             0
Ingress DSCP24 418262395     0
Ingress DSCP25 0             0
Ingress DSCP26 3004424       0
Ingress DSCP27 0             0
Ingress DSCP28 0             0
Ingress DSCP29 0             0
Ingress DSCP30 0             0
Ingress DSCP31 0             0
Ingress DSCP32 0             0
Ingress DSCP33 0             0
Ingress DSCP34 108686        0
Ingress DSCP35 0             0
Ingress DSCP36 0             0
Ingress DSCP37 0             0
Ingress DSCP38 0             0
Ingress DSCP39 0             0
Ingress DSCP40 0             0
Ingress DSCP41 0             0
Ingress DSCP42 0             0
Ingress DSCP43 0             0
Ingress DSCP44 0             0
Ingress DSCP45 0             0
Ingress DSCP46 1550533909    0
Ingress DSCP47 0             0
Ingress DSCP48 302513870     0
Ingress DSCP49 0             0
Ingress DSCP50 0             0
Ingress DSCP51 0             0
Ingress DSCP52 0             0
Ingress DSCP53 0             0
Ingress DSCP54 2             0
Ingress DSCP55 0             0
Ingress DSCP56 0             0
Ingress DSCP57 0             0
Ingress DSCP58 0             0
Ingress DSCP59 0             0
Ingress DSCP60 0             0
Ingress DSCP61 0             0
Ingress DSCP62 0             0
Ingress DSCP63 0             0
Ingress COS0 60324229464   0
Ingress COS1 11572964454   0
Ingress COS2 3263465       0
Ingress COS3 0             0
Ingress COS4 0             0
Ingress COS5 0             0
Ingress COS6 0             0
Ingress COS7 111835556     0
Egress DSCP0 45720354616   0
Egress DSCP1 0             0
Egress DSCP2 0             0
Egress DSCP3 0             0
Egress DSCP4 168           0
Egress DSCP5 0             0
Egress DSCP6 4933          0
Egress DSCP7 0             0
Egress DSCP8 9659414       0
Egress DSCP9 0             0
Egress DSCP10 4600740       0
Egress DSCP11 0             0
Egress DSCP12 0             0
Egress DSCP13 0             0
Egress DSCP14 0             0
Egress DSCP15 0             0
Egress DSCP16 813172776     0
Egress DSCP17 0             0
Egress DSCP18 2791702990    0
Egress DSCP19 0             0
Egress DSCP20 0             0
Egress DSCP21 0             0
Egress DSCP22 0             0
Egress DSCP23 0             0
Egress DSCP24 509168402     0
Egress DSCP25 0             0
Egress DSCP26 711251        0
Egress DSCP27 0             0
Egress DSCP28 0             0
Egress DSCP29 0             0
Egress DSCP30 0             0
Egress DSCP31 0             0
Egress DSCP32 417763        0
Egress DSCP33 0             0
Egress DSCP34 62350         0
Egress DSCP35 0             0
Egress DSCP36 0             0
Egress DSCP37 0             0
Egress DSCP38 82288         0
Egress DSCP39 0             0
Egress DSCP40 0             0
Egress DSCP41 0             0
Egress DSCP42 0             0
Egress DSCP43 0             0
Egress DSCP44 0             0
Egress DSCP45 0             0
Egress DSCP46 560167476     0
Egress DSCP47 0             0
Egress DSCP48 41119286503   0
Egress DSCP49 0             0
Egress DSCP50 8             0
Egress DSCP51 0             0
Egress DSCP52 0             0
Egress DSCP53 0             0
Egress DSCP54 68            0
Egress DSCP55 0             0
Egress DSCP56 22250         0
Egress DSCP57 0             0
Egress DSCP58 0             0
Egress DSCP59 0             0
Egress DSCP60 0             0
Egress DSCP61 0             0
Egress DSCP62 0             0
Egress DSCP63 269           0
Egress COS0 45776301142   0
Egress COS1 15476033      0
Egress COS2 3604874412    0
Egress COS3 505191634     0
Egress COS4 562358        0
Egress COS5 555846854     0
Egress COS6 41079634368   0
Egress COS7 22250         0

Re: QoS Dropping Packets

martynch1 — Thu, 25 Oct 2018 18:13:21 GMT

Could a mis-configured QoS affect ports that do not have QoS policies attached to them?

Port        Align-Err     FCS-Err    Xmit-Err     Rcv-Err UnderSize OutDiscards
Gi1/0/1             0           0    25452737           0          0     25452737
Gi1/0/2             0           0    33403581           0          0     33403581
Gi1/0/3             0           0      279166           0          0       279166
Gi1/0/4             0           0    19374337           0          0     19374337
Gi1/0/5             0           0    82146339           0          0     82146339
Gi1/0/6             0           0       59278           0          0        59278
Gi1/0/7             0           0           0           0          0            0
Gi1/0/8             0           0    18373082           0          0     18373082
Gi1/0/9             0           0    11634632           0          0     11634632
Gi1/0/10            0           0   198306645           0          0    198306645
Gi1/0/11            0           0     8007183           0          0      8007183
Gi1/0/12            0           0   720977758           0          0    720977758
Gi1/0/13            0           0           0           0          0            0
Gi1/0/14            0           0     8308167           0          0      8308167
Gi1/0/15            0           0      129370           0          0       129370
Gi1/0/16            0           0     3987453           0          0      3987453
Gi1/0/17            0           0    12564574           0          0     12564574
Gi1/0/18            0           0    22917583           0          0     22917583
Gi1/0/19            0           0     3900292           0          0      3900292
Gi1/0/20            0           0     1726647           0          0      1726647
Gi1/0/21            0           0    34887365           0          0     34887365
Gi1/0/22            0           0    13796343           0          0     13796343
Gi1/0/23            0           0     5894413           0          0      5894413
Gi1/0/24            0           0           0           0          0            0
Gi1/0/25            0           0           0           0          0            0
Gi1/0/26            0           0    23166434           0          0     23166434
Gi1/0/27            0           0           0           0          0            0
Gi1/0/28            0           0           0           0          0            0
Gi1/0/29            0           0    17422847           0          0     17422847
Gi1/0/30            0           0    29595513           0          0     29595513
Gi1/0/31            0           0           0           0          0            0
Gi1/0/32            0           0           0           0          0            0
Gi1/0/33            0           0           0           0          0            0
Gi1/0/34            0           0     5007989           0          0      5007989
Gi1/0/35            0           0           0           0          0            0
Gi1/0/36            0           0    19793040           0          0     19793040
Gi1/0/37            0           0    12853882           0          0     12853882
Gi1/0/38            0           0           0           0          0            0
Gi1/0/39            0           0    13975161           0          0     13975161
Gi1/0/40            0           0        9036           0          0         9036
Gi1/0/41            0           0           0           0          0            0
Gi1/0/42            0           0           0           0          0            0
Gi1/0/43            0           0           0           0          0            0
Gi1/0/44            0           0           0           0          0            0
Gi1/0/45            0           0      452156           0          0       452156
Gi1/0/46            0           0           0           0          0            0
Gi1/0/47            0           0           0           0          0            0
Gi1/0/48            0           0           0           0          0            0