Site Network Slowness, Possible QoS Issue, Packet Drops on Policy-map

[Verbatim]

Related post: https://community.cisco.com/t5/other-network-architecture-subjects/accounting-for-packet-drops-qos/td-p/4851641

Network at site is experiencing slow speed / performance.

Network Map

Network map above; ISP provides via MPLS; there are two routers for the site, A and B; A has a 500m link to the ISP, B, a 200m link.

A bandwidth for past 48 hours:

B for past 24 hours:

Policy map on A router gig 0/0/0 interface will be at the end of this post. The 347 packet queue limit seems low, but that number is not set in the router config; assuming it is automatically generated? Showing significant dropped packet count on A router, gig 0/0/0 policy-map. Suggestions for what to investigate? Thank you!

rrx99-b45a#show policy-map interface gigabitEthernet 0/0/0
GigabitEthernet0/0/0

Service-policy output: QOS-WAN-500MB

Class-map: class-default (match-any)
18276624119 packets, 6314737075542 bytes
5 minute offered rate 185000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 347 packets
(queue depth/total drops/no-buffer drops) 0/534251/0
(pkts output/bytes output) 18269047366/6313603330772
shape (average) cir 500000000, bc 2000000, be 2000000
target shape rate 500000000

Service-policy : QOS-WAN-ASR-1G

queue stats for all priority classes:
Queueing
queue limit 512 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 3667277/1026644786

Class-map: QOS-PRIORITY (match-any)
3667277 packets, 1026644786 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp cs5 (40) ef (46)
Priority: 20% (100000 kbps), burst bytes 2500000, b/w exceed drops: 0

Class-map: QOS-REAL-TIME (match-any)
13038095 packets, 12391711302 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp cs4 (32) af41 (34)
Queueing
queue limit 125 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 13038095/12391711302
bandwidth remaining 20%

Class-map: QOS-SIGNALING (match-any)
7136843 packets, 2324671686 bytes
5 minute offered rate 156000 bps, drop rate 0000 bps
Match: dscp cs3 (24) af31 (26) cs6 (48) cs7 (56)
Queueing
queue limit 125 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 6911694/2308910996
bandwidth remaining 20%

Class-map: QOS-TIME-SENSITIVE (match-any)
18 packets, 2708 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp cs2 (16) af21 (18)
Queueing
queue limit 125 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 18/2708
bandwidth remaining 10%

Class-map: QOS-BULK (match-any)
57491 packets, 7941630 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: dscp 7 af11 (10)
Queueing
queue limit 125 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 57491/7941630
bandwidth remaining 1%

Class-map: class-default (match-any)
938634208 packets, 324039883126 bytes
5 minute offered rate 24000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 4000 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 938634208/324039883126
bandwidth remaining 49%

[Leo Laohoo]

How big is this link, 15 Mbps? 

What is the model of the router?

From the first graph, I can see the utilization throughout the day is dead and ramps up from 22:00 until 10:00 the next morning.  Is that the backup happening?  

[Joseph W. Doherty · Accepted solution]

"Showing significant dropped packet count on A router . . ."

My math might be wrong, but I show overall drop percent less than .003%.  As up to 1% is (traditionally) considered acceptable, why do you consider this significant?

"The 347 packet queue limit seems low . . ."

Perhaps it is, or perhaps not.

The "ideal" buffer size would be half the calculated BDP (bandwidth delay product).  Unfortunately, most Cisco routers will only support queuing by packet counting (some [ASRs?] do now also support queue limits by bytes or ms), so how many packets should allow would depend on packet sizes.

[Verbatim]

I was rushing, didn't do the math. Would have helped if I'd cleared counters; you're quite right, we had zero drops for the last business day. So the performance issue is something else. Good to know about the packet queue limit.

Thank you.

[Joseph W. Doherty]

BTW, although that one interface doesn't show significant drops, your WAN vendor might be dropping additional packets (especially if the link has a lessor than full bandwidth cap [which you might use a shaper to avoiding bumping into, but I believe many Cisco shapers only count L3 bytes while WAN vendors might also count L2 bytes - and if so, you can overrun your WAN limit]).

There are other possible issues to a "slow" network across a WAN, such as additional WAN latency and/or vendor WAN links not meeting their performance specifications.

[Verbatim]

Well, if we are only using 15% of the theoretical maximum of the link and bumping into an ISP imposed limit, that would be a problem. Oversubscription possibly? Thanks for pointing that out.

[Joseph W. Doherty]

Ah, but why only using 15%?

"Hidden" drops can tank throughput and/or 5 minute (?) averages also often hide what's happening down at the ms level, like microbursts.

Insufficient information to even speculate, but when poor performance, stats may be misleading.