06-27-2017 08:55 PM - edited 03-05-2019 08:46 AM
HI Champs,
I am struggling to isolate a problem but not exactly sure if this is related to hardware or Memory issue.. So here is the problem..
Problem - I have Cisco 881 router and have 4M MPLS pipe from provider. I am running QoS across the network. Below are the queue that are used.. EF for Voice, Two for Business Data and One for default traffic.. Any internet traffic from Internet router is sent as default traffic. The problem is when customer starts downloading System updates from any public server, the latency for servers defined in Business data class increases i.e. when I ping from branch site to any server located in head office and defined in Business class it shows upto 1000ms of latency. For branch site PE to CE (where the problem is) I am allowing only 10% for default-class but it can burst upto port speed if there is no other traffic. But when there is other Business class traffic, it should still get prioritised from PE to CE..
I have checked CPU and Memory utilization and all looks normal. When customer starts downloading updates, traffic peaks only upto 3.2M so there is sill some room for Business class traffic but still the latency for servers in these class goes high when I ping to any server in this class from remote site.. Voice class doesn't show any latency.. Any idea what this can be.. ?
06-28-2017 12:30 AM
Hello,
post your configuration so we can check what you have configured in detail...
06-28-2017 05:40 PM
Hi Georg,
Please check I have uploaded the configs and other stats.
06-29-2017 12:05 AM
Hello,
post the output of:
show service-policy interface FastEthernet4
show service-policy interface Vlan1
This will show if packets are actually being dropped.
From what I can tell by looking at your configuration, you have effectively created a nested policy within a nested policy. Try and remove the third level in the hierarchy (remove the lines in bold), as the bandwidth percentage command used in the classes should be sufficient:
policy-map QOS-OUT-D1-RT-MPIPN8256047
class DSCP-OUT-D1-RT-INP-MPIPN8256047
police cir 1184000 bc 148000 be 148000
conform-action transmit
exceed-action set-dscp-transmit af42
policy-map QOS-OUT-D2-BA-MPIPN8256047
class DSCP-OUT-D2-BA-INP-MPIPN8256047
police cir 704000 bc 88000 be 88000
conform-action transmit
exceed-action set-dscp-transmit af22
policy-map QOS-OUT-D3-TO-MPIPN8256047
class DSCP-OUT-D3-TO-INP-MPIPN8256047
police cir 472000 bc 59000 be 59000
conform-action transmit
exceed-action set-dscp-transmit af12
policy-map QOS-OUT-MPIPN8256047
class DSCP-OUT-RT-VO-MPIPN8256047
priority 1640 205000
police cir 1640000 bc 205000 be 205000
conform-action set-dscp-transmit ef
exceed-action drop
class DSCP-OUT-D1-RT-MPIPN8256047
bandwidth percent 30
random-detect dscp-based
random-detect dscp 34 39 117 20
random-detect dscp 36 19 38 20
service-policy QOS-OUT-D1-RT-MPIPN8256047
class DSCP-OUT-D2-BA-MPIPN8256047
bandwidth percent 18
random-detect dscp-based
random-detect dscp 18 61 122 20
random-detect dscp 20 34 68 20
service-policy QOS-OUT-D2-BA-MPIPN8256047
class DSCP-OUT-D3-TO-MPIPN8256047
bandwidth percent 10
random-detect dscp-based
random-detect dscp 10 78 156 20
random-detect dscp 12 51 102 20
service-policy QOS-OUT-D3-TO-MPIPN8256047
policy-map SHAPE-OUT-4MB-MPIPN8256047
class class-default
shape average 4000000
service-policy QOS-OUT-MPIPN8256047
06-29-2017 12:27 AM
HI Georg,
I know the nested policies are extra bits and best to remove them but the standard is not defined by me.. so I have to follow which is been used at a large.. Below are the outputs as you have requested.. I was thinking of it as 'Wedged interface buffer leak' as I have noticed drops in the input queue of Fa4 interface(output in the last) but they are not increasing and problem only happens when customer starts downloading external updates and there are heavy number of packets received on WAN interface.. I was not sure if this is the real problem, I just guessed based on input queue drops..
macq-MFMRS8256014#sh policy-map interface fastEthernet 4
FastEthernet4
Service-policy input: TEST_DSCP_IN
Class-map: DSCP_IN_EF (match-any)
1640831 packets, 305171386 bytes
30 second offered rate 0000 bps
Match: ip dscp ef (46)
1640831 packets, 305171386 bytes
30 second rate 0 bps
Class-map: DSCP_IN_AF4 (match-any)
16722 packets, 6426532 bytes
30 second offered rate 8000 bps
Match: ip dscp af41 (34)
16722 packets, 6426532 bytes
30 second rate 8000 bps
Match: ip dscp af42 (36)
0 packets, 0 bytes
30 second rate 0 bps
Class-map: DSCP_IN_AF2 (match-any)
6430 packets, 487571 bytes
30 second offered rate 0000 bps
Match: ip dscp af21 (18)
6430 packets, 487571 bytes
30 second rate 0 bps
Match: ip dscp af22 (20)
0 packets, 0 bytes
30 second rate 0 bps
Class-map: DSCP_IN_AF1 (match-any)
10305757 packets, 4546820775 bytes
30 second offered rate 2394000 bps
Match: ip dscp af11 (10)
9302059 packets, 3336356037 bytes
30 second rate 2299000 bps
Match: ip dscp af12 (12)
1003698 packets, 1210464738 bytes
30 second rate 94000 bps
Class-map: class-default (match-any)
119763 packets, 13483011 bytes
30 second offered rate 2000 bps, drop rate 0000 bps
Match: any
Service-policy output: SHAPE-OUT-4MB-MPIPN8256047
Class-map: class-default (match-any)
11729945 packets, 1787426473 bytes
30 second offered rate 376000 bps, drop rate 0000 bps
Match: any
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 11729945/1787426473
shape (average) cir 4000000, bc 16000, be 16000
target shape rate 4000000
Service-policy : QOS-OUT-MPIPN8256047
queue stats for all priority classes:
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 1760343/330516396
Class-map: DSCP-OUT-RT-VO-MPIPN8256047 (match-any)
1760343 packets, 330516396 bytes
30 second offered rate 86000 bps, drop rate 0000 bps
Match: ip dscp ef (46)
1760343 packets, 330516396 bytes
30 second rate 86000 bps
Priority: 1640 kbps, burst bytes 205000, b/w exceed drops: 0
police:
cir 1640000 bps, bc 205000 bytes, be 205000 bytes
conformed 1760343 packets, 330516396 bytes; actions:
set-dscp-transmit ef
exceeded 0 packets, 0 bytes; actions:
drop
violated 0 packets, 0 bytes; actions:
drop
conformed 86000 bps, exceeded 0000 bps, violated 0000 bps
Class-map: DSCP-OUT-D1-RT-MPIPN8256047 (match-any)
8914 packets, 1430468 bytes
30 second offered rate 3000 bps, drop rate 0000 bps
Match: ip dscp af41 (34)
8914 packets, 1430468 bytes
30 second rate 3000 bps
Match: ip dscp af42 (36)
0 packets, 0 bytes
30 second rate 0 bps
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 8914/1430468
bandwidth 30% (1200 kbps)
Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob
af41 8914/1430468 0/0 0/0 39 117 1/20
af42 0/0 0/0 0/0 19 38 1/20
Service-policy : QOS-OUT-D1-RT-MPIPN8256047
Class-map: DSCP-OUT-D1-RT-INP-MPIPN8256047 (match-all)
8914 packets, 1430468 bytes
30 second offered rate 3000 bps, drop rate 0000 bps
Match: ip dscp af41 (34)
police:
cir 1184000 bps, bc 148000 bytes, be 148000 bytes
conformed 8914 packets, 1430468 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
set-dscp-transmit af42
violated 0 packets, 0 bytes; actions:
set-dscp-transmit af42
conformed 3000 bps, exceeded 0000 bps, violated 0000 bps
Class-map: class-default (match-any)
0 packets, 0 bytes
30 second offered rate 0000 bps, drop rate 0000 bps
Match: any
Class-map: DSCP-OUT-D2-BA-MPIPN8256047 (match-any)
91401 packets, 13343822 bytes
30 second offered rate 1000 bps, drop rate 0000 bps
Match: ip dscp af21 (18)
91401 packets, 13343822 bytes
30 second rate 1000 bps
Match: ip dscp af22 (20)
0 packets, 0 bytes
30 second rate 0 bps
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 91401/13343822
bandwidth 18% (720 kbps)
Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob
af21 91401/13343822 0/0 0/0 61 122 1/20
af22 0/0 0/0 0/0 34 68 1/20
Service-policy : QOS-OUT-D2-BA-MPIPN8256047
Class-map: DSCP-OUT-D2-BA-INP-MPIPN8256047 (match-all)
91401 packets, 13343822 bytes
30 second offered rate 1000 bps, drop rate 0000 bps
Match: ip dscp af21 (18)
police:
cir 704000 bps, bc 88000 bytes, be 88000 bytes
conformed 91401 packets, 13343822 bytes; actions:
transmit
exceeded 0 packets, 0 bytes; actions:
set-dscp-transmit af22
violated 0 packets, 0 bytes; actions:
set-dscp-transmit af22
conformed 1000 bps, exceeded 0000 bps, violated 0000 bps
Class-map: class-default (match-any)
0 packets, 0 bytes
30 second offered rate 0000 bps, drop rate 0000 bps
Match: any
Class-map: DSCP-OUT-D3-TO-MPIPN8256047 (match-any)
9869047 packets, 1442117967 bytes
30 second offered rate 284000 bps, drop rate 0000 bps
Match: ip dscp af11 (10)
9869047 packets, 1442117967 bytes
30 second rate 284000 bps
Match: ip dscp af12 (12)
0 packets, 0 bytes
30 second rate 0 bps
Queueing
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 9869047/1442117967
bandwidth 10% (400 kbps)
Exp-weight-constant: 9 (1/512)
Mean queue depth: 0 packets
dscp Transmitted Random drop Tail drop Minimum Maximum Mark
pkts/bytes pkts/bytes pkts/bytes thresh thresh prob
af11 9698568/1307018565 0/0 0/0 78 156 1/20
af12 170480/135099496 0/0 0/0 51 102 1/20
Service-policy : QOS-OUT-D3-TO-MPIPN8256047
Class-map: DSCP-OUT-D3-TO-INP-MPIPN8256047 (match-all)
9869047 packets, 1442117967 bytes
30 second offered rate 284000 bps, drop rate 0000 bps
Match: ip dscp af11 (10)
police:
cir 472000 bps, bc 59000 bytes, be 59000 bytes
conformed 9698568 packets, 1307018565 bytes; actions:
transmit
exceeded 39616 packets, 24632129 bytes; actions:
set-dscp-transmit af12
violated 130864 packets, 110467367 bytes; actions:
set-dscp-transmit af12
conformed 274000 bps, exceeded 9000 bps, violated 1000 bps
Class-map: class-default (match-any)
0 packets, 0 bytes
30 second offered rate 0000 bps, drop rate 0000 bps
Match: any
Class-map: class-default (match-any)
240 packets, 17820 bytes
30 second offered rate 0000 bps, drop rate 0000 bps
Match: any
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 240/17820
macq-MFMRS8256014#
macq-MFMRS8256014#
macq-MFMRS8256014#show ser
macq-MFMRS8256014#show poli
macq-MFMRS8256014#show policy-m
macq-MFMRS8256014#show policy-map
macq-MFMRS8256014#show policy-map int vlan 1
Vlan1
Service-policy input: QOS-IN-MPIPN8256047
Class-map: DSCP-IN-RT-VO-MPIPN8256047 (match-any)
1761220 packets, 330840666 bytes
5 minute offered rate 76000 bps, drop rate 0000 bps
Match: protocol rtp
1727815 packets, 326742170 bytes
5 minute rate 76000 bps
Match: protocol h323
15835 packets, 1436564 bytes
5 minute rate 0 bps
Match: access-group name DSCP-IN-RT-VO-MPIPN8256047
17570 packets, 2661932 bytes
5 minute rate 0 bps
QoS Set
dscp ef
Packets marked 1761220
Class-map: DSCP-IN-D1-RT-MPIPN8256047 (match-any)
8947 packets, 1467784 bytes
5 minute offered rate 2000 bps, drop rate 0000 bps
Match: access-group name DSCP-IN-D1-RT-MPIPN8256047
8947 packets, 1467784 bytes
5 minute rate 2000 bps
QoS Set
dscp af41
Packets marked 8947
Class-map: DSCP-IN-D2-BA-MPIPN8256047 (match-any)
1158 packets, 151245 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: access-group name DSCP-IN-D2-BA-MPIPN8256047
1158 packets, 151245 bytes
5 minute rate 0 bps
QoS Set
dscp af21
Packets marked 1158
Class-map: DSCP-IN-D3-TO-MPIPN8256047 (match-any)
10062001 packets, 1468987853 bytes
5 minute offered rate 459000 bps, drop rate 0000 bps
Match: access-group name DSCP-IN-D3-TO-MPIPN8256047
10062001 packets, 1468987853 bytes
5 minute rate 459000 bps
QoS Set
dscp af11
Packets marked 10062001
Class-map: class-default (match-any)
0 packets, 0 bytes
5 minute offered rate 0000 bps, drop rate 0000 bps
Match: any
QoS Set
dscp af11
Packets marked 0
macq-MFMRS8256014#
macq-MFMRS8256014#
macq-MFMRS8256014#sh int fa4
FastEthernet4 is up, line protocol is up
Hardware is PQII_PRO_UEC, address is f07f.06ba.dfcc (bia f07f.06ba.dfcc)
Description: MPIPN8256047
Internet address is 10.225.7.102/30
MTU 1500 bytes, BW 4096 Kbit/sec, DLY 100 usec,
reliability 255/255, txload 11/255, rxload 33/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/13/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: Class-based queueing
Output queue: 0/1000/0 (size/max total/drops)
30 second input rate 532000 bits/sec, 152 packets/sec
30 second output rate 189000 bits/sec, 152 packets/sec
12062870 packets input, 4888236844 bytes
Received 145 broadcasts (0 IP multicasts)
0 runts, 0 giants, 3 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
11756188 packets output, 1791075455 bytes, 0 underruns
0 output errors, 0 collisions, 3 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
1 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
06-29-2017 12:53 AM
Hello,
there are no drops in any of the classes you have defined (the input queue drops on the physical interface are minimal and negligible).
I actually don't think the problem is your 881 or any of the QoS configured on the 881, but further upstream. Can you get a hold of the QoS (if any) defined on the Juniper at the core site ?
06-29-2017 06:07 PM
Hey Georg,
Here is the Juniper config.. It is an ATM circuit with RJ45 handoff at customer end but on PE end it is configured as "Ethernet over ATM LLC"
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess vc-cos-mode strict
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-ef priority high
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-ef transmit-weight percent 41
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-ef linear-red-profile lrp-ef
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-af2 priority low
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-af2 transmit-weight percent 30
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-af2 linear-red-profile lrp-af2
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-af1 priority low
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-af1 transmit-weight percent 18
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-af1 linear-red-profile lrp-af1
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-be priority low
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-be transmit-weight percent 11
set interfaces at-0/0/0 atm-options scheduler-maps MPIPN8256047-atmaccess forwarding-class fc-be linear-red-profile lrp-be
set interfaces at-0/0/0 unit 1091 encapsulation ether-over-atm-llc
set interfaces at-0/0/0 unit 1091 vci 1.91
set interfaces at-0/0/0 unit 1091 shaping vbr peak 4m
set interfaces at-0/0/0 unit 1091 shaping vbr sustained 4m
set interfaces at-0/0/0 unit 1091 shaping vbr burst 1
set interfaces at-0/0/0 unit 1091 atm-scheduler-map MPIPN8256047-atmaccess
set interfaces at-0/0/0 unit 1091 family inet mtu 1500
set interfaces at-0/0/0 unit 1091 family inet address 10.225.7.101/30
m10-01-re0> show interfaces at-0/0/0.1091 detail
Logical interface at-0/0/0.1091 (Index 100) (SNMP ifIndex 809) (Generation 6749)
Flags: Point-To-Multipoint SNMP-Traps 0x4000 Encapsulation: Ether-over-ATM-LLC
Traffic statistics:
Input bytes : 2886766117
Output bytes : 9838109560
Input packets: 19553363
Output packets: 20631712
Local statistics:
Input bytes : 3209219
Output bytes : 3297334
Input packets: 46680
Output packets: 47173
Transit statistics:
Input bytes : 2883556898 320048 bps
Output bytes : 9834812226 1114680 bps
Input packets: 19506683 307 pps
Output packets: 20584539 296 pps
Protocol inet, MTU: 1500, Generation: 2508, Route table: 25
Flags: Sendbcast-pkt-to-re, Is-Primary, User-MTU
Addresses, Flags: Is-Preferred Is-Primary
Destination: 10.225.7.100/30, Local: 10.225.7.101, Broadcast: 10.225.7.103, Generation: 1899
VCI 1.91
Flags: Active, Shaping, Multicast
VBR, Peak: 4mbps, Sustained: 4mbps, Burst size: 1
Total down time: 00:31:45 sec, Last down: 66:51:41 ago
EPD threshold: 0, Transmit weight cells: 0
ATM per-VC transmit statistics:
Tail queue packet drops: 29441
Traffic statistics:
Input bytes : 3443743742
Output bytes : 10291252760
Input packets: 19553318
Output packets: 20602220
ATM CoS statistics:
VC CoS mode: strict
fc-be 0:
Priority: low, Transmit weight percent: 11
RED queue depth: 50000, Low PLP threshold: 50, High PLP threshold: 10, Low PLP max threshold: 50,
High PLP max threshold: 10
Output bytes : 9764680283
Output packets: 18046475
Packet Drops : 29441
fc-af1 1:
Priority: low, Transmit weight percent: 18
RED queue depth: 20000, Low PLP threshold: 80, High PLP threshold: 60, Low PLP max threshold: 80,
High PLP max threshold: 60
Output bytes : 1379
Output packets: 15
Packet Drops : 0
fc-af2 2:
Priority: low, Transmit weight percent: 30
RED queue depth: 20000, Low PLP threshold: 100, High PLP threshold: 100, Low PLP max threshold: 100,
High PLP max threshold: 100
Output bytes : 11656530
Output packets: 18681
Packet Drops : 0
fc-ef 3:
Priority: high, Transmit weight percent: 41
RED queue depth: 10000, Low PLP threshold: 100, High PLP threshold: 100, Low PLP max threshold: 100,
High PLP max threshold: 100
Output bytes : 514914568
Output packets: 2537049
Packet Drops : 0
OAM F5 cell statistics:
Total received: 1874, Total sent: 1874
Loopback received: 0, Loopback sent: 0
RDI received: 0, RDI sent: 1874
Last received: Never, Last sent: 66:19:59
AIS received: 1874, AIS sent: 0
Last received: 66:19:59, , Last sent: Never
m10-01-re0>
m10-01-re0> show interfaces at-0/0/0.1091
Logical interface at-0/0/0.1091 (Index 100) (SNMP ifIndex 809)
Flags: Point-To-Multipoint SNMP-Traps 0x4000 Encapsulation: Ether-over-ATM-LLC
Input packets : 19561774
Output packets: 20639042
Protocol inet, MTU: 1500
Flags: Sendbcast-pkt-to-re, Is-Primary, User-MTU
Addresses, Flags: Is-Preferred Is-Primary
Destination: 10.225.7.100/30, Local: 10.225.7.101, Broadcast: 10.225.7.103
VCI 1.91
Flags: Active, Shaping, Multicast
VBR, Peak: 4mbps, Sustained: 4mbps, Burst size: 1
Total down time: 00:31:45 sec, Last down: 66:52:14 ago
EPD threshold: 0, Transmit weight cells: 0
Input packets : 19561662
Output packets: 20610105
OAM F5 cell statistics:
Total received: 1874, Total sent: 1874
Loopback received: 0, Loopback sent: 0
RDI received: 0, RDI sent: 1874
AIS received: 1874, AIS sent: 0
m10-01-re0>
06-28-2017 04:48 AM
Could you also clarify your traffic flows. I.e. how you apply QoS to a mixture of internal (?) business class traffic and Internet traffic?
Also could you clarify where all QoS is deployed. I.e. you mention PE to CE, but you say your (?) only allowing 10% for default class, on PE?
06-28-2017 05:47 PM
Hi Joseph,
Please check config I have uploaded CE config which should give idea of QoS applied to Business and Internet traffic.
Core is Juniper and the PE(at branch site) has four queues for different traffic flows from PE to CE.
EF queue has given 41% of 4M bandwidth and rate-limited.
Two data queues have been given 48% of remaining bandwidth and allowed to burst upto port speed.
Default queue has been given 11% of remaining bandwidth and allowed to burst upto port speed.
06-28-2017 05:39 PM
Sorry.. I forgot to upload the documents.. please find it here. Text file has configuration and other stats. Two .png files shows the latency for BA and RT data queues.. We are monitoring each class traffic from a central monitoring station and these pictures were taken from that server.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide