Solved: yup yup indeed! :) there is a

evan.r.moore · ‎08-29-2014

I am seeing behavior in the lab which I don't understand. I hope someone can help.

I have three ASR 9010 running XR 4.2.3. All have RSP440, MOD-80, and A9K-MPA-4X10GE cards. Node A has a 10 gigabit ethernet link to node B, and node B has a 10 gigabit ethernet link to node C. I have test systems connected to A and C, again with 10 gigabit ethernet links.

If I configure all the ports with input service policies that only remark packets, and no output service policies, I can pass traffic loads from 5 to 9 gigabits, at various small packet sizes, without losing any datagrams. If I add an output service policy to the port on C linked to a test system:

policy-map test
class VOICE
priority level 1
police rate percent 5
conform-action transmit
exceed-action drop
!
!
class BETTER
bandwidth percent 35
!
class GOOD
bandwidth percent 45
!
class class-default
random-detect discard-class 2 4096 bytes 8192 bytes
random-detect discard-class 1 256 bytes 8192 bytes
queue-limit 8192 bytes
!
end-policy-map

then I get packet drops, both from RED and from tail drops.

This may seem perfectly fine to you, until you remember that I'm testing the interfaces at less than 100%. Since there is no congestion, why is the output service policy having any impact at all?

Bitwrangler for FirstLight Fiber Corp

xthuijs · ‎09-01-2014

ah that helps ERM!

ok so here is the deal, by default WRED kicks in at the queue limit as the "max" value and the min is 1/2* the queue limit. If the queue limit is not defined on a class, 100msec as default is taken for that value.

Now you ended up with a queue limit of only 8k, which is very small obviously. And as you can see in the output, the instant queue length is 30 (that is instant read out of the hardware that moment), so there were packets in the queue. That means that the WRED has kicked in and there will be a drop probability assigned to the packet and it might get wacked, and that is what you are seeing here.

Things you can do here is adjust your WRED curves to get less likely drops, adjust the queue limit of the default class (which will also help reducing the drop probability), fact of the matter is your WRED current config is very "tight" which means that if there is the slightest amount of buffering going on, WRED will immediately kick in.

cheers!

xander

View solution in original post

xthuijs · ‎08-31-2014

hi evan,

can you show us the show policy-map interface <name> out from the C device to the tester?

also show me a show qos int te<if of C to tester> out

this way I can see what the hw programmed (with the 2nd command) and what the policy-map is doing.

regards!

xander

evan.r.moore · ‎09-01-2014

As you requested:

RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#show int te0/7/0/3
TenGigE0/7/0/3 is up, line protocol is up
Interface state transitions: 3
Hardware is TenGigE, address is d867.d94d.e0f3 (bia d867.d94d.e0f3)
Layer 1 Transport Mode is LAN
Internet address is 64.17.123.5/30
MTU 9100 bytes, BW 10000000 Kbit (Max: 10000000 Kbit)
reliability 255/255, txload 196/255, rxload 196/255
Encapsulation ARPA,
Full-duplex, 10000Mb/s, link type is force-up
output flow control is off, input flow control is off
loopback not set,
ARP type ARPA, ARP timeout 04:00:00
Last input 00:00:00, output 00:00:00
Last clearing of "show interface" counters 00:03:19
5 minute input rate 7698307000 bits/sec, 6776681 packets/sec
5 minute output rate 7697799000 bits/sec, 6776233 packets/sec
1346430692 packets input, 191193158118 bytes, 0 total input drops
0 drops for unrecognized upper-level protocol
Received 0 broadcast packets, 0 multicast packets
0 runts, 0 giants, 0 throttles, 0 parity
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
1346259548 packets output, 191168855600 bytes, 0 total output drops
Output 0 broadcast packets, 0 multicast packets
0 output errors, 0 underruns, 0 applique, 0 resets
0 output buffer failures, 0 output buffers swapped out
0 carrier transitions

RP/0/RSP0/CPU0:NodeC#show qos inter te0/7/0/3 output
Interface: TenGigE0_7_0_3 output
Bandwidth configured: 10000000 kbps Bandwidth programed: 10000000 kbps
ANCP user configured: 0 kbps ANCP programed in HW: 0 kbps
Port Shaper programed in HW: 0 kbps
Policy: core-link-10g-intrapop Total number of classes: 4
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: qos5dscpEF
QueueID: 196648 (Priority 1)
Queue Limit: 6272 kbytes Abs-Index: 111 Template: 0 Curve: 6
Shape CIR Profile: INVALID
Policer Profile: 47 (Single)
Conform: 500000 kbps (5 percent) Burst: 6250000 bytes (0 Default)
Child Policer Conform: TX
Child Policer Exceed: DROP
Child Policer Violate: DROP
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: qos3dscpBetter
QueueID: 196650 (Priority Normal)
Queue Limit: 52224 kbytes Abs-Index: 158 Template: 0 Curve: 0
Shape CIR Profile: INVALID
WFQ Profile: 3/204 Committed Weight: 877 Excess Weight: 877
Bandwidth: 3500000 kbps, BW sum for Level 0: 8000000 kbps, Excess Ratio: 1
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: qos2dscpGood
QueueID: 196651 (Priority Normal)
Queue Limit: 65536 kbytes Abs-Index: 163 Template: 0 Curve: 0
Shape CIR Profile: INVALID
WFQ Profile: 3/219 Committed Weight: 1119 Excess Weight: 1119
Bandwidth: 4500000 kbps, BW sum for Level 0: 8000000 kbps, Excess Ratio: 1
----------------------------------------------------------------------
Level: 0 Policy: core-link-10g-intrapop Class: class-default
QueueID: 196652 (Priority Normal)
Queue Limit: 8 kbytes (8192 bytes) Abs-Index: 6 Template: 0 Curve: 0
Shape CIR Profile: INVALID
WFQ Profile: 3/0 Committed Weight: 1 Excess Weight: 1
Bandwidth: 0 kbps, BW sum for Level 0: 8000000 kbps, Excess Ratio: 1
WRED Type: Discard-class based Table: 0 Curves: 3
Default RED Curve Profile: 6/0/0 Thresholds Min : 7 (8) kbytes Max: 8 (8) kbytes
WRED Curve: 1 Profile: 6/7/4 Thresholds Min : 3 (4) kbytes Max: 8 (8) kbytes
Match: 2
WRED Curve: 2 Profile: 6/15/3 Thresholds Min : 0 (0) kbytes Max: 8 (8) kbytes
Match: 1
----------------------------------------------------------------------
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#show policy-map int te0/7/0/3

TenGigE0/7/0/3 input: input-classify

Class exp1
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class exp2
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class exp3
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class exp4
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class exp5
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class exp6
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class exp7
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class dscpEF
Classification statistics (packets/bytes) (rate - kbps)
Matched : 194/29458 0
Transmitted : N/A
Total Dropped : N/A
Class dscpScavenger
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class dscpGood
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class dscpBetter
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : N/A
Total Dropped : N/A
Class class-default
Classification statistics (packets/bytes) (rate - kbps)
Matched : 1735410808563/282710291507630 7923363
Transmitted : N/A
Total Dropped : N/A

TenGigE0/7/0/3 output: core-link-10g-intrapop

Class qos5dscpEF
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : 0/0 0
Total Dropped : 0/0 0
Policing statistics (packets/bytes) (rate - kbps)
Policed(conform) : 0/0 0
Policed(exceed) : 0/0 0
Policed(violate) : 0/0 0
Policed and dropped : 0/0
Queueing statistics
Queue ID : 196648
High watermark (Unknown)
Inst-queue-len (packets) : 0
Avg-queue-len (Unknown)
Taildropped(packets/bytes) : 0/0
Queue(conform) : 0/0 0
Queue(exceed) : 0/0 0
RED random drops(packets/bytes) : 0/0

Class qos3dscpBetter
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : 21893601/3196465746 0
Total Dropped : 0/0 0
Queueing statistics
Queue ID : 196650
High watermark (Unknown)
Inst-queue-len (packets) : 0
Avg-queue-len (Unknown)
Taildropped(packets/bytes) : 0/0
Queue(conform) : 21893601/3196465746 0
Queue(exceed) : 0/0 0
RED random drops(packets/bytes) : 0/0

Class qos2dscpGood
Classification statistics (packets/bytes) (rate - kbps)
Matched : 0/0 0
Transmitted : 0/0 0
Total Dropped : 0/0 0
Queueing statistics
Queue ID : 196651
High watermark (Unknown)
Inst-queue-len (packets) : 0
Avg-queue-len (Unknown)
Taildropped(packets/bytes) : 0/0
Queue(conform) : 0/0 0
Queue(exceed) : 0/0 0
RED random drops(packets/bytes) : 0/0

Class class-default
Classification statistics (packets/bytes) (rate - kbps)
Matched : 1475073517/215360733700 7919974
Transmitted : 1470089735/214633101528 7918540
Total Dropped : 271052/39573592 1519
Queueing statistics
Queue ID : 196652
High watermark (Unknown)
Inst-queue-len (packets) : 30
Avg-queue-len (Unknown)
Taildropped(packets/bytes) : 0/0
Queue(conform) : 1470089735/214633101528 7918540
Queue(exceed) : 0/0 0
RED random drops(packets/bytes) : 271052/39573592

WRED profile for WRED Curve 1
RED Transmitted (packets/bytes) : N/A
RED random drops(packets/bytes) : 271052/39573592
RED maxthreshold drops(packets/bytes): N/A
WRED profile for WRED Curve 2
RED Transmitted (packets/bytes) : N/A
RED random drops(packets/bytes) : 0/0
RED maxthreshold drops(packets/bytes): N/A
RP/0/RSP0/CPU0:NodeC#
RP/0/RSP0/CPU0:NodeC#

In this particular run, taildrops haven't yet occurred, but RED drops have. If the interface was actually congested, this would make complete sense. As the show interface output indicates, I'm not even at 80% load, so I don't understand why the service-policy is active.

I'll also add that these are JDSU test sets generating constant load traffic on a single flow, so I don't think it should be a "microburst" situation.

Thanks for taking a look.

ERM

Bitwrangler for FirstLight Fiber Corp

xthuijs · ‎09-01-2014

ah that helps ERM!

ok so here is the deal, by default WRED kicks in at the queue limit as the "max" value and the min is 1/2* the queue limit. If the queue limit is not defined on a class, 100msec as default is taken for that value.

Now you ended up with a queue limit of only 8k, which is very small obviously. And as you can see in the output, the instant queue length is 30 (that is instant read out of the hardware that moment), so there were packets in the queue. That means that the WRED has kicked in and there will be a drop probability assigned to the packet and it might get wacked, and that is what you are seeing here.

Things you can do here is adjust your WRED curves to get less likely drops, adjust the queue limit of the default class (which will also help reducing the drop probability), fact of the matter is your WRED current config is very "tight" which means that if there is the slightest amount of buffering going on, WRED will immediately kick in.

cheers!

xander

evan.r.moore · ‎09-01-2014

TAC just responded to my case (Mark Sanchez for the win). He reminds me that of course RED is dropping packets when the port is less than 100%; that's what RED is meant to do.

Sometimes you just need someone to tell you what's already in front of your eyes.

ERM

Bitwrangler for FirstLight Fiber Corp

xthuijs · ‎09-01-2014

yup yup indeed! :) there is a great Dutch saying: "double screwed is fixed for sure"

applies here also :) glad it is resolved ERM!

cheers

xander

QoS without congestion on ASR 9K