IOS-XR QoS ECN feature on NCS5500

arajhans · ‎05-31-2019

On IOS-XR, Quality of Service has an extension to WRED (Weighted Random Early Detection) called Explicit Congestion Notification (ECN). ECN will mark packets instead of dropping them when the average queue length exceeds a specific threshold value. When configured, ECN helps routers and end hosts to understand that the network is congested and slow down sending packets. However, If the number of packets in the queue is above the maximum threshold, packets are dropped based on the drop probability. This is the identical treatment a packet receives when WRED is enabled without ECN configured on the router.

This feature was enabled on IOS-XR 6.6.2 release for NCS-5500 series routers.

Scope: This document explains with some test results about ECN’s ability to manage excessive drops in case of congestion.

Note: For optimal profile, network operator needs to review various factors such as Bandwidth percentage, WRED values, etc. With manageable congestion you can achieve ‘zero’ loss for important traffic streams. Below results indicate that with ECN, we can reduce loss in imp streams for sure.

Congestion profile
Topology
Traffic distribution
Configs used
Testcases and Results
Understanding results
Test logs
More CLI’s
Disclaimers [ECN on NCS5500]
More references
Reviewers

Congestion profile

15.1G traffic on 100G ingress and 10G at egress on NCS-5508 router.

Topology

Ttraffic flow is from TGN port 7/1 to 3/1]

TGN1(3/1)<-->Hu0/1/0/1(NCS6K-2T)TenGigE0/7/0/0<-->TenGigE0/0/0/1/0(NCS5508)Hu0/0/0/0 <--->TGN(7/1)

100.3.2.1 100.3.2.2 100.3.3.1 100.3.3.2 100.3.4.2 100.3.4.1

Traffic distribution

discard-class	class-map	traffic-class	dscp	TOS(Dec)	stream	traffic load	frame size	Scheduling	BW %age
0	DSCP_P0	5	ef	184		2	1500	continuos+ burst size 10000 packets	10
0	DSCP_P1	4	cs6 cs7	192 224	cs6	1.5	1500		5
0	DSCP_P2	3	cs3	96		1.5	1500		10
1	DSCP_P3	2	af21 af22 af23	88	af23	3	1500		30
1	DSCP_P4	1	af11	40		1	1500		5
1	DSCP_P5	0	default	0		6	1500		40
	Total					15G			100

Configs used

At Ingress: (NCS5508)Hu0/0/0/0

Input: class-map	Input: policy-map
	policy-map twt-IN-1	policy-map twt-IN-2
class-map match-any DSCP_P0	class DSCP_P0	class DSCP_P0
match dscp ef	set traffic-class 5	set traffic-class 5
end-class-map	set discard-class 0	!
!	!	class DSCP_P1
class-map match-any DSCP_P1	class DSCP_P1	set traffic-class 4
match dscp cs6 cs7	set traffic-class 4	!
end-class-map	set discard-class 0	class DSCP_P2
!	!	set traffic-class 3
class-map match-any DSCP_P2	class DSCP_P2	!
match dscp cs3	set traffic-class 3	class DSCP_P3
end-class-map	set discard-class 0	set traffic-class 2
!	!	!
class-map match-any DSCP_P3	class DSCP_P3	class DSCP_P4
match dscp af21 af22 af23	set traffic-class 2	set traffic-class 1
end-class-map	set discard-class 1	!
!	!	class class-default
class-map match-any DSCP_P4	class DSCP_P4	!
match dscp af11	set traffic-class 1	end-policy-map
end-class-map	set discard-class 1	!
!	!
	class class-default
	!
	end-policy-map
	!

At Egress: TenGigE0/0/0/1/0(NCS5508)

Output:Class-map	Output: Policy-map
	policy-map twt-OUT-1	policy-map twt-OUT-2	policy-map twt-OUT-3	policy-map twt-OUT-4	policy-map twt-OUT-5	policy-map twt-OUT-6
class-map match-any P0	class P0	class P0	class P0	class P0	class P0	class P0
match traffic-class 5	bandwidth percent 10	bandwidth percent 10	bandwidth percent 10	bandwidth percent 10	bandwidth percent 10	bandwidth percent 10
end-class-map	random-detect default	random-detect discard-class 0 20 ms 120 ms	random-detect discard-class 0 20 ms 120 ms	random-detect discard-class 0 20 ms 120 ms	random-detect default	random-detect default
!	!	!	!	random-detect ecn	!	random-detect ecn
class-map match-any P1	class P1	class P1	class P1	!	class P1	!
match traffic-class 4	bandwidth percent 5	bandwidth percent 5	bandwidth percent 5	class P1	bandwidth percent 5	class P1
end-class-map	random-detect default	random-detect discard-class 0 20 ms 120 ms	random-detect discard-class 0 20 ms 120 ms	bandwidth percent 5	random-detect default	bandwidth percent 5
!	!	!	!	random-detect discard-class 0 20 ms 120 ms	!	random-detect default
class-map match-any P2	class P2	class P2	class P2	random-detect ecn	class P2	random-detect ecn
match traffic-class 3	bandwidth percent 10	bandwidth percent 10	bandwidth percent 10	!	bandwidth percent 10	!
end-class-map	random-detect default	random-detect discard-class 0 10 ms 60 ms	random-detect discard-class 0 10 ms 60 ms	class P2	random-detect default	class P2
!	!	!	!	bandwidth percent 10	!	bandwidth percent 10
class-map match-any P3	class P3	class P3	class P3	random-detect discard-class 0 10 ms 60 ms	class P3	random-detect default
match traffic-class 2	bandwidth percent 30	bandwidth percent 30	bandwidth percent 30	random-detect ecn	bandwidth percent 30	random-detect ecn
end-class-map	random-detect default	random-detect discard-class 1 5 ms 30 ms	random-detect discard-class 1 5 ms 30 ms	!	random-detect default	!
!	!	!	random-detect ecn	class P3	random-detect ecn	class P3
class-map match-any P4	class P4	class P4	!	bandwidth percent 30	!	bandwidth percent 30
match traffic-class 1	bandwidth percent 5	bandwidth percent 5	class P4	random-detect discard-class 1 5 ms 30 ms	class P4	random-detect default
end-class-map	random-detect default	random-detect discard-class 1 2 ms 12 ms	bandwidth percent 5	!	bandwidth percent 5	!
!	!	!	random-detect discard-class 1 2 ms 12 ms	class P4	random-detect default	class P4
	class class-default	class class-default	random-detect ecn	bandwidth percent 5	random-detect ecn	bandwidth percent 5
	bandwidth percent 40	bandwidth percent 40	!	random-detect discard-class 1 2 ms 12 ms	!	random-detect default
	random-detect default	random-detect discard-class 1 10 ms 60 ms	class class-default	!	class class-default	!
	!	!	bandwidth percent 40	class class-default	bandwidth percent 40	class class-default
	end-policy-map	end-policy-map	random-detect discard-class 1 10 ms 60 ms	bandwidth percent 40	random-detect default	bandwidth percent 40
	!	!	random-detect ecn	random-detect discard-class 1 10 ms 60 ms	random-detect ecn	random-detect default
			!	!	!	!
			end-policy-map	end-policy-map	end-policy-map	end-policy-map

Testcases and Results

test #	Description	Input policy	Output policy	traffic	Congestion
1	only ingress policy	twt-IN-1	None	all	151%
2	output policy with default random-detect values	twt-IN-1	twt-OUT-1	all	151%
3	output policy optimized random detect values	twt-IN-1	twt-OUT-2	all	151%
4	output policy optimized random detect values	twt-IN-1	twt-OUT-2	only DC1	101%
5	output policy optimized random detect values + ECN on DC1	twt-IN-1	twt-OUT-3	all	151%
6	output policy optimized random detect values + ECN on DC1	twt-IN-1	twt-OUT-3	only DC1	101%
7	output policy optimized random detect values + ECN on DC0	twt-IN-1	twt-OUT-4	all	151%
8	output policy default random detect values + ECN on DC1	twt-IN-1	twt-OUT-5	all	151%
9	output policy default random detect values + ECN on DC1	twt-IN-1	twt-OUT-5	only DC1	101%
10	output policy default random detect values + ECN on DC0	twt-IN-1	twt-OUT-6	all	151%

Test cases: 5, 6, 7, 10: covers main functionality for ECN

class stream	Gbps (Tx)	tests cases results (Drops in %age)
		TC1	TC2	TC3	TC4	TC5	TC6	TC7	TC8	TC9	TC10
DSCP_P0	2	15.8	81.6	38.6	-	45.2	-	42.1	81.6	81.6	45.2
DSCP_P1	1.5	15.8	85.7	67	-	70.7	-	69	85.7	85.7	70.7
DSCP_P2	1.5	16	81.6	49.6	-	49.7	-	22.9	81.6	81.5	49.6
DSCP_P3	3	19.7	65.3	22.5	20.1	0	0	22.7	65.3	64.8	0
DSCP_P4	1	15.9	85.7	94.5	91.7	89.8	88.9	94.5	85.7	85.7	89.7
DSCP_def	6	58.8	57.1	14	0	23.3	0	19.1	57.1	56.8	23.3
Total:	15G

Understanding results

Traffic and input-policy is constant across all testcases but drops vary. Various config tweaks in output policy shows how drops can be manipulated for priority traffic.

TC1: This is basic test with no output policy, drops are seen across all traffic classes in general.

TC2: output policy has system-default random detect values. Clearly this is not optimal to minimize drops as user has not considered the traffic input rates into each class.

TC3: output policy with defined threshold limit under random detect. This shows better results for priority classes traffic streams, drops.

TC4: Traffic enabled for only discard-class 1 streams [still congested at 101%]. Output policy with defined threshold limit under random detect. This shows better results for priority classes traffic streams drops.

TC5: Compare the result with TC3 numbers. Output policy with defined threshold limit under random detect, enable ECN for discard-class 1 traffic classes [P3, P4 and default]. ECN kicks in, P3 class drops are zero and P4, and default drops have reduced.

TC6: Compare the results with TC4 numbers. Traffic enabled for only discard-class 1 streams [still congested at 101%]. output policy with defined threshold limit under random detect, enable ECN for discard-class 1 traffic classes [P3, P4 and default]. ECN kicks in, P3 and default class drops are zero and P4 drops have reduced.

TC7: Compare the result with TC3 numbers. Output policy with defined threshold limit under random detect, enable ECN for discard-class 0 traffic classes [P0, P1 and P2]. ECN kicks in, P2 class drops improved, P0 & P1 went up. This indicates the wred needs some re-tuning wrt threshold values for traffic coming into these class streams.

TC8 & TC9: Compare with TC2. With system default random detect threshold values, ECN doesn’t help much irrespective of the DC on which the config is applied. This shows that wred tuning is still required.

TC10: Compare with TC3. Output policy with defined threshold limit under random detect, enable ECN for discard-class 0 and 1 [i.e. all streams used in the test]. ECN helps for DC-1 streams P3, P4, default and for DC-0 streams the drops go slightly up. User needs to fine tune the random detect threshold as per operational need on priority streams.

Test logs

QOS-ECN feature is introduced on NCS5500 platform starting release ios-xr 6.6.1, however there was no easy CLI to verify the ECN status on the router except the below logs. New CLI’s are introduced in release 6.6.2 [mentioned later in this document].

TC5: Configure input policy twt-IN-1 and output policy twt-OUT-3, ECN enabled for discard-class 1 traffic class [P3, P4 and class-default]

Verify discard drop are happening on LC at fia [LC fabricq ingress ASIC]

RP/0/RP0/CPU0:xrg-301-ncs5508#sh controllers fia statistics instance all location 0/0/CPU0 | i Dscrd | ex : 0

IQM1 EnqueueDscrdPktCnt : 508165034922

IQM1 QueueEnqDscrdByteCnt : 775968008319786

IQM1 QueueEnqDscrdPktCnt : 508165034920

IQM1 TotDscrdByteCnt : 71851069816600

RP/0/RP0/CPU0:xrg-301-ncs5508#sh controllers fia statistics instance all location 0/0/CPU0 | i Dscrd | ex : 0

IQM1 EnqueueDscrdPktCnt : 508189826863

IQM1 QueueEnqDscrdByteCnt : 776005865810676

IQM1 QueueEnqDscrdPktCnt : 508189827001

IQM1 TotDscrdByteCnt : 71854567701971

RP/0/RP0/CPU0:xrg-301-ncs5508#

Verify ECN is enabled [checking for one traffic-class]

[Note: voq ID used below may change depending on what queue ID system assigns for each traffic class. Use “sh policy-map interface tenGigE 0/0/0/1/0 output” to find the corresponding queue ID(VOQ ID) for the traffic-class]

RP/0/RP0/CPU0:xrg-301-ncs5508#show controller fia diagshell 0 "diag cosq voq id=1576 detail=1 core=1" loc 0/0/CPU0 | be WRED --à voq id 1576 corresponds to class-default

WRED info: Green : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)

Yellow: enable(True) min_thresh(5013504) max_thresh(30015488) drop_probability(9) -> Yellow for DC 1

Red : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)

Black : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)

Guaranteed info: guaranteed queue size in bytes(0) guaranteed queue size in BDs(0)

Tail drop info:

Green : max queue size in bytes(60030976), max queue size in BDs(234496)

Yellow: max queue size in bytes(60030976), max queue size in BDs(234496)

Red : max queue size in bytes(60030976), max queue size in BDs(234496)

Black : max queue size in bytes(60030976), max queue size in BDs(234496)

Fair adaptive tail drop info:

Enable: True

Green : alpha(-2)

Yellow: alpha(-2)

Red : alpha(-2)

Black : alpha(-2)

ECN max queue size in bytes(5013504), max queue size in BDs(524288)

ECN wred info: enable(True) min_thresh(0) max_thresh(30015488) drop_probability(11)

RP/0/RP0/CPU0:xrg-301-ncs5508#

Steps to verify drops reduced [using router CLI’s]

With ECN enabled for just one class - P3 on router, disable ECN bit from traffic generator for this stream [spirent]
Make a note of drops

RP/0/RP0/CPU0:xrg-301-ncs5508#sh policy-map interface tenGigE 0/0/0/1/0 output | b P3

Class P3

Classification statistics (packets/bytes) (rate - kbps)

Matched : 100000/150000000 0

Transmitted : 83134/124701000 0

Total Dropped : 16866/25299000 0 >>>>>> 16866 pkts drop

Queueing statistics

Queue ID : 1578

Taildropped(packets/bytes) : 16866/25299000

Enable ECN bit on Spirent now for the traffic stream, then check the counters:

RP/0/RP0/CPU0:xrg-301-ncs5508#clear qos counters interface all

Send traffic

RP/0/RP0/CPU0:xrg-301-ncs5508#sh policy-map interface tenGigE 0/0/0/1/0 output | b P3

Class P3

Classification statistics (packets/bytes) (rate - kbps)

Matched : 100000/150000000 0

Transmitted : 90407/135610500 0

Total Dropped : 9593/14389500 0 >>>>>> 9593 pkts drop

Queueing statistics

Queue ID : 1578

Taildropped(packets/bytes) : 9593/14389500

Conclusion: Drops are reduced for ECN marked packets in this case. ECN will mark packets instead of dropping them when the average queue length exceeds a specific threshold value. Some packets are still dropped as the packets in queue are more than the max threshold.

DC 0 behavior: [color code for discard class 0 – green]

RP/0/RP0/CPU0:xrg-301-ncs5508#show controller fia diagshell 0 "diag cosq voq id=1581 detail=1 core=1" loc 0/0/CPU0 | be WRED

WRED info:

Green : enable(True) min_thresh(2506752) max_thresh(15007744) drop_probability(9) >>>>>Green

Yellow: enable(False) min_thresh(0) max_thresh(0) drop_probability(0)

Red : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)

Black : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)

Guaranteed info: guaranteed queue size in bytes(0) guaranteed queue size in BDs(0)

Tail drop info:

Green : max queue size in bytes(30015488), max queue size in BDs(117248)

Yellow: max queue size in bytes(30015488), max queue size in BDs(117248)

Red : max queue size in bytes(30015488), max queue size in BDs(117248)

Black : max queue size in bytes(30015488), max queue size in BDs(117248)

Fair adaptive tail drop info:

Enable: True

Green : alpha(-2)

Yellow: alpha(-2)

Red : alpha(-2)

Black : alpha(-2)

ECN max queue size in bytes(2506752), max queue size in BDs(524288)

ECN wred info: enable(True) min_thresh(0) max_thresh(15007744) drop_probability(11)

RP/0/RP0/CPU0:xrg-301-ncs5508#

Note: color code for discard class 0 – yellow

More CLI’s

Starting IOS-XR 6.6.2 release, easier CLI’s are available

Sample logs below [not from original tests]

RP/0/RP0/CPU0:head#show qos interface tenGigE 0/0/0/6 output

NOTE:- Configured values are displayed within parentheses

Interface TenGigE0/0/0/6 ifh 0x1c8 -- output policy

NPU Id: 0

Total number of classes: 2

Interface Bandwidth: 10000000 kbps

Policy Name: pm-out-queue

VOQ Base: 1112

Accounting Type: Layer1 (Include Layer 1 encapsulation and above)

------------------------------------------------------------------------------

Level1 Class = cm-tc-1

Egressq Queue ID = 1113 (LP queue)

Queue Max. BW. = 1001956 kbps (10 %)

Queue Min. BW. = 0 kbps (default)

Inverse Weight / Weight = 1 / (BWR not configured)

Guaranteed service rate = 1000000 kbps

TailDrop Threshold = 5013504 bytes / 40 ms (default)

Default RED profile

WRED Min. Threshold = 1253376 bytes (10 ms)

WRED Max. Threshold = 2506752 bytes (20 ms)

WRED ECN = Enabled

Level1 Class = class-default

Egressq Queue ID = 1112 (Default LP queue)

Queue Max. BW. = no max (default)

Queue Min. BW. = 0 kbps (default)

Inverse Weight / Weight = 1 / (BWR not configured)

Guaranteed service rate = 5000000 kbps

TailDrop Threshold = 6258688 bytes / 10 ms (default)

WRED not configured for this class

RP/0/RP0/CPU0:head#

RP/0/RP0/CPU0:head#show policy-map interface tenGigE 0/0/0/6 output

TenGigE0/0/0/6 output: pm-out-queue

Class cm-tc-1

Classification statistics (packets/bytes) (rate - kbps)

Matched : 85528554/87581239296 4830672

Transmitted : 16240891/16630672384 966585

Total Dropped : 69287663/70950566912 3864087

Queueing statistics

Queue ID : 1113

Taildropped(packets/bytes) : 69287663/70950566912

WRED profile for

RED Transmitted (packets/bytes) : N/A

RED random drops(packets/bytes) : N/A

RED maxthreshold drops(packets/bytes) : N/A

RED ecn marked & transmitted(packets/bytes): N/A --> WRED counters are not supported on NCS5500, however this line entry confirms ECN is enabled.

Class class-default

Classification statistics (packets/bytes) (rate - kbps)

Matched : 0/0 0

Transmitted : 0/0 0

Total Dropped : 0/0 0

Queueing statistics

Queue ID : 1112

Taildropped(packets/bytes) : 0/0

Policy Bag Stats time: 1551105209930 [Local Time: 02/25/19 14:33:29.930]

RP/0/RP0/CPU0:head#

Disclaimers [ECN on NCS5500]

Not supported:	ECN at ingress
Not supported:	WRED not supported in priority level, hence no ECN
Not supported:	WRED ECN stats not supported on NCS5500
Supported:	Two level policy at egress, child may have ECN

More references

Modular QoS Configuration Guide for Cisco NCS 5500 Series Routers, IOS XR Release 6.0.x - https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/qos/60x/b-ncs5500-qos-configuration-guide-60x/b-qos-cg-ncs5500_chapter_00.html
Understanding QOS, default marking behavior and troubleshooting –https://community.cisco.com/t5/service-providers-documents/asr9000-xr-understanding-qos-default-marking-behavior-and/ta-p/3128709

Reviewers

Xander Thuijs xthuijs@cisco.com [DISTINGUISHED ENGINEER]