cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
3166
Views
0
Helpful
0
Comments
arajhans
Cisco Employee
Cisco Employee

On IOS-XR, Quality of Service has an extension to WRED (Weighted Random Early Detection) called Explicit Congestion Notification (ECN). ECN will mark packets instead of dropping them when the average queue length exceeds a specific threshold value. When configured, ECN helps routers and end hosts to understand that the network is congested and slow down sending packets. However, If the number of packets in the queue is above the maximum threshold, packets are dropped based on the drop probability. This is the identical treatment a packet receives when WRED is enabled without ECN configured on the router.

This feature was enabled on IOS-XR 6.6.2 release for NCS-5500 series routers.

Scope: This document explains with some test results about ECN’s ability to manage excessive drops in case of congestion.

Note: For optimal profile, network operator needs to review various factors such as Bandwidth percentage, WRED values, etc. With manageable congestion you can achieve ‘zero’ loss for important traffic streams. Below results indicate that with ECN, we can reduce loss in imp streams for sure.

Congestion profile

15.1G traffic on 100G ingress and 10G at egress on NCS-5508 router.

Topology

Ttraffic flow is from TGN port 7/1 to 3/1]

TGN1(3/1)<-->Hu0/1/0/1(NCS6K-2T)TenGigE0/7/0/0<-->TenGigE0/0/0/1/0(NCS5508)Hu0/0/0/0 <--->TGN(7/1)

100.3.2.1         100.3.2.2                     100.3.3.1                  100.3.3.2                                 100.3.4.2                     100.3.4.1

Traffic distribution

discard-class

class-map

traffic-class

dscp

TOS(Dec)

stream

traffic load

frame size

Scheduling

BW %age

0

DSCP_P0

5

ef

184

 

2

1500

continuos+ burst size 10000 packets

10

0

DSCP_P1

4

cs6 cs7

192 224

cs6

1.5

1500

5

0

DSCP_P2

3

cs3

96

 

1.5

1500

10

1

DSCP_P3

2

af21 af22 af23

88

af23

3

1500

30

1

DSCP_P4

1

af11

40

 

1

1500

5

1

DSCP_P5

0

default

0

 

6

1500

40

 

Total

 

 

 

 

15G

 

 

100

 

Configs used

  • At Ingress: (NCS5508)Hu0/0/0/0

 

Input: class-map

 

Input: policy-map

 

 

policy-map twt-IN-1

policy-map twt-IN-2

class-map match-any DSCP_P0

 

 class DSCP_P0

 class DSCP_P0

 match dscp ef

 

  set traffic-class 5

  set traffic-class 5

 end-class-map

 

  set discard-class 0

 !

!

 

 !

 class DSCP_P1

class-map match-any DSCP_P1

 

 class DSCP_P1

  set traffic-class 4

 match dscp cs6 cs7

 

  set traffic-class 4

 !

 end-class-map

 

  set discard-class 0

 class DSCP_P2

!

 

 !

  set traffic-class 3

class-map match-any DSCP_P2

 

 class DSCP_P2

 !

 match dscp cs3

 

  set traffic-class 3

 class DSCP_P3

 end-class-map

 

  set discard-class 0

  set traffic-class 2

!

 

 !

 !

class-map match-any DSCP_P3

 

 class DSCP_P3

 class DSCP_P4

 match dscp af21 af22 af23

 

  set traffic-class 2

  set traffic-class 1

 end-class-map

 

  set discard-class 1

 !

!

 

 !

 class class-default

class-map match-any DSCP_P4

 

 class DSCP_P4

 !

 match dscp af11

 

  set traffic-class 1

 end-policy-map

 end-class-map

 

  set discard-class 1

!

!

 

 !

 

 

 

 class class-default

 

 

 

 !

 

 

 

 end-policy-map

 

 

 

!

 

 

  • At Egress: TenGigE0/0/0/1/0(NCS5508)

Output:Class-map

Output: Policy-map

 

policy-map twt-OUT-1

policy-map twt-OUT-2

policy-map twt-OUT-3

policy-map twt-OUT-4

policy-map twt-OUT-5

policy-map twt-OUT-6

class-map match-any P0

 class P0

 class P0

 class P0

 class P0

 class P0

 class P0

 match traffic-class 5

  bandwidth percent 10

  bandwidth percent 10

  bandwidth percent 10

  bandwidth percent 10

  bandwidth percent 10

  bandwidth percent 10

 end-class-map

  random-detect default

  random-detect discard-class 0 20 ms 120 ms  

  random-detect discard-class 0 20 ms 120 ms 

  random-detect discard-class 0 20 ms 120 ms 

  random-detect default

  random-detect default

!

 !

 !

 !

random-detect ecn

 !

random-detect ecn

class-map match-any P1

 class P1

 class P1

 class P1

 !

 class P1

 !

 match traffic-class 4

  bandwidth percent 5

  bandwidth percent 5

  bandwidth percent 5

 class P1

  bandwidth percent 5

 class P1

 end-class-map

  random-detect default

  random-detect discard-class 0 20 ms 120 ms 

  random-detect discard-class 0 20 ms 120 ms 

  bandwidth percent 5

  random-detect default

  bandwidth percent 5

!

 !

 !

 !

  random-detect discard-class 0 20 ms 120 ms 

 !

  random-detect default

class-map match-any P2

 class P2

 class P2

 class P2

random-detect ecn

 class P2

random-detect ecn

 match traffic-class 3

  bandwidth percent 10

  bandwidth percent 10

  bandwidth percent 10

 !

  bandwidth percent 10

 !

 end-class-map

  random-detect default

  random-detect discard-class 0 10 ms 60 ms 

  random-detect discard-class 0 10 ms 60 ms 

 class P2

  random-detect default

 class P2

!

 !

 !

 !

  bandwidth percent 10

 !

  bandwidth percent 10

class-map match-any P3

 class P3

 class P3

 class P3

  random-detect discard-class 0 10 ms 60 ms 

 class P3

  random-detect default

 match traffic-class 2

  bandwidth percent 30

  bandwidth percent 30

  bandwidth percent 30

random-detect ecn

  bandwidth percent 30

random-detect ecn

 end-class-map

  random-detect default

  random-detect discard-class 1 5 ms 30 ms 

  random-detect discard-class 1 5 ms 30 ms 

 !

  random-detect default

 !

!

 !

 !

random-detect ecn

 class P3

random-detect ecn

 class P3

class-map match-any P4

 class P4

 class P4

 !

  bandwidth percent 30

 !

  bandwidth percent 30

 match traffic-class 1

  bandwidth percent 5

  bandwidth percent 5

 class P4

  random-detect discard-class 1 5 ms 30 ms 

 class P4

  random-detect default

 end-class-map

  random-detect default

  random-detect discard-class 1 2 ms 12 ms 

  bandwidth percent 5

 !

  bandwidth percent 5

 !

!

 !

 !

  random-detect discard-class 1 2 ms 12 ms 

 class P4

  random-detect default

 class P4

 

 class class-default

 class class-default

random-detect ecn

  bandwidth percent 5

random-detect ecn

  bandwidth percent 5

 

bandwidth percent 40

  bandwidth percent 40

 !

  random-detect discard-class 1 2 ms 12 ms 

 !

  random-detect default

 

  random-detect default

  random-detect discard-class 1 10 ms 60 ms

 class class-default

 !

 class class-default

 !

 

!

 !

  bandwidth percent 40

 class class-default

bandwidth percent 40

 class class-default

 

 end-policy-map

 end-policy-map

  random-detect discard-class 1 10 ms 60 ms

  bandwidth percent 40

  random-detect default

bandwidth percent 40

 

!

!

random-detect ecn

  random-detect discard-class 1 10 ms 60 ms

random-detect ecn

  random-detect default

 

 

 

 !

 !

!

!

 

 

 

 end-policy-map

 end-policy-map

 end-policy-map

 end-policy-map

 

Testcases and Results

test #

Description

Input policy

Output policy

traffic

Congestion

1

only ingress policy

twt-IN-1

None

all

151%

2

output policy with default random-detect values

twt-IN-1

twt-OUT-1

all

151%

3

output policy optimized random detect values

twt-IN-1

twt-OUT-2

all

151%

4

output policy optimized random detect values

twt-IN-1

twt-OUT-2

only DC1

101%

5

output policy optimized random detect values + ECN on DC1

twt-IN-1

twt-OUT-3

all

151%

6

output policy optimized random detect values + ECN on DC1

twt-IN-1

twt-OUT-3

only DC1

101%

7

output policy optimized random detect values + ECN on DC0

twt-IN-1

twt-OUT-4

all

151%

8

output policy default random detect values + ECN on DC1

twt-IN-1

twt-OUT-5

all

151%

9

output policy default random detect values + ECN on DC1

twt-IN-1

twt-OUT-5

only DC1

101%

10

output policy default random detect values + ECN on DC0

twt-IN-1

twt-OUT-6

all

151%

 

 

Test cases: 5, 6, 7, 10: covers main functionality for ECN

 

class stream

Gbps (Tx)

tests cases results (Drops in %age)

 

 

TC1

TC2

TC3

TC4

TC5

TC6

TC7

TC8

TC9

TC10

DSCP_P0

2

15.8

81.6

38.6

-

45.2

-

42.1

81.6

81.6

45.2

DSCP_P1

1.5

15.8

85.7

67

-

70.7

-

69

85.7

85.7

70.7

DSCP_P2

1.5

16

81.6

49.6

-

49.7

-

22.9

81.6

81.5

49.6

DSCP_P3

3

19.7

65.3

22.5

20.1

0

0

22.7

65.3

64.8

0

DSCP_P4

1

15.9

85.7

94.5

91.7

89.8

88.9

94.5

85.7

85.7

89.7

DSCP_def

6

58.8

57.1

14

0

23.3

0

19.1

57.1

56.8

23.3

Total:

15G

                   

 

Understanding results

Traffic and input-policy is constant across all testcases but drops vary. Various config tweaks in output policy shows how drops can be manipulated for priority traffic.

TC1: This is basic test with no output policy, drops are seen across all traffic classes in general.

TC2: output policy has system-default random detect values. Clearly this is not optimal to minimize drops as user has not considered the traffic input rates into each class.

TC3: output policy with defined threshold limit under random detect. This shows better results for priority classes traffic streams, drops.

TC4: Traffic enabled for only discard-class 1 streams [still congested at 101%]. Output policy with defined threshold limit under random detect. This shows better results for priority classes traffic streams drops.

TC5: Compare the result with TC3 numbers. Output policy with defined threshold limit under random detect, enable ECN for discard-class 1 traffic classes [P3, P4 and default].  ECN kicks in, P3 class drops are zero and P4, and default drops have reduced.

TC6: Compare the results with TC4 numbers. Traffic enabled for only discard-class 1 streams [still congested at 101%]. output policy with defined threshold limit under random detect, enable ECN for discard-class 1 traffic classes [P3, P4 and default].  ECN kicks in, P3 and default class drops are zero and P4 drops have reduced.

TC7: Compare the result with TC3 numbers. Output policy with defined threshold limit under random detect, enable ECN for discard-class 0 traffic classes [P0, P1 and P2].  ECN kicks in, P2 class drops improved, P0 & P1 went up. This indicates the wred needs some re-tuning wrt threshold values for traffic coming into these class streams.

TC8 & TC9: Compare with TC2. With system default random detect threshold values, ECN doesn’t help much irrespective of the DC on which the config is applied. This shows that wred tuning is still required.

TC10:   Compare with TC3. Output policy with defined threshold limit under random detect, enable ECN for discard-class 0 and 1 [i.e. all streams used in the test].  ECN helps for DC-1 streams P3, P4, default and for DC-0 streams the drops go slightly up. User needs to fine tune the random detect threshold as per operational need on priority streams.

 

Test logs

QOS-ECN feature is introduced on NCS5500 platform starting release ios-xr 6.6.1, however there was no easy CLI to verify the ECN status on the router except the below logs. New CLI’s are introduced in release 6.6.2 [mentioned later in this document].

TC5: Configure input policy twt-IN-1 and output policy twt-OUT-3, ECN enabled for discard-class 1 traffic class [P3, P4 and class-default]

 

  1. Verify discard drop are happening on LC at fia [LC fabricq ingress ASIC]

RP/0/RP0/CPU0:xrg-301-ncs5508#sh controllers fia statistics instance all location 0/0/CPU0 | i Dscrd | ex :                0                                                                                 

  IQM1 EnqueueDscrdPktCnt              :     508165034922                                                         

  IQM1 QueueEnqDscrdByteCnt            :  775968008319786                                                                               

  IQM1 QueueEnqDscrdPktCnt             :     508165034920                                                                               

  IQM1 TotDscrdByteCnt                 :   71851069816600                                                                               

RP/0/RP0/CPU0:xrg-301-ncs5508#sh controllers fia statistics instance all location 0/0/CPU0 | i Dscrd | ex :                0                                                                                 

  IQM1 EnqueueDscrdPktCnt              :     508189826863                                                                               

  IQM1 QueueEnqDscrdByteCnt            :  776005865810676                                                                               

  IQM1 QueueEnqDscrdPktCnt             :     508189827001                                                                               

  IQM1 TotDscrdByteCnt                 :   71854567701971                                                                               

RP/0/RP0/CPU0:xrg-301-ncs5508#                                                                                    

  1. Verify ECN is enabled [checking for one traffic-class]                                    

[Note: voq ID used below may change depending on what queue ID system assigns for each traffic class. Use “sh policy-map interface tenGigE 0/0/0/1/0 output” to find the corresponding queue ID(VOQ ID) for the traffic-class]                                                                                                    

RP/0/RP0/CPU0:xrg-301-ncs5508#show controller fia diagshell 0 "diag cosq voq id=1576 detail=1 core=1" loc 0/0/CPU0 | be WRED       --à voq id 1576 corresponds to class-default                          

        WRED info:                                                                                   Green : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                        

       Yellow: enable(True) min_thresh(5013504) max_thresh(30015488) drop_probability(9)                                                                    -> Yellow for DC 1            

       Red   : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                 

       Black : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                 

       Guaranteed info: guaranteed queue size in bytes(0) guaranteed queue size in BDs(0)                                                                                

       Tail drop info:                                                                                     

       Green : max queue size in bytes(60030976), max queue size in BDs(234496)                                                                                  

       Yellow: max queue size in bytes(60030976), max queue size in BDs(234496)                                                                                  

       Red   : max queue size in bytes(60030976), max queue size in BDs(234496)                                                                                  

       Black : max queue size in bytes(60030976), max queue size in BDs(234496)                                              

        Fair adaptive tail drop info:                                                                                     

                Enable: True                                                                               

                Green : alpha(-2)                                                                                 

                Yellow: alpha(-2)                                                                                 

                Red   : alpha(-2)                                                                                 

                Black : alpha(-2)                                                                                 

        ECN max queue size in bytes(5013504), max queue size in BDs(524288)                                                                                  

        ECN wred info: enable(True) min_thresh(0) max_thresh(30015488) drop_probability(11)                                                                          

RP/0/RP0/CPU0:xrg-301-ncs5508#                                                               

             

  1. Steps to verify drops reduced [using router CLI’s]
  • With ECN enabled for just one class - P3 on router, disable ECN bit from traffic generator for this stream [spirent]
  • Make a note of drops

RP/0/RP0/CPU0:xrg-301-ncs5508#sh policy-map interface tenGigE 0/0/0/1/0 output | b P3                                                                                

Class P3                                                                                     

  Classification statistics          (packets/bytes)     (rate - kbps)                                                                                      

   Matched             :              100000/150000000            0                                                                                      

   Transmitted         :               83134/124701000            0                                                                                      

   Total Dropped       :               16866/25299000             0 >>>>>> 16866 pkts drop                                              

  Queueing statistics                                                                               

    Queue ID                             : 1578                                                                                  

    Taildropped(packets/bytes)           : 16866/25299000                                                                               

  • Enable ECN bit on Spirent now for the traffic stream, then check the counters:

RP/0/RP0/CPU0:xrg-301-ncs5508#clear qos counters interface all

  • Send traffic                                                          

RP/0/RP0/CPU0:xrg-301-ncs5508#sh policy-map interface tenGigE 0/0/0/1/0 output | b P3                                                                                

Class P3                                                                                     

  Classification statistics          (packets/bytes)     (rate - kbps)                                                                                      

    Matched             :              100000/150000000            0                                                                                      

    Transmitted         :               90407/135610500            0                                                                                      

    Total Dropped       :                9593/14389500             0 >>>>>> 9593 pkts drop                                                                                   

  Queueing statistics                                                                               

    Queue ID                             : 1578                                                                                  

    Taildropped(packets/bytes)           : 9593/14389500                                                                         

Conclusion: Drops are reduced for ECN marked packets in this case. ECN will mark packets instead of dropping them when the average queue length exceeds a specific threshold value. Some packets are still dropped as the packets in queue are more than the max threshold.                         

                                                                                      

DC 0 behavior: [color code for discard class 0 – green]          

                                                                                      

RP/0/RP0/CPU0:xrg-301-ncs5508#show controller fia diagshell 0 "diag cosq voq id=1581 detail=1 core=1" loc 0/0/CPU0 | be WRED                                                                                

        WRED info:                                                                                  

                Green : enable(True) min_thresh(2506752) max_thresh(15007744) drop_probability(9)                                                                           >>>>>Green    

                Yellow: enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                 

                Red   : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                 

                Black : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                 

        Guaranteed info: guaranteed queue size in bytes(0) guaranteed queue size in BDs(0)                                                                                

        Tail drop info:                                                                                    

                Green : max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                 

                Yellow: max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                 

                Red   : max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                 

                Black : max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                 

        Fair adaptive tail drop info:                                                                                     

                Enable: True                                                                               

                Green : alpha(-2)                                                                                 

                Yellow: alpha(-2)                                                                                 

                Red   : alpha(-2)                                                                                 

                Black : alpha(-2)                                                                                 

        ECN max queue size in bytes(2506752), max queue size in BDs(524288)                                                                                  

        ECN wred info: enable(True) min_thresh(0) max_thresh(15007744) drop_probability(11)                                                                                              

RP/0/RP0/CPU0:xrg-301-ncs5508#                                                                                    

Note: color code for discard class 0 – yellow

 

More CLI’s

Starting IOS-XR 6.6.2 release, easier CLI’s are available

Sample logs below [not from original tests]

RP/0/RP0/CPU0:head#show qos interface tenGigE 0/0/0/6 output

NOTE:- Configured values are displayed within parentheses

Interface TenGigE0/0/0/6 ifh 0x1c8  -- output policy

NPU Id:                        0

Total number of classes:       2

Interface Bandwidth:           10000000 kbps

Policy Name:                   pm-out-queue

VOQ Base:                      1112

Accounting Type:               Layer1 (Include Layer 1 encapsulation and above)

------------------------------------------------------------------------------

Level1 Class                             =   cm-tc-1

Egressq Queue ID                         =   1113 (LP queue)

Queue Max. BW.                           =   1001956 kbps (10 %)

Queue Min. BW.                           =   0 kbps (default)

Inverse Weight / Weight                  =   1 / (BWR not configured)

Guaranteed service rate                  =   1000000 kbps

TailDrop Threshold                       =   5013504 bytes / 40 ms (default)

 

Default RED profile

WRED Min. Threshold                      =   1253376 bytes (10 ms)

WRED Max. Threshold                      =   2506752 bytes (20 ms)

 

WRED ECN                                 =   Enabled

 

Level1 Class                             =   class-default

Egressq Queue ID                         =   1112 (Default LP queue)

Queue Max. BW.                           =   no max (default)

Queue Min. BW.                           =   0 kbps (default)

Inverse Weight / Weight                  =   1 / (BWR not configured)

Guaranteed service rate                  =   5000000 kbps

TailDrop Threshold                       =   6258688 bytes / 10 ms (default)

WRED not configured for this class

RP/0/RP0/CPU0:head#

RP/0/RP0/CPU0:head#show policy-map interface tenGigE 0/0/0/6 output

TenGigE0/0/0/6 output: pm-out-queue

Class cm-tc-1

  Classification statistics          (packets/bytes)     (rate - kbps)

    Matched             :            85528554/87581239296          4830672

    Transmitted         :            16240891/16630672384          966585

    Total Dropped       :            69287663/70950566912          3864087

  Queueing statistics

    Queue ID                             : 1113

    Taildropped(packets/bytes)           : 69287663/70950566912

 

    WRED profile for

    RED Transmitted (packets/bytes)            : N/A

    RED random drops(packets/bytes)            : N/A

    RED maxthreshold drops(packets/bytes)      : N/A

    RED ecn marked & transmitted(packets/bytes): N/A    --> WRED counters are not supported on NCS5500, however this line entry confirms ECN is enabled.

Class class-default

  Classification statistics          (packets/bytes)     (rate - kbps)

    Matched             :                   0/0                    0

    Transmitted         :                   0/0                    0

    Total Dropped       :                   0/0                    0

  Queueing statistics

    Queue ID                             : 1112

    Taildropped(packets/bytes)           : 0/0

Policy Bag Stats time: 1551105209930  [Local Time: 02/25/19 14:33:29.930]

RP/0/RP0/CPU0:head#

 

 

Disclaimers [ECN on NCS5500]

Not supported:

ECN at ingress

Not supported:

WRED not supported in priority level, hence no ECN

Not supported:

WRED ECN stats not supported on NCS5500

Supported:

Two level policy at egress, child may have ECN

 

 

More references

Reviewers

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Quick Links