XR-vm - CLI's Description show context  look for any process crash, review time stamp[if it is too old, then no immediate action needed] show redundancy verify if standby state  is Ready and NSR-Ready show proc cpu | exclude " 0%" look for processes hogging CPU show memory summary location all gives a dump of overall system memory state with availability show hw-module fpd all HW level FPD programming state, look for 'NEED UPGD' ; if found please upgrade FPD for that card  show configuration commit list 5 view last 5 commits  show version | in uptime verify system up time  show media verify filesystem space  show watchdog memory-state location all    detailed memory status show rcc ipv4 unicast all route consistency check  [unicast only] show rcc ipv6 unicast all RCC for v6  show lcc ipv4 unicast all Label Consistency Check  show lcc ipv6 unicast all LCC for v6 show cef resource location <Line card> | ex GREEN  LC forwarding resources [if GREEN, we are good, any other color then the resources are getting over-utilized]  Show CEF tables summary location <Line card> show LC forwarding CEF tables for the LC show cef resource hardware ingress location 0/0/CPU0  | ex GREEN LC forwarding at hardware level - ingress  how cef resource hardware egress location 0/0/CPU0  | ex GREEN LC forwarding at hardware level - egress show health gsp    inter-process communication - GSP health  show health cfgmgr configuration manager health show alarms brief system active review output for alarm description requiring immediate action show health sysdb system database [to store processes db] SYSDB health show asic-errors all summary location  look for any fast increment ASIC error counters  show processes blocked location all | ex lpts look for any processes in Mutex/reply blocked state for extended period of time Other Useful CLI's   monitor interface * OR <interface name> to review interface traffic incoming/outgoing - real time config t - lpts pifib hardware police location 0/0/CPU0 flow ssh known rate <>  to configure LPTS policer rate for incoming ssh sessions through LC interface  admin config - fpd auto-upgrade enable  to enable auto-upgrade functionality for FPD versions during SW image upgrade show install active summary  to view list of active packages  show install inactive summary  to view list of inactive packages  install remove inactive  removed all inactive packages [ helps to clear disk space] install commit to checkpoint current software version and packages installed  show install request  to verify any ongoing install activity progress to move between different vm's [e.g. host to sysadmin to xr-vm] Ctrl+o , press again to go to next VM in sequence Admin-vm CLI's   show controller fabric plane all detail fabric plane details -verify if all planes are UP UP show controllers fabric health overall fabric health show platform cards list  show context  look for any process crash show install committed summary show all packages installed and committed, verify if it matches with show install active summary show environment temperatures shows temperature of all hw components [cards] show environment fan fan status - speed, etc show environment power power status show inventory all all cards inventory [ you can find card serial numbers here] show vm verify if all VM's are in 'running' state  ... View more

On IOS-XR, Quality of Service has an extension to WRED (Weighted Random Early Detection) called Explicit Congestion Notification (ECN). ECN will mark packets instead of dropping them when the average queue length exceeds a specific threshold value. When configured, ECN helps routers and end hosts to understand that the network is congested and slow down sending packets. However, If the number of packets in the queue is above the maximum threshold, packets are dropped based on the drop probability. This is the identical treatment a packet receives when WRED is enabled without ECN configured on the router. This feature was enabled on IOS-XR 6.6.2 release for NCS-5500 series routers. Scope: This document explains with some test results about ECN’s ability to manage excessive drops in case of congestion. Note: For optimal profile, network operator needs to review various factors such as Bandwidth percentage, WRED values, etc. With manageable congestion you can achieve ‘zero’ loss for important traffic streams. Below results indicate that with ECN, we can reduce loss in imp streams for sure. Congestion profile 15.1G traffic on 100G ingress and 10G at egress on NCS-5508 router. Topology Ttraffic flow is from TGN port 7/1 to 3/1] TGN1(3/1)<-->Hu0/1/0/1(NCS6K-2T)TenGigE0/7/0/0<-->TenGigE0/0/0/1/0(NCS5508)Hu0/0/0/0 <--->TGN(7/1) 100.3.2.1         100.3.2.2                     100.3.3.1                  100.3.3.2                                 100.3.4.2                     100.3.4.1 Traffic distribution discard-class class-map traffic-class dscp TOS(Dec) stream traffic load frame size Scheduling BW %age 0 DSCP_P0 5 ef 184   2 1500 continuos+ burst size 10000 packets 10 0 DSCP_P1 4 cs6 cs7 192 224 cs6 1.5 1500 5 0 DSCP_P2 3 cs3 96   1.5 1500 10 1 DSCP_P3 2 af21 af22 af23 88 af23 3 1500 30 1 DSCP_P4 1 af11 40   1 1500 5 1 DSCP_P5 0 default 0   6 1500 40   Total         15G     100   Configs used At Ingress: (NCS5508)Hu0/0/0/0   Input: class-map   Input: policy-map     policy-map twt-IN-1 policy-map twt-IN-2 class-map match-any DSCP_P0    class DSCP_P0  class DSCP_P0  match dscp ef     set traffic-class 5   set traffic-class 5  end-class-map     set discard-class 0  ! !    !  class DSCP_P1 class-map match-any DSCP_P1    class DSCP_P1   set traffic-class 4  match dscp cs6 cs7     set traffic-class 4  !  end-class-map     set discard-class 0  class DSCP_P2 !    !   set traffic-class 3 class-map match-any DSCP_P2    class DSCP_P2  !  match dscp cs3     set traffic-class 3  class DSCP_P3  end-class-map     set discard-class 0   set traffic-class 2 !    !  ! class-map match-any DSCP_P3    class DSCP_P3  class DSCP_P4  match dscp af21 af22 af23     set traffic-class 2   set traffic-class 1  end-class-map     set discard-class 1  ! !    !  class class-default class-map match-any DSCP_P4    class DSCP_P4  !  match dscp af11     set traffic-class 1  end-policy-map  end-class-map     set discard-class 1 ! !    !        class class-default        !        end-policy-map       !     At Egress: TenGigE0/0/0/1/0(NCS5508) Output:Class-map Output: Policy-map   policy-map twt-OUT-1 policy-map twt-OUT-2 policy-map twt-OUT-3 policy-map twt-OUT-4 policy-map twt-OUT-5 policy-map twt-OUT-6 class-map match-any P0  class P0  class P0  class P0  class P0  class P0  class P0  match traffic-class 5   bandwidth percent 10   bandwidth percent 10   bandwidth percent 10   bandwidth percent 10   bandwidth percent 10   bandwidth percent 10  end-class-map   random-detect default   random-detect discard-class 0 20 ms 120 ms     random-detect discard-class 0 20 ms 120 ms    random-detect discard-class 0 20 ms 120 ms    random-detect default   random-detect default !  !  !  ! random-detect ecn  ! random-detect ecn class-map match-any P1  class P1  class P1  class P1  !  class P1  !  match traffic-class 4   bandwidth percent 5   bandwidth percent 5   bandwidth percent 5  class P1   bandwidth percent 5  class P1  end-class-map   random-detect default   random-detect discard-class 0 20 ms 120 ms    random-detect discard-class 0 20 ms 120 ms    bandwidth percent 5   random-detect default   bandwidth percent 5 !  !  !  !   random-detect discard-class 0 20 ms 120 ms   !   random-detect default class-map match-any P2  class P2  class P2  class P2 random-detect ecn  class P2 random-detect ecn  match traffic-class 3   bandwidth percent 10   bandwidth percent 10   bandwidth percent 10  !   bandwidth percent 10  !  end-class-map   random-detect default   random-detect discard-class 0 10 ms 60 ms    random-detect discard-class 0 10 ms 60 ms   class P2   random-detect default  class P2 !  !  !  !   bandwidth percent 10  !   bandwidth percent 10 class-map match-any P3  class P3  class P3  class P3   random-detect discard-class 0 10 ms 60 ms   class P3   random-detect default  match traffic-class 2   bandwidth percent 30   bandwidth percent 30   bandwidth percent 30 random-detect ecn   bandwidth percent 30 random-detect ecn  end-class-map   random-detect default   random-detect discard-class 1 5 ms 30 ms    random-detect discard-class 1 5 ms 30 ms   !   random-detect default  ! !  !  ! random-detect ecn  class P3 random-detect ecn  class P3 class-map match-any P4  class P4  class P4  !   bandwidth percent 30  !   bandwidth percent 30  match traffic-class 1   bandwidth percent 5   bandwidth percent 5  class P4   random-detect discard-class 1 5 ms 30 ms   class P4   random-detect default  end-class-map   random-detect default   random-detect discard-class 1 2 ms 12 ms    bandwidth percent 5  !   bandwidth percent 5  ! !  !  !   random-detect discard-class 1 2 ms 12 ms   class P4   random-detect default  class P4    class class-default  class class-default random-detect ecn   bandwidth percent 5 random-detect ecn   bandwidth percent 5   bandwidth percent 40   bandwidth percent 40  !   random-detect discard-class 1 2 ms 12 ms   !   random-detect default     random-detect default   random-detect discard-class 1 10 ms 60 ms  class class-default  !  class class-default  !   !  !   bandwidth percent 40  class class-default bandwidth percent 40  class class-default    end-policy-map  end-policy-map   random-detect discard-class 1 10 ms 60 ms   bandwidth percent 40   random-detect default bandwidth percent 40   ! ! random-detect ecn   random-detect discard-class 1 10 ms 60 ms random-detect ecn   random-detect default        !  ! ! !        end-policy-map  end-policy-map  end-policy-map  end-policy-map   Testcases and Results test # Description Input policy Output policy traffic Congestion 1 only ingress policy twt-IN-1 None all 151% 2 output policy with default random-detect values twt-IN-1 twt-OUT-1 all 151% 3 output policy optimized random detect values twt-IN-1 twt-OUT-2 all 151% 4 output policy optimized random detect values twt-IN-1 twt-OUT-2 only DC1 101% 5 output policy optimized random detect values + ECN on DC1 twt-IN-1 twt-OUT-3 all 151% 6 output policy optimized random detect values + ECN on DC1 twt-IN-1 twt-OUT-3 only DC1 101% 7 output policy optimized random detect values + ECN on DC0 twt-IN-1 twt-OUT-4 all 151% 8 output policy default random detect values + ECN on DC1 twt-IN-1 twt-OUT-5 all 151% 9 output policy default random detect values + ECN on DC1 twt-IN-1 twt-OUT-5 only DC1 101% 10 output policy default random detect values + ECN on DC0 twt-IN-1 twt-OUT-6 all 151%     Test cases: 5, 6, 7, 10: covers main functionality for ECN   class stream Gbps (Tx) tests cases results (Drops in %age)     TC1 TC2 TC3 TC4 TC5 TC6 TC7 TC8 TC9 TC10 DSCP_P0 2 15.8 81.6 38.6 - 45.2 - 42.1 81.6 81.6 45.2 DSCP_P1 1.5 15.8 85.7 67 - 70.7 - 69 85.7 85.7 70.7 DSCP_P2 1.5 16 81.6 49.6 - 49.7 - 22.9 81.6 81.5 49.6 DSCP_P3 3 19.7 65.3 22.5 20.1 0 0 22.7 65.3 64.8 0 DSCP_P4 1 15.9 85.7 94.5 91.7 89.8 88.9 94.5 85.7 85.7 89.7 DSCP_def 6 58.8 57.1 14 0 23.3 0 19.1 57.1 56.8 23.3 Total: 15G                       Understanding results Traffic and input-policy is constant across all testcases but drops vary. Various config tweaks in output policy shows how drops can be manipulated for priority traffic. TC1: This is basic test with no output policy, drops are seen across all traffic classes in general. TC2: output policy has system-default random detect values. Clearly this is not optimal to minimize drops as user has not considered the traffic input rates into each class. TC3: output policy with defined threshold limit under random detect. This shows better results for priority classes traffic streams, drops. TC4: Traffic enabled for only discard-class 1 streams [still congested at 101%]. Output policy with defined threshold limit under random detect. This shows better results for priority classes traffic streams drops. TC5: Compare the result with TC3 numbers. Output policy with defined threshold limit under random detect, enable ECN for discard-class 1 traffic classes [P3, P4 and default].  ECN kicks in, P3 class drops are zero and P4, and default drops have reduced. TC6: Compare the results with TC4 numbers. Traffic enabled for only discard-class 1 streams [still congested at 101%]. output policy with defined threshold limit under random detect, enable ECN for discard-class 1 traffic classes [P3, P4 and default].  ECN kicks in, P3 and default class drops are zero and P4 drops have reduced. TC7: Compare the result with TC3 numbers. Output policy with defined threshold limit under random detect, enable ECN for discard-class 0 traffic classes [P0, P1 and P2].  ECN kicks in, P2 class drops improved, P0 & P1 went up. This indicates the wred needs some re-tuning wrt threshold values for traffic coming into these class streams. TC8 & TC9: Compare with TC2. With system default random detect threshold values, ECN doesn’t help much irrespective of the DC on which the config is applied. This shows that wred tuning is still required. TC10:   Compare with TC3. Output policy with defined threshold limit under random detect, enable ECN for discard-class 0 and 1 [i.e. all streams used in the test].  ECN helps for DC-1 streams P3, P4, default and for DC-0 streams the drops go slightly up. User needs to fine tune the random detect threshold as per operational need on priority streams.   Test logs QOS-ECN feature is introduced on NCS5500 platform starting release ios-xr 6.6.1, however there was no easy CLI to verify the ECN status on the router except the below logs. New CLI’s are introduced in release 6.6.2 [mentioned later in this document]. TC5: Configure input policy twt-IN-1 and output policy twt-OUT-3, ECN enabled for discard-class 1 traffic class [P3, P4 and class-default]   Verify discard drop are happening on LC at fia [LC fabricq ingress ASIC] RP/0/RP0/CPU0:xrg-301-ncs5508#sh controllers fia statistics instance all location 0/0/CPU0 | i Dscrd | ex :                0                                                                                    IQM1 EnqueueDscrdPktCnt              :     508165034922                                                            IQM1 QueueEnqDscrdByteCnt            :  775968008319786                                                                                  IQM1 QueueEnqDscrdPktCnt             :     508165034920                                                                                  IQM1 TotDscrdByteCnt                 :   71851069816600                                                                                RP/0/RP0/CPU0:xrg-301-ncs5508#sh controllers fia statistics instance all location 0/0/CPU0 | i Dscrd | ex :                0                                                                                    IQM1 EnqueueDscrdPktCnt              :     508189826863                                                                                  IQM1 QueueEnqDscrdByteCnt            :  776005865810676                                                                                  IQM1 QueueEnqDscrdPktCnt             :     508189827001                                                                                  IQM1 TotDscrdByteCnt                 :   71854567701971                                                                                RP/0/RP0/CPU0:xrg-301-ncs5508#                                                                                     Verify ECN is enabled [checking for one traffic-class]                                     [Note: voq ID used below may change depending on what queue ID system assigns for each traffic class. Use “sh policy-map interface tenGigE 0/0/0/1/0 output” to find the corresponding queue ID(VOQ ID) for the traffic-class]                                                                                                     RP/0/RP0/CPU0:xrg-301-ncs5508#show controller fia diagshell 0 "diag cosq voq id=1576 detail=1 core=1" loc 0/0/CPU0 | be WRED       --à voq id 1576 corresponds to class-default                                   WRED info:                                                                                   Green : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                Yellow: enable(True) min_thresh(5013504) max_thresh(30015488) drop_probability(9)                                                                    -> Yellow for DC 1                    Red   : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                         Black : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                         Guaranteed info: guaranteed queue size in bytes(0) guaranteed queue size in BDs(0)                                                                                        Tail drop info:                                                                                             Green : max queue size in bytes(60030976), max queue size in BDs(234496)                                                                                          Yellow: max queue size in bytes(60030976), max queue size in BDs(234496)                                                                                          Red   : max queue size in bytes(60030976), max queue size in BDs(234496)                                                                                          Black : max queue size in bytes(60030976), max queue size in BDs(234496)                                                       Fair adaptive tail drop info:                                                                                                      Enable: True                                                                                                Green : alpha(-2)                                                                                                  Yellow: alpha(-2)                                                                                                  Red   : alpha(-2)                                                                                                  Black : alpha(-2)                                                                                          ECN max queue size in bytes(5013504), max queue size in BDs(524288)                                                                                           ECN wred info: enable(True) min_thresh(0) max_thresh(30015488) drop_probability(11)                                                                           RP/0/RP0/CPU0:xrg-301-ncs5508#                                                                              Steps to verify drops reduced [using router CLI’s] With ECN enabled for just one class - P3 on router, disable ECN bit from traffic generator for this stream [spirent] Make a note of drops RP/0/RP0/CPU0:xrg-301-ncs5508#sh policy-map interface tenGigE 0/0/0/1/0 output | b P3                                                                                 Class P3                                                                                        Classification statistics          (packets/bytes)     (rate - kbps)                                                                                          Matched             :              100000/150000000            0                                                                                          Transmitted         :               83134/124701000            0                                                                                          Total Dropped       :               16866/25299000             0 >>>>>> 16866 pkts drop                                                 Queueing statistics                                                                                    Queue ID                             : 1578                                                                                       Taildropped(packets/bytes)           : 16866/25299000                                                                                Enable ECN bit on Spirent now for the traffic stream, then check the counters: RP/0/RP0/CPU0:xrg-301-ncs5508#clear qos counters interface all Send traffic                                                           RP/0/RP0/CPU0:xrg-301-ncs5508#sh policy-map interface tenGigE 0/0/0/1/0 output | b P3                                                                                 Class P3                                                                                        Classification statistics          (packets/bytes)     (rate - kbps)                                                                                           Matched             :              100000/150000000            0                                                                                           Transmitted         :               90407/135610500            0                                                                                           Total Dropped       :                9593/14389500             0 >>>>>> 9593 pkts drop                                                                                      Queueing statistics                                                                                    Queue ID                             : 1578                                                                                       Taildropped(packets/bytes)           : 9593/14389500                                                                          Conclusion: Drops are reduced for ECN marked packets in this case. ECN will mark packets instead of dropping them when the average queue length exceeds a specific threshold value. Some packets are still dropped as the packets in queue are more than the max threshold.                                                                                                                 DC 0 behavior: [color code for discard class 0 – green]                                                                                                  RP/0/RP0/CPU0:xrg-301-ncs5508#show controller fia diagshell 0 "diag cosq voq id=1581 detail=1 core=1" loc 0/0/CPU0 | be WRED                                                                                         WRED info:                                                                                                   Green : enable(True) min_thresh(2506752) max_thresh(15007744) drop_probability(9)                                                                           >>>>>Green                     Yellow: enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                                  Red   : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                                  Black : enable(False) min_thresh(0) max_thresh(0) drop_probability(0)                                                                                          Guaranteed info: guaranteed queue size in bytes(0) guaranteed queue size in BDs(0)                                                                                         Tail drop info:                                                                                                     Green : max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                                  Yellow: max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                                  Red   : max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                                  Black : max queue size in bytes(30015488), max queue size in BDs(117248)                                                                                          Fair adaptive tail drop info:                                                                                                      Enable: True                                                                                                Green : alpha(-2)                                                                                                  Yellow: alpha(-2)                                                                                                  Red   : alpha(-2)                                                                                                  Black : alpha(-2)                                                                                          ECN max queue size in bytes(2506752), max queue size in BDs(524288)                                                                                           ECN wred info: enable(True) min_thresh(0) max_thresh(15007744) drop_probability(11)                                                                                               RP/0/RP0/CPU0:xrg-301-ncs5508#                                                                                     Note: color code for discard class 0 – yellow   More CLI’s Starting IOS-XR 6.6.2 release, easier CLI’s are available Sample logs below [not from original tests] RP/0/RP0/CPU0:head#show qos interface tenGigE 0/0/0/6 output NOTE:- Configured values are displayed within parentheses Interface TenGigE0/0/0/6 ifh 0x1c8  -- output policy NPU Id:                        0 Total number of classes:       2 Interface Bandwidth:           10000000 kbps Policy Name:                   pm-out-queue VOQ Base:                      1112 Accounting Type:               Layer1 (Include Layer 1 encapsulation and above) ------------------------------------------------------------------------------ Level1 Class                             =   cm-tc-1 Egressq Queue ID                         =   1113 (LP queue) Queue Max. BW.                           =   1001956 kbps (10 %) Queue Min. BW.                           =   0 kbps (default) Inverse Weight / Weight                  =   1 / (BWR not configured) Guaranteed service rate                  =   1000000 kbps TailDrop Threshold                       =   5013504 bytes / 40 ms (default)   Default RED profile WRED Min. Threshold                      =   1253376 bytes (10 ms) WRED Max. Threshold                      =   2506752 bytes (20 ms)   WRED ECN                                 =   Enabled   Level1 Class                             =   class-default Egressq Queue ID                         =   1112 (Default LP queue) Queue Max. BW.                           =   no max (default) Queue Min. BW.                           =   0 kbps (default) Inverse Weight / Weight                  =   1 / (BWR not configured) Guaranteed service rate                  =   5000000 kbps TailDrop Threshold                       =   6258688 bytes / 10 ms (default) WRED not configured for this class RP/0/RP0/CPU0:head# RP/0/RP0/CPU0:head#show policy-map interface tenGigE 0/0/0/6 output TenGigE0/0/0/6 output: pm-out-queue Class cm-tc-1   Classification statistics          (packets/bytes)     (rate - kbps)     Matched             :            85528554/87581239296          4830672     Transmitted         :            16240891/16630672384          966585     Total Dropped       :            69287663/70950566912          3864087   Queueing statistics     Queue ID                             : 1113     Taildropped(packets/bytes)           : 69287663/70950566912       WRED profile for     RED Transmitted (packets/bytes)            : N/A     RED random drops(packets/bytes)            : N/A     RED maxthreshold drops(packets/bytes)      : N/A     RED ecn marked & transmitted(packets/bytes): N/A    --> WRED counters are not supported on NCS5500, however this line entry confirms ECN is enabled. Class class-default   Classification statistics          (packets/bytes)     (rate - kbps)     Matched             :                   0/0                    0     Transmitted         :                   0/0                    0     Total Dropped       :                   0/0                    0   Queueing statistics     Queue ID                             : 1112     Taildropped(packets/bytes)           : 0/0 Policy Bag Stats time: 1551105209930  [Local Time: 02/25/19 14:33:29.930] RP/0/RP0/CPU0:head#     Disclaimers [ECN on NCS5500] Not supported: ECN at ingress Not supported: WRED not supported in priority level, hence no ECN Not supported: WRED ECN stats not supported on NCS5500 Supported: Two level policy at egress, child may have ECN     More references Modular QoS Configuration Guide for Cisco NCS 5500 Series Routers, IOS XR Release 6.0.x - https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5500/qos/60x/b-ncs5500-qos-configuration-guide-60x/b-qos-cg-ncs5500_chapter_00.html Understanding QOS, default marking behavior and troubleshooting –https://community.cisco.com/t5/service-providers-documents/asr9000-xr-understanding-qos-default-marking-behavior-and/ta-p/3128709 Reviewers Xander Thuijs xthuijs@cisco.com [DISTINGUISHED ENGINEER] ... View more