Introduction   There are a couple different variants of telnet/console hanging. This is mainly when using telnet to connect to a serial port over network. In some cases the XR telnet is hung, and other cases it is the internal Serial to XR Console session.     There can be nested telnets: Server                  Serial                  XR Console                      XR telnet destination                            <— internal telnet-->     <—------------------ telnet to XR Console—————>                                                 <—— XR telnet to XR connected destination —> Escape Sequence List   The following are the escape sequences for the various telnets: 1) XR telnet escape:   ^^q = CTRL-SHIFT-6 followed by q 2) Serial to XR console escape:   ^q  = CTRL-SHIFT-Q  (CTRL-q seems to work also) 3) Server telnet to Serial escape   ^]   = CTRL-] Details   1) Kill the XR Telnet first if it exists. - if you login to XR main console, if pressing keys cause garbage characters to appear, then you are probably still connected via the XR Telnet process OR,  -if you can get to an XR aux port, you can see a telnet process running for the connection you made: bash-4.3# ps -ef | grep telnet  root 21303 20957 0 19:36 pts/2 00:00:02 telnet 5.11.15.251 0x1 -m 0x0 -p ^^q   -> you may or may not get telnet prompt, this may exit immediately quit <Enter> <Enter> ctrl+]   -> exit Server to XR session telnet back to Serial and go to (2) 2) Kill the Serial to XR Console  - After killing the XR telnet process in (1), or if pressing [Enter] repeatedly shows nothing on the the screen ctrl+Q   -> upper case Q, though lower case seems to work sometimes quit [Enter] [Enter] 3) Kill Server to Serial Telnet ctrl+] quit [Enter] Telnet back for prompt.  Hit [Enter] multiple times NOTE:  in some cases, even after you get XR Console back, double characters or garbage character appear.  Ctrl+], quit, [Enter] to exit Server to Serial Telnet, and then telnet back NOTE: sometimes the telnet prompt does not appear when hitting escape sequence.  Type quit, [Enter] anyway. NOTE: The above can be used on Admin Console also NOTE: Sometimes you need to exit the Server to XR Console session (^], quit) then telnet back in to clear the next level. ... View more

This document explains the basic troubleshooting commands that should be run to verify the health of the XRv9000 router. Majority of these commands are implemented within the generic show tech-support command for ease of troubleshooting. Please refer to the command outputs to verify the specific expected output for a virtual routing platform. 1. show version +++++++++++++++ show version [12:39:55.884 ECT Wed Sep 14 2016] +++++++++++++++ Cisco IOS XR Software, Version 6.1.2.04I Copyright (c) 2013-2016 by Cisco Systems, Inc. Build Information:  Built By     : deenayak  Built On     : Sat Sep  3 09:58:27 PDT 2016  Build Host   : iox-lnx-073  Workspace    : /auto/iox-lnx-073-san2/production/6.1.2.04I.DT_IMAGE/xrv9k/workspace  Version      : 6.1.2.04I <--- Image version  Location     : /opt/cisco/XR/packages/ cisco IOS-XRv 9000 () processor <------ Platform type System uptime is 15 hours, 43 minutes   2. show install active summary +++++++ show install active summary [12:39:56.324 ECT Wed Sep 14 2016] ++++++++ Node 0/RP0/CPU0 [RP]   Boot Partition: xr_lv0   Active Packages: 1         xrv9k-xr-6.1.2.04I version=6.1.2.04I [Boot image] 3. Show sdr +++++++++++++++++ show sdr [12:39:56.740 ECT Wed Sep 14 2016] +++++++++++++++++ Type                  NodeName       NodeState      RedState       PartnerName -------------------------------------------------------------------------------- RP                    0/RP0/CPU0     IOS XR RUN     ACTIVE         NONE           R-IOSXRV9000-RP       0/RP0          OPERATIONAL                   N/A            4. admin show platform +++++++++++ admin show platform [12:39:57.109 ECT Wed Sep 14 2016] ++++++++++++ Node name       Node type       Partner name    SW status       IP address --------------- --------------- --------------- --------------- --------------- 0/RP0/CPU0      RP (ACTIVE)     NONE            FINAL Band      192.0.0.4 5. show redundancy +++++++++++++ show redundancy [12:39:57.508 ECT Wed Sep 14 2016] ++++++++++++++ Redundancy information for node 0/RP0/CPU0: ========================================== Node 0/RP0/CPU0 is in ACTIVE role Node 0/RP0/CPU0 has no valid partner Details -------- Current active rmf state:      NSR not ready since Standby is not Present Reload and boot info ---------------------- IN reloaded Tue Sep 13 20:56:48 2016: 15 hours, 43 minutes ago Active node booted Tue Sep 13 20:56:48 2016: 15 hours, 43 minutes ago Active node reload "User initiated forced reload" 6. show logging <Capture Syslog data> 7. Routing info <collect IGP and BGP specific info, e.g. show isis neighbors, show ospf summary, show ospf neighbor> 8. show placement ++++++++++++++ show placement [12:40:01.923 ECT Wed Sep 14 2016] ++++++++++++++ Locations: 0/RP0/CPU0 Role: REDUNDANCY_ACTIVE    Primary: yes    Paired: no -------------------------------------------------------------------------------------------- Programs running on Node 0/RP0/CPU0 ------------ ---------------------------------------------------------------------More--(94%) sdr_instmgr snmpd ipv4_static ospfv3 isis_uv lldp_mgr lldp_agent bpm ospfv3_uv lspv_server ospf_uv   9. show dsc all 10. show ipv4 traffic IP statistics:   Rcvd:  3516705 total, 973277 local destination          0 format errors, 0 bad hop count          0 unknown protocol, 0 not a gateway          0 security failures, 0 bad source, 0 bad header          0 with options, 0 bad, 0 unknown   Opts:  0 end, 0 nop, 0 basic security, 0 extended security          0 strict source rt, 0 loose source rt, 0 record rt          0 stream ID, 0 timestamp, 0 alert, 0 cipso   Frags: 1138 reassembled, 92 timeouts, 0 couldn't reassemble, 10279 fragments received          24791 fragmented, 0 fragment count, 0 fragment max drop   Bcast: 0 sent, 0 received   Mcast: 3789700 sent, 2512381 received    Lisp: 0 encapped in v4, 0 decapped from v4          0 encapped in v6, 0 decapped from v6          0 encap errors, 0 decap errors    Drop: 0 encapsulation failed, 0 no route, 0 too big    Sent: 4726972 total ICMP statistics:   Sent: 0 admin unreachable, 0 network unreachable         0 host unreachable, 0 protocol unreachable         0 port unreachable, 0 fragment unreachable         2 time to live exceeded, 0 reassembly ttl exceeded         2 echo request, 0 echo reply         0 mask request, 0 mask reply         0 parameter error, 0 redirects         4 total   Rcvd: 0 admin unreachable, 0 network unreachable         0 host unreachable, 0 protocol unreachable         19 port unreachable, 0 fragment unreachable         0 time to live exceeded, 0 reassembly ttl exceeded         0 echo request, 2 echo reply         0 mask request, 0 mask reply         0 redirect, 0 parameter error         0 source quench, 0 timestamp, 0 timestamp reply         0 router advertisement, 0 router solicitation         21 total, 0 checksum errors, 0 unknown UDP statistics:         1213323 packets input, 2290536 packets output         0 checksum errors, 23317 no port         0 forwarded broadcasts TCP statistics:         64153 packets input, 62974 packets output         0 checksum errors, 65 no port   11. show ipv6 traffic 12.  show route vrf all afi-all summary 13. show ipv4 vrf  all int brief 14. show int desc 15. sh cdp nei +++++++++++++++ show cdp nei [12:40:14.322 ECT Wed Sep 14 2016] +++++++++++++++ Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge                   S - Switch, H - Host, I - IGMP, r - Repeater Device ID       Local Intrfce    Holdtme Capability Platform  Port ID CRS-ABR1-R2     Te0/0/0/0        133     R          CRS       Te0/0/0/4        CRS-ABR1-R2     Te0/0/0/2        137     R          CRS       Te0/0/0/2        CRS-ABR1-R2     Te0/0/0/3        141     R          CRS       Te0/0/0/3        ssCE1-R9.cisco. Gi0/0/0/7        133     R          IOS-XRv 9 Gi0/0/0/0        ssPE4-R8.cisco. Gi0/0/0/6        134     R          IOS-XRv 9 Gi0/0/0/0        16. show route summary ++++++++++++ show route summary [12:40:16.576 ECT Wed Sep 14 2016] ++++++++++++ Route Source                     Routes     Backup     Deleted     Memory(bytes) connected                        140        4          0           34560         local                            144        0          0           34560         local LSPV                       1          0          0           240           static                           3          0          0           720           bgp 100                          2014       0          0           483584        ospf area1                       14028      0          0           20803216      isis 1                           1010       14         0           362240        dagr                             0          0          0           0             Total                            17340      18         0           21719120 ... View more

Basic debugging All XRv9k datapath commands for clear/show/debug are available under the following CLI set: show controller dpa ... debug controller dpa ... clear controller dpa ... Datapath Architecture The control and dataplanes are separated into distinct linux containers. The dataplane is a part of the Universal Virtual Forwarder (UVF) container. Its composed of two parts: 1. DataPath Agent (DPA) that communicates with a controller in the XR container to get the route/feature configuration that gets programmed into the datapath. 2. The datapath forwarding code which is comprised of DPDK (Intel's DataPath Dev Kit), VPP (infra code) and the XRv9k dataplane code.  The Punt/Inject packets travel from the SPP to the DPA.   Show commands for VPP Graph Nodes  There are many VPP graph nodes under the RX, FWD, TM, TX and DPA blocks. A node processes a frame of packets (upto 256 packets) using batch processing. Batch processing gives cache efficiency gains. The infra (VPP) code is always running in the background even if there are no packets to process, therefore the CPUs will always run at 100%. This is by design.   VPP graph node show commands show controller dpa graph ... The running state of the VPP graph nodes is displayed by show controller dpa graph runtime  Per node counters are displayed with show controller dpa graph counters  Inter-core frame queues are displayed with show controller dpa graph frame-queue  Examples Punt/Inject Packet Path Use the following CLI to troubleshoot the punt/inject packet path: show spp sid [table|stats|node-counters|client] show netio forwarder stats show lpts pifib hardware entry [stat|brief|policer|context info] show lpts pifib stat Packet Tracing Packets arriving from the line or inject packets may be traced as follows: debug controllers dpa packet-trace [line|inject] <# to trace> show controllers dpa packet-trace  clear controllers dpa packet-trace ... View more

View the demo profile at this link -  https://dcloud-cms.cisco.com/?p=23387 Overview The Cisco IOS XRv 9000 router implements the feature set of Cisco IOS XR software. Running on virtualized general x86 compute platforms, it complements existing physical Cisco router platforms that rely on Cisco IOS XR software, such as Cisco Network Convergence System routers, Cisco ASR 9000 Series routers, and Cisco Carrier Routing System (CRS) platforms. Now, service providers can enhance their operational excellence and offerings based on physical routers and easily move them to a virtual form factor. The Cisco IOS XRv 9000 router offers greater agility, improved network efficiency, lower capital and operational expenditures, and the ability to efficiently scale network capacity up and down, based on demand. This lab demonstrates how to install the IOS XRv 9000 in a VMware ESXi environment. Scenarios Scenario 1: Install IOS XRv in VMware ESXi Environment Scenario 2: Verify Configuration Requirements Required Optional Laptop Cisco AnyConnect® Components PC Workstation at 198.18.133.252 Includes VMware vCenter Client VMware ESXi 6.0 Features IOS XRv9k Virtualized instance of IOS XR Zero-touch provisioning functionality Provides an environment to run chef, native applications, and container applications ... View more

View the demo profile at this link -  https://dcloud-cms.cisco.com/demo/ios-xrv-9k-config-lab-v1 Overview The Cisco® IOS XRv 9000 router implements the feature set of Cisco IOS XR software. Running on virtualized general x86 compute platforms, it complements existing physical Cisco router platforms that rely on Cisco IOS XR software, such as Cisco Network Convergence System routers, Cisco ASR 9000 Series routers, and Cisco Carrier Routing System (CRS) platforms. Now, service providers can enhance their operational excellence and offerings based on physical routers and easily move them to a virtual form factor. The Cisco IOS XRv 9000 router offers greater agility, improved network efficiency, lower capital and operational expenditures, and the ability to efficiently scale network capacity up and down, based on demand. Scenarios Scenario 1: Start Simulation Scenario 2: Navigate VMs Scenario 3: Configure vRR Scenario 4: Configure vPE Requirements Required Optional Laptop Cisco AnyConnect® Components PC Workstation at 198.18.133.252 VM Maestro RouteM Features IOS XRv9k Virtualized instance of IOS XR Zero-touch provisioning functionality Provides an environment to run chef, native applications, and container applications ... View more

Introduction Bundle Consistency Checker is an on-the-router mechanism to check and verify the correct programming for bundle interface ingress and egress hardware chains on the CRS platform. The hardware chain comprises of the information required for hardware forwarding at the PSE ASIC level. The ingress chain information is available for all nodes configured for IPv4 whereas the egress chain information is populated for active member nodes only. The Bundle Consistency Checker (BCC) script is supported for Bundle-Ether as well as Bundle-POS interfaces. Implementation Script run time options  There is an option to run multiple bundles at same time by using –t option. Please mention –t 5 in command prompt if you want to run 5 threads in parallel. Script will spawn 5 child processes at same time. If one child process is completed then script will launch another child process immediately and maintain 5 threads till last available bundle. Validation time for local run is optimized as compared with external validation through telnet. A single bundle verification requires approximately 15-20% of CPU load. For a local run, max thread count is limited to 4 to avoid CPU hog.   Supported Hardware        set Topaz_Type [ list "MSC-X" "LSP-X" "FP-X-L" "MSC-X-L" "FP-X"]         set Taiko_Type [ list "MSC-140G" "FP-140G"]         set Metro_Type [ list "MSC-B" "MSC" ] Logging 10 log levels 0-9 are defined. If verbose level set to N then all messages with verbose level 0 through N will be displayed. Verbose level is used to restrict the messages in output screen. Log file will have all the messages irrespective of given verbose level.  Default verbose level is 1. Verbose level: 0 - Only final output will be displayed if input parameter validation is passed 1 – First level of debug messages displayed in output screen. Default Level 2 – Second level of debug messages displayed. 3 – Array variable and its value displayed while setting it.      4-9  Future use.   How to run the script 1. Local run BCC TCL script is located under /pkg/lib/tcl directory. Script run will create log files in the current directory, therefore write permission is required.   #tclsh /pkg/lib/tcl/bundle_consistency_checker.tcl all Command line arguments: -v : Verbose level. Default is 1 -l : log file name. Default is YYYYMMDD-HHMMSS-bcc.log -t : Thread. Number of threads to run in parallel Lastly, specify the bundle id that needs to be verified. Last parameter can be all – All bundles in given router will be verified BE524 – Bundle-Ether 524 will be verified BP514 – Bundle-POS 514 will be verified “BE524 BP514” – Both bundle Bundle-Ether 524 and Bundle-POS 514 will be verified   2. Execute the script through telnet Copy the script to directory (/root/bcc) Chmod 755 /root/bcc/bundle_consistency_checker.tcl [root@LFS bcc]# ./root/bcc/bundle_consistency_checker.tcl –i <Router ip/name> -o <port number> -u <username> -p <password> all [root@LFS bcc]# ./bundle_consistency_checker.tcl -i crs-vega1-lnx -o 5001 -u cisco -p cisco all Command line arguments: -i : Router IP/ Name -o : Port number -u : Username -p  : Password -v : Verbose level. Default is 1 -l : log file name. Default is YYYYMMDD-HHMMSS-bcc.log -t : Thread. Number of threads to run in parallel. Default is 1. Lastly, specify the bundle id that needs to be verified.  Last parameter can be all – All bundles in given router will be verified BE524 – Bundle-Ether 524 will be verified BP514 – Bundle-POS 514 will be verified “BE524 BP514” – Both bundle Bundle-Ether 524 and Bundle-POS 514 will be verified  Sample Output   ... View more

Categories of ASIC errors  Single Bit Errors Multiple Bit Errors Parity Errors CRC Errors Generic Errors Barrier Errors Unexpected Errors Link Errors OOR Thresh BP Errors IO Errors Ucode Errors Config Errors Indirect Errors ASIC Reset Errors Non Error   Error Threshold and Recovery Error threshold and recovery mechanism is defined by the ASIC type as well as by the type of error. Triggers ASIC Reset ASIC is reset after predefined number of errors. Predefined thresholds define when a ASIC would reset. Following syslogs are seen depending on the type of ASIC and type of error CIH-2-ASIC_ERROR_HARD_RESET  XXX error occurred causing halt Linecard Reload In some cases, ASIC resets does not cure the problem and ASIC would continue to reset. ASIC reset thresholds define thresholds for ASIC resets/unit of time until the LC is reloaded. ASIC_ERROR_REQUEST_RELOAD_BOARD device is not recovered from fault and reload is requested.  Linecard Shutdown If LC reload does not fix the issue, the LC would be shut down. LC reload threshold is defined by admin config, e.g. hw-module reset daily threshold 5 location all hw-module reset hourly threshold 2 location all <1-10>   number of resets after which the card will be placed in IN-RESET state   nolimit  disable checking of reset threshold limit (default threshold limit is 5 for one hour, 8 for one day)   Syslog: SHUTDOWN_ON_MULTIPLE_UNGRACEFUL_RELOAD   Examples The card is reset  on the 6 th occurrence of the  ASIC Hard_Reset: LC/1/5/CPU0:Oct 19 13:35:43.548 : ingressq[225]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A mbe error has occurred causing  halt. 0x130f000c   LC/1/5/CPU0:Oct 19 13:36:30.877 : ingressq[225]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A mbe error has occurred causing  halt. 0x130f000c   LC/1/5/CPU0:Oct 19 13:53:02.170 : ingressq[225]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A mbe error has occurred causing  halt. 0x130f000c   LC/1/5/CPU0:Oct 19 13:56:52.929 : ingressq[225]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A mbe error has occurred causing  halt. 0x130f000c   LC/1/5/CPU0:Oct 19 13:58:03.238 : ingressq[225]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A mbe error has occurred causing  halt. 0x130f000c   LC/1/5/CPU0:Oct 19 14:01:17.599 : ingressq[225]: %PLATFORM-CIH-1-ASIC_ERROR_REQUEST_RELOAD_BOARD : ingressq[0]: device is not recovered from fault - and reload is requested. LC/1/5/CPU0:Oct 19 14:01:17.619 : ingressq[225]: %PLATFORM-CIH-2-ASIC_ERROR_HARD_RESET : ingressq[0]: A mbe error has occurred causing  halt. 0x130f000c   LC/1/5/CPU0:Oct 19 14:01:22.775 : sysmgr[82]: %OS-SYSMGR-2-MANAGED_REBOOT : reboot to be managed by process (platform_mgr_common) reason (ASIC seal instance 0 in critical alarm) LC/1/5/CPU0:Oct 19 14:04:23.607 : sysmgr[82]: %OS-SYSMGR-5-NOTICE : Card is COLD started   Enhancements   CSCuv58131   : Bringdown Fabric board on hitting FGID SRAM parity error This is specific for the Fabric boards. Fixed in 5.3.3 ( SMU available for 5.3.1 ) CSCuw99811   : Reload taiko MSC on hitting QDRAM MBE on Seal ASIC. Fixed in 5.3.3   Scope of ASIC Errors and Recovery  Single event  Transient errors that may not persist for long duration. Least impactful recovery mechanism is recommended (memory scrubbing or hard reset) for recovering from such errors. Repeated Errors There are cases where Single event recovery mechanism fails to recover the system. If this error count reaches a threshold within a well defined time window, more impactful recovery action will be initiated (e.g. PON reset, board reload etc). Repeated Error threshold is monitored per second and per day (E.g. For MBE errors, the threshold is 5/sec or 20/day) Configurable System Level Reset Thresholds Five hard resets or PON reset of the ASIC per day results in card reload Five card reloads per day results in card shutdown   System Error Thresholds   Error Category Threshold SBE 20/sec or 80/day MBE 5/sec or 20/day Parity 5/sec or 20/day OOR 1500/5min or 6000/day BP 1500/min or 1200/day INDIRECT 1500/5min or 6000/day LINK Error 20/day   Error Handling   Error type Behavior Current S/W action (for single event) SBE Single bit errors. Detected and corrected by h/w Optional re-write to the impacted address MBE Multi-bit errors. Detected by h/w ASIC Reset Parity Parity errors. Detected by h/w ASIC Reset Link Errors External interface link errors. Detected by h/w Link retrain, ASIC Reset or card reload Out of Resource Internal/external resource (e.g. packet memory). Detected by h/w ASIC reset or card reload Indirect Error Error due to peer ASIC. Detected by h/w. Cross card boundary, impacting System/Network, and worst when originated at IngressQ Card reload Misc Config errors, BP errors etc. Detected by h/w ASIC reset or card reload   MBE/Parity Error Handling Changes (Release 6.1.1 or with a 5.3.3 SMU installed)   Error type Initial Recovery Attempt Repeated Occurrence response MBE Hard Reset/PON Reset/None Shut down the board on 2nd occurrence in 3 months. Parity – General Hard Reset/PON Reset/None Shut down the board on 2nd occurrence in 3 months. Parity – L2 TCAM Scrub the location Hard reset on threshold (2/sec or 8/day). Shut down the board if threshold reached second time in 24 hour window. Parity – L3 TCAM Scrub the location Shut down the board on 2 nd occurrence in 24 hour window. PLL Loss of Lock Current recovery: PON reset New recovery: Error will be ignored per recommendation from vendor on Beluga/Pogo/Seal/Superstar ASICs. N/A   ... View more