I. Introduction This document will help to resolve fabric issues reported in the Cisco Nexus7000 platform. This  document covers the most common types of fabric CRC errors.  Troubleshooting fabric CRCs requires data collecting, data analysis, and  then performing a process of eliminating to isolate the most-likely  failing component. The  “General CRC  troubleshooting guideline” section given below will  establish a general  framework for troubleshooting these issues. Then  the case study sections  will provide some examples on how a similar  problem could be  troubleshot.  And finally the” monitoring fabric CRCs”  section defines  an alternative way for detecting and monitoring fabric  CRCs. II. Fabric CRC detection overview: High-level diagram of Nexus 7018 Fabric with M1 linecards Legend: Stage1 (S1), Stage2 (S2) and Stage3 (S3) are the three stages of the Nexus7000 fabric. Octopus is the Queue Engine Santa Cruz  (SC) is the Fabric ASIC Instance 1 and 2 are the two Santa Cruz instances on the XBAR. The  above is an overview of the components involved when a packet traverses  fabric.  To make it  simple, this document considers only one XBAR.  Please keep in mind that  most of the Nexus7000 switches have three or  more XBARs installed. Assuming  a  unidirectional flow from Module #1 to Module #2, the ingress  Octopus-1  on Mod 1 will perform error checking on packets it receives  from the  south, and the egress Octopus-1 on Mod 2 from the north. If  CRC is  detected in stage 3. problem could have happened in stage 1 or  stage 2,  too, since no CRC check is done in those stages. So the devices in involved in the path are the ingress Octopus, chassis, crossbar fabric, and egress Octopus. In M1/Fab1 architecture, CRCs are detected only on the egress linecard (S3). Sample Error Message: %OC_USD-SLOT1-2-RF_CRC: OC1 received packets with CRC error from MOD 15 through XBAR slot 1/inst 1 The above message is reported by module 1, indicating it received packets with wrong CRC from Module 15 via xbar 1/instance 1 III. Understanding different Fabric CRC errors: (1) CRC error with single source module, receive module, and XBAR instance %OC_USD-SLOT1-2-RF_CRC: OC1 received packets with CRC error from MOD 15 through XBAR slot 1/inst 1 This  means that the module in slot 1 detected a CRC error coming from module  15 through XBAR 1/instance 1.  Going forward we will refer to the  module where the CRC errors were  coming from as the ingress module (mod  15 in this case) and the module  that reported the problem as the  egress module (mod 1).  XBAR #1 is the  cross bar we received the packet  through.  There are two instances per  XBAR, so in this case module 1  detected CRC errors coming from module 15  through XBAR 1 instance 1 (2) CRC error with single source module, receive module, but no XBAR instance %OC_USD-SLOT4-2-RF_CRC: OC2 received packets with CRC error from MOD 1 In  this message,  module 4 reported the CRC error coming from module 1.   You will notice  that the XBAR info is missing, why ?  The system is  unable to ascertain  the XBAR the packet traversed. There are many  reasons, but the most  common are that first the information in the  fabric header of the packet  could be corrupt, so the source module  can’t be determined. Second, the  XBAR that was traversed has been  removed from the system since the  error incremented.  Thus it wasn’t  reported in the hourly syslog  message. (3) CRC error with no receive module %OC_USD-2-RF_CRC: OC1 received packets with CRC error from MOD 16 through XBAR slot 1/inst 1 Here,  some device  detected a CRC from Module 16 through XBAR 1. There is,  however, no  receiver module, why ? When the SUP detects a CRC coming  from the fabric  the slot info is not logged.  So when you see no slot  info then the SUP  detected the problem.  Does this mean that the SUP is  bad ? Not  necessarily, just like with a module reporting the problem  you have  multiple components that could have caused the problem:   module 16, the  chassis (not as likely), XBAR 1, or the SUP. (4) CRC error with multiple possible source modules %OC_USD-SLOT6-2-RF_CRC: OC2 received packets with CRC error from MOD 11 or 12 or 14 or 15 or 16 or 17 or 18 The  source module is gleaned from learning the ingress Octopus that sourced  the bad packet. The  driver that raises an interrupt to log this error  message does not  always know the ingress Octopus the bad packet  originated from. This is  because some of the bits being used to  represent the ingress Octopus are  not used.If the system determines  multiple modules might have these  unused bits turned on the system has  to assume anyone of them could be  the source and as a result we include  all of those modules in the error  message.The system found that module  13 couldn’t have this conflict due  to those bits not being used, thus  it wasn’t logged as a potential  source. IV. Fabric CRC Troubleshooting approach: New linecards (M2) and fabric2 (FAB2) detect CRCs in S1, S2 or S3 making it much easier to isolate the faulty component. Investigating in detail and finding pattern in the failure and log messages will help to isolate the faulty component Some of the questions to ask: Was the error message a one-time event or have multiple CRC error messages been logged ? How frequent are the CRC error messages  being logged ? Do we see them every hour, once a day, once a month, etc. Are the CRC errors ALL coming from the same ingress module ? Are the CRC errors ALL reported on the same egress module ? Are the CRC errors coming from multiple ingress modules AND reported on multiple egress modules ? If multiple modules are reporting CRC errors is there a common source module or XBAR module ? Answers  to the above  questions should allow you to approach troubleshooting  from an angle  which is more likely to lead to faster resolution. V. General CRC Troubleshooting Guidelines: Find the common modules (including XBARs) that are reported in the Fabric CRC error messages. Build  a theory and  test it. That is, after finding the common modules pick  the most likely  cause of the problem and shutdown (in case of XBAR),  move it to a known  good slot, reseat, and replace it while monitoring  to see if the problem  goes away. Shutdown/reseat/replace modules one at a time. This makes it easier to isolate the faulty part. When  you shutdown,  move, reseat, or replace a part look for any changes in  the problem’s  symptom.  You may have to revise your action plan after  you learn more  from each step taken. If  multiple parts  have been replaced and the problem still persists:  (1)  the new parts  could be bad so test them in a working slot (2) multiple  XBARs could be  bad (3) a bad chassis slot could be the cause. VI. Case Studies: (1) Ingress module corrupting the packets Logs: %OC_USD-SLOT1-2-RF_CRC: OC2 received packets with CRC error from MOD 7 %OC_USD-SLOT3-2-RF_CRC: OC2 received packets with CRC error from MOD 7 %OC_USD-SLOT1-2-RF_CRC: OC2 received packets with CRC error from MOD 7 %OC_USD-SLOT3-2-RF_CRC: OC2 received packets with CRC error from MOD 7 %OC_USD-SLOT1-2-RF_CRC: OC2 received packets with CRC error from MOD 7 %OC_USD-SLOT3-2-RF_CRC: OC2 received packets with CRC error from MOD 7 Problem: For the last few hours CRC errors have been seen on module 1 and 3 coming from Module 7 and Module 7 only. Most likely cause of the problem: 1.  There is a bad or mis-seated XBAR corrupting packets going to module 7 2.  Module 7 is bad or mis-seated Process to isolate the faulty component: Shutdown XBARs one-by-one monitoring to see if the problem goes away Reseat the ingress module 7 and monitor Replace the module 7 and monitor If  you have three XBARs installed this gives you N+1 redundancy.Therefore,  you should be able to shut them down one at a time (and never more than  one shut at any given time) with only minimal impact to see if the  problem goes away. N7K(config)# poweroff xbar 1 <monitor> N7K(config)# no poweroff xbar 1 N7K(config)# poweroff xbar 2 <monitor> N7K(config)# no poweroff xbar 2 N7K(config)# poweroff xbar 3 <monitor> N7K(config)# no poweroff xbar 3 In this particular case study, shutting down the XBARs did not resolve the problem. As  there are two modules reporting CRC errors, it is unlikely that two  modules reporting the errors (mod 1 & 3) are the cause. Our  next  step then should be to reseat module 7 (ingress module) , because  it is  the most likely faulty component. Mis-seated linecards can cause  this  problem, and it is recommended to reseat the module before replacement. After  reseating  module 7, and monitoring we still find that CRC errors are  incrementing  on the fabric. A Cisco TAC should be opened (but can  always be opened  earlier) at this point to replace/EFA module 7 since a  reseat didn’t  resolve the problem. In  our case study,  the replacement of module 7 stopped the fabric crc  error messages and  the packet loss our customer was seeing. (2)  Mis-seated XBAR injecting corrupt packets Logs: %OC_USD-SLOT11-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT12-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT13-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT15-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT2-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT4-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT5-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT6-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT7-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 %OC_USD-SLOT8-2-RF_CRC: CRC error from MOD 12 through XBAR slot 3/inst 1 Problem:  Multiple modules are reporting CRC errors from Module 12 going through the    XBAR 3. Most likely cause of the problem: 1.  XBAR 3 is bad or mis-seated 2.  Module 12 is mis-seated or faulty Process to isolate the faulty component: 1.  Shutdown XBAR 3 and monitor 2.  Reseat the ingress module 12 and monitor 3.  Replace  module 12 and monitor In  our case, we  shutdown XBAR 3, using the procedure previously described  (in first case  study), and monitored for further errors.  It was found  that errors  ceased when XBAR 3 was shutdown. At this point, XBAR 3 was  reseated,  taking care to ensure that no pins are bent on the midplane  and that the  module is properly inserted.  After re-enabling XBAR 3 it  was found  that the problem has never reoccurred. This problem can be attributed to a mis-seated XBAR module. (3) Faulty Egress module corrupts packets from the Fabric Logs: %OC_USD-SLOT6-2-RF_CRC:   OC1 received packets with CRC error from MOD 1 or 2 or 7 or 13 or 17   through XBAR  slot 1/inst 1 and slot 2/inst 1 and slot 3/inst 1 %OC_USD-SLOT6-2-RF_CRC:   OC2 received packets with CRC error from MOD 1 or 2 or 3 or 7 or 15 or   17 through XBAR slot 2/inst 1 and slot 3/inst 1 %OC_USD-SLOT6-2-RF_CRC:   OC1 received packets with CRC error from MOD 1 or 2 or 5 or 7 or 16 or   17 through XBAR slot 1/inst 1 and slot 2/inst 1 and slot 3/inst 1 Problem: Module 6 is reporting packets with CRC errors being received from multiple linecards and XBARs Most likely cause of the problem: Module 6 is mis-seated or bad Process to isolate the faulty component: Reseat module 6 and monitor Replace module 6 and monitor Module  6 is the most  likely cause of the fault because it is the one common  module in all  the errors messages. Of all the modules listed in the  error messages. the one  that most consistently shows up is module 6.   Therefore, we try to  reseat module 6 to see if that resolves the issue  before replacing it. In  our case, we  reseated module 6 but the errors still persisted. So the  next step  should be to open a TAC Case to have module 6 replaced.  And  after replacing module 6 the errors were not reported any longer. VII. Troubleshooting Commands: Some of the commands used to troubleshoot/debug: show clock sh mod xbar show hardware fabric-utilization detail  show hardware fabric-utilization detail timestamp show hardware internal xbar-driver all event-history errors show hardware internal xbar-driver all event-history msgs show system internal xbar-client internal event-history msgs show system internal xbar all show module internal event-history xbar 1 show module internal activity xbar 1 show module internal event-history xbar 2 show module internal activity xbar 2 show module internal event-history xbar 3 show module internal activity xbar 3 show module internal event-history xbar 4 show module internal activity xbar 4 show module internal event-history xbar 5 show module internal activity xbar 5 show logging onboard internal xbar show logging onboard internal octopus show tech detail For comments and feedback, please email Yogesh at yramdoss@cisco.com ... View more

At  Cisco TAC, when the phones ring for P1/P2 outages, a typical problem we hear from customers is about connectivity issues between end-hosts and/or end-users' application performance issues. In an enterprise core/distribution and datacenter network, where virtual switching solutions (like, VSS or vPC) are highly deployed with large number of port-channels / redundant links, it is challenging and critical to trace the path of the packet to identify the device, link and port causing the above-mentioned problems. How does a TAC engineer really find the device / link / port in issue ? The answer is ... by using the innovative tools discussed here. (1) NetDR / EthAnalyzer Net Driver (NetDR) tool is used in Catalyst 6500 and Cisco 7600 platforms to capture the traffic sent to (and received from) the CPU. EthAnalyzer performs the same functions in Nexus platforms and is an implementation of TShark in NX-OS. When poor performance is experienced with specific application, using these tools (in the devices along the traffic path) helps us to confirm that specific traffic is getting hardware-switched and NOT processed by the CPU. Here is a sample NetDR capture - reporting layer2, layer3 and layer4 information along with ingress interface. (2) Access-List / VLAN Access-List Capture (ACL / VACL Capture) This tool allows network administrators to replicate network traffic, especially when there is a need to monitor subset of traffic on high bandwidth links (thats how it differs from traditional R/ER/SPAN). VACL Capture is supported only on Catalyst 6500 / Cisco 7600 platforms. ACL Capture is supported only on Nexus7000 M1-series modules in 5.2(1) release onwards. The examples discussed are very simple but powerful in terms of narrowing down the device(s) It can be very invaluable if number of packets for a specific flow can be counted at each layer-2 hop, specifically for faster problem resolution. (3) Embedded Logic Analyzer Module (ELAM) First of all, ELAM is NOT a supported feature and is very specific to the hardware architecture. Anything and everything about it may change from version to version without any notice. ELAM is supported in Catalyst 6500, Cisco 7600, Nexus 7000 platforms. ELAM is used by TAC as a capturing tool to validate: (a) Was the packet received ? (b) What did the packet look like ? (c) Was the packet rewritten/alterted ? If so, how ? (d) Where was the packet sent ? The following diagram shows the workflow of using the ELAM tool: As you can see, the first step is to identify the ingress forwarding engine, which requires knowledge on hardware architecure. It is highly recommened for customers to review a detailed document on ELAM available here. Miscellaneous : A relevant TS Newsletter Article has discussed different packet capturing tools available in Cisco devices: http://www.cisco.com/web/services/news/ts_newsletter/archives/201206.html A Cisco Live 2013 Breakout session BRKARC-2011 discusses and provides an Overview of Troubleshooting Tools in Cisco Switches and Routers. Session Slides (in PDF) Session Video (in MP4) To access contents, please visit http://www.ciscolive365.com and register. References: VACL Capture for Granular Traffic Analysis with Catalyst 6500 running Cisco IOS Software: http://www.cisco.com/en/US/tech/tk389/tk814/technologies_configuration_example09186a00808122ac.shtml Nexus7000 NX-OS Security Configuration Guide - ACL Capture: http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/security/configuration/guide/b_Cisco_Nexus_7000_NX-OS_Security_Configuration_Guide__Release_5.x_chapter_01110.html#con_1210975 Nexus7000 Series Switch ACL Capture Example: http://www.cisco.com/en/US/products/ps9402/products_tech_note09186a0080c021d5.shtml ... View more

Introduction This document disucsses on high CPU condition seen in Catalyst 6500 platforms due to IOS Server Load Balancing (SLB) feature. Problem Description: Catalyst 6500 reporting high CPU due to interrupts and "IP Input" process. C6K-A#show process cpu sorted | exclude 0.00 CPU utilization for five seconds: 98%/57%; one minute: 99%; five minutes: 97% PID Runtime(ms)  Invoked      uSecs   5Sec   1Min   5Min TTY Process 227   816331652 12632718       4266 32.10% 31.69% 31.07%   0 IP Input Troubleshooting and Symptoms: (1) The IOS SLB is configured with "nat server"" ip slb serverfarm TEST-FARM1    nat server    nat client POOL1 <snip> (2) Switch do NOT have any sw-installed Netflow entries: C6K-A#show mls netflow ip sw-installed Displaying Netflow entries in Active Supervisor EARL in module 5 No Entries C6K-A#show mls netflow ip sw-installed module 1 <<=== Ingress DFC module Displaying Netflow entries in Active Supervisor EARL in module 1 No Entries (3) SLB has active connections: C6K-A#show ip slb connections vserver         prot client                real               state        nat ------------------------------------------------------------------------------- VSERVER-NAME    TCP  <client-ip>:<port>    <real-ip>:<port>   <TCP-State>  S,C <snip> Root-Cause and Resolution: When the client sends the traffic to virtual IP address, the load-balancer (in this case, IOS SLB) will NAT the traffic, as the real/physical severs are NOT aware of the virtual IP address. Cat6500 with "nat server" configuration, the switch is NOT capable of creating hardware shortcuts. As a result, the traffic will be process/software switched. This is done by punting the traffic to the CPU and it can be verified by: (Here, 10.50.50.2 is IP address of a virtual server) C6K-A#show tcam int vlan <client-vlan> acl in ip | inc 10.50.50.2     punt         udp any host 10.50.50.2     redirect     tcp any host 10.50.50.2 fragments     policy-route tcp any host 10.50.50.2 eq <port#>     <snip> To resolve this issue, it is recommended to configure IOS SLB in "Dispatched" mode. http://www.cisco.com/en/US/products/hw/modules/ps2706/products_configuration_example09186a0080094066.shtml http://www.cisco.com/en/US/products/hw/routers/ps341/products_tech_note09186a0080134735.shtml Further Information: To know more on IOS SLB feature, please refer the 12.2SX Feature Configuration Guide: http://www.cisco.com/en/US/docs/ios/12_2sx/feature/guide/slbsxf7.html Please be aware that starting from 12.2(33)SXJ IOS SLB is NOT supported: http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/release/notes/features.html#wp4798296 For feedback and comments, please contact Yogesh at yramdoss@cisco.com ... View more

An updated version of this document is available at: http://www.cisco.com/en/US/products/hw/routers/ps332/products_tech_note09186a0080c178fb.shtml We apologize for any inconvinience. ... View more

Introduction The purpose of this document is to provide a step-by-step process to determine if a N7K-M132XP-12 or N7K-M132XP-12L module in Nexus7000 switch need to be RMAed or not. This document lists the symptoms and relevant troubleshooting steps in determining the health of the module. Scenario 1: N7K-M132XP-12 or N7K-M132XP-12L “TestPortLoopback” diagnostic test failed Symptoms: Diag failure, the following syslog is observed: 2011 Dec 10 11:47:11 %DIAG_PORT_LB-2-PORTLOOPBACK_TEST_FAIL: Module:18 TestPortLoopback failed 10 consecutive times. Faulty module:Module 18 affected ports:23 Error:Loopback test failed. Packets lost on the LC at the Queueing engine ASIC N7K2# show diagnostic result module 18 Current bootup diagnostic level: complete Module 18: 10 Gbps Ethernet Module Test results: (. = Pass, F = Fail, I = Incomplete, U = Untested, A = Abort, E = Error disabled) 1) EOBCPortLoopback--------------> . 2) ASICRegisterCheck-------------> E 3) PrimaryBootROM----------------> . 4) SecondaryBootROM--------------> . 5) PortLoopback: Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ----------------------------------------------------- U U I I I I I I U U I . I . I . Port 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------------------------------------- U U . . U U E . U U I I I I I I 6) RewriteEngineLoopback: Port 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ----------------------------------------------------- . . . . . . . . . . . . . . . . Port 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ----------------------------------------------------- . . . . . . . . . . . . . . . . N7K2# show module Mod Ports Module-Type Model Status --- ----- -------------------------------- ------------------ ------------ 16 32 10 Gbps Ethernet Module N7K-M132XP-12 ok 17 32 10 Gbps Ethernet Module N7K-M132XP-12 ok 18 32 10 Gbps Ethernet Module N7K-M132XP-12 ok <snip> Mod Online Diag Status --- ------------------ 16 Fail 17 Pass 18 Fail Checklist: This is likely due to CSCtn81109 or CSCti95293 To verify if you hit these bugs, perform the following checks: (1) Check NX-OS version if match with ddts found version. Both bugs are fixed and verified in 5.2(4) and later releases. (2) Understand when diag message was observed, “show log” will give the time stamp of diag test failure, then check if there is any CPU issue happening near the sametime. Sometimes when the CPU is overwhelmed it causes the diag port loopback test failed, it is not definite, but a good data point to collect. (3) Collect additional Logs: “tac-pac bootflash:tech.txt”  <<== This command may take several minutes to complete “show tech module 1” “show tech gold” “show hardware internal errors module 1 | diff -s” <<== execute this command few times (4) We can clear the diagnostic result and re-run them while CPU is not overwhelmed: # show diagnostic result module 1 # diagnostic clear result module all (config)# no diagnostic monitor module 1 test 5 (check the test number using "show diagnostic content module X") (config)# diagnostic monitor module 1 test 5 # diagnostic start module 1 test 5 # show diagnostic result module 1 test 5 (could take a few minutes before test completed) # show module internal exceptionlog module 1 # show module internal event-history errors # show hardware internal errors module 1 If the module is recovered and diag test pass, highly likely this is due to the DDTS' mentioned above; as actual hardware failure should fail diag consistently. If the module failed diag consistently, Please contact Cisco TAC for further analysis. Scenario 2: M1 modules gets reset and/or link flaps Symptoms: 2012  Jun 13 15:51:30 MDT Q93-7010-A %$ VDC-1 %$  %DIAG_PORT_LB-2-PORTLOOPBACK_TEST_FAIL: Module:3 TestPortLoopback failed  10 consecutive times. Faulty module: affected  ports:3,5,7,11,13,15,19,21,23,27,29,31 Error:Loopback test failed.  Packets lost on the LC at the MAC ASIC 2012 Jun 13 15:51:30 MDT Q93-7010-A %$ VDC-1 %$  %DIAG_PORT_LB-2-PORTLOOPBACK_TEST_FAIL: Module:3 TestPortLoopback failed  10 consecutive times. Faulty module: affected  ports:4,6,8,12,14,16,20,22,24,26,28,30,32 Error:Loopback test failed.  Packets lost on the LC at the Queueing engine ASIC Checklist: This is highly likely due to CSCtt43115 It is NOT a hardware failure, and no RMA/EFA required. Collect all the logs reported and sequence of events occurred. - show tech detail - show accounting log - show logging Please make sure the configurations (specifically SPAN) and symptoms match with that of mentioned in the DDTS Release-note enclosure. Note: This issue is applicable to all M1 module types. Relevant Field Notice: http://www.cisco.com/en/US/ts/fn/634/fn63495.html Scenario 3: N7K-M132XP-12 consistent device base process crashed Symptoms: Nexus7000 switch reports: %SYSMGR-SLOT8-2-SERVICE_CRASHED: Service "lamira_usd" (PID 1944) hasn't caught signal 6 (core will be saved). Here, Lamira is the L3/L4 forwarding engine. The above menssage indicates that this ASIC has crashed and dumped the core. Checklist: In most cases, lamira_usd crash is caused by a reoccurring TCAM parity error, the fix was to RMA/EFA the card. This checklist provides instruction on how to confirm that lamira_usd crash is caused by multiple uncorrectable TCAM parity errors, which can lead to a fast resolution to the case. (1) Collect following logs, by attaching to the module: # attach module <#> module-<#># show logging onboard exception-log module-<#># show logging onboard internal lamira Note: We have to collect the execption-log from the onboard log. “show module internal exceptionlog module <mod>” won't suffice (2) Note the time lamira_usd crashed (3) Now check the onboard exception-log around the time of the LC crash (10-20 minutes) // Five minutes before the lamira_usd crashed Exception Log Record : Tue Jul 10 04:05:40 2012 (840169 us) Device Id : 81 Device Name : Lamira Device Error Code : c5101210(H) Device Error Type : ERR_TYPE_HW Device Error Name : NULL Device Instance : 1 Sys Error : Generic failure Errtype : INFORMATIONAL PhyPortLayer : Ethernet Port(s) Affected : Error Description : LM_INT_CL1_TCAM_B_PARITY_ERR <== uncorrectable parity error in TCAM B DSAP : 211 UUID : 382 Time : Tue Jul 10 04:05:40 2012 (840168 usecs 4FFC0C84(H) jiffies) (4) Compare the TCAM exception above with the exception log record at the time of the crash Exception Log Record : Tue Jul 10 04:17:02 2012 (270399 us) <== 12 minutes after the TCAM parity error Device Id : 34304 Device Name : 0x8600 Device Error Code : 7e010000(H) Device Error Type : NULL Device Error Name : NULL Device Instance : 0 Sys Error : (null) Errtype : CATASTROPHIC PhyPortLayer : 0x0 Port(s) Affected : Error Description : lamira_usd hap reset <== lamira crashed because the number of TCAM parity errors in TCAM B violated the HA Policy(HAP) DSAP : 0 UUID : 16777216 Time : Tue Jul 10 04:17:02 2012 (270399 usecs 4FFC0F2E(H) jiffies) (5) If the lamira crash was caused by HAP reset due to multiple TCAM parity errors then the LC should be RMAd/EFAd. Otherwise, if lamira crashed for some other reason, continue your normal troubleshooting. The onboard lamira log (command #2) will help Cisco TAC to root-cause it. Scenario 4: All M1 modules fail specific diagnostic tests, like PortLoopback or RewriteEngineLoopback test In Nexus7000, if there is any issue between active supervisor engine and an xbar module, and as a result diagnositic packets are dropped, the supeervisor engine may report diagnostic test failure for multiple/all ports in multiple/all modules. This issue requires manual investigation and isolation of faulty sup engine. The condition which caused the tests to go into errdisabled state may be transient. It is recommended to run the tests on-demand and see if the condition is persistent. First, try to clear the ErrDisabled state of the test: N7K# diagnostic clear result module 1 test ?   <1-6>  Test ID(s)   all    Select all To run on-demand test: N7K# diagnostic start module <mod#> test <test#> To stop the test: N7K# diagnostic stop module <mod#> test <test#> As a corrective action, the sup engine do not trigger failover or reset to recover from this condtion. To request corrective action, an enhancement request has been filed - CSCth03474 - n7k/GOLD:Improve Fault Isolation of N7K-GOLD Addtional Information: Software Advisory on CCO: http://www.cisco.com/web/software/datacenter/N7K/nxos_521_sw_advisory.html If you have any comments/feedback, please send email to Yogesh ( yramdoss@cisco.com ) and Vincent ( vng@cisco.com ) ... View more

I. Introduction   This document provides common steps and commands used to retrieve core files in Nexus switching platforms -  Nexus7000, Nexus5000, Nexus 4000, Nexus 3000 and Nexus2000.   II. Technical Background Information:   NX-OS is a full-featured, modular, and scalable Cisco networking operating system available for entire Nexus switching platforms. For more information, please review the information and documents provided at: http://www.cisco.com/en/US/products/ps9372/index.html   NX-OS runs on Linux Kernel. From kernel's perspective all the NX-OS processes are run in the "User" space. Available DRAM are split into two regions: (1) Kernel space (a.k.a Low Memory region) and (2) User space (a.k.a High Memory region).   Kernel   The kernel needs memory to store its own text, data, and  Kernel Loadable Modules (KLMs). KLMs are pieces of code that are loaded  into the kernel (as opposed to being a separate user process). An  example of kernel memory usage is when an inband port driver allocates  memory to receive packets.   User processes   This memory is used by Cisco NX-OS processes (along with Kernel processes that are not integrated into the kernel)                      DRAM     III. What is a Core file ?   Core file is same as crashinfo file generated in Cisco IOS platforms, but with more log and system files bundled together into a tar file.   Core file is generated when a process crash or experience an exception.   Contents of a core file - generated by Nexus7000 switch:     IV. Crash / Exception:   When a specific process (called as Service) crahes, the device should report a log message, as follows:   Scenario 1:   %SYSMGR-2-SERVICE_CRASHED: Service "vpc" (PID 5883) hasn't caught signal 11 (core will be saved)   Here, service "vpc" has crashed and a core file will be saved.   Scenario 2:   The device may report message, with no core file created.   %SYSMGR-2-SERVICE_CRASHED: Service "stp" (PID 4668) hasn't caught signal 9 (no core).   Here, sevice "stp" crashed but has not generated any core file.   V. Retreiving Core files:   For Scenario 1 (as mentioned above):   If there is a process crash/exception reported and the switch has NOT reloaded (since the exception/crash), then do "show cores" to get list of cores.   N7K# show cores VDC Module Instance Process-name     PID       Date(Year-Month-Day Time) --- ------ -------- --------------- -------- ------------------------- 1   6       1         vpc             4763     2011-01-10 11:33:01 1   6       1         vpc             5883     2011-01-10 11:33:05   Please do "show cores vdc-all" to see core files in all VDCs.   The above results indicate that the exception was reported for "vpc" service in VDC #1, Module #6. The results provide different Process ID (PID) - 4763 and 5883 - the specific process had at exception, with timestamps. Instance number will be useful to identify the core files when a specific process with same PID (for the same VDC) experience multiple exceptions.   Please be aware that "show cores" command do NOT provide any information,  if the switch has rebooted since the exception.   To copy the core files to FTP or TFTP server, follow the steps:   N7K# copy core:?    core: Enter URL "core://<module-number>/<process-id>[/instance-num]“   N7K# copy core://6/4763/1 ?    bootflash: Select destination filesystem    ftp:       Select destination filesystem    scp:       Select destination filesystem    sftp:      Select destination filesystem    slot0:     Select destination filesystem    tftp:      Select destination filesystem    usb1:      Select destination filesystem    usb2:      Select destination filesystem   The above command collects all relevant info (system info, log files etc.) from the switch and bundles them into .tar file. It is NOT recommended to copy files directly from different filesystems manually.   If the switch has rebooted, do following command to see if there are core files generated earlier:   N7K# dir logflash://sup- 1 /core  100499456   Aug 29 22:36:54 2011 0x501_ethpm_core.16574    8638991   Aug 29 22:45:14 2011 0x501_ethpm_core.4165.gz      37139   Aug 29 22:36:54 2011 0x501_ethpm_log.16574    7699061   Aug 29 22:36:32 2011 0x501_ethpm_log.16576.tar.gz    8208542   Aug 29 22:36:32 2011 0x501_ethpm_log.4165.tar.gz    7698622   Aug 29 22:45:30 2011 1314657930_0x501_ethpm_log.16576.tar.gz    8208230   Aug 29 22:45:30 2011 1314657930_0x501_ethpm_log.4165.tar.gz   If there is Supervisor failover occurred, please check the other/standby sup for core files.   N7K# dir logflash://sup- 2 /core   In Nexus5000, Nexus4000, Nexus3000 and Nexus2000 platforms, as there is no supervisor engine redundancy, there will not be any failover.   Note:   In Nexus5000, Nexus4000, Nexus3000 and Nexus2000 platforms the core files are stored in the "volatile:" and not in the "logflash:" file system.   N3k-3# dir volatile:?   volatile:///   volatile://module-1/   volatile://sup-1/   volatile://sup-active/   volatile://sup-local/   Please be aware that contents of "volatile:" file system are flushed on reload.   For Scenario 2 (as mentioned above):   N7K# show process log vdc-all VDC Process         PID     Normal-exit Stack Core   Log-create-time --- --------------- ------ ----------- ----- ----- --------------- 1 installer       10544             N     N     N Thu Jun 10 17:49:21 2010 1 ethpm           16574             N     Y     N Mon Aug 29 22:36:15 2011   Here, the "ethpm" sevice crashed and g enerated "Stack" (flagged with Y) but no "Core" file (flagged with N). At the same, for the "installer" process, neither "Stack" nor "Core" file is generated.   For the "installer" process, furher information can be obtained by:   N7K# show process log pid 10544 Service: installer Description: Installer Started at Thu Jun 10 17:45:42 2010 (483528 us) Stopped at Thu Jun 10 17:49:21 2010 (719259 us) Uptime: 3 minutes 39 seconds Start type: SRV_OPTION_RESTART_STATELESS (23) Death reason: SYSMGR_DEATH_REASON_FAILURE_NOCALLHOME (12) Last heartbeat 0.00 secs ago RLIMIT_AS: 69909875 System image name: n7000-s1-dk9.4.2.4.bin System image version: 4.2(4) S32 Exit code: SYSMGR_EXITCODE_FAILURE_NOCALLHOME (20) PID: 10544 SAP: 0 UUID: 0   For the "ethpm" process, the stack trace can be obtained by:   N7K# show process log pid 16574 Service: ethpm Description: Test Ethernet Port Manager Executable: /isan/bin/ethpm Started at Mon Aug 29 22:36:15 2011 (188136 us) Stopped at Mon Aug 29 22:36:15 2011 (746741 us) Uptime: 0 seconds Start type: SRV_OPTION_RESTART_STATEFUL (24) Death reason: SYSMGR_DEATH_REASON_FAILURE_SIGNAL (2) <snip> Virtual Memory:    CODE     08048000 - 08356C90    DATA     08357000 - 08369BA8    BRK       083F0000 - 086F9000    STACK     BFBB25C0    TOTAL     98996 KB <snip> Memory Map: 08048000 ethp 08357000 ethp 4143F000 ld-2.8.s 41459000 ld-2.8.s 4145 A000 ld-2.8.s 4145D000 libc-2.8.s 41596000 libc-2.8.s 41598000 libc-2.8.s 4159E0 <snip> Register Set:    EBX BFBB0ADC         ECX 00000000         EDX 00000002    ESI BFBB15B0         EDI 00000009         EBP BFBB1148 <snip> Stack: 6976 bytes. ESP BFBB0A80, TOP BFBB25C0 0xBFBB0A80: 0000001F 00000000 00000000 00000001 ................ <snip>   VI. Why the core file is missing ? :   If the switch does not have enough space in the specific filesystem (logflash: or volatile: depending on the platform), then the core file may not be successfully generated/stored.   N7K# dir logflash://sup-1/ Usage for logflash://sup-1   498237440 bytes used 7394926592 bytes free 7893164032 bytes total   To check the free space available in different file systems, you can also do:   N7K# show system internal flash Mount-on                  1K-blocks      Used   Available   Use%  Filesystem /                            409600     61372      348228     15   /dev/root /proc                             0         0           0      0   proc /sys                              0         0           0      0   none /isan                       1048576    339184      709392     33   none .... /bootflash                  1809684    673252     1044504     40   /dev/hda3 .... /logflash                   7708168     95004     7221608      2   /dev/hde1 /bootflash_sup-remote       1809688    672952     1044808     40   127.1.1.2:/bootflash/ /logflash_sup-remote        7708168     34976     7281640      1   127.1.1.2:/logflash/   Same set of commands, from a Nexus3000 switch:   N3K# dir volatile://sup-1/ Usage for volatile://sup-1           0 bytes used   104857600 bytes free   104857600 bytes total   N3K# sh system internal flash Mount-on                  1K-blocks      Used   Available   Use%  Filesystem /                            204800    112436       92364     55   /dev/root /proc                             0         0           0      0   proc /post                          2048         4        2044      1   none /sys                              0         0           0      0   none ..... /volatile                    102400         0      102400      0   none /debug                        20480         8       20472      1   none ..... /bootflash                  1609984    582492      945708     39   /dev/sda3     VII. Logs/Files to Capture:   If further analysis required on process exception / core files, please open a Service Request and send following logs:   - show cores vdc-all - Core files saved using "copy core://<module-number>/<process-id>[/instance-num]..." command - show process log vdc-all - show process log details - show logging onboard internal reset-reason - show logging onboard stack-trace - show logging onboard kernel-trace - show module internal exceptionlog module <mod#>   Please make sure all these logs are captured to a file(s), as the logs may go several pages.   VIII. Further Information:   Cisco Nexus 7000 Series NX-OS Troubleshooting Guides: http://docwiki.cisco.com/wiki/Cisco_Nexus_7000_Series_NX-OS_Troubleshooting_Guide Cisco Nexus 7000 Series Switches Configuration Guides: http://www.cisco.com/en/US/products/ps9402/products_installation_and_configuration_guides_list.html Cisco Nexus 7000 Series Switches Command Reference Guides: http://www.cisco.com/en/US/products/ps9402/prod_command_reference_list.html   For comments and feedback, please contact the author Yogesh Ramdoss at yramdoss@cisco.com ... View more

  Introduction Understanding “Input Discard” Interface Counter in Nexus 3000 Nexus 3000 - Product Overview The Cisco Nexus 3000 Switch is a high-performance, high-density, ultra-low-latency Ethernet switch. This compact one-rack-unit (1RU) form factor 10 Gigabit Ethernet switch provides line-rate Layer 2 and Layer 3 switching. The switch runs the industry-leading Cisco NX-OS Software operating system, providing customers with comprehensive features and functionality that are widely deployed globally. “Input Discard” Interface Counter in Nexus 3000 As in other Catalyst and Nexus products, Nexus 3000 also has “show interface ethernet x/y” command.   N3k-A# show interface eth1/1 Ethernet1/1 is up Hardware: 10/100/1000 Ethernet, address: 547f.ee3f.9c68 (bia 547f.ee3f.9c68) MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA Port mode is trunk full-duplex, 1000 Mb/s Beacon is turned off Input flow-control is off, output flow-control is off Switchport monitor is off EtherType is 0x8100 Last link flapped 3d18h Last clearing of "show interface" counters 3d18h 30 seconds input rate 2272 bits/sec, 284 bytes/sec, 3 packets/sec 30 seconds output rate 8304 bits/sec, 1038 bytes/sec, 13 packets/sec Load-Interval #2: 5 minute (300 seconds) input rate 2.15 Kbps, 3 pps; output rate 7.58 Kbps, 12 pps RX 0 unicast packets 1344749 multicast packets 0 broadcast packets 1344749 input packets 98148908 bytes 0 jumbo packets 0 storm suppression packets 0 giants 0 input error 0 short frame 0 overrun 0 underrun 0 watchdog 0 if down drop 0 input with dribble 3 input discard(includes ACL drops) 0 Rx pause TX 0 unicast packets 4772275 multicast packets 0 broadcast packets 4772275 output packets 342602114 bytes 0 jumbo packets 0 output errors 0 collision 0 deferred 0 late collision 0 lost carrier 0 no carrier 0 babble 0 Tx pause 1 interface resets N3K-A#   In releases earlier to 5.0(3)U2(1), the “show interface ethernet x/y” report input discard counters just as “input discard” without “(includes ACL drops)”.   Bug CSCto53539 CSCto53539 - Interface discard counters increment even when packets are not forwarded CSCto53539 has ONLY changed how the counter is reported in the “show interface eth x/y” results: http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCto53539   This bug is fixed in 5.0(3)U2(1) or later releases.   The redirection of certain protocol packets e.g. PIM/IGMP happens in N3K ASIC via an ACL entry. The packet is qualified using the ACL match actions ( e.g., IP protocol == PIM ) and one of the actions of this ACL would be to copy the packet to CPU. Since the protocol packets are typically required to be handled only by the CPU and are not expected to get forwarded, this ACL also had a DROP action. This way a copy of the packet would go to CPU and then the packet would get dropped in the hardware emulating the desired behavior. However, drop action from an ACL on an incoming packet is counted as indiscards on the receive interface. Because of this all the legitimate CPU bound protocol packets that get redirected this way would contribute to the indiscards. Considering this an additional description was added in CLI to say "includes ACL drops" for such packets. Getting further details on Input Discard counter   To get more granular details, it is recommended to use following command:   To get the statistics from Eth1/1:   N3k-A# show hardware internal interface indiscard-stats front-port 1 +---------------------+-------+----------------+---------------------+ | Counter Description | Count | Last Increment | Last Increment Time | +---------------------+-------+----------------+---------------------+ IPv4 Discards 0 0 STP Discards 2 2 04-23-2012 17:47:04.558 Policy Discards 0 0 ACL Drops 0 0 Receive Drops 2 2 04-23-2012 17:47:04.558 Vlan Discards 0 0 +---------------------+-------+---------------+----------------------+   Please be aware of CSCtu29771, due to which the above command reports all counters as zero, even though the “show interface” reports non-zero input discards. This bug is fixed in 5.0(3)U3(1) or later releases.   Release note for CSCtu29771: http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtu29771   Details of the counters   IPv4 Discards IPv4 Discards represent errors at IP layer e.g., IP checksum error   STP Discards STP Discards are incremented when packets are received when interface is not in STP forwarding state   Policy Discards Policy Discards are incremented when there are discards because of input policy on the interface e.g., storm control   ACL drops ACL drops indicate that incoming packets match an ACL entry with a drop action. This can increment under following conditions: (1) When user configured ACL denies the traffic. (2) When interface receives the link local multicast packets (EIGRP/PIM/BFD hello’s).              Reason being, internally the packet is getting copied to CPU, and dropped by the ACL Drop.        (3) "STP Discards" action is hit.   Through an enhancement (CSCtx63122), from 5.0(3)U3(1) onwards Nexus 3000 will NOT increase "ACL Drops" counter (and hence "input discard" under "show interface") due to control-plane packets like IGMP/PIM/BFD etc. except Precision Time Protocol (PTP) packets.   Receive Drops This counter is a wrapper counter which represents a condition where no output port could be determined for this packet. From ASIC perspective, Receive Drop is almost a catch-all for all conditions. Some of the common conditions could be: - Vlan check failed - MTU check failed - "ACL Drops" action is hit - Traffic destined to a non-existing route - Traffic destined to an IP address for which ARP entry is incomplete - Multicast traffic getting dropped as the (S, G) has null OIF It will be challenging to come up with a comprehensive list of all possible conditions. This counter is incremented if the forwarding result at the end of the ingress pipeline (in the forwarding decision making process) is zero, which means, no egress port identified.   Sample Scenario Assume that there is a static mac-address entry for mac-address M2 in vlan V2 mapped to port P2. If a frame with source mac-address M2 received in vlan V2 on P1. In this condition, source mac-address learning cannot happen for this frame and this frame will be dropped.   Vlan Discard Vlan Discard indicates vlan-based discards e.g., vlan tagged packet ingressing on a port which is not member of the vlan.   Further Enhancement Enhancement request CSCtx60271 has been filed modify the counters considered for "input discards" counters under "show interface".   With this fix, now the "input discards" counter does not include following counters: - ACL (RACL/VACL/PACL) drops - IGMP/BFD packets dropped by CoPP - while the packet rate is higher than 1000 packets/second - DeJavu scenario - packets received and sent out on the same port or port-channel, with no modifications to the packet.   Release note for CSCtx60271: http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtx60271   N3k-A# show queuing interface eth 1/1 Ethernet1/1 queuing information: TX Queuing qos-group sched-type oper-bandwidth 0 WRR 100 RX Queuing qos-group 0 HW MTU: 1500 (1500 configured) drop-type: drop, xon: 0, xoff: 0 Statistics: Ucast pkts sent over the port : 0 Ucast bytes sent over the port : 0 Mcast pkts sent over the port : 0 Mcast bytes sent over the port : 0 Ucast pkts dropped : 0 Ucast bytes dropped : 0 Mcast pkts dropped : 0 Mcast bytes dropped : 0 N3k-A#   Further Information on Nexus 3000 Nexus 3000 – Products Data Sheet http://www.cisco.com/en/US/products/ps11541/products_data_sheets_list.html Software Download page http://www.cisco.com/cisco/software/navigator.html?mdfid=283734368&i=rm Relevant Document Nexus 3500 Series Switch Platform System Health Check Process http://www.cisco.com/en/US/products/ps13383/products_tech_note09186a0080c1e9f2.shtml For comments and feedback, please contact the author Yogesh Ramdoss at yramdoss@cisco.com ... View more