Nexus 5Ks to 2960-X stack CRC errors

[neteng2323]

I have 2 5Ks in a peer-link config. They are the core\distribution switch for a building. I have 5 2960-X stacks connected to them in a 20G port channel.  Each leg of the port channel connects to 1 5K in a VPC.  For some reason, no matter how I configure these connections I'm getting a lot of INPUT and CRC errors on the second 5K (5K2) consistent across all stacks.  I don't think this is a isolated layer 1 issue.

Things I've tried...

Changing SFPs

Changing Ports

Changing Cables

Changing PC configs on Stacks

Changing VPC\PC configs on 5Ks

The stacks are passing traffic, and port channels stay up.  The connection seems ok, although pings look a little dirty at times.

[Philip D'Ath]

Are both ends reporting the same duplex?

Are the 5k's using a reasonable software version?

[neteng2323]

I don't see any duplex or speed mismatch errors on either side... Here's results from show ver

Software
BIOS: version 3.6.0
loader: version N/A
kickstart: version 7.0(7)N1(1)
system: version 7.0(7)N1(1)
Power Sequencer Firmware:
Module 1: version v3.0
Module 2: version v1.0
Module 3: version v5.0
Microcontroller Firmware: version v1.2.0.1
SFP uC: Module 1: v1.1.0.0
QSFP uC: Module not detected
BIOS compile time: 05/09/2012
kickstart image file is: bootflash:///n5000-uk9-kickstart.7.0.7.N1.1.bin
kickstart compile time: 8/24/2015 22:00:00 [08/25/2015 02:56:43]
system image file is: bootflash:///n5000-uk9.7.0.7.N1.1.bin
system compile time: 8/24/2015 22:00:00 [08/25/2015 08:46:21]

Hardware
cisco Nexus5548 Chassis ("O2 32X10GE/Modular Universal Platform Supervisor")
Intel(R) Xeon(R) CPU with 8253868 kB of memory.
Processor Board ID FOC18254FBP

Device name: GSODMASWI01
bootflash: 2007040 kB

Kernel uptime is 0 day(s), 22 hour(s), 57 minute(s), 21 second(s)

Last reset at 387947 usecs after Mon Dec 28 16:35:48 2015

Reason: Reset Requested by CLI command reload
System version: 7.0(7)N1(1)

[Philip D'Ath]

The switch running the software currently recommend by Cisco.  If you swap the port channel members over on the 5k's, does the issue stay on the same 5k port or move to the other 5k?

[Rajeshkumar Gatti · Accepted solution]

The 5ks do cut through switching by default. If the 5k receives a bad/corrupted frame from downstream switch, it will NOT drop it but stomp it and forward it to the upstream device in the path. The corresponding CRC error counters will increment on the receiving port at the first hop.  The upstream switch will also do the same (increment CRC counter) when it receives this corrupted frame which will make troubleshooting difficult if you are not familiar with this concept. So even if you change the cable/sfp etc on this intermediate device, it will not solve the issue. What you need to find is the entry point of these bad CRC packets.

See this cisco live presentation slide 54-76 (BRKCRS-3145) on how you can trace the source of the bad packets.

The cisco live presentations are also available at http://www.ciscolive.com. You have to create a login which is free and you can have access to a ton of good information. This particular slide deck can be searched using BRKCRS-3145.

Hope this helps.

-Raj

[neteng2323]

I did some reading on this... So if the 5Ks are the core of the network for this building and the stacks are the edge (access layer), and the input and crc errors are being seen on the stack uplink interfaces that would point to the 5K as the source, correct?  From the 5K back to the campus core it is layer 3 so layer 2 stops at the 5K. We didn't see these errors before when we had a 6509 in as the core\distribution switch.

[Rajeshkumar Gatti · Accepted solution]

From what you are saying it appears that the 5k is receiving this bad packets and sending it down to the stack switches. What you need to find is which port it hits first cuz once it enters the switch then it gets stomped and gets propagated to wherever it is destined for which in this case appears to be hosts connected to the switch stack. Use the slide deck and you will be able to identify and mitigate this issue. Mitigation may include swapping cables/SFP etc at the entry point of these bad packets.

[neteng2323]

Thank you, this was it!  I found the only interface on the 5K with RX  input and crc errors, and traced that back to a bad fiber patch.  Shutting the port it was connected to, cleared up the errors on all other switches which gave me my answer right away.