Cisco UCS fabric port alerts

[Quintin.Mayo]

Hi,

We have a Cisco UCS 6248UP device running software version: 3.1(3c). When investigating we found this specific alert "TCA: etherNiErrStats crcDelta for sys/chassis-2/slot-2/fabric/port3 current value = 5, raised above esc value 5". Errors are on interfaces Chassis2 Slot 2 ports 1-3. They would clear for about 5 minutes, and then start popping errors again.This is  between the FIs and the IO modules, core switch does not appear to be having any issues. I have attached the output for both the A-B FI's for the below commands. Can someone assist with the analysis of the output and appropriate plan of action to be taken? Any assistance would be greatly appreciated.

- show hardware internal carmel crc

- UCS-B(nxos)# show hardware internal carmel eye

- UCS-B(nxos)# show hardware internal carmel counters interrupt | egrep "(statistics|mtu_vio)"

Thanks,

. 

[Wes Austin]

Based on the fact this is occuring on both fabrics, you likely have an MTU violation occurring in your upstream network. When >1500 MTU packets are being stomped with CRC, they are being forwarded down to UCS-FI where we are reporting them.

What does your QoS classes look like on your UCS?

You can probably increase your best-effort class to 9216 MTU, but may also investigate upstream to understand what devices are sending >1500 MTU to your UCS environment.

[Quintin.Mayo]

HI,

Thanks for the advice. There isn't a QoS policy in place. Would you recommend us create a QoS policy with Priority?  Also, can you inform on the Best Effort, Burst: 9215, Rate: line-rate, Host Control: None or Full setting? Once again, the assistance is greatly appreciated.

Thanks

[Wes Austin]

This would be a QoS policy on the Fabric Interconnects, not the vNIC.

LAN Tab > QoS System Class > Best Effort > Adjust MTU from 1500 to 9216.

While this should not cause disruption, recommended to change this during maintenance window.

[Quintin.Mayo]

Hi,

We have another question we moved off of our 5548 directly to a new core switch last Thursday. Before last Thursday the FI's went to the 5548s before going to the core switch. The 5548s didn't have MTU configured so pretty sure default MTU is 1500, the old core switches were set to MTU 1500. The new core switches MTU is not configured and most likely default to MTU 1500. So with all equipment potentially rocking MTU 1500 is the recommendation to configure just the UCS to 9216 MTU or on all equipment involved (i.e core switch, etc)?

Thanks

[Wes Austin]

If you have interfaces that are using MTU >1500 upstream, you would need end to end 9000 MTU to avoid the CRC errors.

If you just want to clear the faults in UCSM, you can make the adjustment in UCSM. If this adjustment yields positive results and the errors stop, you know that you are sending jumbo MTU from some source in the network down to some destination(s) within the UCS domain.

Your mileage may vary.

[Quintin.Mayo]

Hi,

Thanks for all the direction. Just to confirm we can resolve the UCS side errors by adjusting to 9000 MTU thru QoS and that won't have an impact or issue with the upstream switch still at 1500 MTU or do they need to be done on both sides...? A little confused on that. Also, if all equipment is 1500 MTU why did this happen?

Thanks,

[Wes Austin]

Without looking into your environment, it is hard to understand why this is occurring and confirm adjusting the QoS will resolve the issue. I would recommend you open a TAC case if you need further investigation.

Based on the information you provided, one possible solution is to adjust your Best Effort class in QoS System Class to 9216 MTU. You would need to work with your networking team to understand what traffic is being sent to the UCS domain.