Cisco UCS VIC adapter link down errors

[midhun.cp]

Hi All
I have seen so many cases where blades will report link down errors as below.

2016 Nov 28 19:05:22  %UCSM-3-LINK_DOWN: [F0283][major][link-down][sys/chassis-2/blade-1/fabric-A/path-1/vc-8907] ether VIF 8907 on server 2 / 1 of switch A down, reason: Error disabled
2016 Nov 28 19:05:22  %UCSM-3-LINK_DOWN: [F0283][major][link-down][sys/chassis-2/blade-1/fabric-A/path-1/vc-713] ether VIF 713 on server 2 / 1 of switch A down, reason: Error disabled

Sometimes adapter restarted messages causing the links to go down but blade not rebooted.

Description: Adapter 2/1/1 restarted

A reboot or re-discovery of the blade will resolve the issue. My question is how to identify whether it is an issue with the hardware or driver mismatch/operating system?

What keywords/errors to check in Mezzanine logs to confirm hardware failure? For one of the case I could not see any ERROR,PANIC, CRASH, CORE messages on obfl logs.

I know few bugs such as below can cause this issue.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCus07107
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy43473

Sometimes I can see below messages in obfl logs. Does this mean hardware issue?

mcp.cfgblk ERROR: Failed to read config block

Looking for an overall idea for analyzing the logs to isolate hardware issue.

Thank you.

[midhun.cp]

Any thoughts guys?

[Wes Austin]

Hello,

Ideally the first thing you can check is your OS software/hardware compatibility to ensure that is in spec.

Do you see this issue on the same blade? or is it multiple blades across the environment?

If the adapter restarted, I would suspect you see link down messages for all veth interfaces and not just 1-2 on a single side (A or B).

The error message you mention does not indicate any failure. You also mention two bugs, but are you on impacted firmware versions for those defects? Do you see the same errors messages as mentioned in the defects.

You would check OBLF and SEL logs for signs of hardware problems with the motherboard or other components and check the MEZZ logs for any particular errors with the VIC adapter.

HTH,

Wes