CSCvj32984 - VIC 1340 on 3.2(3a) utilizing FC Storage under high load hangs/halts OS

[Uday]

Hi,

I have seen a similar issue ( not exactly the same ) with one of our customers who is running on .3.2(2d) version, is that an affected version too?

[cvalencia01]

What version of ESXi is affected? Is it all versions or just a specific version?

[Uday]

Hi,

It was 6.0U2 , several of his hosts went down.

[Andrew Kelly]

Hello,

Any Operating System could be impacted by this issue. 

Thank you,

Andrew

[ajkuftic]

Are there any non-VMware OSes reported to have had the issue? I get the issue is driver-firmware related and that any OS is possibly affected, but have there been any reports of the issue with Microsoft or Linux operating systems so far?

[Andrew Kelly]

Hello Uday,

CSCvj32984 only impacts VIC 13xx running adapter firmware 4.2(3a). This firmware is part of the 3.2(3a) and 3.2(3b) UCSM Blade and Rack bundles. The 4.2(3a) adapter firmware was also included in the 3.1(3a) and 3.1(3b) Standalone M5 Racks bundles. 

It is possible to see some of these same symptoms, such as some type of aborts in ESXi logs, with a different root cause. Open a TAC case and attach the UCS and Operating System logs so we can take a closer look.

Thank you,

Andrew

[ajkuftic]

If we upgrade to 3.2(3a) but exclude the adapter firmware, we shouldn't see this issue manifest, correct? We'd like to get the 3.2(3a) BIOS loaded for the Spectre microcode update, but don't want to introduce an FC issue at the same time.

[Andrew Kelly]

Hello,

The recommendation would be to upgrade the entire blade or rack server to an earlier release that is not exposed to CSCvj32984 but still has the Spectre fixes. 

For instance, the 3.2(2f) release includes the Spectre fixes and is not exposed to CSCvj32984.

Check this out for more info on Spectre:

https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180104-cpusidechannel 

Thank you,

Andrew

[ajkuftic]

Just to confirm, if we're running 3.2(3a) at an infrastructure level, we can run 3.2(2f) on the blades and be OK.

That sounds a like a perfectly fine plan, but I just wanted to make sure this was the case.

[Andrew Kelly]

Hello,

Yes - with UCS Infrastructure running 3.2(3a) the blades are still supported on 3.2(2f).

Cross firmware compatibility info is always documented in the release notes. Here is the link to the section within the 3.2 UCS Manager Release Notes:

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/release/notes/CiscoUCSManager-RN-3-2.html#concept_4B24EBA51A3940EEA3AB50860D963F57

Thank you,
Andrew

[squebel]

@Andrew Kelly we just experienced this bug last week and had multiple ESXi 6.0U3 hosts hang so hard that we had to force power cycle them. VIC firmware was already at 4.2(3b) BUT the FIs and IOMs were only at 3.2(2b). Do you think we experienced the bug because the full UCS stack was not up to at least 3.2(3d)?

[Andrew Kelly]

Hello,

Once the VIC is updated you are no longer exposed to CSCvj32984. This is because the firmware fix for the defect is included in the VIC firmware. There is no change to the FI or IOM infrastructure components. 

I would gather logs and reach out to Cisco TAC and VMware to analyze the hang experienced. 

Thank you,

Andrew

[squebel]

Thanks. We do have a TAC case open as well as a case with Dell/EMC and they have analyzed the logs and of course the finger pointing has commenced. The original poster mentioned having the same issue even on 3.2(3d) which was the same for us. Everything in the logs looks very similar to this bug. It may not be the same thing but it's very close.

I will say this too: we upgraded another Dell SC 8000 array a few weeks prior and had no issues. They were still UCS blades attached via fibre channel BUT they were using VIC 1240s. The only stack we've seen this problem with was fibre channel on VIC 1340's even on the proper firmware.

[clindoan1]

Hi Andrew,

   We are still using the older firmware version for our customers on the FC Adapter 1340. Are firmware versions 1.6.0.36 and 2.3.0.14 impacted by issue discussed in ticket CSCvj32984? Our company have released UCS bundle version 3.2(3a) to our customers for the spectre/meltdown fixed but DO NOT upgrade the VIC 1340 firmware, meaning stay at FW version 1.6.0.36 and 2.3.0.14. Can you please give me your comment regarding to the scenario we have. Thanks.

Regards,

-Clint

[Stephan Cook]

What fnic drivers were they using?