Post upgrade errors

[marcusauman]

I upgraded a UCS B series system from 2.1(2a) to 2.2(3d) and i've receiving some errors and I was hoping someone could help me resolve them. Below are the errors i'm seeing.

- Code F0315 - Service profile failed to associate to server

- FC uplink down on VSAN 1001 (storage is still working)

- When i go to the service profile and look at the FSM its stuck at 98% and says "Failed: Local storage Controller is in activating state sys/chassis/-X/blade-X/board/storage-sas-1

Any ideas? 

[Walter Dey]

Hi Marcus

What kind of blade is it ? I assume you had installed the OS on the local disk ?

Is this the only blade with this error ?

Can you please post the firmware information of this blade ?

[DAVE GENTON]

  Hey Walter Marcus is tied up so I'm looking into this issue...   No the disk is not local, it has no local drives and boots from FC SAN.   As Marcus stated the blade boots up via FC despite the error stating FC link is down on respective VSAN.   Blades are all up and running now as they were prior to this upgrade with versions in original post.  The only difference is these errors continually running out for every service profile stating that it cannot associate due to firmware in activating state.  

 Post upgrade this error and firmware of local storage controller stuck in activating state is for EVERY Service Profile, affecting every blade in production, with these "errors".

  Firmware for the local storage controller appears to be stuck in 'activating' state for some reason.  I have never saw this and did this upgrade many times, and gave the version to Marcus to use in this instance..   Blades are all B200-M2's controller is LSI MegaRAID SAS 2004 ROMB rev B2,  all normal stuff I see each and every week..  Customer is running in production with upgrade completed they requested however they want the errors cleaned up and for them to stop before moving forward.   Not finding anything on this online, hoping someone here/tac can assist. 

Screenshots attached of each status/error and versions grouped below

Upgraded 2.1(2a) to 2.2(3d)

Both running and startup version for controller are 20.11.1-0135

bootloader version 4.35.00_4.12.05.00_0x05270000

Activate Status: Activating

[Walter Dey]

Hi Dave

I have seen this error before: FC uplink down on VSAN 1001 (storage is still working)

It is informational only, very misleading ! everything is working ok.

Regarding the other problem: Do I understand you ok, the blades are working, but the SP association is failing ? Can you please post the Firmware (BIOS, CIMC,....) of a blade.

[DAVE GENTON]

Walter that is correct and I and Marcus have both posted the versions, multiple times.  The system was upgraded to 2.2(3d) and the result of that upgrade has caused all of these errors:

ERROR:

Service Profiles failed to associate to blade - this is on every blade in the system

REASON:

  Local Storage Controller firmware is stuck in 'activating' state.

  As it says, its 98% complete and then hangs there on every blade.  

  They did indeed boot up since they are FC SAN Boot and it did say FC uplink down for vsan but its not.   I've done this same code upgrade on other systems without this issue, but now its continuing to throw out errors customer wants fixed and I dont blame them.  All other firmware updates completed without issue, so you know all the versions.

  Anyone familiar with this or aware of issue ???

thanks in advance,

Dave

[Venelin Georgiev]

Hi Marcus

Two things  that may related to the issues you hit.

1) There is a issue in the way that "Management Firmware Profiles" on 2.1 and 2.2 are handled by the UCSM. If you have used "Management Firmware Profiles" and they are grayed out on Firmware Policies of the service profile (see the screenshot) then Remove Management Firmware Policy and that will allow the blade to associate.

What happens behind the scene is: The UCSM actually considers the settings in the grayed out Mgmt FW policy and sets the FW of the blade CIMS to 2.1 and then the blade is stuck at associating state. The error code in that case is F0315.

So check that and just click the Clear Button.

2) Before a week we have hit another nasty bug.- CSCui90080

Today Cisco made it customer readable you can find it here (https://tools.cisco.com/bugsearch/bug/CSCui90080/?reffering_site=dumpcr) This one is related to B230M2 but it may afect some other Blades also, I am not sure...

Try to acknowledge the blade to see if that will fix the Storage Controller issue. If you are hitting that CSCui90080 the a simple re-ack will help to resolve the issue

I hope that helps

[Walter Dey]

Just for the understanding:

- In the past, Management Firmware Packages were used essentially to upgrade CIMC

- as per the screenshot, MFP's became obsolete

- when a SP was referencing nevertheless a MFP (and had in Addition a reference to a Host Firmware package), the MFP was overriding the HFP; resulting that CIMC (and all the other components) was not updated.