UCS FI upgrade issue IOM becomes inoperable probably due to wrong procedure

[dganta]

HI Team,

We are doing an installation of UCS Bseries servers with 5108 chassis and 6296UP Fabric Interconnects. Was trying to upgrade the firmware version of all the components to 3.1(1e) from 2.2(2C). During the upgrade of the secondary FI-B after checking the FSM progress to 100 immediately upgraded the Primary FI A and changed the cluster lead to B, however I missed an important step of checking the FSM status of the IOM modules and also did a immediate cluster failover to B with the IOM not getting upgraded. Right now I see that the IOM module shows inoperable and the IOM module continuously flaps. Would like to know how can I resolve this issue. Would like to get your help on how to get the IOM module which is connected to FI-B up

[Walter Dey]

Questions:

- did you read http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Firmware-Mgmt/3-1/b_UCSM_GUI_Firmware_Management_Guide_3_1.html

- did you read http://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/CLI-User-Guides/Firmware-Mgmt/3-1/b_UCSM_CLI_Firmware_Management_Guide_3_1.html

- I assume you did a manual upgrade, not a autoinstall infrastructure (which I would recommend)

- did you upgrade UCSM first ?

- did you update AND activate the IOM ?

- which version is the IOM now ?

[dganta]

HI,

Yes went through that. I upgraded the UCSM first and then the FI when we activate on the FI it will upgrade the IPMs as well. There are 2 chassis with 2 IOMs in each the other 3 IOMS and the 2nd IOM on the same chassis is upgradedd to 3.1(1e) only this IOM module is showing the problem and is still on 2.2 version and right now the biggest concern is not the version but the IOM module is inoperable so want to make the IOM module up irresepective of the version and then take it from there.

[Walter Dey]

Try to do a IOM reset (soft), and if this doesn't help, I would remove the IOM from the chassis, wait 30 seconds, then reinsert it.

If this doesn't help, I would open a TAC case; maybe if you have a spare IOM, try it.

[dganta]

Hi Walter,

Thanks can I only upgrade the firmwqare on the IOM as the iOM is running a 2.2 version where as all the other IOMs are in 3.1(e) or should I be doing it with the FI's. Right now FI is in 3.1(1e) I am planning to restore it to a previous 2.2 version to see whether the IOM becomes operable if it does not work with soft or hard reset

[Walter Dey]

Hi 

I would be careful with downgrading ! 

What is the running / startup / backup version of the failing IOM ?

Usually, the FI will enforce the version on the IOM, and download it automatically.

On the regular upgrade path, firmware is downloaded to IOM and then activated with attribute: pending next boot. Once the Northbound FI is updated and boots, it will also boot the IOM.

Therefore try to do soft and/or hardware reset.

[Qiese Dides]

Hi Dganta,

Did you try to re-seat the IO Module then attempt to re-acknowledge the chassis once the upgrade gets complete. This will help re-initialize the link communication between the IOM and the Fabric Interconnect which then should get it out of that inoperable state.

Regards,

Qiese Dides

[dganta]

Hi Qiese,

Yes I did try that however the IOM module still shows inoperable and I can see that all the ports on the IOM module -B blinks continuously. As the IOM is inoperable the upgrade is stuck. Finally we decommisioned and recommisioned the chassis and then the IOM module itself is not getting detected so I had to configure all my service profiles with only 1 IOM module as the 2nd IOM module is not detected. As of now the blades are all up with service profiles but with only 1 IOM module. I do not know whether it has got to do with the upgrade procedure or some hardware issue with the IOM module

[Qiese Dides]

Can you upload a screenshot of the error message?