Cisco UCS C240 M4 - RAID arrays vanished after firmware update

[mchappell]

This morning I received an alert that my backup server (standalone UCS C240 M4 running windows) had run out of disc space on C and had shut down.  The C drive is a mirrored set of 120Gb SSD discs and the storage array is a number of 1TB sata drives in RAID 5 (about 8TB total).

I found information from Cisco that SSDs can appear to fail after 40k usage hours and that the issue can be resolved through upgrading the firmware.  So I proceeded to upgrade the firmware from 2.0 to 3.0.2.  The firmware upgrade appeared to go through with no issues, all steps passed.  Upon restart however, the system failed to see any of the RAID volumes and all the LEDs on my drives are now dark.

Looking at the the raid controller settings, it looks like nothing is there and I do not have the ability to change anything in the config.

So now my backup server is completely dead in the water and as such, we are not able to backup our infrastructure.  I'm fearing that I'm going to have rebuild this entire server by hand which will be a huge pain. This is the 4th firmware update I've performed on a UCS and this is the first time I've had this issue.

Any help or suggestions are appreciated.  Thanks

[Wes Austin]

Did you upgrade the RAID controller firmware when you upgraded the other components? 

Did you upgrade the disk firmware when you upgraded the other components?

Are you able to revert the upgrade and see if the controller/disks show correctly?

When you boot the server do you see information about the RAID controller/disks when the server is going through POST? Are you able to check the disk status from the LSI Webbios?

Any screenshots or captures you can provide would be helpful

[mchappell]

Hi, thanks for responding.

I did upgrade the RAID controller and the disk firmware when the other components were upgraded.

I have not yet tried to revert the firmware upgrade to v2.0(?). Where would that be available?

There are currently NO physical discs being found by the system, no SSD drives, and no SATA drives on the controller.  I am attaching screenshots of everything you've asked.

I am so hoping that there is a resolution to this.  At this point, we are even willing to start over completely in order to rebuild from the ground up, but we need the system to actually SEE the discs first.

[mchappell]

Attached is the screenshot from the Megaraid config under "F2-Operations"  None of these choices provided do anything.

[mchappell]

Reverting the firmware had no change.  There are ZERO hard disks detected by the unit at all.  Choosing "upgrade HDD" detects no devices to upgrade.  This is extremely frustrating and has put our organization in a very bad situation.

[Wes Austin · Accepted solution]

Have you tried to reseat any of the disks to see if they show up after? 

Are you able to remove power from the server for 30 seconds and see if the disks show back up after power cycle?

You can look in the CIMC web interface under the Sensors information and see if the HDD/SSD are being detected in the disks slots or not. I suspect if a power cycle does not resolve, there may be an issue with the backplane or the cable that connects the controller to the backplane.

Depending on your support contract, it would be advantageous to open a TAC case to investigate as well.  

[mchappell]

Hi,

All drives have been re-seated with no change.  I removed power (disconnected power cables) for more than 2 minutes, with no change.

I do not see any HDD/SSD in the disk slots under CIMC Sensors.

So all signs are pointing to this being a hardware issue with the backplane.