C220 M3 Dead CIMC? (Server won't POST)

[DRAGONKZ]

I'm just after some advice regarding my C220 M3 server which I'm using as part of my home lab.

I bought it a few weeks ago and upgraded it from original release firmware (2011 era) to the latest firmware and everything has been working fine.

I shut it down the today to add a VIC 1225 CNA and since having done so the server no longer POSTS.

I have removed the VIC and still have the same problem. (I also tried a second VIC 1225 which had the same symptoms)

When I power on the server the fans essentially run at full speed, there is no display, and there is no "beep" that normally occurs for connected USB devices. (I left it for 30 mins at one point and it just had the fans going at full speed with no display the whole time)

I also get no sign of life from my CIMC management port in terms of activity lights or ping, and connecting via serial also displays nothing.

The LEDs on the front panel are all green, and performing an action that should normally throw an amber light (like removing a PSU) doesn't result in this... all the LEDs stay green.

I'm assuming that my motherboard/CIMC has coincidently died at the same time I installed the VIC?

Are there any other forms of troubleshooting that I can try, or am I going to need to start looking at getting another board (or buying a smartnet contract).

Thanks 

[Leo Laohoo]

If you connect to a console cable at the back (not the front), what do you get during boot-up?

[DRAGONKZ]

I used the console cable from my Cisco SG3x0 series switches with an Aten USB to serial convertor.

(The above works on my Dell 8024F switches with no problems)

I get no sign of life from the console port... nothing is displayed on the session console and no activity lights on the console cable port either.

[Kirk J]

Greetings.

The status LED lights not reflecting device changes like the PSU, does not sound promising :(

Try doing the J37 jumber CMOS/BIOS clear action, seen at 

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C220/install/C220/install.html#78729

Pull absolutely all components out but the CPUs and a DIMM in each CPU bank (A1, D1).

Make sure the PCI-E riser card is reseated...

Are any of the systemboard status lights amber (see https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/c/hw/C220/install/C220/replace.html#21803 )

Kirk...

[DRAGONKZ]

Resetting the CMOS shows no change, as does pulling all the RAM bar 1 stick per CPU.

All lights are green when the power is on. (Board, front panel, PSUs and LOMs)

Does the CIMC fail to operate if there is a CPU failure? (If so, does it only need CPU1 as I could swap the CPUs or pull 1 and test one by one)

I’m guessing it’s CPU or board failure :(

[Kirk J]

Even if the CPUs have issues, the CIMC itself (which has a small CPU, memory, storage of its own) should still come up.

I would try swapping CPUs, to see if some equipment is causing the systemboard to be in an odd state.

My guess would be this is a systemboard issue.

Kirk...

[DRAGONKZ]

Ok, I’ll try that and report back.

In terms of a board replacement, I imagine it’s the same as other tier 1 OEMs processes?

Ie, remove components from dead board, remove dead board, install new board, install components, re-configure bios settings, upgrade firmware, import RAID config... and ideally good to go.

Will a replacement board be smart enough to see/import a RAID1 FlexFlash config, or would that need to be redone?

Thanks

[Brandon Rumer (Cisco)]

I think I am wrong on this recommendation, but I didn't see it here so I might as well put it out there. I assume that since you've swapped hardware you completely unplugged the server? I've seen multiple C220m3 servers have a problem (don't have the bug ID handy right now) and the fix is to UNPLUG the entire server for a few mins, then plug everything back up.

[DRAGONKZ]

I did try that to no avail, thanks for the suggestion thought!

[DRAGONKZ]

Long story short, Cisco wanted USD $200,000 for a replacement board since there was no contract, and the server was too old to purchase a new contract... so I've sourced a second hand board from the USA and it just arrived today.

It's the same revision as mine, and is only a few hundred numbers higher than mine serial number wise.

I'm hoping the swap-over should be painless, and the server will just boot as per normal.

One further question I have is regarding the chassis serial number in the BIOS... can this be updated on this "new" board to reflect my chassis serial number?

(I know in the HP and Dell world it can either be done via the BIOS, or via special toolkits...etc)

Is there a way to do this in the Cisco world?

Thanks

[Wes Austin]

The chassis serial number is burned into the hardware and does not transfer over. Once you boot the server with the replacement CPU, components, etc....the serial number will be the new board.

[DRAGONKZ]

Sorry, I used the wrong terminology.

I’ve now got the system running again, and can access the CIMC, so what I was referring to were the “Product Name” and “PID” values displayed in the CIMC summary tab.

Is there a way to change these values to reflect what that should be (and used to be)?

Thanks

Edit - I’m not too fussed about the serial number, but out of curiosity what do the techs do when replacing boards? Surely when these things are being replaced under warranty/contract the serial number has a way of being updated?

[Wes Austin]

When the board is replaced, for support, we update the contracts to reflect the replaced SN so support is not denied if a case is opened at a later date.

Are you saying the PID on your new server is not the same PID as your old server? The PID is also burned into the SEEPROM and cannot be changed. The value should be UCSC-C220-M3.

[DRAGONKZ]

I bought the board second hand from a refurbisher in the USA and had it shipped to AUS.

There are are barely any boards available second hand so there is not much choice when it comes to replacing. ($200,000 US for a board through Cisco entitlement team is a tad unrealistic... :) )

From a pure inspection perspective, there is no way of telling what system the board is from unless you fire it up and check the CIMC.

Media engines and Nexus VSAs were all based on the same C220 M3 system, they just appear to have had different Product names and PIDs.

In this instance the board appears to be from a “Cisco Nexus Virtual Services Appliance” with a pid of “N1K-1110-X”.

The board part number is identical, and assuming the serial numbers for systems are done sequently, this boards number and MAC address are within 60-80 from mine which means they were likely made on the same day/week. (Ie, only the last 2 digits of the serial number and MAC address are different, and they aren’t too far off each other)

Considering the part number is the same, that means in the Cisco spare part warehouses there has to be multiple piles of boards, all with the same part number, but they are seperated because they need to have the PID/description of the correct device they are going in to?

At the end of the day it’s a cosmetic issue, it just strikes me as strange as all the other tier1 OEMs have processes to change the baked in info via CLI or GUI tools.

Thanks