Recover FI

[Michael King]

We took a extended power failure, and one of our FI's didn't come back.

We reset the power on it, hoping it would come back, but it didn't.

Connecting a console on it, and booting it again has exposed the problem as a corrupt image.  The question I have is which one?  Am I past the 3 special firmwares I have to create a TAC case on.

I can tell that it successfully boots the kickstart image:

 Booting kickstart image: bootflash:/installables/switch/ucs-6100-k9-kickstart.5
.2.3.N2.2.23f.bin....
...............................................................................
..............................................Image verification OK

It appears it's the actual image:

Uncompressing system image: bootflash:/installables/switch/ucs-6100-k9-system.5.2.3.N2.2.23f.bin
Loading plugin 0: core_plugin...
Loading plugin 1: eth_plugin...
Loading plugin 2: fc_plugin...
/bin/dd: invalid number `Error'
gzip: stdin: unexpected end of file

Then LOTS of errors, before I finally end up here:

2015 Dec 1 13:25:46 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: sam_core_mon crashed with crash type:256
2015 Dec 1 13:25:46 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2015 Dec 1 13:25:46 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: sam_core_mon - pmon
2015 Dec 1 13:25:47 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Running in PIO stats mode - carmelusd

---- Basic System Configuration Dialog ----

 Enter the configuration method. (console/gui) ?

It's a lab system, so I have the luxury of screwing around with it without it impacting anything.

My thoughts are:

1. Disconnect the L1/L2 cables (actually, all cables)
2. run thru the console setup
3. Connect to the GUI
4. upgrade (?) to the same version already on it
5. Factory reset it
6. Reconnect L1/L2 (and all cables)
7. cross fingers?

Sound like a plan, or am I missing an easier way?

[Wes Austin · Accepted solution]

Hey Michael,

It sounds like you are hitting one of these defects:

https://tools.cisco.com/bugsearch/bug/CSCut63966/?reffering_site=dumpcr

https://tools.cisco.com/bugsearch/bug/CSCus79160/?reffering_site=dumpcr

You can attempt the workarounds you outlined, but if you still see all services crashing upon reboot, you will need to repair the filesystem. You can open a TAC case to accomplish this.

Let me know if you have further questions.

--Wes

[Qiese Dides]

Hi Michael,

So that is a way to do it. When you plug in L1 and L2 cables and ask the Subordinate FI to join a cluster it would automatically upgrade the firmware if it was out of date. The method you are doing can work, but you might have to rebuild the Fabric Interconnect from scratch if it doesnt. That would require getting the kickstart, system, and manager images. This is shown in the defect below;

https://tools.cisco.com/bugsearch/bug/CSCut63966/?reffering_site=dumpcr

If you're method doesn't work and it is still failing you can message me directly and I can help get that started for you.

Regards,

Qiese Dides

[gruberwilli]

Hi,

after an upgrade attempt, my FI 6248up is becoming inaccessible via console. It is simply doing nothing. I don't believe, that a such equipment is not recoverable. I did recovery of many other switches, but they gave me at least console access. This is not the case. Is there any procedure I can follow to revive this system? It will be very appreciated.

TIA, Willi