Cisco UCS Fabric Intercconect boot failure

[muusitalo]

Hi!

We have an old UCS system that had been unused for two years. When we tried to brig this old system up, the Fabric Interconnect switches did not boot up correctly. Both switches acted similarly; all ports on both switches remained orange.

HAs anybody any info on how I should proceed in bringing these switches in working order?

Here are the bootup messages from one switch:

Booting kickstart image: bootflash:/installables/switch/ucs-6100-k9-kickstart.5
.2.3.N2.2.23d.bin....
...............................................................................
..............................................Image verification OK

ÿUsage: init 0123456SsQqAaBbCcUu
INIT: [ 10.658342] I2C - Mezz absent
Starting system POST.....
Executing Mod 1 1 SEEPROM Test:...done (0 seconds)
Executing Mod 1 1 GigE Port Test:....done (32 seconds)
Executing Mod 1 1 PCIE Test:.................done (0 seconds)
Mod 1 1 Post Completed Successfully
POST is completed
can't create lock file /var/lock/mtab~207: No such file or directory (use -n flag to override)
S10mount-ramfs.supnuovaca Mounting /isan 3000m
Mounted /isan
Creating /callhome..
Mounting /callhome..
Creating /callhome done.
Callhome spool file system init done.
nohup: redirecting stderr to stdout
autoneg unmodified, ignoring
autoneg unmodified, ignoring
Checking all filesystems..... done.
Checking NVRAM block device ... done
The startup-config won't be used until the next reboot.
.
Loading system software
Uncompressing system image: bootflash:/installables/switch/ucs-6100-k9-system.5.2.3.N2.2.23d.bin

Loading plugin 0: core_plugin...
Loading plugin 1: eth_plugin...
Loading plugin 2: fc_plugin...

/bin/dd: invalid number `Error'

gzip: stdin: unexpected end of file
/bin/tar: Child returned status 1
/bin/tar: Error exit delayed from previous errors
ethernet end-host mode on CA
FC end-host mode on CA
n_port virtualizer mode.
---------------------------------------------------------------
INIT: Entering runlevel: 3
touch: cannot touch `/var/lock/subsys/netfs': No such file or di
/isan/bin/first-setup.core: line 2040: /isan/etc/common_defs: No such file or directory
/isan/bin/muxif_config: fex vlan id: -f,4042
Set name-type for VLAN subsystem. Should be visible in /proc/net/vlan/config
Added VLAN with VID == 4042 to IF -:muxif:-
cp: cannot stat `/isan/plugin_img/fex.bin': No such file or directory
ERROR: Failed to upgrade SAM CLI commands. Details in /tmp/sam_upgrade_samcli.log
2020 Oct 22 07:11:05 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: loading cmd files begin - clis
2020 Oct 22 07:11:08 %$ VDC-1 %$ Oct 22 07:11:08 %KERN-0-SYSTEM_MSG: [ 10.658342] I2C - Mezz absent - kernel
2020 Oct 22 07:11:15 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: loading cmd files end - clis
2020 Oct 22 07:11:15 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: CLIS: init begin - clis
2020 Oct 22 07:11:20 %$ VDC-1 %$ %SNMPD-2-CRITICAL: SNMP log critical : load_mib_module :Error, while loading the mib module /isan/lib/libsvc_sam_extSnmpPlugin.so (/isan/lib/libsvc_sam_extSnmpPlugin.so: cannot open shared object file: No such file or directory)
2020 Oct 22 07:11:33 %$ VDC-1 %$ %PFMA-2-PS_FAIL: Power supply 1 failed or shutdown (Serial number ART1611X08G)
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:256
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:256
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:256
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:256
2020 Oct 22 07:12:01 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:02 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:256
2020 Oct 22 07:12:02 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:02 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:256
2020 Oct 22 07:12:02 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:02 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:256
2020 Oct 22 07:12:02 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:03 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:256
2020 Oct 22 07:12:03 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:03 %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:256
2020 Oct 22 07:12:03 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:03 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:256
2020 Oct 22 07:12:03 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:04 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:256
2020 Oct 22 07:12:04 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:04 %$ VDC-1 %$ %CALLHOME-2-EVENT: sfcbd crashed with crash type:256
2020 Oct 22 07:12:04 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:04 %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256
2020 Oct 22 07:12:04 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:05 %$ VDC-1 %$ %CALLHOME-2-EVENT: sam_core_mon crashed with crash type:256
2020 Oct 22 07:12:05 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:05 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:256
2020 Oct 22 07:12:05 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:05 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:256
2020 Oct 22 07:12:05 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:256
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:256
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:256
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:256
2020 Oct 22 07:12:06 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:07 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:256
2020 Oct 22 07:12:07 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:07 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:256
2020 Oct 22 07:12:07 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:07 %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:256
2020 Oct 22 07:12:07 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:256
2020 Oct 22 07:12:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:256
2020 Oct 22 07:12:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: sfcbd crashed with crash type:256
2020 Oct 22 07:12:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256
2020 Oct 22 07:12:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: sam_core_mon crashed with crash type:256
2020 Oct 22 07:12:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:256
2020 Oct 22 07:12:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:256
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:256
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:256
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:256
2020 Oct 22 07:12:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:256
2020 Oct 22 07:12:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:256
2020 Oct 22 07:12:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:256
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:256
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:256
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:256
2020 Oct 22 07:12:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:13 %$ VDC-1 %$ %CALLHOME-2-EVENT: sfcbd crashed with crash type:256
2020 Oct 22 07:12:13 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:13 %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256
2020 Oct 22 07:12:13 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:13 %$ VDC-1 %$ %CALLHOME-2-EVENT: sam_core_mon crashed with crash type:256
2020 Oct 22 07:12:13 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:14 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_controller crashed with crash type:256
2020 Oct 22 07:12:14 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:14 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_controller - pmon
2020 Oct 22 07:12:14 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:256
2020 Oct 22 07:12:14 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:14 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_dme - pmon
2020 Oct 22 07:12:14 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:256
2020 Oct 22 07:12:14 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:14 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_dcosAG - pmon
2020 Oct 22 07:12:15 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:256
2020 Oct 22 07:12:15 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:15 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_bladeAG - pmon
2020 Oct 22 07:12:15 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:256
2020 Oct 22 07:12:15 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:15 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_portAG - pmon
2020 Oct 22 07:12:15 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:256
2020 Oct 22 07:12:15 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:15 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_hostagentAG - pmon
2020 Oct 22 07:12:16 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:256
2020 Oct 22 07:12:16 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:16 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_nicAG - pmon
System is coming up ... Please wait ...
2020 Oct 22 07:12:16 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:256
2020 Oct 22 07:12:16 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:16 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_extvmmAG - pmon
2020 Oct 22 07:12:16 %$ VDC-1 %$ %CALLHOME-2-EVENT: httpd.sh crashed with crash type:256
2020 Oct 22 07:12:16 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:16 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: httpd.sh - pmon
2020 Oct 22 07:12:17 %$ VDC-1 %$ %VDC_MGR-2-VDC_ONLINE: vdc 1 has come online
2020 Oct 22 07:12:17 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:256
2020 Oct 22 07:12:17 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:17 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_cliD - pmon
2020 Oct 22 07:12:18 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:256
2020 Oct 22 07:12:18 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:18 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_pamProxy - pmon
2020 Oct 22 07:12:18 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: sfcbd crashed with crash type:256
2020 Oct 22 07:12:18 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:19 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: sfcbd - pmon
2020 Oct 22 07:12:20 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:256
2020 Oct 22 07:12:20 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:20 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: dhcpd - pmon
2020 Oct 22 07:12:21 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: sam_core_mon crashed with crash type:256
2020 Oct 22 07:12:21 switch %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2020 Oct 22 07:12:21 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: sam_core_mon - pmon
System is coming up ... Please wait ...
touch: missing file operand
Try `touch --help' for more information.
/isan/bin/first-setup.core: line 1243: bond0: command not found
/isan/bin/first-setup.core: line 1244: bond0: command not found
/isan/bin/first-setup.core: line 1245: bond0: command not found
/isan/bin/first-setup.core: line 1247: bond0: command not found
/isan/bin/first-setup.core: line 2101: -t: command not found
nohup: appending output to `nohup.out'
nohup: cannot run command `/isan/bin/initial_setup.sh': No such file or directory
up: error fetching interface information: Device not found
2020 Oct 22 07:12:21 switch %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Running in PIO stats mode - carmelusd

---- Basic System Configuration Dialog ----

This setup utility will guide you through the basic configuration of
the system. Only minimal configuration including IP connectivity to
the Fabric interconnect and its clustering mode is performed through these steps.

Type Ctrl-C at any time to abort configuration and reboot system.
To back track or make modifications to already entered values,
complete input till end of section and answer no when prompted
to apply configuration.

Enter the configuration method. (console/gui) ?

[Steven Tardy]

The software crash is often a indicator of a flash filesystem corruption.

This filesystem corruption is most often a software issue and a filesystem check (`fsck`) resolves it.

You'll need TAC, a physical console/serial cable, and a TFTP server from which to copy a few files.

The required files are not generally/publicly available hence the need for TAC.