I know I made some mistakes doing this upgrade but you live and you learn. Biggest mistake was not verifying our support was current.
Normal upgrade process followed.
- Take a backup of the config
- Take a full state backup.
- Download upgrade packages from Cisco.
- Download the packages to the UCS
- Confirm all faults are cleared (I was able to get it to 0 0 0 0)
- Run the Auto Infrastructure Firmware update
- UCSM updated fine.
- IOM's updated fine.
- Remote interconnect seemed to upgrade fine but didn't come backup.
- Connected console cable and rebooted again.
- Watching the post resulted in the below stream of process failures.
- Stopped at Basic Configuration Wizard.
- Ran through the basic configuration
- It found the working peer and asked to join the cluster
- I said yes
- It complained that the firmware didn't match.
- In order to get to a working state again I agreed to the firmware "update" or downgrade to match the working peer.
- Completed successfully
- After rebooting I get the same messages below and end up at the basic configuration wizard.
So is it dead? Like I mentioned above we inadvertently let our support lapse (working on renewing already before this happened), so I can't call support. Luckily this environment is not really used yet. So I can limp along until support is renewed but I am hoping someone here can help in the meantime.
2016 Oct 21 13:09:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:08 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_controller - pmon
2016 Oct 21 13:09:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dme crashed with crash type:32512
2016 Oct 21 13:09:08 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:08 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_dme - pmon
2016 Oct 21 13:09:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_dcosAG crashed with crash type:32512
2016 Oct 21 13:09:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:09 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_dcosAG - pmon
2016 Oct 21 13:09:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_bladeAG crashed with crash type:32512
2016 Oct 21 13:09:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:09 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_bladeAG - pmon
2016 Oct 21 13:09:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_portAG crashed with crash type:32512
2016 Oct 21 13:09:09 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:09 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_portAG - pmon
2016 Oct 21 13:09:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_hostagentAG crashed with crash type:32512
2016 Oct 21 13:09:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:10 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_hostagentAG - pmon
2016 Oct 21 13:09:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_nicAG crashed with crash type:32512
2016 Oct 21 13:09:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:10 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_nicAG - pmon
2016 Oct 21 13:09:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_extvmmAG crashed with crash type:32512
2016 Oct 21 13:09:10 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:10 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_extvmmAG - pmon
2016 Oct 21 13:09:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_cliD crashed with crash type:32256
2016 Oct 21 13:09:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:11 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_cliD - pmon
2016 Oct 21 13:09:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: svc_sam_pamProxy crashed with crash type:32512
2016 Oct 21 13:09:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:11 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: svc_sam_pamProxy - pmon
2016 Oct 21 13:09:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: sfcbd crashed with crash type:256
2016 Oct 21 13:09:11 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:11 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: sfcbd - pmon
2016 Oct 21 13:09:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: dhcpd crashed with crash type:32512
2016 Oct 21 13:09:12 %$ VDC-1 %$ %CALLHOME-2-EVENT: SW_CRASH
2016 Oct 21 13:09:12 %$ VDC-1 %$ %USER-2-SYSTEM_MSG: Restart count exhausted for process: dhcpd - pmon
