07-11-2010 06:07 PM - edited 03-01-2019 09:41 AM
Hi all,
Had a software failure while upgrading the fabric interconnects. Does anyone know the process to recover from a software failure.
Ive copied up the kickstart and system images, from the loader prompt ive booted the kickstart image and then loaded the system image but the fabric interconnect still reboots to the "switch(boot)" prompt.
Thanks in Advance
Darren
Solved! Go to Solution.
07-11-2010 07:29 PM
You're going to need the UCSM bin file as well. You can download this from CCO.
"ucs-manager-k9.1.3.1c.bin"
1. Once booted to the "switch(boot)" prompt copy all the files to /bootflash
switch(boot)# conf t
switch(boot)(config)# interface mgmt 0
switch(boot)(config-if)# ip address [IP address] [Mask]
switch(boot)(config-if)# no shut
switch(boot)(config-if)# exit
switch(boot)(config)# ip default-gateway [Gateway IP]
switch(boot)(config)# exit
switch(boot)# copy scp://username@x.x.x.x/images/kickstart-image-version.bin bootflash:
switch(boot)# copy scp://username@x.x.x.x/images/system-image-version.bin bootflash:
switch(boot)# copy scp://username@x.x.x.x/images/ucs-manager-version.bin bootflash:
2. Rename ucs-manager-version.bin file to the name that the reserved system image name "nuova-sim-mgmt-nsg.0.1.0.001.bin".
switch(boot)# copy bootflash:ucs-manager-verison.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin
3. Reboot
switch(boot)# exit
4. CTRL-L to break into loader and boot into kickstart and system images at the same time, space between
loader> boot kickstart-image-version.bin system-image-version.bin
5. When you get the system back up, re-join your cluster (or restore your config) and ensure you set the boot variables as previusly mentioned.
Let me know how you go with the above.
Robert
07-11-2010 06:21 PM
The full restore procedure is found here:
The reason your system is still not loading the system software is because the boot variables haven't been set. Once you've loaded your kickstart and system image from the Loader prompt restored your config (or at least setup basic Mgmt IP access), and open up UCSM (via GUI) and set the boot variables by:
Navigate to the Equipment Tab - Firmware Management - and then "Activate" the Kickstart and System software versions (corresponding to the same versions you currently have running) for both FI's. Though there will be no actual new firmware activated, this will set your boot variables.
Let me know if you have any issues.
Robert
07-11-2010 06:58 PM
Hi Robert,
Thanks for the tips, but unfortunately im still stuck on the switch(boot): prompt.
The bootflash: only has 2 files in it, the kickstart and system image.
Each time i get the FI to the boot prompt, i can ping the device, but not access via the GUI. (i dont have a local copy of UCSM installed)
What files should i have in the bootflash: for the FI to correctly load.
below is a log of the FI as it loads the kickstart image from the Loader prompt.
loader> boot bootflash:ucs-6100-k9-kickstart.4.1.3.N2.1.3c.bin
Booting kickstart image: bootflash:ucs-6100-k9-kickstart.4.1.3.N2.1.3c.bin....
....................................................................Image verif
ication OK
Starting kernel...
Usage: init 0123456SsQqAaBbCcUu
INIT: version 2.85 booting
Starting Nexus5010 POST...
Executing Mod 1 1 SEEPROM Test......done
Executing Mod 1 1 GigE Port Test.......done
Executing Mod 1 1 Inband GigE Test.....done
Executing Mod 1 1 NVRAM Test....done
Executing Mod 1 1 PCIE Test..............................done
Mod 1 1 Post Completed Successfully
Mod 2 Post Completed Successfully
POST is completed
Creating /callhome..
Mounting /callhome..
Creating /callhome done.
Callhome spool file system init done.
Checking all filesystems..... done.
Warning: switch is starting up with default configuration
Checking NVRAM block device ... done
The startup-config won't be used until the next reboot.
.
Loading system software
INIT: Sending processes the KILL signalkernel daemon...don
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)#
From here i can the load the system image, but it always fails and drops me back to the same prompt, output show below
Uncompressing system image: bootflash:/ucs-6100-k9-system.4.1.3.N2.1.3c.bin
Loading plugin 0: core_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/0, tar_log = /isan/plugin_extract_log/0
Loading plugin 1: eth_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/1, tar_log = /isan/plugin_extract_log/1
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0.0.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so.0 is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/lib/libplugin_sysreg.so is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/ is excluded from linking
plugin_file_is_excluded_from_exec_path: /boot/etc/plugin_exclude.conf is excluded from linking
Loading plugin 2: fc_plugin...
plugin_link_to_exec_path: plugin_path = /isan/plugin/2, tar_log = /isan/plugin_extract_log/2
plugin_file_is_excluded_from_exec_path: /boot/ is excluded from linking
INIT: Sending processes the TERM signal /boot/lib/ is excluded from lin
INIT: (boot)# l
INIT: Sending processes the KILL signaldaemon...don
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
switch(boot)#
please let me know if you require any further information.
Thanks in Advance
Darren
07-11-2010 07:29 PM
You're going to need the UCSM bin file as well. You can download this from CCO.
"ucs-manager-k9.1.3.1c.bin"
1. Once booted to the "switch(boot)" prompt copy all the files to /bootflash
switch(boot)# conf t
switch(boot)(config)# interface mgmt 0
switch(boot)(config-if)# ip address [IP address] [Mask]
switch(boot)(config-if)# no shut
switch(boot)(config-if)# exit
switch(boot)(config)# ip default-gateway [Gateway IP]
switch(boot)(config)# exit
switch(boot)# copy scp://username@x.x.x.x/images/kickstart-image-version.bin bootflash:
switch(boot)# copy scp://username@x.x.x.x/images/system-image-version.bin bootflash:
switch(boot)# copy scp://username@x.x.x.x/images/ucs-manager-version.bin bootflash:
2. Rename ucs-manager-version.bin file to the name that the reserved system image name "nuova-sim-mgmt-nsg.0.1.0.001.bin".
switch(boot)# copy bootflash:ucs-manager-verison.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin
3. Reboot
switch(boot)# exit
4. CTRL-L to break into loader and boot into kickstart and system images at the same time, space between
loader> boot kickstart-image-version.bin system-image-version.bin
5. When you get the system back up, re-join your cluster (or restore your config) and ensure you set the boot variables as previusly mentioned.
Let me know how you go with the above.
Robert
07-11-2010 10:11 PM
Hi Robert,
The process you outlined has worked in recovering the FI's.. however.. im unable to http or https to the Cluster address after running through the initial express setup on both FI's.. would you have any idea?
I can ping the cluster address
Do i need to activate the new firmware via the CLI?
both FI's look ready to go as per the below output
show system version expand
UCSM:
Running-Vers: 1.3(1c)
Activate-Status: Ready
Fabric Interconnect A:
Running-Kern-Vers: 4.1(3)N2(1.3c)
Running-Sys-Vers: 4.1(3)N2(1.3c)
Startup-Kern-Vers:
Startup-Sys-Vers:
Act-Kern-Status: Ready
Act-Sys-Status: Ready
Bootloader-Vers: v1.3.0(09/08/09)
Fabric Interconnect B:
Running-Kern-Vers: 4.1(3)N2(1.3c)
Running-Sys-Vers: 4.1(3)N2(1.3c)
Startup-Kern-Vers:
Startup-Sys-Vers:
Act-Kern-Status: Ready
Act-Sys-Status: Ready
Bootloader-Vers: v1.3.0(09/08/09)
Cheers
Darren
07-12-2010 10:40 PM
Robert,
I wanted to thank you for your assistance with this. I also wanted to add the additional processess i went through to rectify the issues
1. Boot the Fabric Interconnect (FI) to the "switch(boot)" prompt. If presented with the "loader" prompt and no kickstart image in the Bootflash: you can boot the switch to the "switch(boot)" prompt by booting from a TFTP server
2. Configure the MGMT 0 interface on the FI
switch(boot)#conf t
switch(boot)(config)#interface mgmt 0
switch(boot)(config-if)#ip address [IP address] [Mask]
switch(boot)(config-if)#no shut
switch(boot)(config-if)#exit
switch(boot)(config)#ip default-gateway [Gateway IP]
switch(boot)(config)#exit
switch(boot)#
3. Copy the Kickstart and System Images to the Bootflash: and also copy the UCS Manager Version to the Bootflash: (i used FTP, but SCP, TFTP, FTP is available)
switch(boot)#copy ftp://username@[FTP IP Address]/kickstart-image-version.bin bootflash:
switch(boot)#copy ftp://username@[FTP IP Address]/system-image-version.bin bootflash:
switch(boot)#copy ftp://username@[FTP IP Address]/ucs-manager-version.bin bootflash:
4. Rename the ucs-manager-version.bin file to the name that the reserved system image uses "nuova-sim-mgmt-nsg.0.1.0.001.bin" (Huge thx to Robert for this step)
switch(boot)#copy bootflash:ucs-manager-version.bin bootflash:nuova-sim-mgmt-nsg.0.1.0.001.bin
5. Reboot the FI
switch(boot)#exit
6. In my experience, the switch rebooted to the loader screen, if it doesnt, you can press CTRL-L (as per Roberts instructions) or CRTL-1 or CRTL-SHIFT-R (as per the configuration guide 1.3(1c). At this prompt boot into the kickstart and system images at the same time. Remember to put a space between the file names
loader> boot kickstart-image-version.bin system-image-version.bin
6. Once the FI reboots the option to start the setup is presented. selct the GUI option and then enter the FI details
IP Address:
Subnet Mask:
Default Gateway:
7. Open a web browser and connect via HTTPS: to the IP address configured in the setup
8. Launch the Express Setup option from the browser and complete the Express Setup.
9. I found from here that i then couldnt launch the UCS manager as after the Express setup was completed, it weas inaccessible. I then connected to the FI via console and could log in. From here i uploaded the UCS-Bundle-Version.bin file
FI-A#scope firmware
FI-A /Firmware # download image ftp://username@[IP Address]/UCS-Bundle-Version.bin file
password: FTP Password
Check the image has complete its download
FI-A /Firmware # show image
10. Open a web browser and connect via HTTPS: to the MGMT 0 IP address abd press Launch.
From here you should be able to enter into the UCS manager and see the FI. You're not done here, the FI still needs to have the newly loaded firmware set as the startup image.
11. Navigate to the Equipment Tab >> Firmware Management and then Activate the Kickstart and System software versions that were uploaded earlier in the process. There is no new firmware to update, this will just set the boot variables for the FI so that it boots correctly and doesnt stop at the "loader" prompt again.
Outside of this, I had a few other issues with multiple FI's that failed and then resetting the cluster information, but this process is in the configuration guides.
Once again a HUGE thanks to Robert for his imput earlier that got me underway.
Cheers
Darren
09-01-2010 09:50 AM
I had the same problem. The guide work fine for me, except for point 11
"11. Navigate to the Equipment Tab >> Firmware Management and then Activate the Kickstart and System software versions that were uploaded earlier in the process. There is no new firmware to update, this will just set the boot variables for the FI so that it boots correctly and doesnt stop at the "loader" prompt again."
The Firmware quickstart and system are configured, but I don't know how to configure automatic boot?
Thanks
Cristhian Santelices O.
09-01-2010 05:40 PM
Cristhian,
Even though the variables may be set in UCSM, activate both FI's again. This should re-assign the NXOS boot variables and fix your issue.
Have you already done this twice?
If not, let me know.
Regards,
Robert
09-02-2010 07:47 AM
Hello Robert
This is my screenshot
I don't know how to Activate the Kickstart and System software versions??
The UCS boot to:
Loader Version pr-1.3
loader>
and manually I boot
loader>boot ucs-6100-k9-kickstart.4.1.3.N2.1.3m.bin ucs-6100-k9-system.4.1.3.N2.1.3m.bin
I don't know how to activate boot through GUI or CLI ?
Thanks and regards
Cristhian
09-02-2010 03:57 PM
Hi Cristhian,
From that screen shot, you need to click on "activate firmware" from the menu bar.
From the next screen you can select the startup versions for each of your Fabric Interconnects.
Cheers
Darren
09-06-2010 06:57 AM
Hi Darren
I try to activate firmware, but the combox not available to select a "Startup Version" of firmware for UCS and not show any firmware available.
You can help me about
Thanks and regards
09-06-2010 07:05 AM
Hi,
Can i assume that there is nothing in the drop down box for the start up files for you to select?
Are the files visible on the FI via SSH?
If there isnt, the only thing i could suggest is trying to upload the files again, or to go to the TAC for further assistance
09-06-2010 12:30 PM
Hi Darren
I uploaded de kickstart and system image via GUI and I have boot the system automatically.
Thanks again
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide