Introduction
There are multiple reasons a 3850 may fail to boot correctly including a corrupt boot image, a corrupt packages.conf file, missing files, etc. Below are a few different possible recovery methods to try. I will also explain the two possible mode options, Install and bundle and why you might want to use one or the other.
Install vs. Bundle Mode
There are a few difference in the two modes, I would recommend reading over the config guide for more in-depth details. The recommended mode during operation is INSTALL mode because it allows for more features and requires fewer resources when booting.
++Install Mode
This is the out-of-the-box mode that your switch will be in. INSTALL mode uses a package provisioning file named packages.conf to boot the switch.
If you happen to be in bundle mode upon boot, you can simply boot your switch in install mode by booting the software package provisioning file that resides in flash. If packages.conf doesn’t exist in flash, you need to expand the bundle into the flash file system by running
Switch# software expand file flash: cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin to flash:
Once this completes, you will have all the needed files in flash. You can then change the boot statement to boot to packages.conf
Switch#Config t
Switch(config)# no boot system
Switch(config): boot system switch all flash:packages.conf (do not modify this file, unless necessary)
Switch#write memory
The provisioning file contains a list of software packages to boot, mount, and run. The ISO file system in each installed package is mounted to the root file system directly from flash.
NOTE **Auto-upgrade is disabled, by default. (once in install mode - execute the following command in global config: software auto-upgrade enable )
NOTE **Auto-upgrade includes an auto-copy process and an auto-extract process.
++Bundle Mode
As noted previously, bundle mode consumes more memory than booting in install mode because the packages are extracted from the bundle and copied to the RAM. If you decide to convert to bundle mode, you will first need to download the .bin file from CCO if you don’t already have it in flash. Once in flash, you can simply change your boot statement to point to the (.bin) file:
Switch#Config t
Switch(config)# no boot system
Switch(config): boot system switch all flash: cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
Switch#write memory
The provisioning file contained in a bundle is used to decide which packages to boot, mount, and run. Packages are extracted from the bundle and copied to RAM.
NOTE **Auto install and smart install functionality is not supported in bundle boot mode.
Recovery Methods
USB
The 3850 has a USB port on the front that can be used for both console access and also the ability to utilize a flash drive for image backup and recovery.
If you happen to be stuck at the switch: prompt with a corrupt image or .conf file, you can easily boot to a file stored on the USB drive.
1. Verify that the flashdrive is recognized and the .bin file exists
switch: dir usbflash0:
Directory of usbflash0:/
74 -rw- 223734376 cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
2. Boot to the USB image
switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
Corrupt packages.conf
I’ve seen instances in which packages.conf continually calls files that no longer exist in flash. You can boot to an image from ROMMON just fine, however upon reload it will call packages.conf again and fail to boot. If this happens, I recommend backing up the existing packages.conf file by renaming it or deleting all together. The previous step is mandatory as the next step will fail if a .conf file already exists. You can then run an BUNDLE extract which will create a new packages.conf file.
1. Once booted up (in BUNDLE mode) verify the files in flash
Switch#dir flash:
Directory of flash:/
15500 -rwx 1243 Aug 1 2013 07:04:02 +00:00 packages.conf
2. Copy or rename the existing packages.conf file
Switch#cp flash:packages.conf flash:packages.conf.badop flash:packages.conf flash:packages.conf.bad
Destination filename [packages.conf.bad]?
Copy in progress...C
1243 bytes copied in 0.140 secs (8879 bytes/sec)
Switch#dir flash:
Directory of flash:/
15500 -rwx 1243 Aug 1 2013 07:04:02 +00:00 packages.conf
15502 -rw- 1243 Aug 1 2013 11:53:51 +00:00 packages.conf.bad
3. Delete packages.conf
Switch#del flash:packages.conf
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]
4. Expand BUNDLE to create new packages.conf
Switch#software expand running switch 1 to flash:
Preparing expand operation ...
[1]: Expanding the running bundle
[1]: Copying package files
[1]: Package files copied
[1]: Finished expanding the running bundle
5. Verify boot
Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist
Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
6. Reload Switch
switch#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]
Emergency Recovery
If all else fails, the 3850 has a “trap door” method of recovering the system. All you need is a terminal connected to the management port of the 3850 running a tftp server. Download a valid image file from CCO and store it in the root of the tftp server.
On the switch, you are most likely stuck at the switch: prompt. If however you are in some sort of boot loop, you can use the “mode” button on the front of the switch to break the cycle. Simply hold the button for roughly 10 seconds and the switch should react by breaking the cycle and stopping at a switch: prompt.
1. Set the switch IP
switch: set IP_ADDR 192.0.2.123/255.255.255.0
2. Set the default gateway
switch: set DEFAULT_ROUTER 192.0.2.1
3.Test connectivity by pinging terminal (that contains the tftp server)
switch: ping 192.0.2.1
ping 192.0.2.1 with 32 bytes of data ...
Host 192.0.2.1 is alive.
4. Verify that the emergency files exist in the switches file system
switch: dir sda9:
Directory of sda9:/
2 drwx 1024 .
2 drwx 1024 ..
11 -rwx 18958824 cat3k_caa-recovery.bin
36903936 bytes available (20866048 bytes used)
5. Run the emergency install feature
switch: emergency-install tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
The bootflash will be erased during install operation, continue (y/n)?Y
Starting emergency recovery (tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin)...
Reading full image into memory......................done
Nova Bundle Image
--------------------------------------
Kernel Address : 0x6042f5d8
Kernel Size : 0x317ccc/3243212
Initramfs Address : 0x607472a4
Initramfs Size : 0xdc6546/14443846
Compression Format: .mzip
Bootable image at @ ram:0x6042f5d8
Bootable image segment 0 address range [0x81100000, 0x81b80000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x811060f0
Loading Linux kernel with entry point 0x811060f0 ...
Bootloader: Done loading app on core_mask: 0xf
### Launching Linux Kernel (flags = 0x5)
Initiating Emergency Installation of bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
Downloading bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Validating bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Installing bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Verifying bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Package cat3k_caa-base.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-drivers.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-infra.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg is Digitally Signed
Package cat3k_caa-platform.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-wcm.SPA.10.0.111.0.pkg is Digitally Signed
Preparing flash...
Syncing device...
Emergency Install successful... Rebooting
Restarting system.
Reference:
3850 Deployment guide:
http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps12686/deployment_guide_c07-727067.html#wp9000153
