cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Recovering from a 3850 boot failure.

8178
Views
0
Helpful
0
Comments

Introduction

There are multiple reasons a 3850 may fail to boot correctly including a corrupt boot image, a corrupt packages.conf file, missing files, etc.  Below are a few different possible recovery methods to try.  I will also explain the two possible mode options, Install and bundle and why you might want to use one or the other.

Install vs. Bundle Mode

There are a few difference in the two modes, I would recommend reading over the config guide for more in-depth details. The recommended mode during operation is INSTALL mode because it allows for more features and requires fewer resources when booting.

++Install Mode

This is the out-of-the-box mode that your switch will be in.  INSTALL mode uses a package provisioning file named packages.conf to boot the switch. 

If you happen to be in bundle mode upon boot, you can simply boot your switch in install mode by booting the software package provisioning file that resides in flash. If packages.conf doesn’t exist in flash, you need to expand the bundle into the flash file system by running

Switch# software expand file flash: cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin to flash:

Once this completes, you will have all the needed files in flash. You can then change the boot statement to boot to packages.conf

Switch#Config t

Switch(config)# no boot system

Switch(config): boot system switch all flash:packages.conf  (do not modify this file, unless necessary)

Switch#write memory

The provisioning file contains a list of software packages to boot, mount, and run. The ISO file system in each installed package is mounted to the root file system directly from flash.

NOTE **Auto-upgrade is disabled, by default. (once in install mode - execute the following command in global config: software auto-upgrade enable )

NOTE **Auto-upgrade includes an auto-copy process and an auto-extract process.

++Bundle Mode

As noted previously, bundle mode consumes more memory than booting in install mode because the packages are extracted from the bundle and copied to the RAM.  If you decide to convert to bundle mode, you will first need to download the .bin file from CCO if you don’t already have it in flash.  Once in flash, you can simply change your boot statement to point to the (.bin) file:

Switch#Config t

Switch(config)# no boot system

Switch(config): boot system switch all flash: cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin

Switch#write memory

The provisioning file contained in a bundle is used to decide which packages to boot, mount, and run. Packages are extracted from the bundle and copied to RAM.

NOTE **Auto install and smart install functionality is not supported in bundle boot mode.

Recovery Methods

USB

The 3850 has a USB port on the front that can be used for both console access and also the ability to utilize a flash drive for image backup and recovery.

If you happen to be stuck at the switch: prompt with a corrupt image or .conf file, you can easily boot to a file stored on the USB drive.

1. Verify that the flashdrive is recognized and the .bin file exists

switch: dir usbflash0:

Directory of usbflash0:/

74  -rw-  223734376  cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin

2. Boot to the USB image

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin

Corrupt packages.conf

I’ve seen instances in which packages.conf continually calls files that no longer exist in flash.  You can boot to an image from ROMMON just fine, however upon reload it will call packages.conf again and fail to boot.  If this happens, I recommend backing up the existing packages.conf file by renaming it or deleting all together.  The previous step is mandatory as the next step will fail if a .conf file already exists.  You can then run an BUNDLE extract which will create a new packages.conf file. 

1. Once booted up (in BUNDLE mode) verify the files in flash

Switch#dir flash:

Directory of flash:/

15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf

2. Copy or rename the existing packages.conf file

Switch#cp flash:packages.conf flash:packages.conf.badop flash:packages.conf flash:packages.conf.bad

Destination filename [packages.conf.bad]?

Copy in progress...C

1243 bytes copied in 0.140 secs (8879 bytes/sec)

Switch#dir flash:

Directory of flash:/

15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf

15502  -rw-        1243   Aug 1 2013 11:53:51 +00:00  packages.conf.bad

3. Delete packages.conf

Switch#del flash:packages.conf

Delete filename [packages.conf]?

Delete flash:/packages.conf? [confirm]

4. Expand BUNDLE to create new packages.conf

Switch#software expand running switch 1 to flash:

Preparing expand operation ...

[1]: Expanding the running bundle

[1]: Copying package files

[1]: Package files copied

[1]: Finished expanding the running bundle

5. Verify boot

Switch#show boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable does not exist

Boot Variables on next reload:

BOOT variable = flash:packages.conf;

Manual Boot = no

Enable Break = no

6. Reload Switch

switch#reload

Reload command is being issued on Active unit, this will reload the whole stack

Proceed with reload? [confirm]

Emergency Recovery

If all else fails, the 3850 has a “trap door” method of recovering the system.  All you need is a terminal connected to the management port of the 3850 running a tftp server.  Download a valid image file from CCO and store it in the root of the tftp server.

On the switch, you are most likely stuck at the switch: prompt.  If however you are in some sort of boot loop, you can use the “mode” button on the front of the switch to break the cycle.  Simply hold the button for roughly 10 seconds and the switch should react by breaking the cycle and stopping at a switch: prompt.

1. Set the switch IP

switch:  set IP_ADDR 192.0.2.123/255.255.255.0

2. Set the default gateway

switch: set DEFAULT_ROUTER 192.0.2.1

3.Test connectivity by pinging terminal (that contains the tftp server)

switch: ping 192.0.2.1

ping 192.0.2.1 with 32 bytes of data ...

Host 192.0.2.1 is alive.

4. Verify that the emergency files exist in the switches file system

switch: dir sda9:

Directory of sda9:/

    2  drwx  1024       .

    2  drwx  1024       ..

   11  -rwx  18958824   cat3k_caa-recovery.bin

36903936 bytes available (20866048 bytes used)

5. Run the emergency install feature

switch: emergency-install tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin

The bootflash will be erased during install operation, continue (y/n)?Y

Starting emergency recovery (tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin)...

Reading full image into memory......................done

Nova Bundle Image

--------------------------------------

Kernel Address    : 0x6042f5d8

Kernel Size       : 0x317ccc/3243212

Initramfs Address : 0x607472a4

Initramfs Size    : 0xdc6546/14443846

Compression Format: .mzip

Bootable image at @ ram:0x6042f5d8

Bootable image segment 0 address range [0x81100000, 0x81b80000] is in range [0x80180000, 0x90000000].

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x811060f0

Loading Linux kernel with entry point 0x811060f0 ...

Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Initiating Emergency Installation of bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin

Downloading bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Validating bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Installing bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Verifying bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Package cat3k_caa-base.SPA.03.02.02.SE.pkg is Digitally Signed

Package cat3k_caa-drivers.SPA.03.02.02.SE.pkg is Digitally Signed

Package cat3k_caa-infra.SPA.03.02.02.SE.pkg is Digitally Signed

Package cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg is Digitally Signed

Package cat3k_caa-platform.SPA.03.02.02.SE.pkg is Digitally Signed

Package cat3k_caa-wcm.SPA.10.0.111.0.pkg is Digitally Signed

Preparing flash...

Syncing device...

Emergency Install successful... Rebooting

Restarting system.

Reference:

3850 Deployment guide:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps12686/deployment_guide_c07-727067.html#wp9000153


CreatePlease to create content
Content for Community-Ad
July's Community Spotlight Awards