Unable to change boot file on Cisco 3850

[whiteleyjl]

I was working on a Cisco 3850 24 port switch today and I read that it doesn't use the normal "boot system flash:XYZ.bin" but instead it's something like this:

"software install file flash:XYZ.bin new"

That changes the install package or something which makes it boot in the newly selected package which contains the new IOS.  Anyway, when i put in that command I get something about "Failed to ...." or something.  I'm sorry but I'm at home now and I don't have the device with me and it just occured to me to post this on the forum for possible help.  Either way, it's specifically says "Failed..." as the first word which is not what it should normally say.

I used these directions:

http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps12686/deployment_guide_c07-727067.html#wp9000169

I am in Install mode.  Can anyone help me figure out why this is happening before my outage window on Sunday night?  I've downloaded the new version of the IOS from Cisco.com and verified it is currently located in the flash of this device. 

Thanks for any help you can provide!

[Marvin Rhoads]

Have you verified your image file is the proper one? Checked the MD5 checksum?

You have a pretty new switch - do you have Smartnet support? The Cisco TAC can assist in real time with your upgrade if you open a case.

[Akzhol Konurbaev]

Hi everyone! 

i can't install IOS 

cat3k_caa-universalk9ldpe.SPA.03.07.02.E.152-3.E2.bin

Validating bundle tftp://10.5.5.1/cat3k_caa-universalk9ldpe.SPA.03.07.02.E.152-3.E2.bin..
Installing bundle tftp://10.5.5.1/cat3k_caa-universalk9ldpe.SPA.03.07.02.E.152-3.E2.bin.
Verifying bundle tftp://10.5.5.1/cat3k_caa-universalk9ldpe.SPA.03.07.02.E.152-3.E2.bin
Package cat3k_caa-base.SPA.03.07.02E.pkg is Digitally Signed
Package cat3k_caa-drivers.SPA.03.07.02E.pkg is Digitally Signed
Package cat3k_caa-infra.SPA.03.07.02E.pkg is Digitally Signed
Package cat3k_caa-iosd-universalk9.SPA.152-3.E2.pkg is Digitally Signed
Package cat3k_caa-platform.SPA.03.07.02E.pkg is Digitally Signed
Package cat3k_caa-wcm.SPA.10.3.120.0.pkg is Digitally Signed
Preparing flash...
Syncing device...
Emergency Install successful... Rebooting
Restarting system.

Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++
Base ethernet MAC Address: bc:c4:93:38:1c:00
Initializing Flash...

flashfs[7]: 0 files, 1 directories
flashfs[7]: 0 orphaned files, 0 orphaned directories
flashfs[7]: Total bytes: 6784000
flashfs[7]: Bytes used: 1024
flashfs[7]: Bytes available: 6782976
flashfs[7]: flashfs fsck took 1 seconds....done Initializing Flash.
Getting rest of image
Reading full image into memory....done
Reading full base package into memory...: done = 82016892
Bundle Image
--------------------------------------
Kernel Address    : 0x6e4463ge
Kernel Size       : 0x3e5hrea9/4172713
Initramfs Address : 0x57456ef8
Initramfs Size    : 0xdf1b84/1445788
Compression Format: .mzip

Bootable image at @ ram:0x6042d34c
Bootable image segment 0 address range [0x81100000, 0x820a0000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@boot_system: 377
Loading Linux kernel with entry point 0x816733a0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

%IOSXEBOOT-4-ABORT: (rp/0): Failed to process provisioning file (/tmp/packages.conf).
%IOSXEBOOT-4-ABORT: (rp/0): Failed to process provisioning file (/tmp/packages.conf).

Booting...Initializing RAM +++++++@@@@@@@@...++++++++++++++++++++++++++++++++
Base ethernet MAC Address: bc:c4:93:38:1c:00
I

Error loading "flash:packages.conf"
Boot process failed...

The system is unable to boot automatically. The
BOOT environment variable needs to be set to a
bootable image.

switch: dir

List of filesystems currently registered:

              xmodem[0]: (read-only)
                null[1]: (read-write)
                  bs[3]: (read-only)
               flash[18]: (read-only)
                tftp[19]: (read-write)

switch: delete flash:packages.conf
Are you sure you want to delete "flash:packages.conf" (y/n)?y
File "flash:packages.conf" not deleted -- read only file system

switch: dir flash:
Directory of flash:/

 24241  drwx  4096       .
     2  drwx  4096       ..
 24242  -rw-  2097152    nvram_config
     0  -rw-  2097152    cat3k_caa-base.SPA.03.07.02E.pkg
     0  -rw-  2097152    cat3k_caa-drivers.SPA.03.07.02E.pkg
     0  -rw-  2097152    cat3k_caa-infra.SPA.03.07.02E.pkg
     0  -rw-  2097152    cat3k_caa-iosd-universalk9.SPA.152-3.E2.pkg
     0  -rw-  2097152    cat3k_caa-platform.SPA.03.07.02E.pkg
     0  -rw-  2097152    cat3k_caa-wcm-ldpe.SPA.10.3.120.0.pkg
 24249  -rw-  1250       packages.conf 

Why Flash mode read-only ? 

[dimitri.perrin1]

Hi All

C3850 48 PoE+ - show version within the attached file

I have the same issue than Akzhol Konurbaev.

Any news about this ???

I've tried several methods, as described there :

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html

I join the file with my différents actions and results.

Should be better to format all the FS, in my opinion, but i'd prefer to have a procedure...

Anyone can help ???

Thanks for your attention and help  :-)

[Morteza Rahmani Amoli]

Hi everyone,

If you faced this problem, the best solution is upgrading the switch using a USB Flash.

You can download the last vesrion or new one, copy to flash memory and then upgrade the switch.

If the booting of switch is not successful you go to rommon after 5 booting failure with this prompt:

Switch:

Now you can attach your USB Flash to the switch an do this:

switch: boot usbflash0:XXXX.bin

Which xxxx.bin is the IOS-XE that you copied this to flash memory.

Then your switch should boot with this image. After booting the switch you have this prompt:

Switch>enable
Switch#

now you must copy this image to flash of switch:

Switch#copy usbflash0:XXXX.bin flash

Now the boot options must be checked :

Switch#show run | inc boot

If you don't see "boot system switch all flash:XXXX.bin" under "show run | inc boot', you should enter this in config mode.

After reload the switch must be booted with this version.

Kind regards,

Morteza

[nfarrelly]

I have the same issue. Tried emergency install, everything mentioned above - Nothing works. The problem is the flash: is read only. I would like to delete the packages.conf file as I think it is corrupt.

Is there anything I can do??

[KHAJA FAZLE USMANI]

Hi,

I have same problem as well in 3850 switch. flash: read only file system, want to delete packages.conf file, unable to delete.

here is the out of set:

switch: set
ABNORMAL_RESET_COUNT=5
AUTOREBOOT_RESTORE=0
BOOT=flash:packages.conf
BSI=0
BYPASS_STARTUP_CONFIG=1
CFG_MODEL_NUM=WS-C3850-48F-E
CLEI_CODE_NUMBER=IPMW700ARB
DEFAULT_ROUTER=192.0.2.122
ECI_CODE_NUMBER=466864
IP_ADDR=192.0.2.123/255.255.255.0
LINUX_COREMASK=15
MAC_ADDR=88:90:8D:71:9F:00
MANUAL_BOOT=no
MODEL_NUM=WS-C3850-48P
MODEL_REVISION_NUM=S0
MOTHERBOARD_ASSEMBLY_NUM=73-14442-10
MOTHERBOARD_REVISION_NUM=A0
MOTHERBOARD_SERIAL_NUM=FOC19126A4E
POE1_ASSEMBLY_NUM=73-14095-01
POE1_REVISION_NUM=B0
POE1_SERIAL_NUM=FOC191160L8
POE2_ASSEMBLY_NUM=73-14095-01
POE2_REVISION_NUM=B0
POE2_SERIAL_NUM=FOC191160L1
RANDOM_NUM=1572356125
RECOVERY_BUNDLE=sda9:cat3k_caa-recovery.bin
STKPWR_ASSEMBLY_NUM=73-11956-08
STKPWR_REVISION_NUM=B0
STKPWR_SERIAL_NUM=FOC191200AJ
SWITCH_NUMBER=
SWITCH_PRIORITY=
SYSTEM_SERIAL_NUM=FCW1912C16E
TAN_NUM=800-37564-03
TAN_REVISION_NUMBER=B0
TEMPLATE=advanced
TERMLINES=0
USB_ASSEMBLY_NUM=73-12923-05
USB_REVISION_NUM=C0
USB_SERIAL_NUM=FOC19122U1Z
VERSION_ID=V04

switch:

[Brian Luger]

I'm having the same issue, it's a 3850-48P-S. I've followed the document below for the emergency-install procedure and It still comes back :

Verification Error: bootloader failed self-verification!

switch:

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/3se/system_management/configuration_guide/b_sm_3se_3850_cg/b_sm_3se_3850_cg_chapter_01110.html#concept_6E610BCFF5174E498454F09CB6489799

Thank you.

[KHAJA FAZLE USMANI]

I have tried emergencyinstall procedure also before. my problem is that the flash: is in "read-only" mode.

How can i get flash: in "Read-Write" mode. so that the fresh IOS can be installed.

Thanks for your inputs..

[joleloves12]

I've just done well. don't care about flash. Go the lastest version of your ios of your switch, and try  to boot from the ios on usbflash. It means that you have to copy the images ios on usb bootable.

once done, put the image on the "boot system", save and verify before reload the switch.

Hope it'd would help you!

[john.pan.s]

Thank you Morteza this was so helpful for me cause I was able to boot at least but when I reload the switch it again went to switch: mode. Also it seems by this method it runs in bundle mode instead of install mode. what I did is that I did boot again from usbflash but this time I used Software expand running to flash: xxxxx.bin command to install the image ( cause you can't use software install file command ).  

[SantiagoR64237]

How and where do you get the last version? How much?

I'm new with Cisco products and trying to get some knowledge.

Thanks!

[InayathUlla Sharieff]

Joshua,

Plesae find quick guide on upgrading and booting, see below as a reference.

Recovering from a 3850 boot failure.


 There are multiple reasons a 3850 may fail to boot correctly including a corrupt boot image, a corrupt packages.conf file, missing files, etc.  Below are a few different possible recovery methods to try.  I will also explain the two possible mode options, Install and bundle and why you might want to use one or the other.



Install vs. Bundle Mode



There are a few difference in the two modes, I would recommend reading over the config guide for more in-depth details. The recommended mode during operation is INSTALL mode because it allows for more features and requires fewer resources when booting.



++Install Mode

This is the out-of-the-box mode that your switch will be in.  INSTALL mode uses a package provisioning file named packages.conf to boot the switch.

If you happen to be in bundle mode upon boot, you can simply boot your switch in install mode by booting the software package provisioning file that resides in flash. If packages.conf doesn�t exist in flash, you need to expand the bundle into the flash file system by running



Switch# software expand file flash: cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin to flash:



Once this completes, you will have all the needed files in flash. You can then change the boot statement to boot to packages.conf



Switch#Config t
Switch(config)# no boot system
Switch(config): boot system switch all flash:packages.conf  (do not modify this file, unless necessary)
Switch#write memory



The provisioning file contains a list of software packages to boot, mount, and run. The ISO file system in each installed package is mounted to the root file system directly from flash.



NOTE **Auto-upgrade is disabled, by default. (once in install mode - execute the following command in global config: software auto-upgrade enable )

NOTE **Auto-upgrade includes an auto-copy process and an auto-extract process.



++Bundle Mode

As noted previously, bundle mode consumes more memory than booting in install mode because the packages are extracted from the bundle and copied to the RAM.  If you decide to convert to bundle mode, you will first need to download the .bin file from CCO if you don�t already have it in flash.  Once in flash, you can simply change your boot statement to point to the (.bin) file:



Switch#Config t
Switch(config)# no boot system
Switch(config): boot system switch all flash: cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
Switch#write memory



The provisioning file contained in a bundle is used to decide which packages to boot, mount, and run. Packages are extracted from the bundle and copied to RAM.



NOTE **Auto install and smart install functionality is not supported in bundle boot mode.





Recovery Methods



USB

The 3850 has a USB port on the front that can be used for both console access and also the ability to utilize a flash drive for image backup and recovery.



If you happen to be stuck at the switch: prompt with a corrupt image or .conf file, you can easily boot to a file stored on the USB drive.



1. Verify that the flashdrive is recognized and the .bin file exists

switch: dir usbflash0:
Directory of usbflash0:/

74  -rw-  223734376  cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin



2. Boot to the USB image

switch: boot usbflash0:cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin





Corrupt packages.conf

I�ve seen instances in which packages.conf continually calls files that no longer exist in flash.  You can boot to an image from ROMMON just fine, however upon reload it will call packages.conf again and fail to boot.  If this happens, I recommend backing up the existing packages.conf file by renaming it or deleting all together.  NOTE: The previous step is mandatory as the next step will fail if a .conf file already exists.  You can then run an BUNDLE extract which will create a new packages.conf file.



1. Once booted up (in BUNDLE mode) verify the files in flash

Switch#dir flash:
Directory of flash:/



15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf



2. Copy or rename the existing packages.conf file

Switch#cp flash:packages.conf flash:packages.conf.badop flash:packages.conf flash:packages.conf.bad
Destination filename [packages.conf.bad]?
Copy in progress...C
1243 bytes copied in 0.140 secs (8879 bytes/sec)
Switch#dir flash:
Directory of flash:/
15500  -rwx        1243   Aug 1 2013 07:04:02 +00:00  packages.conf
15502  -rw-        1243   Aug 1 2013 11:53:51 +00:00  packages.conf.bad



3. Delete packages.conf

Switch#del flash:packages.conf
Delete filename [packages.conf]?
Delete flash:/packages.conf? [confirm]



4. Expand BUNDLE to create new packages.conf

Switch#software expand running switch 1 to flash:
Preparing expand operation ...
[1]: Expanding the running bundle
[1]: Copying package files
[1]: Package files copied
[1]: Finished expanding the running bundle



5. Verify boot

Switch#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no



6. Reload Switch

switch#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]





Emergency Recovery

If all else fails, the 3850 has a �trap door� method of recovering the system.  All you need is a terminal connected to the management port of the 3850 running a tftp server.  Download a valid image file from CCO and store it in the root of the tftp server.

On the switch, you are most likely stuck at the switch: prompt.  If however you are in some sort of boot loop, you can use the �mode� button on the front of the switch to break the cycle.  Simply hold the button for roughly 10 seconds and the switch should react by breaking the cycle and stopping at a switch: prompt. The following steps will walk you through the recovery:



1. Set the switch IP

switch:  set IP_ADDR 192.0.2.123/255.255.255.0



2. Set the default gateway

switch: set DEFAULT_ROUTER 192.0.2.1



3.Test connectivity by pinging terminal (that contains the tftp server)

switch: ping 192.0.2.1
ping 192.0.2.1 with 32 bytes of data ...
Host 192.0.2.1 is alive.



4. Verify that the emergency files exist in the switches file system

switch: dir sda9:
Directory of sda9:/

    2  drwx  1024       .
    2  drwx  1024       ..
   11  -rwx  18958824   cat3k_caa-recovery.bin

36903936 bytes available (20866048 bytes used)


5. Run the emergency install feature

switch: emergency-install tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin
The bootflash will be erased during install operation, continue (y/n)?Y
Starting emergency recovery (tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin)...
Reading full image into memory......................done
Nova Bundle Image
--------------------------------------
Kernel Address    : 0x6042f5d8
Kernel Size       : 0x317ccc/3243212
Initramfs Address : 0x607472a4
Initramfs Size    : 0xdc6546/14443846
Compression Format: .mzip

Bootable image at @ ram:0x6042f5d8
Bootable image segment 0 address range [0x81100000, 0x81b80000] is in range [0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point: 0x811060f0
Loading Linux kernel with entry point 0x811060f0 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Initiating Emergency Installation of bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin

Downloading bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...

Validating bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Installing bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Verifying bundle tftp://192.0.2.1/cat3k_caa-universalk9.SPA.03.02.02.SE.150-1.EX2.bin...
Package cat3k_caa-base.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-drivers.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-infra.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-iosd-universalk9.SPA.150-1.EX2.pkg is Digitally Signed
Package cat3k_caa-platform.SPA.03.02.02.SE.pkg is Digitally Signed
Package cat3k_caa-wcm.SPA.10.0.111.0.pkg is Digitally Signed
Preparing flash...
Syncing device...
Emergency Install successful... Rebooting
Restarting system.


Please let me know if you have any further questions.

HTH

Regards

Inayath

[Amos Mandishona]

Thanks the above Procedure works like a Champs

Regards

Amos

[plee_northdevonnhs]

Thank you Inayath!

You have saved me hours of research/testing. Your information above can only be described as a golden nugget. The packages.conf file seems to be corrupt on every model I have tried an upgrade on and I have tried several switches across the following model types 3850+3650. Will there be a Cisco fix on next release?