3850 XOS upgrade fails

[a.otto]

Catalyst 3850

Hello,

I'm trying to upgrade a stack of two 3850's from 03.07.04E to 16.06.05 using the command:
software install file usbflash0:cat3k_caa-universalk9.16.06.05.SPA.bin switch 1-2 new force

Unfortunately the upgrade process stops after copying the file to all switches and the error message is not very meaningful:

LAB-3850-1.Gustavsburg#$-universalk9.16.06.05.SPA.bin switch 1-2 new force
Preparing install operation ...
[1]: Copying software from active switch 1 to switch 2
[1]: Finished copying software to switch 2
[1 2]: Starting install operation
[1]: % Operation install cannot be performed: install in progress. Operation aborted.
[1]: % An internal error was encountered. Operation aborted.
[2]: % An internal error was encountered. Operation aborted.

I tried to reload the whole stack but no change. Google search was not helpful at all. Does anyone have an idea?

[Leo Laohoo]

Take switch 2 off (power it down completely)
Reboot switch 1
Unpack.

[a.otto]

Done, no success.



LAB-3850-1.Gustavsburg#sh switch
Switch/Stack Mac Address : 2852.61a3.bd80 - Local Mac Address
Mac persistency wait time: Indefinite

H/W Current
Switch# Role Mac Address Priority Version State
----------------------------------------------------------------------------
---------
*1 Active 2852.61a3.bd80 15 V08 Ready
2 Member 0000.0000.0000 0 Provisioned

LAB-3850-1.Gustavsburg#$aa-universalk9.16.06.07.SPA.bin new auto-copy
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

FAILED: switch local is not booted with provisioning file
LAB-3850-1.Gustavsburg#

[Leo Laohoo · Accepted solution]

software install file flash:cat3k_caa-universalk9.16.06.07.SPA.bin new force

Does this command make any difference?

Another option is to boot into ROMMON and then try: 

emergency-install flash:cat3k_caa-universalk9.16.06.07.SPA.bin

NOTE:  This process may erase the contents of the flash.  Let it do so. 

[a.otto]

Hi Leo,

The software command is no longer available as the switch was already
running Everest. I tried the emergency recovery from flash but if failed in
the first hand after initializing the Kernel:
switch: emergency-install flash:cat3k_caa-universalk9.16.06.07.SPA.bin

The bootflash will be erased during install operation, continue (y/n)?y
Starting emergency recovery
(flash:cat3k_caa-universalk9.16.06.07.SPA.bin)...
Reading full image into memory.......................done
Bundle Image
--------------------------------------
Kernel Address : 0x5342f5d4
Kernel Size : 0x41884e/4294734
Initramfs Address : 0x53847e24
Initramfs Size : 0xe77d99/15170969
Compression Format: .mzip

Bootable image at @ ram:0x5342f5d4
Bootable image segment 0 address range [0x81100000, 0x82140000] is in range
[0x80180000, 0x90000000].
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
File "sda9:cat3k_caa-recovery.bin" uncompressed and installed, entry point:
0x8166a600
Loading Linux kernel with entry point 0x8166a600 ...
Bootloader: Done loading app on core_mask: 0xf

### Launching Linux Kernel (flags = 0x5)

Initiating Emergency Installation of bundle
flash:cat3k_caa-universalk9.16.06.07.SPA.bin

Emergency Install failed - bundle
flash:cat3k_caa-universalk9.16.06.07.SPA.bin does not exist

Restarting system.



I tried again using USBFLASH0: instead and this time it went successful.
Afterwards I changed the boot variable and reloaded again:

Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
LAB-3850-1.Gustavsburg#



Again, thanks a lot for your great support and your patience. This was a
great exercise for me as well!

Best regards

Axel

[Leo Laohoo]

---

@a.otto wrote:
I tried the emergency recovery from flash but if failed in
the first hand after initializing the Kernel:

---

Axel, 

There are two reasons why this failed: 

• I was literally drunk
• "emergency-install" should always point to the USB because the process will erase the flash first and then unpack.  Nothing to unpack if the flash is wiped clean. 

Finally, I think I know the reason why it has been failing:  3650/3850 had several design defects and one of those is the size of the flash.  There is not enough to put three or more unpacked firmware and most of the the time it barely has room for two.  Looking at the "dir flash-1:" output shows the switch to have 2 Gb flash but only >400 Mb free space.  

This is why erasing the flash and then unpacking it worked.  

Happy to see it is finally working.