C3850 upgrade failure

[krzysztofmaciejewskiit]

I wanted to upgrade the C3850 in stack from version 16.12.8 to version 16.12.11 in BUNDLE mode.
The steps I followed:
   1. copy tftp: flash: .... -> successfully
   2. no boot system
   3. boot system switch all flash:cat3k_caa-universalk9.16.12.11.SPA.bin
   4. write
   5. show boot -> everything was ok
   6. reload

After the upgrade only the Active Unit updated and the Member went into rommon. I wanted to upload new software to Member over USB, because when I did a “dir flash:” on it, I didn't see the uploaded software in its flash. However, I only had a 64GB flash drive, which I could not format as FAT32, and on NTFS and exFAT the switch was throwing errors. I ended up doing an Active Unit rollback because the maintenance window was ending.

What do you advise, is it better to convert the software mode to INSTALL from BUNDLE? On other switches I've upgraded, I always did the upgrade in INSTALL mode and all was well.
However, did I perhaps do something wrong?

Just to add, in the logs right after the failed upgrade in rommon I saw some log about lack of space. 
copy flash:cat3k_caa-universalk9.16.12.11.SPA.bin flash-2:cat3k_caa-universalk9.16.12.11.SPA.bin
Destination filename [cat3k_caa-universalk9.16.12.11.SPA.bin]?
%Error copying flash:cat3k_caa-universalk9.16.12.11.SPA.bin (Not enough space on device)


I'm also wondering what command to use to clear unused files. The software clean command does not work.

[Kasun Bandara]

@krzysztofmaciejewskiit hi, hope below link will explain you how to proceed with upgrade from bundle mode to install mode. now cisco recommends to use install mode since efficient memory and process management capabilities.

https://genkiganko.medium.com/how-to-upgrade-your-cisco-3650-3850-catalyst-switches-dec09610d767

[krzysztofmaciejewskiit]

Yes it was my next idea for a second approach, but I would also like to know the cause of this problem. From what I see in the instructions, perhaps I used the wrong command?
Maybe I should use this command software expand file flash:cat3k_caa-universalk9.16.03.03.SPA.bin to flash:

[Kasun Bandara]

@krzysztofmaciejewskiit hi, yes. when you copying manually, if any of member switches dont have enough space to copy OS, it will fail to upgrade.,

[krzysztofmaciejewskiit]

I don't really understand what the reason for this lack of space is. As all files seem to be needed. Also this switch has not been touched since new, so no one has put anything in there. I tried to delete unused packages however the command software clean (https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html#toc-hId-1661774022) does not work for me.

[Leo Laohoo]

Cisco 3850: IOS-XE/Firmware Upgrade

[krzysztofmaciejewskiit]

When I read carefully I noticed this command ‘install remove inactive’, it seems that this is what I am looking for. Since my goal is to switch from BUNDLE to INSTALL mode and upgrade the software from 16.12.08 to 16.12.11:
1. So, I will first remove inactive packages with the ‘install remove inactive’ command.
2. then I'll make packages from the current software .bin file with the command ‘software expand file flash:cat3k_caa-universalk9.16.12.11.SPA.bin to flash:’.
3. then I will convert the mode from BUNDLE to INSTALL by setting BOOT = flash:packages.conf to the variable
After successfully converting from BUNDLE to INSTALL it will update the software with the command ‘install add file flash:cat3k_caa-universalk9.16.12.11.SPA.bin activate commit’.

Will this procedure be correct?

[Leo Laohoo]

---

@krzysztofmaciejewskiit wrote:
2. then I'll make packages from the current software .bin file with the command ‘software expand file flash:cat3k_caa-universalk9.16.12.11.SPA.bin to flash:’.
3. then I will convert the mode from BUNDLE to INSTALL by setting BOOT = flash:packages.conf to the variable
After successfully converting from BUNDLE to INSTALL it will update the software with the command ‘install add file flash:cat3k_caa-universalk9.16.12.11.SPA.bin activate commit’.

---

WTF is this?  

Either someone did not read my process or someone really did not read.  Period.

[krzysztofmaciejewskiit]

Of course I have read and taken out what I need. It seems to me that you also do not fully understand the problem I am considering here. Because this post is not just about me wanting to update the software on the C3850 switch in INSTALL mode. This post is about the failed first attempt to update the C3850 stack in BUNDLE mode due to lack of space in the second unit and about the initial ignorance of how to clear unused packages, about converting from BUNDLE to INSTALL mode and finally about updating in INSTALL mode after converting modes. Nevertheless, thank you for your help.

[Leo Laohoo]

---

@krzysztofmaciejewskiit wrote:

1. So, I will first remove inactive packages with the ‘install remove inactive’ command.
2. then I'll make packages from the current software .bin file with the command ‘software expand file flash:cat3k_caa-universalk9.16.12.11.SPA.bin to flash:’.
3. then I will convert the mode from BUNDLE to INSTALL by setting BOOT = flash:packages.conf to the variable
After successfully converting from BUNDLE to INSTALL it will update the software with the command ‘install add file flash:cat3k_caa-universalk9.16.12.11.SPA.bin activate commit’.

---

This process is totally different to "This post is about the failed first attempt to update the C3850 stack in BUNDLE mode due to lack of space in the second unit and about the initial ignorance of how to clear unused packages, about converting from BUNDLE to INSTALL mode and finally about updating in INSTALL mode after converting modes."

Only Step 1 is valid and makes sense.  Step 2 & Step 4 are redundant with Step 2 "dangerous".  Step 3 is out of place or in the wrong spot.  

NOTE:  With Step 2, there will be no opportunity to do Step 3 because the switch will reboot before Step 3 can be done.  Hence, I never encourage anyone to use the "software expand" command.

[krzysztofmaciejewskiit]

Thank you for sharing what's wrong, this is important for me in my learning process. I will take your advice into consideration when updating.

[MOEL]

Step-by-Step Upgrade Process

1. Download the new image: Obtain the desired Cisco IOS XE software package (e.g., cat3k_caa-universalk9.SPA.03.06.06.E.152-2.E6.bin) and place it in the TFTP server.

1. Extract the bundle image: Use the software expand command to extract the bundle image to flash memory:

3850# software expand running to flash:

This command will create a running-packages.conf file and extract the necessary packages to flash memory. 3. Verify the boot variables: Use the show boot command to verify the boot variables, including the BOOT variable, CONFIG_FILE variable, and BOOTLDR variable.

4. 4.   Remove the old boot command: Use the no boot system command to remove the old boot command:

3850(config)# no boot system switch all flash:cat3k_caa-universalk9.SPA.03.02.03.E.152-2.E3.bin (old version)

1. Set the new boot command: Use the boot system command to set the new boot command, pointing to the extracted bundle image:

1. 3850(config)# boot system switch all flash:cat3k_caauniversalk9.SPA.03.06.06.E.152-2.E6.bin (new version)

1. Verify the boot variables again: Use the show boot command to verify the boot variables once more.
2. Reload the switch: Use the reload command to reload the switch:

3850# reload

The switch will boot with the new image.

Important Notes

• The Bundle mode consumes more memory than the Install mode because the packages are extracted from the bundle and copied to RAM.
• The Auto-upgrade feature is disabled by default and is not available in Bundle mode.
• Ensure you have sufficient flash memory space available before upgrading.

Verification

After the upgrade, verify the switch is running the new image by using the show version command:

3850# show version

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.06.06.E RELEASE SOFTWARE (fc2)

[krzysztofmaciejewskiit]

Why do I have to create packages from a .bin file and then set the BOOT variable to a .bin file again? Shouldn't we aim to convert from BUNDLE to INSTALL mode?

[Deepak Kumar]

Hi, 

It depends on what mode is suitable for you, I prefer to Install mode due to additional benefits but if you want to go with bundle mode it is also ok. It is official support and no harm with it. 

As you said you had run through the rollback process and I am hoping that both switches are in a stack and working now. You can follow the following steps for:

1) Clean all unnecessary files from Flash-1 and Flash-2. I will check for any big crash files, old firmware versions, etc. Delete all those files from both files. 

2) Always run the "verify /md5 flash-x:"image name"" command from both flashes. If the values match with the Cisco Website, you are good to go. 

3) If I am updating a switch in bundle mode and if I have any doubt about failure. Especially when I am working from remotely then I always go with failsafe mode (this is my term), which means it always adds old firmware as a backup image as below:

no boot system switch all flash:cat3k_caa-universalk9.<old image>.bin

boot system switch all flash:cat3k_caa-universalk9.<new image>.bin

boot system switch all flash:cat3k_caa-universalk9.<old image>.bin

These steps saved me many times, back in the day. 

[krzysztofmaciejewskiit]

I was working remotely, but fortunately had a technician with a laptop and console in backup.
This way with setting up two software is great, I hadn't thought of that.

Tell me, what would happen if one unit updated and the other didn't? Does the whole stack then get up with one version, e.g. the one before the upgrade. Or does one unit get up with the higher version and the other due to the fail get up with the old software.
If the software is not consistent then, from what I have observed, the stack will not be available.