Solved: 3850 install to bundle in one reload

waqas gondal · ‎01-06-2021

Hi, Happy New Year!

I was wondering if it is possible to change the boot mode from install mode to bundle mode without reloading twice (once to change the boot mode and again to upgrade).

I ask because we have a fleet of 3850 stacks running 3.3.5SE which is very buggy and Cisco won't allow downloading it anymore. That version has an issue where the flash on some switches is corrupted so we have to boot from usb in bundle mode first. Only after the upgrade is the bug fixed and we can continue to 16.x

Is there an easier way to do this?

Thanks,

Waqas

Leo Laohoo · ‎01-06-2021

@waqas gondal wrote:

I ask because we have a fleet of 3850 stacks running 3.3.5SE which is very buggy and Cisco won't allow downloading it anymore. That version has an issue where the flash on some switches is corrupted so we have to boot from usb in bundle mode first. Only after the upgrade is the bug fixed and we can continue to 16.x

That does not sound good.

IMPORTANT: Before I start, what 16.X.X firmware are you planning to go?

If I was to recover this stack, this is what I would be doing:

1. Remove the switch with a corrupted flash content from the stack because they will be of no use.

2. For the remainder of the stack, clean the flash first (important).

software clean force

3. Copy the firmware filename to the 3850 and "unpack":

software install file flash:IOS-XE_filename.bin new force verb on-reboot

NOTE:

The last option, "on-reboot", instructs the stack DO NOT REBOOT after the unpacking.
The "on-reboot" option is a bug -- It is meant to be "no-reboot" but Cisco left it that way (and I am happy to remind people about it).

4. Make sure the stack will reboot the new package.conf file by verifying the boot variable string:

show boot

5. Verify the contents of the packages.conf file.

more flash-X:packages.conf | begin for NOVA

IMPORTANT: Check every stack member for the contents of the packages.conf file. EVERY SWITCH MEMBER OF THE STACK.

6. Reboot the stack.

Ok. At this stage, the stack is rebooting but the switch with a corrupted flash is dead in the water. So the following steps below will be a continuation ...

6. The fastest way to copy the firmware file into a switch that is in ROMMON is via a supported USB thumb drive.

7. Initiate the Emergency Install so the switch can boot into the correct 16.X.X firmware.

emergency-install flash:IOS-XE_filename.bin

8. Wait for the switch to boot up. Once it boots up, power it down, attach the stacking cables and power the switch up.

That should be it.

View solution in original post

Leo Laohoo · ‎01-06-2021

@waqas gondal wrote:

I ask because we have a fleet of 3850 stacks running 3.3.5SE which is very buggy and Cisco won't allow downloading it anymore. That version has an issue where the flash on some switches is corrupted so we have to boot from usb in bundle mode first. Only after the upgrade is the bug fixed and we can continue to 16.x

That does not sound good.

IMPORTANT: Before I start, what 16.X.X firmware are you planning to go?

If I was to recover this stack, this is what I would be doing:

1. Remove the switch with a corrupted flash content from the stack because they will be of no use.

2. For the remainder of the stack, clean the flash first (important).

software clean force

3. Copy the firmware filename to the 3850 and "unpack":

software install file flash:IOS-XE_filename.bin new force verb on-reboot

NOTE:

The last option, "on-reboot", instructs the stack DO NOT REBOOT after the unpacking.
The "on-reboot" option is a bug -- It is meant to be "no-reboot" but Cisco left it that way (and I am happy to remind people about it).

4. Make sure the stack will reboot the new package.conf file by verifying the boot variable string:

show boot

5. Verify the contents of the packages.conf file.

more flash-X:packages.conf | begin for NOVA

IMPORTANT: Check every stack member for the contents of the packages.conf file. EVERY SWITCH MEMBER OF THE STACK.

6. Reboot the stack.

Ok. At this stage, the stack is rebooting but the switch with a corrupted flash is dead in the water. So the following steps below will be a continuation ...

6. The fastest way to copy the firmware file into a switch that is in ROMMON is via a supported USB thumb drive.

7. Initiate the Emergency Install so the switch can boot into the correct 16.X.X firmware.

emergency-install flash:IOS-XE_filename.bin

8. Wait for the switch to boot up. Once it boots up, power it down, attach the stacking cables and power the switch up.

That should be it.

waqas gondal · ‎01-06-2021

Thanks Leo!!

The version I plan on going to is 16.9.6 (current recommended by Cisco). The flash bug is in the IOS apparently so an upgrade should fix the problem when it boots from usb.

I have about 183 switch stacks of different sizes so isolating the ones which have the bug would take a while. Plus the 3.3.5 is end of support this June so time is limited.

Cheers,

Waqas

Leo Laohoo · ‎01-06-2021

@waqas gondal wrote:

The version I plan on going to is 16.9.6 (current recommended by Cisco).

Bad choice.

16.9.X has a memory leak bug affecting the "repm" (replication module/manager): CSCvv66845.
Ok if the switch is standalone but the bug appears to affect stacks running 16.9.X. As far as I know (2 TAC cases attributed to this bug alone), this bug is not fixed in 16.9.6.

16.12.X has several critical bug involving SNMP which then "extends" to PoE: CSCvv28324 (>2 TAC Case). Applying the SMU will not work (1 TAC Case).

You're better off using the latest 16.6.X.

Leo Laohoo · ‎01-06-2021

@waqas gondal wrote:

I have about 183 switch stacks of different sizes

1. 3.X.X has a "time-bomb" bug

If you attempt to unpack (the firmware) to a stack with a very high uptime (>2 years) there is a strong chance the "unpacking" process will stop with some funny error message. Do not be alarmed by it. It is an "undocumented feature".

If this happens, reboot the entire stack and re-run unpacking command again and it would all be good.

2. Micro-code Upgrade

You are going from 3.X.X to 16.X.X. That is a major upgrade step. There will be a mandatory (one-time only) microcode upgrade. This will take between 14 to 24 minutes. This is normal.

3. Release Notes -- Read it! Your job will depend entirely upon it

I cannot emphasis this. Read the Release Notes very, very carefully. 3650/3850 running 16.X.X is fraught with danger (i. e. memory leaks and CPU hogs). Read the Release Notes so you will know what you will be up against.

4. Dot1X + 16.X.X + Link Flapping = Unstable stack

If you have Dot1X, make sure to "police" your downstream links. If you have links that are flapping, I can guarantee different stack members will go into a round-robin crash every 20 hours.

waqas gondal · ‎01-06-2021

Thanks Leo,

You think if we reload the stack before we begin the upgrade we could avoid the corrupted flash bug? I wish I had a stack I could test on.

Leo Laohoo · ‎01-06-2021

@waqas gondal wrote:

You think if we reload the stack before we begin the upgrade we could avoid the corrupted flash bug?

No. You have no choice. You need to upgrade the firmware.

waqas gondal · ‎01-06-2021

Aight