3850 incomplete software install, what path forward?

[markmayfield]

I was working to upgrade a 4-switch 3850 stack from cat3k_caa-universalk9.SPA.03.03.01.SE.150-1.EZ1.bin (Install mode) to 03.07.04, and it failed to copy to one of the switches because it had an additional image still on that one. 

I cleared that out to make space, but then I tried to run a software clean; sat with no output for over an hour, and became disconnected when I had to shut down my laptop.

Then I tried the install process again; no luck - it sits with no output.  

I found all those VTY sessions remained hung on the switch, so I cleared them with the "clear tcp tcb xxxx" command, and ran the install again, this time in verbose mode - it sits with no output.

A show ver results in this:

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 56 WS-C3850-48P BUNDLE UNKNOWN IMAGE NAME UNKNOWN BUNDLE
2 56 WS-C3850-48P BUNDLE UNKNOWN IMAGE NAME UNKNOWN BUNDLE
3 56 WS-C3850-48P BUNDLE UNKNOWN IMAGE NAME UNKNOWN BUNDLE
4 56 WS-C3850-48P BUNDLE UNKNOWN IMAGE NAME UNKNOWN BUNDLE

Which is a bit unnerving.  It's still up and working, but I don't really want to reboot from this state without being onsite since it's a couple hours drive away. 

I'd prefer to find a way to get a reasonable recovery without a site visit, if anyone's seen this happen before.

Thanks,

Mark  

[Leo Laohoo]

Hmmmm ... that's a disturbing output.
The issue is we don't know if the firmware was actually loaded completely into each stack member.
I would recommend that you attempt to re-install the firmware again, HOWEVER, instead of instructing the unpacking of the firmware to ALL SWITCHES, specify the unpacking on a per-switch-member basis and pay attention to the output.
Does this make any sense?

[markmayfield]

Thanks for the idea. 

It doesn't seem to help though.  I tried on each member, each from it's own bin file copy in flash, and it just hung with no output each time.  Verify returns OK on the bin file.

I tried one with the "new" modifier to create a new package provisioning file, and that didn't help either.

Mark

[Leo Laohoo]

What is the uptime of the all the switches?

[markmayfield]

They are all up 1 year, 6 weeks.

Any thoughts if it's safe to delete packages.conf and try starting over that way?

Thanks,

Mark

[Leo Laohoo]

First off, I've never seen a behaviour like this ...
But then again the current stack is running 3.3.1 ... and the uptime is >1 year. I cannot begin to tell you which one is worst, the uptime or the version number.
Either way, the only way out is to reboot the entire stack. If the new version kicks in, then good. If the old version kicks in, then do the firmware upgrade again but a reboot is definitely required so it will clean/clear out all the "gunks" hiding.

[markmayfield]

Yeah...

I'm considering setting it to boot from the .bin during my maintenance window, hoping that gets it online for sure, then transition it back to install mode.  I'm just not excited about the 2+ hour drive in a rush I'm looking at if that doesn't work.

Right now I renamed the packages.conf on one stack member and am re-uploading the new code and planning to do an expand and see how that looks.

I searched a bunch, and cant find any reference to this state, which surprised me to the point of starting this post.  However it shakes out, I'll try and be sure to update the thread.  

Thanks

Mark

[markmayfield]

I was originally hoping to set the switches to boot in bundle mode from the newer code, and get this done in two reboots, but the switch that had failed the original copy wouldn't report additional free space on the flash, even after deleting other files.  

This left me with the best choice being to set the boot variable for all switches to the older .bin file (03.03.01.SE) and reboot them in bundle mode.  I did this, and all four switches came up fine, and the problem switch properly reported free flash space.

Now I had room to copy the new .bin and run a verify on each copy.  I then set the boot variable to that version (03.07.04E).  Note: This boot to upgrade the code took 27 minutes.

After that, did the "software expand running" and reset the boot variable to packages.conf, and reloaded one more time to get up and running in install mode.  Overall, slightly more than an hour of downtime.