New 3850 stack, upgrade problem (3.x -> 16.x)

ianatkinson · ‎11-01-2017

Hi everyone,

We recently bought 5 switches, they are all WS-C3850-48P-S

I started to unpack them and make a stack out of them, when I got to the third one it wouldn’t stack which seemed odd.

I noticed that the first two were running OS 16.3.3 and the third one was running OS 03.06.06E, looking at the boxes the first two were marked July 2017 and the third one May 2017. I guess between those dates there was some big change in the OS? In total of the 5 switches 2 are May ones and 3 are July ones.

Looking at these release notes here for 16.3.x:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-3/release_notes/ol-16-3-3850.html

It states:

"Previously, Cisco IOS XE Denali 16.x.x releases supported this feature only on switches running an IOS XE Denali 16.x.x image joining an existing stack with a different Cisco IOS XE Denali 16.x.x image version. Starting with this release, the active switch can resolve a mismatch across Cisco IOS XE Release 3.xE and Cisco IOS XE Denali 16.3.x releases.”

That suggests to me that the newer switches running 16.3.3 should be able to update these older switches running 03.06.06E?

It states in the notes that this isn’t active by default and you need to add 'software auto-upgrade enable’ to the running config. I’ve done that and also ensured that all the switches are in install mode not bundle mode and finally that the licenses match (ipbase permanent) which seems to be all the pre-requisites. Hardware wise they seem identical and are all version 7 from the look of it.

Try as I might though I can not get the newer switches to upgrade the old ones. If I boot all 3 switches together it pauses the boot sequence saying there are switches in the stack with incompatible OS and only continues to boot when I disconnect the older switch.

If I get the new ones booted up and then connect the older one no auto upgrade takes place, it shows the switch and assigns it a number but says ‘OS-Mismatch’ like this:

H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 0008.3237.8100 1 V07 Ready
2 Standby 0008.3237.8d00 1 V07 Ready
3 Member 009a.d2ee.1c00 0 OS-Mismatch

I also tried issuing the 'request platform software package install auto upgrade’ command which didn’t seem to do anything, it pauses for a couple of seconds but there are no messages or anything like that.

The final thing I thought to try was booting the older switch first and then adding in the newer switch. This actually caused the old switch to downgrade the new switch to to the 3.x release which I wasn’t expecting at all! Oops :(

I’m a bit stuck now, am I doing something wrong here or is this procedure not possible? I really need to get them all stacked and on the 16.3.3 release if I can. Other than the notices mentioned above very little seems to get logged about it.

We don't have a cisco support contract so I'm not able to download any software for these, I need to get the switches to just upgrade on the bench. I don't have any experience with this new Linux/iOS type ones but when I've previously done this with 3750 series switches they seemed to upgrade each other with no problems.

Thanks,

Ian

Bob Loblaw · ‎11-01-2017

Maybe try just isolating the switch with the outdated OS, and updating it locally? Transferring the image from one of the newer switches via tftp to your PC, and then xferring that over to the old switch. Then try stacking them while they're on the exact same OS.

If this post was any helpful please rate it so, thanks !

ianatkinson · ‎11-06-2017

Hi Bob,

Thanks for the suggestion, it's frustrating that it isn't doing this automatically given the cost but I guess I can try it manually if that's what's required!

I've never upgraded a switch in this way before, I read the docs and it's suggesting that I need a .bin file to put on there?

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/117552-technote-cat3850-00.html#anc8

As these are in the new install mode they don't seem to have that on there just the package files:

new_gsw_acad#dir flash:

Directory of flash:/

38814 -rw- 2097152 Nov 1 2017 12:42:04 +00:00 nvram_config

38803 -rw- 15954560 Jul 3 2017 05:07:27 +00:00 cat3k_caa-guestshell.16.03.03.SPA.pkg

38804 -rw- 22173639 Jul 3 2017 05:07:24 +00:00 cat3k_caa-rpbase.16.03.03.SPA.pkg

38805 -rw- 264899192 Jul 3 2017 05:07:26 +00:00 cat3k_caa-rpcore.16.03.03.SPA.pkg

38806 -rw- 9091712 Jul 3 2017 05:07:25 +00:00 cat3k_caa-srdriver.16.03.03.SPA.pkg

38807 -rw- 191324788 Jul 3 2017 05:07:25 +00:00 cat3k_caa-wcm.16.03.03.SPA.pkg

38808 -rw- 13404796 Jul 3 2017 05:07:25 +00:00 cat3k_caa-webui.16.03.03.SPA.pkg

38809 -rw- 4737 Jul 3 2017 05:07:49 +00:00 packages.conf

38810 -rw- 407 Nov 1 2017 12:36:09 +00:00 bootloader_evt_handle.log

69841 drwx 4096 Jul 3 2017 05:19:51 +00:00 core

77601 drwx 4096 Jul 3 2017 05:10:05 +00:00 .prst_sync

11 drwx 4096 Jul 3 2017 05:10:07 +00:00 .rollback_timer

7762 drwx 4096 Nov 1 2017 12:36:31 +00:00 dc_profile_dir

7764 drwx 4096 Jul 3 2017 05:10:17 +00:00 gs_script

38811 -rw- 65301 Nov 1 2017 12:36:48 +00:00 memleak.tcl

15521 drwx 4096 Jul 3 2017 05:10:47 +00:00 .installer

38812 -rw- 34 Jul 3 2017 05:19:57 +00:00 pnp-tech-time

38813 -rw- 21443 Jul 3 2017 05:20:06 +00:00 pnp-tech-discovery-summary

69843 drwx 4096 Oct 30 2017 16:10:42 +00:00 onep

38815 -rw- 736 Nov 1 2017 12:41:52 +00:00 vlan.dat

1624104960 bytes total (1019162624 bytes free)

new_gsw_acad#

Do you suppose this would work if I just copy over all those files? Or if I change it into bundle mode will it maybe generate the required .bin file? As I say we don't have a support contract so I can't download this.

Ian

Bob Loblaw · ‎11-06-2017

If the .bin exists on any of the other switches you can move it to a TFTP server and then TFTP it to the switch that needs it locally.

Let's say SW1 has the image

SW1# copy flash:cat3k.bin tftp:

SW1#Address or name of remote host: Your TFTP server IP(Computer with TFTP client installed)

SW1#Destination filename: Just hit enter

Then on SW that is connected to your TFTP

SW3#copy tftp:cat3k.bin flash:

Same as before.

You might need to connect yourself to the SW and assign yourself a static IP on the same subnet as one of the interfaces on the SW that way it's able to communicate with your computer.

Hope this helps.

If this post was any helpful please rate it so, thanks !

ianatkinson · ‎11-07-2017

Hi,

I don't think any of the 16.x switches have a bin file on, they seem to have left the factory without it. The 3.x switches also don't have a bin file, it seems to me that they now come in the install mode without the bin by default?

I don't think you can change to bundle mode and make a bin appear either, it says in the docs to specifically make sure you have a bin file available before switching to this mode so it must rely on it; it's not going to make one.

I did wonder if maybe the reason my auto-upgrade isn't working is because the bin file is required, however since there is no bin on the 3.x switch and it downgraded the 16.x switch then it can't be required for the auto upgrade process?

I think at this point my only option is to downgrade all the switches and just run them on 3.x however I will need to add more switches to the stack next year so it might cause problems down the line. Maybe next year's switches will be able to upgrade all of this years successfully, if not we'd have to look into buying the support thing so I can get at the files.

Ultimately if I need the bin file and it's not on the switches then I'm stuck with that as my only option.

This does seem a bit crappy to me though, you spend £40k on some switches which are supposed to have stacking capabilities and can't stack them without spending even more on downloads :/

karin.westmoreland! · ‎02-27-2018

Change your priority on the switches make the master 15, the other switches 10, 5. The other switches will assume the IOS of the master switch.

Leo Laohoo · ‎11-01-2017

Take the 3rd switch off the stack and upgrade it individually.