Upgraded 3650 slave by mistake

[shower]

I'm struggling to learn Cisco 3650 stacked switches and need help. I successfully upgraded a 3650 switch with version 16.12.12 software. Unfortunately, I started with the slave. All went well until I upgraded the master. Now I can't reach either switch or any device connected to the switches. I power cycled both switches, and they came back up as follows:

Master: SYS light solid green. No other lights.

Slave: STBY, SYS, and ACTV lights all solid green. No other lights.

And I still can't reach the switches, or any device connected to the switches.

Can anyone help? Thanks!

[Leo Laohoo]

Two options: 

Option 1:  

conf t
 software auto-upgrade enable
end
wr

Option 2: 

In Enable Mode, enter the following command: 

install autoupgrade

[shower]

Thanks, but I can't reach either switch. I worked remotely and putty in. I imagine someone will have to go onsite and connect to the console port, but I'm hoping to avoid that. There are people on site who can power cycle the switch, or switches, and tell us the status of the LEDs but that's all.

[Jens Albrecht]

Looks like you broke the stack but without console access you don't know the exact condition right now.

Based on the fact that you work remote but have people on-site you can try the following:

• Power down the slave and check whether you can connect to the master remotely
• If not, power cycle the master while the slave is still down, then check connectivity to the master again
• If you can still not connect remotely, you probably have to go on-site and use the console port
• If you can connect to the master, use option 2 that @Leo Laohoo mentioned to upgrade the master
  Once the master is upgraded and back online, power on the slave and check whether it rejoins the stack

In case you have some kind of terminal server at your site you can also check whether you can use it to connect to the console port of your devices.

HTH!

[pieterh]

If there is still some network connection possible then As alternative to the terminal server mentioned by @Jens Albrecht  you can remote desktop into a PC with serial connection to the console

[shower]

Thanks all. I made one mistake in my original post: the Master's system LED never stops flashing, so I believe that means it did not pass POST.

We tried the power down and cycle sequence suggested by @jens with no luck. Also tried disconnecting the Stack cables, power cycling, and tried to reach either switch. No luck.

We'll try going through the console port next.

[Leo Laohoo]

Console into the switch and reboot. 

Post the entire bootup process.

[shower]

Thanks @leo. Here is what I see:

Booting...
Interface GE 0 link down***ERROR: PHY link is down
flash: cat3k_caa-universalk.9.16.12.12.SPA.bin: no such file or directory
Cannot open file flash: cat3k_caa-universalk.9.16.12.12.SPA.bin

The system is unable to boot automatically. the boot environment variable needs to be set to a bootable image.

[Jens Albrecht · Accepted solution]

So I assume you are in rommon mode now.
What is the content of the flash? Does it contain any IOS bin file?

If yes, try to manually boot it with "boot flash:<filename.bin>".
If not, prepare an USB stick with the IOS and boot from there.

[pieterh]

Re-heck the image name

In my memory ...universallk<dot>9.. .. does not sound right.

it should have been k9 without <dot>

[shower]

Success! We had to boot the master from the USB stick, then save the new software to flash, then write it to memory. Then powered up the slave and it joined the stack. As a final test we power cycled both switches and checked to see that they both were running the new software.

Than you for all of your help!!!

[Jens Albrecht]

You're welcome! Glad to hear that your stack is finally back online!