3850 wiped after IOS-XE upgrade

[newberntac]

I upgraded the IOS on one of my 3850s this past weekend.  I went from 16.9.4 to 16.12.7, which from everything I read, should not have been an issue. I had a major issue - the entire config was wiped from my switch.  Fortunately, I had a saved config in PI that I could put back.  I will not be upgrading any others to 16.12.7 until I either know what happened or am required to!

Now I'm seeing a lot of people reporting that 16.12.x is extremely unstable.  I have a few other switches running 16.12.5b with no issues, but wanted to ask if anyone else could provide information on the instability they are seeing, any work arounds I may need to be aware of - this switch is in a 24x7 location and not easily downgraded (may be a few weeks).

[Reza Sharifi]

You may want to try and load Gibraltar-16.12.5b on a test switch, as it is the suggested version by Cisco.

HTH

[newberntac]

Yeah, I've got 16.12.5b running successfully on a few switches.  There were PSIRTs identified on that version, so I was trying to be smart and upgrade while I had the time..... Didn't work so well

[marce1000]

 - Make sure config-register  setting has production value settings at all times , some values make the device ignore startup-config (e.g) : use for instance :

   #config-register 0x102

   #no system ignore startupconfig

 M.

[newberntac]

I would need to check this on the next switch, right?  This one's already re-configured.

[Leo Laohoo]

I have never seen a behaviour like that before. 

On that switch, post the complete output to the command "sh boot". 

[newberntac]

---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

As I mentioned, I used Prime Infrastructure to deploy and activate the new IOS.  I've done it quite a few times, even on this switch, with no issues.  That's why I'm thinking it's something on the IOS or the path I took - straight from 16.9.4 to 16.12.7

[Leo Laohoo]

Granted I only have a stack on 16.12.7, and I never use PI or DNAC to upgrade the firmware of my switch stacks, I have not seen a behaviour like this before.  

Please post the complete output to the following commands: 

• sh version | i Configuration register
• sh romvar

Try upgrading a stack without using PI (see attachment).

[newberntac]

WITHOUT PRIME   LOL  That's asking a lot from a small shop.  Here's the requested output - 

sh ver | i Configuration register
Configuration register is 0x142 (will be 0x102 at next reload)
SWITCH#sh romvar
ROMMON variables:
ABNORMAL_RESET_COUNT="0"
BOOT="flash:packages.conf;"
CFG_MODEL_NUM="WS-C3850-48P-S"
CLEI_CODE_NUMBER="IPMW700ARB"
DC_COPY="yes"
DEFAULT_ROUTER="10.1.1.1"
D_STACK_DOMAIN_NUM="1"
ECI_CODE_NUMBER="466864"
EI_NOPROG="1"
IP_ADDR="10.1.1.24/255.255.255.0"
LICENSE_BOOT_LEVEL="ipbasek9,all:C3850_48;"
LINUX_CMDLINE="rw console=ttyS0,9600,n8"
LINUX_COREMASK="15"
MAC_ADDR="20:4C:9E:FE:14:80"
MANUAL_BOOT="no"
MODEL_NUM="WS-C3850-48P"
MODEL_REVISION_NUM="S0"
MOTHERBOARD_ASSEMBLY_NUM="73-14442-10"
MOTHERBOARD_REVISION_NUM="A0"
MOTHERBOARD_SERIAL_NUM="FOC1902ACT0"
POE1_ASSEMBLY_NUM="73-14095-01"
POE1_REVISION_NUM="B0"
POE1_SERIAL_NUM="FOC19026QVE"
POE2_ASSEMBLY_NUM="73-14095-01"
POE2_REVISION_NUM="B0"
POE2_SERIAL_NUM="FOC19026QZZ"
RECOVERY_BUNDLE="sda9:cat3k_caa-recovery.bin"
STKPWR_ASSEMBLY_NUM="73-11956-08"
STKPWR_REVISION_NUM="B0"
STKPWR_SERIAL_NUM="FOC190295NV"
SWITCH_IGNORE_STARTUP_CFG="1"
SWITCH_NUMBER="1"
SYSTEM_SERIAL_NUM="FOC1903U09M"
TAN_NUM="800-37564-03"
TAN_REVISION_NUMBER="B0"
TEMPLATE="advanced"
TERMLINES="0"
USB_ASSEMBLY_NUM="73-12923-05"
USB_REVISION_NUM="C0"
USB_SERIAL_NUM="FOC19026SFU"
VERSION_ID="V04"
BAUD="9600"
BSI="0"
RANDOM_NUM="1003989753"
AUTO_SWITCH_CONSOLE_DISABLE="0"

[Reza Sharifi]

Configuration register is 0x142 (will be 0x102 at next reload)

The config-register needs to be  0x102. 

0x142 will ignore to load the start-up config

A reboot should fix that, if not manually change the config-register and reload.

HTH

[newberntac]

And how would that have been changed?  I don't go around making changes like this.....

[Reza Sharifi]

Hard to tell, but it is possible that the upgrade somehow caused the config-register to change.

[Leo Laohoo]

---

@newberntac wrote:

Configuration register is 0x142 (will be 0x102 at next reload)

---

There is the problem right there.  Password recovery was enabled and this bypasses/ignores startup-config.  Should be OK after a reboot. 

---

@newberntac wrote:

WITHOUT PRIME   LOL  That's asking a lot from a small shop.

---

I have several hundred stacks of 2960S/2960X, 3750X, 3850 and 9300.  I never use PI nor DNAC to upgrade the firmware.  

Why not? 

1. Because when the upgrade fails (when using PI &/or DNAC), it will fail in the most spectacular way.  
2. Because PI and/or DNAC cannot "act" when an error occurs.  

Look at the attachment I have posted (above).  Read it.  It is not complicated (look, I even have pictures!).