3850 lost config after reboot

[hmc250000]

Our 3850 (WS-C3850-12S) was rebooted this morning and somehow lost it's config. Could this be a bug in the IOS software? Or is it an incorrect config register? version listed below.

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 20 WS-C3850-12S 03.03.03SE cat3k_caa-universalk9 INSTALL

Configuration register is 0x142 (will be 0x102 at next reload)

[balaji.bandi]

below thread help you :

https://community.cisco.com/t5/switching/why-sw-3850-loses-config-after-reload-even-though-config-is/td-p/2471639

[hmc250000]

I saw that but it's confusing to me.

would this fix it?

"no system ignore startupconfig switch all"

[balaji.bandi]

The configuration-register value needs to be 0x102

++ Boot the switch into ROMMON mode.

Connect to the console of the switch
Power off the switch
While holding the MODE button down, restore power to the switch
Keep pressing the MODE button and watch the console output
After approx. 10-15 seconds, the switch should end up at the ROMMON/boot loader switch: prompt
Release the MODE button
Type command flash_init
Type command set
Look for if SWITCH_IGNORE_STARTUP_CFG bit is set to 1
Type command unset SWITCH_IGNORE_STARTUP_CFG
Reload the switch

change some config, write and reload check is the config stay as it is or lost. ?

[Reza Sharifi]

Hi,

Is there any crash file on flash?

"dir" or "sh flash" should show you the crash file if there is any. Also, anything on the logs?

Can you post the output of "sh ver"?

Configuration register is 0x142 (will be 0x102 at next reload)

Also, 0x2142 means the switch will ignore the start-up config. So, you do want to make sure the config-register is 0x2102 before rebooting. 

HTH

[hmc250000]

I would like to know if it's ok to run a simple command in the running IOS to fix this. 

Like "no system ignore startupconfig switch all" with ""switch all config-register 0x102""

or config-register 0x2102. Unfortunately this is a critical prod switch and I will not have much time to reboot and play around to see what will work.

#show flash:
-#- --length-- ---------date/time--------- path
2 2097152 Mar 11 2022 12:41:04.0000000000 +00:00 nvram_config
3 79121160 Sep 09 2014 15:53:02.0000000000 +00:00 cat3k_caa-base.SPA.03.03.03SE.pkg
4 6484668 Sep 09 2014 15:53:03.0000000000 +00:00 cat3k_caa-drivers.SPA.03.03.03SE.pkg
5 34522096 Sep 09 2014 15:53:02.0000000000 +00:00 cat3k_caa-infra.SPA.03.03.03SE.pkg
6 34821452 Sep 09 2014 15:53:02.0000000000 +00:00 cat3k_caa-iosd-universalk9.SPA.150-1.EZ3.pkg
7 25131920 Sep 09 2014 15:53:02.0000000000 +00:00 cat3k_caa-platform.SPA.03.03.03SE.pkg
8 77312832 Sep 09 2014 15:53:03.0000000000 +00:00 cat3k_caa-wcm.SPA.10.1.130.0.pkg
9 1248 Sep 09 2014 15:53:10.0000000000 +00:00 packages.conf
10 924 Sep 25 2014 19:36:01.0000000000 +00:00 express_setup.debug
11 676 Jan 28 2021 18:07:32.0000000000 +00:00 vlan.dat
1220993024 bytes available (262144000 bytes used)

#sh ver
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2014 by Cisco Systems, Inc.
Compiled Sun 27-Apr-14 18:33 by prod_rel_team

Cisco IOS-XE software, Copyright (c) 2005-2014 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.1, RELEASE SOFTWARE (P)

SHB013 uptime is 8 hours, 49 minutes
Uptime for this control processor is 8 hours, 51 minutes
System returned to ROM by reload
System restarted at 05:43:03 UTC Fri Mar 11 2022
System image file is "flash:packages.conf"
Last reload reason: reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipservices
License Type: Permanent
Next reload license Level: Ipservices

cisco WS-C3850-12S (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FOC1834U0RH
2 Virtual Ethernet interfaces
16 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
1550272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of at webui:.

Base Ethernet MAC Address : bc:16:f5:03:b7:00
Model Revision Number : A0
Motherboard Revision Number : A0
Model Number : WS-C3850-12S

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 20 WS-C3850-12S 03.03.03SE cat3k_caa-universalk9 INSTALL

Configuration register is 0x142 (will be 0x102 at next reload)

#

[Reza Sharifi]

This should work:

no system ignore startupconfig switch all

config-register 0x102

reload in a maintenance window.

HTH

[balaji.bandi]

Like "no system ignore startupconfig switch all" with ""switch all config-register 0x102""

or config-register 0x2102. Unfortunately this is a critical prod switch and I will not have much time to reboot and play around to see what will work.

sure noted.

Like "no system ignore startupconfig switch all" with ""switch all config-register 0x102""

reboot 

[Richard Burts]

From the output of show version in one of the responses I focus on this line

Configuration register is 0x142 (will be 0x102 at next reload)

So it looks like the config register was changed at some point (as part of password recovery?) and the config command to change it back has already been executed. According to this output at the next reboot things should be back to normal and the switch should boot the startup config. No further commands needed at this point.

[hmc250000]

From the output of show version in one of the responses I focus on this line

Configuration register is 0x142 (will be 0x102 at next reload)

So it looks like the config register was changed at some point (as part of password recovery?) and the config command to change it back has already been executed. According to this output at the next reboot things should be back to normal and the switch should boot the startup config. No further commands needed at this point.

HTH

Rick

No, it did not change after we rebooted the switch. It lost the config again. I think another admin did perform a password recovery in the past.

[Richard Burts]

Thank you for the update. The results you report are quite surprising. The output showing the config register indicated that at the next reboot it should be back to normal. But at the reboot it lost its config (again). I wonder if someone might have removed/deleted the startup config which would produce the same symptoms. Assuming that you will reboot the switch (at least at some time, and I would hope you might reboot soon to test this out) I suggest following these steps:

1) do show version and down at the bottom check what it says about the config register.

2) do show startup and verify that the startup config does exist and has appropriate commands in it.

3) connect to the console and capture all console output during the boot process

4) reboot the switch

5) update this discussion with the results

Thinking about what might have caused this and wonder about the possibility that someone may have set this switch up to boot a config file different from startup. Please check the switch config looking for anything that might alter the normal boot process.

[hmc250000]

1) do show version and down at the bottom check what it says about the config register.

Configuration register is 0x142 (will be 0x102 at next reload)

2) do show startup and verify that the startup config does exist and has appropriate commands in it.

running config and startup config are identical. 

Could this possibly be a bug in the IOS software?

I don't know when we will have a maintenance window again to verify the rest. Might be months from now.

[Richard Burts]

Thanks for the additional information, and especially for the picture verifying what it says about the next reload. And for verifying that startup config does exist and is the same as running config.
The logic says that at the next reboot it should be ok, should reboot and use startup config. But the logic said that last time and it did not work as expected. So next time is a bit of a question mark. I hope that when that time comes you will use the steps that I suggested (especially important to capture all of the console output during the boot process).

I can think of a couple of things that might have been done that could explain this behavior. If possible I would like to see the complete config (with disguise for passwords, Public IP, etc) to look for possible causes. But it the complete config is not possible then please post the output of these commands

show run | include service

show run | include boot

[Leo Laohoo]

---

@hmc250000 wrote:

No, it did not change after we rebooted the switch. It lost the config again. I think another admin did perform a password recovery in the past.

---

Post the complete output to the command "sh romvar".

[Leo Laohoo]

Is this switch "managed" by PI or DNAC (or something similar)?