Re: 3850s in Stack - packages.conf not found after power outage - Page 2

procopius1980 · ‎09-16-2021

I have a site with three sets of Cisco 3850 Stacks running IOS-XE 16.12.05b. The switches are configured for INSTALL mode, and all three stacks are configured to boot to packages.conf. The switches are on battery backups, but the site has had about four power outages in the past two months that have drained the batteries. When power is restored, each time we have had a different set of member switches fail to fully boot. When you console into the switch, you are in the bootloader interface. If you try to boot again, you get the "packages.conf not found" error.

The solution to get them back online is to connect my laptop to the switches and perform an "emergency-install tftp:\\x.x.x.x\cat3cat3k_caa-universalk9.16.12.05b.SPA.bin command". The switch comes back online within ten minutes. When I issue "show flash-1:, show flash-2:" and so forth, I can see that packages.conf has been copied to each switch in the stack.

This issue has only happened to switches that I have personally installed and updated to 16.12.05b. The majority of the switches in this organization are still running variants of 03.03.x. I am happy to provide the output of any show commands who running-config that may be helpful in determining why this happens. Thank you so much in advance!

procopius1980 · ‎01-25-2023

Hello! I know this is a very old thread, but I have an update:

The same switch stack suffered a failure this morning as well as on January 4. Today, a power failure which exceeded the UPS capacity caused the switch stack to have to be rebooted. When it did, switches 1-2 (it's a 5 switch stack) failed to boot. The customer performed an emergency install to get the stack running. I am investigating the issue, and I may have found the problem.

procopius1980 · ‎01-25-2023

HS-IDF-B#show boot
---------------------------
Switch 3
---------------------------
Current Boot Variables:
BOOT variable = flash:running-packages.conf;

Boot Variables on next reload:
BOOT variable = flash:running-packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

HS-IDF-B#show ver | in INSTALL
1 56 WS-C3850-48P 16.6.9 CAT3K_CAA-UNIVERSALK9 INSTALL
2 56 WS-C3850-48P 16.6.9 CAT3K_CAA-UNIVERSALK9 INSTALL
* 3 56 WS-C3850-48P 16.6.9 CAT3K_CAA-UNIVERSALK9 INSTALL
4 56 WS-C3850-48P 16.6.9 CAT3K_CAA-UNIVERSALK9 INSTALL
5 56 WS-C3850-48P 16.6.9 CAT3K_CAA-UNIVERSALK9 INSTALL

However, a review of the dir command revealed that Flash-1 and Flash-2 do not have running-packages.conf in the directory. Instead they have packages.conf listed.

HS-IDF-B#dir flash-1:
Directory of flash-1:/

24254 -rwx 2097152 Jan 25 2023 10:59:42 -06:00 nvram_config
24243 -rw- 2097152 Jan 25 2023 09:05:54 -06:00 nvram_config_bkup
24244 -rw- 19167872 Jan 25 2023 09:05:30 -06:00 cat3k_caa-guestshell.16.06.09.SPA.pkg
24245 -rw- 30424184 Jan 25 2023 09:05:28 -06:00 cat3k_caa-rpbase.16.06.09.SPA.pkg
24246 -rw- 336573048 Jan 25 2023 09:05:30 -06:00 cat3k_caa-rpcore.16.06.09.SPA.pkg
24247 -rw- 5100160 Jan 25 2023 09:05:28 -06:00 cat3k_caa-srdriver.16.06.09.SPA.pkg
24248 -rw- 16271996 Jan 25 2023 09:05:28 -06:00 cat3k_caa-webui.16.06.09.SPA.pkg
24249 -rw- 4742 Jan 25 2023 09:05:48 -06:00 packages.conf
48481 drwx 4096 Jan 25 2023 09:09:09 -06:00 .installer
24250 -rw- 37 Jan 25 2023 09:09:17 -06:00 bootloader_evt_handle.log
56561 drwx 4096 Jan 25 2023 09:09:24 -06:00 core
64641 drwx 4096 Jan 25 2023 09:09:22 -06:00 .prst_sync
80801 drwx 4096 Jan 25 2023 09:09:25 -06:00 .rollback_timer
11 drwx 4096 Jan 25 2023 09:09:34 -06:00 gs_script
8081 drwx 4096 Jan 25 2023 09:09:34 -06:00 dc_profile_dir
24251 -rw- 76 Jan 25 2023 09:09:36 -06:00 boothelper.log
24252 -rw- 128103 Jan 25 2023 09:09:55 -06:00 memleak.tcl
24242 -rw- 2356 Jan 25 2023 07:43:22 -06:00 vlan.dat

HS-IDF-B#dir flash-2:
Directory of flash-2:/

40414 -rwx 2097152 Jan 25 2023 10:59:42 -06:00 nvram_config
40403 -rw- 2097152 Jan 24 2023 14:51:31 -06:00 nvram_config_bkup
40404 -rw- 19167872 Jan 24 2023 14:51:06 -06:00 cat3k_caa-guestshell.16.06.09.SPA.pkg
40405 -rw- 30424184 Jan 24 2023 14:51:05 -06:00 cat3k_caa-rpbase.16.06.09.SPA.pkg
40406 -rw- 336573048 Jan 24 2023 14:51:06 -06:00 cat3k_caa-rpcore.16.06.09.SPA.pkg
40407 -rw- 5100160 Jan 24 2023 14:51:05 -06:00 cat3k_caa-srdriver.16.06.09.SPA.pkg
40408 -rw- 16271996 Jan 24 2023 14:51:05 -06:00 cat3k_caa-webui.16.06.09.SPA.pkg
40409 -rw- 4742 Jan 24 2023 14:51:24 -06:00 packages.conf
64641 drwx 4096 Jan 24 2023 14:54:55 -06:00 .installer
40410 -rw- 37 Jan 24 2023 14:55:02 -06:00 bootloader_evt_handle.log
72721 drwx 4096 Jan 24 2023 14:55:09 -06:00 core
80801 drwx 4096 Jan 24 2023 14:55:07 -06:00 .prst_sync
11 drwx 4096 Jan 24 2023 14:55:10 -06:00 .rollback_timer
8081 drwx 4096 Jan 24 2023 14:55:19 -06:00 gs_script
16161 drwx 4096 Jan 24 2023 14:55:19 -06:00 dc_profile_dir
40411 -rw- 76 Jan 24 2023 14:55:21 -06:00 boothelper.log
40412 -rw- 128103 Jan 24 2023 14:55:40 -06:00 memleak.tcl
40402 -rw- 2356 Jan 25 2023 07:43:22 -06:00 vlan.dat

1562902528 bytes total (1068498944 bytes free)

Switches 3-5 have running-packages.conf listed, which I think explains why switches 3-5 booted. I think the solution to this problem is to copy running-packes.conf from flash-3 to flash-1 & 2, but wanted to check here to confirm.

Leo Laohoo · ‎01-25-2023

@procopius1980 wrote:

BOOT variable = flash:running-packages.conf;

Someone has changed the boot variable statement. Instead of packages.conf, the boot variable statement is looking for running-packages.conf.

Change it back, from the switch master, and everything should be fine.

procopius1980 · ‎01-25-2023

Thanks! I want to make sure that it's clear that Switches 1-2 have a packages.conf file, and Switches 3-5 have a running-packages.conf file listed in flash. So just to confirm, all I have to do is change the boot variable to packages.conf, and then all 5 switches will boot up fine?

Leo Laohoo · ‎01-25-2023

The boot variable string can all be adjusted from the stack manager.

conf t
 no boot system flash switch all
 boot system flash flash:packages.conf
end
wr

By the way, good choice of firmware version, i. e. 16.6.9.

procopius1980 · ‎01-27-2023

I believe it was you who recommended 16.6.9.

I have changed the boot variable in HS-IDF-B to packages.conf. What I'm concerned out now is what is going to happen to switches 3-5 the next time there is a power outage (and with this customer, it's a question of when, not iff).

Switch #	packages.conf present	running-packages.conf present
Switch 1 (priority 15)	Yes	No
Switch 2 (priority 10)	Yes	No
Switch 3 (priority 5)	No	Yes
Switch 4 (priority 4)	No	Yes
Switch 5 (priority 5)	No	Yes

The re-clarify, I think all I have done is set the boot variable to ensure switches 1-2 boot up, but now 3-5 are pointing at a file that does not exist in their file system.

Okay, I just tried the following:

HS-IDF-B#copy flash-1:packages.conf flash-3:
HS-IDF-B#copy flash-1:packages.conf flash-4:
HS-IDF-B#copy flash-1:packages.conf flash-5:

Now I have packages.conf moved to all 5 switches. I am hopeful this means we finally have this problem fixed on this stack.

Leo Laohoo · ‎01-27-2023

Please post the complete output to the following commands:

sh boot
sh romvar

procopius1980 · ‎01-27-2023

Sure - one moment.

procopius1980 · ‎01-27-2023

HS-IDF-B#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
HS-IDF-B#

HS-IDF-B#show romvar
^
% Invalid input detected at '^' marker.

HS-IDF-B#

Leo Laohoo · ‎01-27-2023

I always get mixed up with the two commands: sh rom-var

procopius1980 · ‎01-27-2023

All good! I should have written that I ?-marked out the command. This is the only thing that pops up:

HS-IDF-B#show rom?
rom-monitor

Leo Laohoo · ‎01-27-2023

I just checked, it should be "sh romvar" but I suspect this command is not yet supported in 16.6.X.

Otherwise, the "sh boot" output looks good.

By the way, here is a partial list of bug attributed to Cisco Smart License:

CSCvv72609, for example, has more than 850 TAC Cases since the bug ID was first created on 15 September 2020. Cisco Smart License support starts from 16.9.X for switches (16.10.X for routers).

Aside from the bugs (above), Cisco Smart License, particularly the "call home" function is affected by FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality. Make sure to do Workaround #2 -- This workaround is safe to perform during business hours and will not cause any adverse reaction.