3850s in Stack - packages.conf not found after power outage

[procopius1980]

I have a site with three sets of Cisco 3850 Stacks running IOS-XE 16.12.05b.  The switches are configured for INSTALL mode, and all three stacks are configured to boot to packages.conf.  The switches are on battery backups, but the site has had about four power outages in the past two months that have drained the batteries.  When power is restored, each time we have had a different set of member switches fail to fully boot.  When you console into the switch, you are in the bootloader interface.  If you try to boot again, you get the "packages.conf not found" error.

The solution to get them back online is to connect my laptop to the switches and perform an "emergency-install tftp:\\x.x.x.x\cat3cat3k_caa-universalk9.16.12.05b.SPA.bin command".  The switch comes back online within ten minutes.  When I issue "show flash-1:, show flash-2:" and so forth, I can see that packages.conf has been copied to each switch in the stack.

This issue has only happened to switches that I have personally installed and updated to 16.12.05b.  The majority of the switches in this organization are still running variants of 03.03.x.  I am happy to provide the output of any show commands who running-config that may be helpful in determining why this happens.  Thank you so much in advance!

[balaji.bandi]

can you post dir output of the master switch see wht files in the there ?

is this occuring after upgrade from 3.3 to 16.12.X ?

[procopius1980]

I absolutely can get that info you for!  And yes, this started happening after updating from 3.3.  I performed a software cleanup before performing the update, but I do not recall if I performed a post cleanup.  

Here's the output:

LSC-IDF-C#dir
Directory of flash:/

7746 -rw- 2097152 Sep 14 2021 23:05:22 -05:00 nvram_config
7747 -rwx 74369716 May 19 2015 09:13:53 -05:00 cat3k_caa-base.SPA.03.02.03.SE.pkg
7748 -rwx 5808828 May 19 2015 09:13:54 -05:00 cat3k_caa-drivers.SPA.03.02.03.SE.pkg
7749 -rwx 32496484 May 19 2015 09:13:53 -05:00 cat3k_caa-infra.SPA.03.02.03.SE.pkg
7750 -rwx 30418104 May 19 2015 09:13:54 -05:00 cat3k_caa-iosd-universalk9.SPA.150-1.EX3.pkg
7751 -rwx 16059104 May 19 2015 09:13:54 -05:00 cat3k_caa-platform.SPA.03.02.03.SE.pkg
7752 -rwx 64586444 May 19 2015 09:13:54 -05:00 cat3k_caa-wcm.SPA.10.0.120.0.pkg
7756 -rw- 4788 May 18 2021 08:48:12 -05:00 packages.conf
7754 -rw- 480018036 May 18 2021 08:42:59 -05:00 cat3k_caa-universalk9.16.12.05b.SPA.bin
7755 -rwx 1224 May 19 2015 09:14:04 -05:00 packages.conf.00-
7753 drwx 4096 Sep 14 2021 22:36:41 -05:00 .installer
7757 -rw- 20095616 May 18 2021 08:47:43 -05:00 cat3k_caa-guestshell.16.12.05b.SPA.pkg
7758 -rw- 32798679 May 18 2021 08:47:43 -05:00 cat3k_caa-rpbase.16.12.05b.SPA.pkg
7759 -rw- 400747128 May 18 2021 08:48:03 -05:00 cat3k_caa-rpcore.16.12.05b.SPA.pkg
7760 -rw- 3359360 May 18 2021 08:48:04 -05:00 cat3k_caa-srdriver.16.12.05b.SPA.pkg
7761 -rw- 23011964 May 18 2021 08:48:10 -05:00 cat3k_caa-webui.16.12.05b.SPA.pkg
7762 -rw- 481 Sep 14 2021 22:36:44 -05:00 bootloader_evt_handle.log
7763 drwx 4096 May 18 2021 09:04:42 -05:00 core
15489 drwx 4096 Sep 14 2021 22:40:54 -05:00 .prst_sync
30978 drwx 4096 May 18 2021 08:55:19 -05:00 .rollback_timer
38722 drwx 4096 May 18 2021 08:55:23 -05:00 gs_script
46466 drwx 4096 May 18 2021 08:55:23 -05:00 tech_support
54210 drwx 4096 Sep 14 2021 22:36:58 -05:00 dc_profile_dir
7764 -rw- 132108 Sep 14 2021 22:37:20 -05:00 memleak.tcl
7765 -rw- 76 Sep 14 2021 22:37:02 -05:00 boothelper.log
7766 -rw- 76 Sep 14 2021 22:25:17 -05:00 boothelper.old
7767 drwx 4096 Jun 17 2021 11:41:39 -05:00 .dbpersist
7768 -rw- 0 Sep 14 2021 22:40:48 -05:00 rdope_out.txt
7769 -rw- 545 Sep 14 2021 22:41:00 -05:00 rdope.log
7770 -rw- 2097152 Sep 14 2021 23:05:22 -05:00 nvram_config_bkup
7771 drwx 4096 May 18 2021 09:04:40 -05:00 onep
7776 -rw- 1364 Jun 25 2021 02:00:58 -05:00 vlan.dat
7778 -rw- 1364 Sep 14 2021 22:43:45 -05:00 stby-vlan.dat

[balaji.bandi]

After Upgrade Looks like you did not clean up, i can see old IOS  XE 3.3 and package.conf files,

Note	Old files listed in the logs should be removed using the request platform software package clean switch all command, after reload.

not sure what is the content of the package.conf file and packages.conf.00-

make sure the packages.conf should have 16.12.X

packages.conf example see as below : (below from my running swich with 16.6.X)

boot rp 0 0 rp_boot cat9k-rpboot.16.06.03.SPA.pkg

[procopius1980]

I was thinking on the same line... that maybe packages.conf was referencing the earlier IOS-XE image.  That does not appear to be the case.  Here's the non-comment portion of my switches' packages.conf.  Regardless, let me do a quick cleanup.

# This is for NOVA
iso rp 0 0 rp_base cat3k_caa-rpbase.16.12.05b.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.12.05b.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.12.05b.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.12.05b.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.12.05b.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.12.05b.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.12.05b.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.12.05b.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.12.05b.SPA.pkg

[procopius1980]

Just performed a request platform software package clean.  Here is the updated output:

LSC-IDF-C#dir
Directory of flash:/

7746 -rw- 2097152 Sep 14 2021 23:05:22 -05:00 nvram_config
7756 -rw- 4788 May 18 2021 08:48:12 -05:00 packages.conf
7753 drwx 4096 Sep 14 2021 22:36:41 -05:00 .installer
7757 -rw- 20095616 May 18 2021 08:47:43 -05:00 cat3k_caa-guestshell.16.12.05b.SPA.pkg
7758 -rw- 32798679 May 18 2021 08:47:43 -05:00 cat3k_caa-rpbase.16.12.05b.SPA.pkg
7759 -rw- 400747128 May 18 2021 08:48:03 -05:00 cat3k_caa-rpcore.16.12.05b.SPA.pkg
7760 -rw- 3359360 May 18 2021 08:48:04 -05:00 cat3k_caa-srdriver.16.12.05b.SPA.pkg
7761 -rw- 23011964 May 18 2021 08:48:10 -05:00 cat3k_caa-webui.16.12.05b.SPA.pkg
7762 -rw- 481 Sep 14 2021 22:36:44 -05:00 bootloader_evt_handle.log
7763 drwx 4096 May 18 2021 09:04:42 -05:00 core
15489 drwx 4096 Sep 14 2021 22:40:54 -05:00 .prst_sync
30978 drwx 4096 May 18 2021 08:55:19 -05:00 .rollback_timer
38722 drwx 4096 May 18 2021 08:55:23 -05:00 gs_script
46466 drwx 4096 May 18 2021 08:55:23 -05:00 tech_support
54210 drwx 4096 Sep 14 2021 22:36:58 -05:00 dc_profile_dir
7764 -rw- 132108 Sep 14 2021 22:37:20 -05:00 memleak.tcl
7765 -rw- 76 Sep 14 2021 22:37:02 -05:00 boothelper.log
7766 -rw- 76 Sep 14 2021 22:25:17 -05:00 boothelper.old
7767 drwx 4096 Jun 17 2021 11:41:39 -05:00 .dbpersist
7768 -rw- 0 Sep 14 2021 22:40:48 -05:00 rdope_out.txt
7769 -rw- 545 Sep 14 2021 22:41:00 -05:00 rdope.log
7770 -rw- 2097152 Sep 14 2021 23:05:22 -05:00 nvram_config_bkup
7771 drwx 4096 May 18 2021 09:04:40 -05:00 onep
7776 -rw- 1364 Jun 25 2021 02:00:58 -05:00 vlan.dat
7778 -rw- 1364 Sep 14 2021 22:43:45 -05:00 stby-vlan.dat

[balaji.bandi]

This looks better, but do you still see the issue of the boot? is this part of the stack?

[procopius1980]

This output is from the stack that lost switches 3-4 the other morning.  I don't have sufficient data to tell if this problem takes place every time there is a buliding power outage.  The last several times it's taken place, we were in a reactionary state where all we had time to do was perform the emergency-install and move on.  This is the first time the incident has occurred where we've stopped to really investigate.  In this case, the MDF and IDF-A did not have a failure, but I also noted that these stacks did not lose power, only IDF-C.  So at this point I don't have any good data.

[balaji.bandi]

Got it, May be if you can make stack switch numbers and priority this will not messup of master and memeber switch due to Power Failures

From master you can see show boot :

#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

[procopius1980]

Here's what I have:

LSC-IDF-C#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
LSC-IDF-C#
LSC-IDF-C#
LSC-IDF-C#show switch
Switch/Stack Mac Address : f078.16c5.2e80 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active f078.16c5.2e80 15 V04 Ready
2 Standby 580a.20e3.7400 14 V03 Ready
3 Member 2c3e.cf23.ea80 13 V03 Ready
4 Member 2c3e.cf12.9600 1 V03 Ready

LSC-IDF-C#

[procopius1980]

I gave the switches priority to ensure Switch1 was the master and Switch2 was the backup.  Thank you for continuing to look into this with me.

[balaji.bandi]

Looks good for me, we  are running 16.12.04 is working as expected from Long,  You may worth try new Code 17.3.X ?

[Leo Laohoo]

---

@procopius1980 wrote:

"packages.conf not found"

---

Post the complete output to the command "sh boot". 

NOTE:  16.12.X is not as cracked up as it meant to be and I would avoid it at all cost.  

[procopius1980]

Thank you, Leo.  Here is the output:

LSC-IDF-C#show run | in boot
boot system switch all flash:packages.conf
license boot level ipbasek9
diagnostic bootup level minimal
LSC-IDF-C#

If 16.12.x is unstable, what firmware family would you recommend?  

In addition, I should say that this does not take place after every power outage or reboot.  I drove to the site back in the June when this first happened.  After getting the downed switches corrected (it was I think a single switch at the MDF), I did some intial digging, and added the boot system command to each stack (it was missing).  Afterwards, I issued a soft reboot on each stack, as well as a hard reboot (pulled the power cables) at the MDF.  Everything came back up.  

[Leo Laohoo]

---

@procopius1980 wrote:

LSC-IDF-C#show run | in boot

That is not the output I want to see.  Post the complete output to the command "sh boot".

Stick with the latest 16.6.X.  

For those who are a glutton for punishment of Cisco Smart License (CSL), go with the latest 16.9.X.  

NOTE:  The last and final IOS-XE releases for the 3650/3850 are 16.6.10, 16.9.8 and 16.12.6.

IMPORTANT:  Read the Release Notes carefully.