upgraded ios stack member provisioned

[n-switcher]

Hi guys,

I have upgraded a stack of 2 x 3850's and after reloading the stack, the 2th switch went into provisioned mode.
I also noticed that the 1th switch shows the new ios version. However when I check the running config I still see this :
boot system switch all flash:cat3k_caa-universalk9.16.06.06.SPA.conf. which does not much the actual running IOS version ? The actual running new IOS version is a higher version. 16.06.06 was the previous one from before the upgrade. Also on other stacks I upgraded I can see this line being configured as "boot system switch all flash:packages.conf"
please advise what should I do to get the stack member back and make sure that the correct IOS version is kept being loaded on both switches when the stack reboots.

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
sh boot
dir
sh switch detail

[n-switcher]

Cisco IOS XE Software, Version 16.12.09
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.12.9, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Mon 25-Mar-23 01:11 by aaasw

Cisco IOS-XE software, Copyright (c) 2005-2023 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 6.08, RELEASE SOFTWARE (P)

switch uptime is 1 day, 15 hours, 19 minutes
Uptime for this control processor is 1 day, 15 hours, 23 minutes
System returned to ROM by Power Failure or Unknown at 13:52:25 EST-DST Sat Oct 14 2023
System restarted at 15:12:03 EST-DST Sat Oct 14 2023
System image file is "flash:cat3k_caa-universalk9.16.06.06.SPA.conf"
Last reload reason: Power Failure or Unknown

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
ipbasek9 Smart License ipbasek9
None Subscription Smart License None

Smart Licensing Status: UNREGISTERED/EVAL MODE

cisco WS-C3850-24P (MIPS) processor (revision AA0) with 794816K/6147K bytes of memory.
Processor board ID FCW2003F13C
13 Virtual Ethernet interfaces
56 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 00:8e:73:7e:05:80
Motherboard Assembly Number : 73-15805-04
Motherboard Serial Number : FOC200353RF
Model Revision Number : AA0
Motherboard Revision Number : B0
Model Number : WS-C3850-24P
System Serial Number : FCW2003F13C

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 32 WS-C3850-24P 16.12.09 CAT3K_CAA-UNIVERSALK9 INSTALL

Configuration register is 0x102

************************************************************************************************************************************************************************************

---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:cat3k_caa-universalk9.16.06.06.SPA.conf;

Boot Variables on next reload:
BOOT variable = flash:cat3k_caa-universalk9.16.06.06.SPA.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

**************************************************************************************************************************************************************************************

Directory of flash:/

85186 -rw- 2097152 Oct 14 2023 13:21:52 -04:00 nvram_config
85187 -rw- 76 Oct 14 2023 13:07:53 -04:00 boothelper.log
85188 -rw- 3861420 Jul 22 2019 07:38:48 -04:00 mycap.pcap
85189 -rw- 62640 May 21 2020 12:27:09 -04:00 IBNS20-Conversion
85215 drwx 4096 Oct 14 2023 12:58:25 -04:00 tech_support
85219 -rw- 4780 Oct 14 2023 12:49:53 -04:00 packages.conf
85208 -rw- 19167872 Aug 23 2019 02:34:15 -04:00 cat3k_caa-guestshell.16.06.06.SPA.pkg
85214 -rw- 27926203 Aug 23 2019 02:34:18 -04:00 cat3k_caa-rpbase.16.06.06.SPA.pkg
85211 -rw- 336437880 Aug 23 2019 02:34:17 -04:00 cat3k_caa-rpcore.16.06.06.SPA.pkg
85190 drwx 4096 Oct 10 2023 04:35:48 -04:00 onep
85244 drwx 4096 Oct 14 2023 13:11:08 -04:00 .dbpersist
85194 drwx 4096 Mar 23 2016 08:10:11 -04:00 virtual-instance
15489 drwx 4096 Mar 23 2016 08:10:11 -04:00 vman_fdb
23233 drwx 4096 Oct 14 2023 13:07:49 -04:00 dc_profile_dir
23235 drwx 4096 Mar 23 2016 08:13:39 -04:00 wnweb_store
85195 -rw- 1216 Oct 14 2023 13:11:49 -04:00 vlan.dat
85199 -rw- 20083332 Oct 7 2023 04:01:16 -04:00 cat3k_caa-guestshell.16.12.09.SPA.pkg
85243 -rw- 66 Oct 14 2023 12:58:45 -04:00 boothelper.old
85193 drwx 4096 Oct 14 2023 13:07:31 -04:00 .installer
85203 -rw- 34844799 Oct 7 2023 04:01:19 -04:00 cat3k_caa-rpbase.16.12.09.SPA.pkg
85200 -rw- 401252988 Oct 7 2023 04:01:18 -04:00 cat3k_caa-rpcore.16.12.09.SPA.pkg
85202 -rw- 3418756 Oct 7 2023 04:01:19 -04:00 cat3k_caa-srdriver.16.12.09.SPA.pkg
85201 -rw- 23014016 Oct 7 2023 04:01:19 -04:00 cat3k_caa-webui.16.12.09.SPA.pkg
85245 -rw- 0 Oct 14 2023 13:11:19 -04:00 rdope_out.txt
85246 -rw- 545 Oct 14 2023 13:11:32 -04:00 rdope.log
85204 -rw- 629 Oct 14 2023 13:07:34 -04:00 bootloader_evt_handle.log
85205 drwx 4096 Oct 14 2023 13:15:15 -04:00 core
85206 drwx 4096 Oct 14 2023 12:58:25 -04:00 .prst_sync
54211 drwx 4096 Jul 22 2019 00:02:51 -04:00 .rollback_timer
61954 drwx 4096 Jul 22 2019 00:02:59 -04:00 gs_script
85207 -rw- 132108 Oct 14 2023 13:08:11 -04:00 memleak.tcl
85210 -rw- 2097152 Oct 14 2023 13:21:53 -04:00 nvram_config_bkup
85213 -rw- 5051008 Aug 23 2019 02:34:18 -04:00 cat3k_caa-srdriver.16.06.06.SPA.pkg
85196 -rw- 4780 Oct 7 2023 04:03:23 -04:00 cat3k_caa-universalk9.16.06.06.SPA.conf
85212 -rw- 16263804 Aug 23 2019 02:34:17 -04:00 cat3k_caa-webui.16.06.06.SPA.pkg
85192 -rw- 4742 Oct 7 2023 04:03:23 -04:00 cat3k_caa-universalk9.16.06.06.SPA.conf.00-

**************************************************************************************************************************************************************************************

Switch/Stack Mac Address : 008e.737e.0580 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active 008e.737e.0580 1 V07 Ready
2 Member 0000.0000.0000 0 Provisioned

Stack Port Status Neighbors
Switch# Port 1 Port 2 Port 1 Port 2
--------------------------------------------------------
1 DOWN DOWN None None

[Leo Laohoo]

---

@n-switcher wrote:
System image file is "flash:cat3k_caa-universalk9.16.06.06.SPA.conf"

---

Who butchered this switch? 

Please post the complete output to the following command:

more flash:packages.conf | begin for NOVA

[n-switcher]

not sure what happened, the upgrade was done via Cisco DNA center, after reloading the stack this was the result...

# This is for NOVA
iso rp 0 0 rp_base cat3k_caa-rpbase.16.12.09.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.12.09.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.12.09.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.12.09.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.12.09.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.12.09.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.12.09.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.12.09.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.12.09.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime: 2023-03-20_03.19
# pkginfo: ReleaseDate: Mon-20-Mar-23-07:23
# pkginfo: .BuildArch: mips64
# pkginfo: BootArchitecture: mips
# pkginfo: .BootArch: mips
# pkginfo: RouteProcessor: edison
# pkginfo: Platform: CAT3K_CAA
# pkginfo: User: aaasw
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.12.09
# pkginfo: .SupportedBoards: edison
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /nobackup/aaasw/release/BLD-V16_12_09_FC2/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64
# pkginfo: .Version: 16.12.09.0.7556.1679307552..Gibraltar
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

[Leo Laohoo · Accepted solution]

Ok, so to fix this, do this: 

conf t
 no boot system flash flash:cat3k_caa-universalk9.16.06.06.SPA.conf
 boot system flash flash:packages.conf
 software auto-install enable
end
wr

And reboot.

[n-switcher]

Ok will this also fix the 2th stack member switch going back into standby mode ? 

[balaji.bandi]

May be for me Looks version missmatch with seconds switch - what suggested is upgrade and reboot

Also connect to second switch console cable and check what is wrong ?

if you enablke auto upgrade option in the Global- the second switch upgrade automatically if the switch upgrade failed ?

[n-switcher]

Switch/Stack Mac Address : 008f.837e.0780 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active 008f.837e.0780 1 V07 Ready
2 Member 0000.0000.0000 0 Provisioned

[balaji.bandi]

not sure we understand this output ?