Lost stack while upgrading c3850

james.king14 · ‎03-29-2023

Once new IOS was upload lost all other switches in stack they went to SWITCH: prompt. I reload on each switch the current IOS. The switch either prompts again with SWITCH or will not take command of "request platform software package install switch all file flash:cat3k_caa-universalk9.16.02.01.SPA.bin new auto-copy"

Switch stack show primary switch ready while other are showing provisioned.

Since these devices are almost of warranty we lost most of network

balaji.bandi · ‎03-29-2023

what was the old version ? Do you have complete boot logs and upgrade logs captured ?

Do you have configuration backup ?

how many switches in the stack ?

post below output :

show version

show switch

dir

Note : i suggest to upgrade to stable version 16.9.6

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

james.king14 · ‎03-29-2023

The version is cat3k_caa-universalk9.16.12.07.SPA.bin
3850 - 48t
Logs are coming working form my phone since LAN has issues.

Getting switch local is not booted with provisioning file

Leo Laohoo · ‎03-29-2023

Post the complete output to the command "dir".

james.king14 · ‎03-29-2023

Here is the output of the “dir” command

dir

Directory of flash:/

61954 -rwx 2097152 Mar 29 2023 16:30:08 +00:00 nvram_config

61991 -rw- 76 Mar 29 2023 16:26:03 +00:00 boothelper.log

61958 drwx 4096 Mar 28 2023 20:53:17 +00:00 tech_support

61995 drwx 4096 Mar 28 2023 21:06:51 +00:00 onep

30978 -rw- 259 Mar 29 2023 16:25:45 +00:00
bootloader_evt_handle.log

61967 drwx 4096 Mar 29 2023 14:55:03 +00:00 core

62003 -rw- 4780 Mar 29 2023 00:12:19 +00:00 packages.conf

61955 -rw- 76 Mar 29 2023 14:55:13 +00:00 boothelper.old

61992 drwx 4096 Mar 29 2023 00:10:07 +00:00 .dbpersist

61970 -rw- 0 Mar 29 2023 16:29:34 +00:00 rdope_out.txt

61994 -rw- 545 Mar 29 2023 16:29:46 +00:00 rdope.log

62004 -rw- 20097668 Mar 29 2023 00:10:46 +00:00
cat3k_caa-guestshell.16.12.07.SPA.pkg

61962 -rw- 1936 Mar 29 2023 16:30:02 +00:00 vlan.dat

62008 -rw- 32795564 Mar 29 2023 00:11:27 +00:00
cat3k_caa-rpbase.16.12.07.SPA.pkg

38721 drwx 4096 Aug 25 2015 09:42:04 +00:00 mnt

15491 drwx 4096 Mar 29 2023 16:26:00 +00:00 dc_profile_dir

61968 -rw- 4780 Mar 29 2023 14:33:32 +00:00 packages.conf.old

62005 -rw- 400857724 Mar 29 2023 00:11:23 +00:00
cat3k_caa-rpcore.16.12.07.SPA.pkg

62007 -rw- 3414660 Mar 29 2023 00:11:25 +00:00
cat3k_caa-srdriver.16.12.07.SPA.pkg

62006 -rw- 23018112 Mar 29 2023 00:11:25 +00:00
cat3k_caa-webui.16.12.07.SPA.pkg

61957 -rw- 34 Jun 4 2019 13:38:16 +00:00 pnp-tech-time

61964 -rw- 11630 Jun 4 2019 13:38:18 +00:00
pnp-tech-discovery-summary

61965 -rw- 36816 Jun 6 2019 15:16:08 +00:00
bkup-running-config.20190606a

30980 -rw- 132108 Mar 29 2023 16:26:21 +00:00 memleak.tcl

69697 drwx 4096 Mar 29 2023 14:58:48 +00:00 .prst_sync

85185 drwx 4096 Nov 21 2020 18:26:17 +00:00 .rollback_timer

92929 drwx 4096 Nov 21 2020 18:26:25 +00:00 gs_script

12 drwx 4096 Mar 29 2023 16:25:42 +00:00 .installer

30982 -rw- 2097152 Mar 29 2023 16:30:09 +00:00 nvram_config_bkup

61999 -rw- 480189009 Mar 29 2023 06:52:25 +00:00
cat3k_caa-universalk9.16.12.07.SPA.bin

1621966848 bytes total (523214848 bytes free)

Switch Ports Model SW Version SW Image
Mode

------ ----- ----- ---------- ----------
----

* 1 56 WS-C3850-48P 16.12.07 CAT3K_CAA-UNIVERSALK9
BUNDLE

Mac persistency wait time: Indefinite

H/W Current

Switch# Role Mac Address Priority Version State

-------------------------------------------------------------------------------------

*1 Active 38ed.186c.9e00 15 V05 Ready

2 Member 0000.0000.0000 0 Provisioned

3 Member 0000.0000.0000 0 Provisioned

4 Member 0000.0000.0000 0 Provisioned

Switch#reload slot 4

Switch 4 is not part of the stack

I have been using this document faithfully and still I cannot get these
devices to stack. My last ditch effort will be to remove the master from
stack and run the commands int his document then add it back to stack and
hope the others follow….

Leo Laohoo · ‎03-30-2023

Post the complete output to the following commands:

sh boot
more flash:packages.conf

james.king14 · ‎03-30-2023

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:cat3k_caa-universalk9.16.12.07.SPA.bin;

Boot Variables on next reload:

BOOT variable = flash:cat3k_caa-universalk9.16.12.07.SPA.bin;

Manual Boot = no

Enable Break = no

Boot Mode = DEVICE

iPXE Timeout = 0

SRH-LAN-STACK#more flash:packages.conf

#! /usr/binos/bin/packages_conf.sh

sha1sum: 97b8f831ba81fcb9fa3234c6d9ab77402f1cc292

# sha1sum above - used to verify that this file is not corrupted.

#

# package.conf: provisioned software file for build 2022-02-02_07.35

#

# NOTE: Editing this file by hand is not recommended. It is generated

# as part of the build process, and is subject to boot-time

# consistency checks. Automatically-produced package files are

# guaranteed to pass those checks. Manually-maintained ones are

# not. Because "nfs" and "mount" directives are processed first,

# regardless of their position in the file, the recommended

# approach is to keep a separate file containing JUST your

# personal "nfs" and "mount" directives, and to append it to the

# automatically-generated file.

#

# Note further that when SHA-1 checksum verification is enabled,

# you will NOT be able to alter this file without updating the

# SHA-1 sum.

#

# This file can contain three types of entries:

#

#

# NFS directives (optional)

# notes: NFS directives are processed before all others (mount, iso).

# Multiple NFS directives may appear so long as they do not

# conflict -- that is, specify the same source or mountpoint.

# syntax: nfs :

# example: nfs 127.0.0.1:/auto/some/nfs/path /auto/some/nfs/path

#

#

# mount directives (optional)

# notes: mount directives are processed after 'nfs' and before 'iso'.

# One mount directive may appear for each F/S/B/P tuple

# syntax: mount FRU SLOT BAY PACKAGE_NAME LINUX_PATH

# example: mount rp 0 0 rp_base
/auto/some/nfs/path/abs_soft/rp_base.ppc

#

# The specified LINUX_PATH may be local [sata disk] or

# reference the NFS mounts since they are processed first.

#

# Mount directives cause the package-specific mount link to

# be set to the specified path instead of to the mountpoint

# in sw for the corresponding ISO.

#

#

# iso directives (mandatory)

# notes: iso directives are processed last: any package for which

# a 'mount' directive does not appear will be mounted.

# One iso directive may appear for each F/S/B/P tuple.

# syntax: iso FRU SLOT BAY PACKAGE_NAME PACKAGE_FILE.bin

# example: iso rp 0 0 rp_base rp_base.ppc.bin

#

# PACKAGE_FILE.bin is a path relative to the packages.conf

# file. Although it supports sub-directories for development

# purposes, in deployment the files will always be managed

# as in the same directory as packages.conf so as to

# guarantee that name collisions cannot occur.

#

# Note that the RP 0/1 distinction is a convenience for development

# and testing as it allows us to have a packages.conf describe a

# SW load that varies depending on whether the RP finds itself in

# slot 0 or 1.

#

# The ISSU process *must* update *both* RP slots simultaneously so that

# the RP will behave predictably whichever slot it finds itself on [e.g.,

# if package X is upgraded, and the RP is ejected and put into either

# slot of a new chassis, we expect to see the upgraded X without regard

# to slot].

#

# This is for NOVA

iso rp 0 0 rp_base cat3k_caa-rpbase.16.12.07.SPA.pkg

iso rp 0 0 rp_core cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 rp_webui cat3k_caa-webui.16.12.07.SPA.pkg

iso rp 0 0 srdriver cat3k_caa-srdriver.16.12.07.SPA.pkg

iso rp 0 0 rp_security cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 guestshell cat3k_caa-guestshell.16.12.07.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.12.07.SPA.pkg

#

# -start- superpackage .pkginfo

#

# pkginfo: Name: rp_super

# pkginfo: BuildTime: 2022-02-02_07.35

# pkginfo: ReleaseDate: Wed-02-Feb-22-04:05

# pkginfo: .BuildArch: mips64

# pkginfo: BootArchitecture: mips

# pkginfo: .BootArch: mips

# pkginfo: RouteProcessor: edison

# pkginfo: Platform: CAT3K_CAA

# pkginfo: User: mcpre

# pkginfo: PackageName: universalk9

# pkginfo: Build: 16.12.07

# pkginfo: .SupportedBoards: edison

# pkginfo: .InstallModel:

# pkginfo: .PackageRole: rp_super

# pkginfo: .RestartRole: rp_super

# pkginfo: CardTypes:

# pkginfo: .CardTypes:

# pkginfo: .BuildPath:
/nobackup/mcpre/release/BLD-V16_12_07_FC2/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64

# pkginfo: .Version: 16.12.07.0.6565.1643816123..Gibraltar

# pkginfo: .InstallVersion: 1.0.0

# pkginfo: .InstallCapCommitSupport: yes

#

# -end- superpackage .pkginfo

#

Second stack:

SRH-LAN-STACK2#sh boot

---------------------------

Switch 1

---------------------------

Current Boot Variables:

BOOT variable = flash:cat3k_caa-universalk9.16.12.07.SPA.bin;

Boot Variables on next reload:

BOOT variable = flash:cat3k_caa-universalk9.16.12.07.SPA.bin;

Manual Boot = no

Enable Break = no

Boot Mode = DEVICE

iPXE Timeout = 0

SRH-LAN-STACK2#more flash:packages.conf

#! /usr/binos/bin/packages_conf.sh

sha1sum: 15336e478c878fae2c06099b43a8197ad7a323d6

# sha1sum above - used to verify that this file is not corrupted.

#

# package.conf: provisioned software file for build 2020-08-17_04.23

#

# NOTE: Editing this file by hand is not recommended. It is generated

# as part of the build process, and is subject to boot-time

# consistency checks. Automatically-produced package files are

# guaranteed to pass those checks. Manually-maintained ones are

# not. Because "nfs" and "mount" directives are processed first,

# regardless of their position in the file, the recommended

# approach is to keep a separate file containing JUST your

# personal "nfs" and "mount" directives, and to append it to the

# automatically-generated file.

#

# Note further that when SHA-1 checksum verification is enabled,

# you will NOT be able to alter this file without updating the

# SHA-1 sum.

#

# This file can contain three types of entries:

#

#

# NFS directives (optional)

# notes: NFS directives are processed before all others (mount, iso).

# Multiple NFS directives may appear so long as they do not

# conflict -- that is, specify the same source or mountpoint.

# syntax: nfs :

# example: nfs 127.0.0.1:/auto/some/nfs/path /auto/some/nfs/path

#

#

# mount directives (optional)

# notes: mount directives are processed after 'nfs' and before 'iso'.

# One mount directive may appear for each F/S/B/P tuple

# syntax: mount FRU SLOT BAY PACKAGE_NAME LINUX_PATH

# example: mount rp 0 0 rp_base
/auto/some/nfs/path/abs_soft/rp_base.ppc

#

# The specified LINUX_PATH may be local [sata disk] or

# reference the NFS mounts since they are processed first.

#

# Mount directives cause the package-specific mount link to

# be set to the specified path instead of to the mountpoint

# in sw for the corresponding ISO.

#

#

# iso directives (mandatory)

# notes: iso directives are processed last: any package for which

# a 'mount' directive does not appear will be mounted.

# One iso directive may appear for each F/S/B/P tuple.

# syntax: iso FRU SLOT BAY PACKAGE_NAME PACKAGE_FILE.bin

# example: iso rp 0 0 rp_base rp_base.ppc.bin

#

# PACKAGE_FILE.bin is a path relative to the packages.conf

# file. Although it supports sub-directories for development

# purposes, in deployment the files will always be managed

# as in the same directory as packages.conf so as to

# guarantee that name collisions cannot occur.

#

# Note that the RP 0/1 distinction is a convenience for development

# and testing as it allows us to have a packages.conf describe a

# SW load that varies depending on whether the RP finds itself in

# slot 0 or 1.

#

# The ISSU process *must* update *both* RP slots simultaneously so that

# the RP will behave predictably whichever slot it finds itself on [e.g.,

# if package X is upgraded, and the RP is ejected and put into either

# slot of a new chassis, we expect to see the upgraded X without regard

# to slot].

#

iso rp 0 0 rp_base cat3k_caa-rpbase.16.03.11.SPA.pkg

iso rp 0 0 rp_core cat3k_caa-rpcore.16.03.11.SPA.pkg

iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.03.11.SPA.pkg

iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.03.11.SPA.pkg

iso rp 0 0 rp_wcm cat3k_caa-wcm.16.03.11.SPA.pkg

iso rp 0 0 rp_webui cat3k_caa-webui.16.03.11.SPA.pkg

iso rp 0 0 srdriver cat3k_caa-srdriver.16.03.11.SPA.pkg

iso rp 0 0 rp_security cat3k_caa-rpcore.16.03.11.SPA.pkg

iso rp 0 0 guestshell cat3k_caa-guestshell.16.03.11.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.03.11.SPA.pkg

#

# -start- superpackage .pkginfo

#

# pkginfo: Name: rp_super

# pkginfo: BuildTime:

# pkginfo: ReleaseDate: Mon-17-Aug-20-08:08

# pkginfo: .BuildArch: mips64

# pkginfo: BootArchitecture: mips

# pkginfo: .BootArch: mips

# pkginfo: RouteProcessor: edison

# pkginfo: Platform: CAT3K_CAA

# pkginfo: User: mcpre

# pkginfo: PackageName: universalk9

# pkginfo: Build: 16.03.11

# pkginfo: .SupportedBoards: edison

# pkginfo: .InstallModel:

# pkginfo: .PackageRole: rp_super

# pkginfo: .RestartRole: rp_super

# pkginfo: CardTypes:

# pkginfo: .CardTypes:

# pkginfo: .BuildPath:
/scratch/mcpre/release/BLD-V16_03_11_FC2/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64

# pkginfo: .Version: 16.03.11.0.2187.1597663389..Denali

#

# -end- superpackage .pkginfo

#

Leo Laohoo · ‎03-30-2023

One stack has 16.12.7 and in Bundle Mode.

The other stack,SRH-LAN-STACK2 , has 16.3.11. I am unable to determine if this stack is in Bundle or Install Mode because of the absence of the output to the command "sh boot".

If both are in Bundle Mode, the correct firmware version will need to be manually copied across. "install auto-upgrade" will not work in Bundle Mode.

Leo Laohoo · ‎03-29-2023

Upgraded to 16.12.1?

NOTE: Last digit is a "1".

Flavio Miranda · ‎03-29-2023

Hi

Break the stack, upgrade them individually and join in stack again.

Then follow this guide

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-2950-series-switches/41845-192.html

Leo Laohoo · ‎03-29-2023

@Flavio Miranda wrote:
Break the stack, upgrade them individually and join in stack again.

Don't bother.

The stack is on 16.12.1. Once the switches are stacked, the stack-mgr process will crash the stack every few days.

16.12.X is THE buggiest trains I've ever known.