topic Re: Unable to join switch to stack in Switching

Unable to join switch to stack

keny_kuruvilla — Wed, 31 Mar 2021 20:56:22 GMT

Hi members,

I have a stack with 4 members and IOS version is Denali 16.3.7. When I try to add a new switch to this stack. I get V-Mismatch error. I even tried converting the new switch to bundle mode and then add to the switch stack and run the autoupgrade feature and still doesn't work.

The following is the error I get on the switch I want to add

Waiting for 120 seconds for other switches to boot
####Switch is in STRAGGLER mode, waiting for active switch to boot
Active switch has booted up, starting discovery phase
#######################################################################################################################
Switch number is 5

Chassis is declared incompatible.
FAILED: Local package info file absent for package: 'guestshell'

Following is the error I get on the working switch stack.

%BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED:Switch 1 R0/0: issu_stack.sh: Incompatible software detected. Details: Chassis 5 is detected INCOMPATIBLE with software version of Active: FAILED: Local package info file absent for package: 'guestshell'

My show switch command is as follows

4 Member xxxx.xxxx.xxxx 7 V05 Ready
5 Member xxxx.xxxx.xxxx 5 V06 V-Mismatch <-- this is the new switch I want to add

Sh version output on the working switch stack is as follows

4 52 WS-C3650-48PD 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL

Any help would be highly appreciated.

Unable to join switch to stack

keny_kuruvilla — Wed, 31 Mar 2021 21:04:11 GMT

Hi members,

The following is the error I get on the switch I want to add

Chassis is declared incompatible.
FAILED: Local package info file absent for package: 'guestshell'

Following is the error I get on the working switch stack.

My show switch command is as follows

4 Member xxxx.xxxx.xxxx 7 V05 Ready
5 Member xxxx.xxxx.xxxx 5 V06 V-Mismatch <-- this is the new switch I want to add

Sh version output on the working switch stack is as follows

4 52 WS-C3650-48PD 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL

Any help would be highly appreciated.

Re: Unable to join switch to stack

balaji.bandi — Wed, 31 Mar 2021 21:07:51 GMT

remove switch 5 from the stack, it got version mismatch.

connect the console to switch 5 check dir? do you have any other version in the switch, clean and make it the same version of the other and join back.

or compare the IOS

show version switch 1

show version switch 5

Re: Unable to join switch to stack

paul driver — Wed, 31 Mar 2021 21:11:57 GMT

Hello

Upgrade the "new" switch to the same image as that is running on the stack and then try adding the new switch into the stack.

"V-Mismatch" means you have a image mis-match between the stack and the new switch.

Re: Unable to join switch to stack

keny_kuruvilla — Wed, 31 Mar 2021 21:12:06 GMT

It is the same image on the new and the working switch stack i.e. 16.3.7

Re: Unable to join switch to stack

keny_kuruvilla — Wed, 31 Mar 2021 21:12:44 GMT

It is the same image on the new and the working switch stack i.e. 16.3.7. I will check the dir and see any other files on the device.

Re: Unable to join switch to stack

keny_kuruvilla — Wed, 31 Mar 2021 21:13:41 GMT

Any idea what this means "FAILED: Local package info file absent for package: 'guestshell'"

Re: Unable to join switch to stack

balaji.bandi — Wed, 31 Mar 2021 21:21:42 GMT

i am thinking the switch may be crashed, generally, we see that error when the switch upgrades time fails.

suggest following my first reply - check any mismatch. try software clean on switch 5

Re: Unable to join switch to stack

Leo Laohoo — Wed, 31 Mar 2021 22:22:22 GMT

@keny_kuruvilla wrote:

Chassis is declared incompatible.

Post the complete output to the following commands taken from the stack and the new switch:

sh version
sh license

Re: Unable to join switch to stack

keny_kuruvilla — Thu, 01 Apr 2021 12:45:18 GMT

The following is the flash out and I believe this is where our problem lies

Directory of flash:/

7746 drwx 4096 Mar 22 2021 07:46:17 -04:00 .installer
7747 -rw- 2097152 Mar 31 2021 10:06:16 -04:00 nvram_config
7748 -rw- 2097152 Mar 31 2021 10:06:16 -04:00 nvram_config_bkup
7749 -rw- 1576 Mar 22 2021 12:02:36 -04:00 vlan.dat

1621966848 bytes total (1017196544 bytes free)
#dir flash-2:
Directory of flash-2:/

46466 drwx 4096 Mar 22 2021 07:48:51 -04:00 .installer
46467 -rw- 2097152 Mar 31 2021 10:06:22 -04:00 nvram_config
46468 -rw- 2097152 Mar 31 2021 10:06:22 -04:00 nvram_config_bkup
46469 -rw- 1576 Mar 22 2021 12:02:36 -04:00 vlan.dat

1622147072 bytes total (1017643008 bytes free)
#dir flash-3:
Directory of flash-3:/

15522 drwx 4096 Mar 22 2021 07:49:14 -04:00 .installer
15525 -rw- 2097152 Mar 31 2021 10:06:16 -04:00 nvram_config
15523 -rw- 1576 Mar 22 2021 12:02:36 -04:00 vlan.dat

1624244224 bytes total (1021837312 bytes free)
#dir flash-4:
Directory of flash-4:/

15490 drwx 4096 Mar 22 2021 07:47:13 -04:00 .installer
85188 -rw- 2097152 Mar 31 2021 10:06:16 -04:00 nvram_config
85186 -rw- 1576 Mar 22 2021 12:02:36 -04:00 vlan.dat

Directory of flash-5:/

31056 -rw- 2097152 Mar 31 2021 15:40:25 -04:00 nvram_config
31043 -rw- 2097152 Mar 31 2021 15:40:26 -04:00 nvram_config_bkup
31042 -rw- 76 Mar 31 2021 15:56:19 -04:00 boothelper.log
31044 -rw- 537489817 Mar 31 2021 15:29:16 -04:00 cat3k_caa-universalk9.16.03.07.SPA.bin
31049 -rw- 15938176 Mar 29 2021 15:46:42 -04:00 cat3k_caa-guestshell.16.03.07.SPA.pkg
31064 -rw- 22304945 Mar 29 2021 15:46:45 -04:00 cat3k_caa-rpbase.16.03.07.SPA.pkg
31050 -rw- 264458872 Mar 29 2021 15:46:43 -04:00 cat3k_caa-rpcore.16.03.07.SPA.pkg
31047 -rw- 4737 Mar 29 2021 15:47:58 -04:00 packages.conf
31051 -rw- 851 Mar 31 2021 15:56:05 -04:00 bootloader_evt_handle.log
31052 drwx 4096 Mar 24 2021 08:52:03 -04:00 core
69841 drwx 4096 Mar 22 2021 08:51:02 -04:00 .prst_sync
85361 drwx 4096 Mar 22 2021 08:51:04 -04:00 .rollback_timer
11 drwx 4096 Mar 31 2021 15:56:18 -04:00 dc_profile_dir
13 drwx 4096 Mar 22 2021 08:51:12 -04:00 gs_script
31054 -rw- 65568 Mar 31 2021 15:56:36 -04:00 memleak.tcl
31055 drwx 4096 Mar 22 2021 08:51:40 -04:00 .installer
31053 drwx 4096 Mar 24 2021 07:47:41 -04:00 onep
31058 -rw- 556 Mar 29 2021 09:30:35 -04:00 vlan.dat
31061 -rw- 9669248 Mar 29 2021 15:46:44 -04:00 cat3k_caa-srdriver.16.03.07.SPA.pkg
31062 -rw- 211694196 Mar 29 2021 15:46:45 -04:00 cat3k_caa-wcm.16.03.07.SPA.pkg
31060 -rw- 13419132 Mar 29 2021 15:46:44 -04:00 cat3k_caa-webui.16.03.07.SPA.pkg

I was checking other switches and I see the .pkg in the flash

Re: Unable to join switch to stack

Leo Laohoo — Sat, 03 Apr 2021 11:02:37 GMT

WTF is going on with Switches 1, 2 & 3. Particularly, where are the IOS files/packages?

Can you post the complete output to the following command:

sh boot
sh version

Re: Unable to join switch to stack

balaji.bandi — Sun, 04 Apr 2021 23:10:58 GMT

as Leo pointed, something is not right on the devices, we did not see IOS XE files in the respected switch,

can you post show version and show boot

Re: Unable to join switch to stack

keny_kuruvilla — Mon, 05 Apr 2021 13:05:08 GMT

Please find the requested output

#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode= DEVICE
iPXE Timeout= 0
Switch# sh ver
Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.7, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Fri 03-Aug-18 23:49 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)

CCN07 uptime is 1 year, 50 weeks, 3 days, 20 hours, 41 minutes
Uptime for this control processor is 1 year, 50 weeks, 3 days, 20 hours, 46 minutes
System returned to ROM by reload
System restarted at 12:16:20 EDT Wed Apr 17 2019
System image file is "flash:packages.conf"
Last reload reason: Reload command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

-----------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
lanbasek9 Permanent lanbasek9

cisco WS-C3650-48PD (MIPS) processor (revision Q0) with 865683K/6147K bytes of memory.
Processor board ID
1 Virtual Ethernet interface
200 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
250456K bytes of Crash Files at crashinfo-2:.
252000K bytes of Crash Files at crashinfo-3:.
250456K bytes of Crash Files at crashinfo-4:.
1609272K bytes of Flash at flash:.
1609272K bytes of Flash at flash-2:.
1611414K bytes of Flash at flash-3:.
1609272K bytes of Flash at flash-4:.
0K bytes of at webui:.
252000K bytes of Crash Files at crashinfo-5:.
1611414K bytes of Flash at flash-5:.

Base Ethernet MAC Address :
Motherboard Assembly Number : 73-15897-06
Motherboard Serial Number :
Model Revision Number : Q0
Motherboard Revision Number : A0
Model Number : WS-C3650-48PD
System Serial Number :

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 52 WS-C3650-48PD 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL
2 52 WS-C3650-48PD 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL
3 52 WS-C3650-48PD 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL
4 52 WS-C3650-48PD 16.3.7 CAT3K_CAA-UNIVERSALK9 INSTALL

Switch 02
---------
Switch uptime : 1 year, 50 weeks, 3 days, 21 hours, 22 minutes

Switch 03
---------
Switch uptime : 1 year, 50 weeks, 3 days, 20 hours, 46 minutes

Switch 04
---------
Switch uptime : 1 year, 50 weeks, 3 days, 20 hours, 46 minutes

Configuration register is 0x102

Switch#more flash:packages.conf
#! /usr/binos/bin/packages_conf.sh

sha1sum: fc0ffb353e67116802fef062e4a67903e00e378f

# sha1sum above - used to verify that this file is not corrupted.

#
# package.conf: provisioned software file for build 2018-08-04_01.05
#

# NOTE: Editing this file by hand is not recommended. It is generated
# as part of the build process, and is subject to boot-time
# consistency checks. Automatically-produced package files are
# guaranteed to pass those checks. Manually-maintained ones are
# not. Because "nfs" and "mount" directives are processed first,
# regardless of their position in the file, the recommended
# approach is to keep a separate file containing JUST your
# personal "nfs" and "mount" directives, and to append it to the
# automatically-generated file.
#
# Note further that when SHA-1 checksum verification is enabled,
# you will NOT be able to alter this file without updating the
# SHA-1 sum.

#
# This file can contain three types of entries:
#

#
# NFS directives (optional)
# notes: NFS directives are processed before all others (mount, iso).
# Multiple NFS directives may appear so long as they do not
# conflict -- that is, specify the same source or mountpoint.
# syntax: nfs <IP ADDRESS>:<REMOTE_PATH> <LOCAL_MOUNTPOINT>
# example: nfs 127.0.0.1:/auto/some/nfs/path /auto/some/nfs/path
#

#
# mount directives (optional)
# notes: mount directives are processed after 'nfs' and before 'iso'.
# One mount directive may appear for each F/S/B/P tuple
# syntax: mount FRU SLOT BAY PACKAGE_NAME LINUX_PATH
# example: mount rp 0 0 rp_base /auto/some/nfs/path/abs_soft/rp_base.ppc
#
# The specified LINUX_PATH may be local [sata disk] or
# reference the NFS mounts since they are processed first.
#
# Mount directives cause the package-specific mount link to
# be set to the specified path instead of to the mountpoint
# in sw for the corresponding ISO.
#

#
# iso directives (mandatory)
# notes: iso directives are processed last: any package for which
# a 'mount' directive does not appear will be mounted.
# One iso directive may appear for each F/S/B/P tuple.
# syntax: iso FRU SLOT BAY PACKAGE_NAME PACKAGE_FILE.bin
# example: iso rp 0 0 rp_base rp_base.ppc.bin
#
# PACKAGE_FILE.bin is a path relative to the packages.conf
# file. Although it supports sub-directories for development
# purposes, in deployment the files will always be managed
# as in the same directory as packages.conf so as to
# guarantee that name collisions cannot occur.
#
# Note that the RP 0/1 distinction is a convenience for development
# and testing as it allows us to have a packages.conf describe a
# SW load that varies depending on whether the RP finds itself in
# slot 0 or 1.
#
# The ISSU process *must* update *both* RP slots simultaneously so that
# the RP will behave predictably whichever slot it finds itself on [e.g.,
# if package X is upgraded, and the RP is ejected and put into either
# slot of a new chassis, we expect to see the upgraded X without regard
# to slot].
#

iso rp 0 0 rp_base cat3k_caa-rpbase.16.03.07.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.03.07.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.03.07.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.03.07.SPA.pkg
iso rp 0 0 rp_wcm cat3k_caa-wcm.16.03.07.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.03.07.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.03.07.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.03.07.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.03.07.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.03.07.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime:
# pkginfo: ReleaseDate: Sat-04-Aug-18-04:05
# pkginfo: .BuildArch: mips64
# pkginfo: BootArchitecture: mips
# pkginfo: .BootArch: mips
# pkginfo: RouteProcessor: edison
# pkginfo: Platform: CAT3K_CAA
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.03.07
# pkginfo: .SupportedBoards: edison
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /auto/mcpbuilds31/release/16.03.07/BLD-V16_03_07_FC4/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64
# pkginfo: .Version: 16.3.7.0.1936.1533369909..Denali
#
# -end- superpackage .pkginfo
#

I am planning to replace the current switches and then added the switch 5 to the new stack.