Re: Stack members come up with default port configurations after reboo

yuri.surcov · ‎11-09-2023

Hi, we have stack of 3 switches, and when switch 1, which is master, reboots: whole stack is rebooted and then switch 2 & 3 come up with default port configuration, switch 1 has all configuration correct(so we have to configure ports on 2 & 3 each time after reboot).

(I did hide last two bytes of MAC addresses):

#show switch
Switch/Stack Mac Address : 4014.82aa.1180 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
-------------------------------------------------------------------------------------
*1 Active 4014.82aa.xxxx 15 V07 Ready
2 Standby 6c03.b5c5.xxxx 10 V07 Ready
3 Member 4014.82a9.xxxx 1 V07 Ready

#show version

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 65 C9300-48T 17.09.03 CAT9K_IOSXE INSTALL
2 65 C9300-48T 17.09.03 CAT9K_IOSXE INSTALL
3 65 C9300-48T 17.09.03 CAT9K_IOSXE INSTALL

I've noticed those log messages:

%HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded

"Bulk Sync Succeed" seem like means that configuration from master actually synced to switch 2 & 3 but in fact it does not happen?

Tris message comes up as well:

%RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)

Reza Sharifi · ‎11-09-2023

Hi,

Can you post the output of "sh ver" from all 3 switches?

Hide the S/N

HTH

yuri.surcov · ‎11-09-2023

>sh version
Cisco IOS XE Software, Version 17.09.03
Cisco IOS Software [Cupertino], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.9.3, RELEASE SOFTWARE (fc6)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2023 by Cisco Systems, Inc.
Compiled Tue 14-Mar-23 18:26 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2023 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 17.9.2r, RELEASE SOFTWARE (P)

xxxxxx uptime is 2 days, 20 hours, 27 minutes
Uptime for this control processor is 2 days, 20 hours, 29 minutes
System returned to ROM by stack merge due to incompatiblity at 22:15:42 UTC Tue Oct 17 2023
System restarted at 01:02:21 UTC Tue Nov 7 2023
System image file is "flash:packages.conf"
Last reload reason: stack merge due to incompatiblity

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-essentials Smart License network-essentials
dna-essentials Subscription Smart License dna-essentials
AIR License Level: AIR DNA Advantage

Smart Licensing Status: Smart Licensing Using Policy

cisco C9300-48T (X86) processor with 1309193K/6147K bytes of memory.
Processor board ID FVH2725L3M5
1 Virtual Ethernet interface
156 Gigabit Ethernet interfaces
24 Ten Gigabit Ethernet interfaces
6 TwentyFive Gigabit Ethernet interfaces
6 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
1638400K bytes of Crash Files at crashinfo:.
1638400K bytes of Crash Files at crashinfo-2:.
1638400K bytes of Crash Files at crashinfo-3:.
11264000K bytes of Flash at flash:.
11264000K bytes of Flash at flash-2:.
11264000K bytes of Flash at flash-3:.

Base Ethernet MAC Address : 40:14:82:xx:xx:xx
Motherboard Assembly Number : 73-19915-04
Motherboard Serial Number : xxxx
Model Revision Number : A0
Motherboard Revision Number : B0
Model Number : C9300-48T
System Serial Number : xxx
CLEI Code Number : INM2900ARG

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 65 C9300-48T 17.09.03 CAT9K_IOSXE INSTALL
2 65 C9300-48T 17.09.03 CAT9K_IOSXE INSTALL
3 65 C9300-48T 17.09.03 CAT9K_IOSXE INSTALL

Switch 02
---------
Switch uptime : 2 days, 20 hours, 29 minutes

Base Ethernet MAC Address : 6c:03:b5:xx:xx:xx
Motherboard Assembly Number : 73-19915-04
Motherboard Serial Number : xxxx
Model Revision Number : A0
Motherboard Revision Number : B0
Model Number : C9300-48T
System Serial Number : xxxx
Last reload reason : NOT ACCESSABLE
CLEI Code Number : INM2900ARG

Switch 03
---------
Switch uptime : 2 days, 20 hours, 29 minutes

Base Ethernet MAC Address : 40:14:82:xx:xx:xx
Motherboard Assembly Number : 73-19915-04
Motherboard Serial Number : xxxx
Model Revision Number : A0
Motherboard Revision Number : B0
Model Number : C9300-48T
System Serial Number : FVH2725L1JQ
Last reload reason : NOT ACCESSABLE
CLEI Code Number : INM2900ARG

yuri.surcov · ‎11-09-2023

I see that switch #1 reload reason is this:

"Last reload reason: stack merge due to incompatiblity"

But I've read that this is same as just "stack merge" .

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300l-switch-stack/218083-troubleshoot-catalyst-9200-9300-reloads.html

Reza Sharifi · ‎11-09-2023

show version should show the config register. example;

Configuration register is 0x102

but I don't see it in your output

HTH

yuri.surcov · ‎11-09-2023

#show boot system
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 2
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
---------------------------
Switch 3
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

Reza Sharifi · ‎11-09-2023

The boot variables look good. What is the config-register number?

yuri.surcov · ‎11-09-2023

I've noticed those two differences between no issue stack and with this issue:

WITH ISSUE:

#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf; ////////////HERE

Boot Variables on next reload:
BOOT variable = flash:packages.conf;

NO ISSUE:

#sh boot system
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist ///// SO THIS STACK HAS nO PROBLEM ATER RELOAD

Boot Variables on next reload:
BOOT variable = flash:packages.conf

and also this:

NO ISSUE STACK switch 1

#show romvar
----------------------------------
ROMMON variables for Active Switch
----------------------------------
ABNORMAL_RESET_COUNT=0
AUTO_SWITCH_CONSOLE_DISABLE=0
BAUD=9600
BOARDID=24661
BOOT=flash:packages.conf
BOOTLDR=
BSI=0
CONFIG_FILE=
CONSOLE_MIRRORING_DISABLE=0
DC_COPY=yes /////////////////HERE
IMAGE_UPGRADE=no

NO ISSUE STACK :

show romvar switch 2
--------------------------------------
NOTE: Variables are updated on reboot.
------------------------------
ROMMON variables for Switch 2
------------------------------
ABNORMAL_RESET_COUNT=1
BAUD=9600
BOARDID=24661
BOOT=flash:packages.conf
BOOTLDR=
BSI=0
CONFIG_FILE=
DC_COPY=yes // switch #2 from stack still has this
IMAGE_UPGRADE=no

*******

STACK WITH ISSUE

#sh romvar switch 1
----------------------------------
ROMMON variables for Active Switch
----------------------------------
ABNORMAL_RESET_COUNT=0
AUTO_SWITCH_CONSOLE_DISABLE=0
BAUD=9600
BOARDID=28757
BOOT=flash:packages.conf;
BSI=0
CONSOLE_MIRRORING_DISABLE=0
DC_COPY=yes //HAS
IMAGE_UPGRADE=no

sh romvar switch 2
--------------------------------------
NOTE: Variables are updated on reboot.
------------------------------
ROMMON variables for Switch 2
------------------------------
ABNORMAL_RESET_COUNT=0
BAUD=9600
BOARDID=28757
BOOT=flash:packages.conf;
BOOTLDR=
BSI=0
CONFIG_FILE= ///MISSING DC_COPY=yes

IMAGE_UPGRADE=no

yuri.surcov · ‎11-09-2023

sh run all | i service

no issue stack has:

service call-home

issue stack:

no service call-home

Leo Laohoo · ‎11-09-2023

Wait, let me get this straight ...

Switch 1 ports have config, however, Switches 2 & 3 ports have no config. Is this correct?

If this is so, then did someone configure the ports of Switch 2 & 3 at all?

yuri.surcov · ‎11-09-2023

"Switch 1 ports have config, however, Switches 2 & 3 ports have no config. Is this correct? "

Correct, I do configure all ports on 1/0/1..., 2/0/1.., 3/0/1... and after reloading Master switch 1config stays only on 1/0/x.

( may be same happens if switches 2 or 3 are rebooted, did not try because of it is in production)

seem like 9300 can not display conf register as it not used anymore:

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9300-series-switches/216850-configuration-register-equivalent-clis-i.html

I could try :

(config)#no system ignore startupconfig : but there is no "startupconfig" word in running configuration, so not sure if this will work

Leo Laohoo · ‎11-09-2023

Alright, let's try a little experiment. Do the following:

conf t
 interface range Gi2/0/1-48, Gi3/0/1-48
 description BOOOOOOOOO!
 SHUTDOWN
end
wr

And then reboot the entire stack.

yuri.surcov · ‎11-09-2023

Thank you very your responses, but next maintenance window(seem like) in 2-3 weeks, so i will collect things and try it at once...

yuri.surcov · ‎11-10-2023

I have another stack of two switches for lab purposes:

#sh romvar switch 2

those two fields are present on lab stack switch #2 and absent on problem stack switch #2:

...

DC_COPY=yes

LICENSE_BOOT_LEVEL=network-essentials,all:C9300_48P;

...

For testing purposes, I've configured

(config)#system ignore startupconfig on lab switch

and then this field appeared in:

#sh romvar switch 2

...

SWITCH_IGNORE_STARTUP_CFG=1

then I did reload lab stack, but configuration persisted on all elements...

Stack members come up with default port configurations after reboot