One stack member switch removed from stack after reload

[VimalDhasmana9182]

Hi Everyone,

In a stack, member switch (Cisco 9300-48P) went into removed state after reload.

Switch# Role     Mac Address          Priority       Version Current State
-------------------------------------------------------------------------------------
*1       Active   4ce1.75f1.a400       1             V03 Ready
2       Member  0000.0000.0000     0         V03 Removed

Is there any possible way to fix it without reload

As I can see in "show logging"

%HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 2: EMP_RELAY: Channel DOWN!

Please share you thoughts !

[MichaelBalzer77353]

Below you will find the full log starting from the afternoon on August 5.

Some other weird behavior because the switch 1 rebooted last Friday at 21:32 for no obvious reasons and now it is as standby and switch 2 is now active. I really don't understand what is going on here. Hopefully the logs below can be helpful.

*Aug  5 14:47:45.306: %SMART_LIC-6-AGENT_ENABLED: Smart Agent for Licensing is enabled 
*Aug  5 14:47:45.369: %REDUNDANCY-3-PEER_MONITOR: PEER_FOUND event on standby
*Aug  5 14:48:08.883: BFD: brace register success
*Aug  5 14:48:09.437: %CRYPTO-4-AUDITWARN: Encryption audit check could not be performed
*Aug  5 14:48:09.437: %CRYPTO_ENGINE-4-CSDL_COMPLIANCE_DISABLED: Cisco PSB security compliance has been disabled
*Aug  5 14:48:09.608: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Aug  5 14:48:10.014: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 1: signal on power supply A is restored
*Aug  5 14:48:10.018: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 2: signal on power supply A is restored
*Aug  5 14:48:12.119: %ONEP_BASE-6-SS_ENABLED: ONEP: Service set Base was enabled by Default
Aug  5 14:48:16.475: %SYS-7-NVRAM_INIT_WAIT_TIME: Waited 0 seconds for NVRAM to be available
Aug  5 14:48:16.928: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-2309131166 has been generated or imported by crypto config
Aug  5 16:48:17: %SYS-5-LOG_CONFIG_CHANGE: Buffer logging: level debugging, xml disabled, filtering disabled, size (65536)
Aug  5 16:48:17: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: SLA-TrustPoint created succesfully
Aug  5 16:48:17: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: TP-self-signed-2309131166 created succesfully
Aug  5 16:48:17: %CRYPTO_ENGINE-5-KEY_REPLACE: A key named TP-self-signed-2309131166 has been replaced by crypto config with new key data 
Aug  5 16:48:17: %SSH-5-DISABLED: SSH 1.99 has been disabled
Aug  5 16:48:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to down
Aug  5 16:48:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface Bluetooth0/4, changed state to down
Aug  5 16:48:25: %SSH-5-ENABLED: SSH 1.99 has been enabled
Aug  5 16:48:25: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Amsterdam], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 17.3.3, RELEASE SOFTWARE (fc7)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2021 by Cisco Systems, Inc.
Compiled Thu 04-Mar-21 08:48 by mcpre
Aug  5 16:48:29: %PKI-4-NOAUTOSAVE: Configuration was modified.  Issue "write memory" to save new certificate
Aug  5 16:48:31: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
Aug  5 16:48:32: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI created succesfully
Aug  5 16:48:32: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI has been generated or imported by pki-sudi
Aug  5 16:48:32: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI0 created succesfully
Aug  5 16:48:33: %PLATFORM-6-RF_PROG_SUCCESS: RF state STANDBY HOT
Aug  5 16:48:35: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named TP-self-signed-2309131166.server has been generated or imported by crypto-engine
Aug  6 21:30:22: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 2: EMP_RELAY: Channel UP!
Aug  6 21:30:22: %PLATFORM-6-HASTATUS: RP switchover, received chassis event to become active
Aug  6 21:30:22: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_NOT_PRESENT)
Aug  6 21:30:22: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_DOWN)
Aug  6 21:30:22: %REDUNDANCY-3-SWITCHOVER: RP switchover (PEER_REDUNDANCY_STATE_CHANGE)
Aug  6 21:30:23: %PLATFORM-6-HASTATUS: RP switchover, sent message became active. IOS is ready to switch to primary after chassis confirmation
Aug  6 21:30:23: %PLATFORM-6-HASTATUS: RP switchover, received chassis event became active
Aug  6 21:30:23: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 1: signal on power supply A is restored
Aug  6 21:30:23: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 2: signal on power supply A is restored
Aug  6 21:30:23: %PLATFORM-6-HASTATUS_DETAIL: RP switchover, received chassis event became active. Switch to primary (count 1)
Aug  6 21:30:23: %HA-6-SWITCHOVER: Route Processor switched from standby to being active
Aug  6 19:30:23.909:  Unable to set IPV4 table id for BT interface

Aug  6 19:30:23.918:  Unable to set IPV6 table id for BT interface

Aug  6 21:30:22: %STACKMGR-4-SWITCH_REMOVED: Switch 2 R0/0: stack_mgr: Switch 1 has been removed from the stack.
Aug  6 21:30:24: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Aug  6 21:30:22: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 1 on Switch 2 is down
Aug  6 21:30:22: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 2 on Switch 2 is down
Aug  6 19:30:24.101: pm_port_em_recovery 
Aug  6 21:30:24: %HMANRP-5-CHASSIS_DOWN_EVENT: Chassis 1 gone DOWN!
Aug  6 21:30:25: %LINK-3-UPDOWN: Interface Lsmpi11/3, changed state to up
Aug  6 21:30:25: %LINK-3-UPDOWN: Interface EOBC11/1, changed state to up
Aug  6 21:30:25: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN11/2, changed state to up
Aug  6 21:30:25: %LINK-3-UPDOWN: Interface LIIN11/2, changed state to up
Aug  6 21:30:25: %LINK-3-UPDOWN: Interface Null0, changed state to up
Aug  6 21:30:25: %LINK-3-UPDOWN: Interface Vlan1, changed state to up
Aug  6 21:30:25: %LINK-3-UPDOWN: Interface TenGigabitEthernet2/1/1, changed state to up
Aug  6 21:30:26: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi11/3, changed state to up
Aug  6 21:30:26: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC11/1, changed state to up
Aug  6 21:30:26: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.
Aug  6 21:30:26: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to down
Aug  6 21:30:26: %LINEPROTO-5-UPDOWN: Line protocol on Interface Null0, changed state to up
Aug  6 21:30:26: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to up
Aug  6 21:30:26: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/1, changed state to up
Aug  6 21:30:26: %SMART_LIC-6-REPORTING_REQUIRED: A Usage report acknowledgement will be required in 50 days.
Aug  6 21:30:33: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Aug  6 21:32:34: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 
Aug  6 21:32:52: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 2 on Switch 2 is up
Aug  6 21:32:56: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 1 on Switch 2 is up
Aug  6 21:32:58: %STACKMGR-4-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 1 has been added to the stack.
Aug  6 21:33:01: %STACKMGR-4-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 1 has been added to the stack.
Aug  6 21:33:02: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 1: signal on power supply A is restored
Aug  6 21:33:05: %PLATFORM_FEP-1-FRU_PS_SIGNAL_OK: Switch 1: signal on power supply A is restored
Aug  6 21:33:08: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!
Aug  6 19:33:09.841: ILP:: POE POST for switch 1: PASS
Dec 12 05:37:13: %IOSXE-3-PLATFORM: Switch 1 R0/0: kernel: pci 0000:00:00.0: of_irq_parse_pci: failed with rc=-22
Dec 12 05:37:13: %IOSXE-3-PLATFORM: Switch 1 R0/0: kernel: pci 0000:00:00.0: of_irq_parse_pci: failed with rc=-22
Aug  6 21:32:56: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is down
Aug  6 21:32:56: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is down
Aug  6 21:32:57: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is up
Aug  6 21:32:57: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is up
Aug  6 21:32:57: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
Aug  6 21:33:00: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
Aug  6 21:33:17: %LINK-3-UPDOWN: Interface GigabitEthernet0/0, changed state to up
Aug  6 21:33:18: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0/0, changed state to up
Aug  6 21:34:12: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/1
Aug  6 21:34:20: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/1, changed state to up
Aug  6 21:34:21: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/1, changed state to up
Aug  6 21:34:29: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 
Aug  6 21:35:05: %IOSXE_REDUNDANCY-6-PEER: Active detected switch 1 as standby.
Aug  6 21:35:05: %STACKMGR-6-STANDBY_ELECTED: Switch 2 R0/0: stack_mgr: Switch 1 has been elected STANDBY.
Aug  6 21:35:15: %REDUNDANCY-3-REDUNDANCY_ALARMS: Unable to deassert REDUNDANCY alarm
Aug  6 21:35:15: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

Aug  6 21:35:15: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

Aug  6 21:35:20: %REDUNDANCY-3-REDUNDANCY_ALARMS: Unable to deassert REDUNDANCY alarm
Aug  6 21:35:20: %REDUNDANCY-3-REDUNDANCY_ALARMS: Unable to deassert REDUNDANCY alarm
Aug  6 21:35:52: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Aug  6 21:35:53: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Aug  6 21:37:12: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 
Aug  7 21:30:25: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Communications failure 
Aug  7 21:37:21: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 
Aug  8 21:37:30: %SMART_LIC-3-COMM_FAILED: Communications failure with the Cisco Smart License Utility (CSLU) : Unable to resolve server hostname/domain name 

[Leo Laohoo]

I want to see those two files, @MichaelBalzer77353.

[MichaelBalzer77353]

Could you kindly provide me with your email address where I can send you these files?

[Leo Laohoo]

---

@MichaelBalzer77353 wrote:

Could you kindly provide me with your email address where I can send you these files?

---

Those files do not contain anything confidential.  
If it still not possible, contact Cisco TAC and get them to analyze the crashinfo files.

Another option is use a search engine and search for the following term:  cisco "stack merge" bug

[MichaelBalzer77353]

Looks like this could be the bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvq56135

I am going to open a TAC case.

[MichaelBalzer77353]

Here are the output of the commands requested above:

sw#dir flash-2:
Directory of flash-2:/

8106    -rw-          2097152   Aug 5 2021 12:09:30 +02:00  nvram_config_bkup
8112    -rw-          2097152   Aug 5 2021 12:09:30 +02:00  nvram_config
40481   drwx             4096   Aug 5 2021 11:10:46 +02:00  .installer
8109    -rw-              556   Aug 5 2021 11:09:30 +02:00  vlan.dat
48579   drwx             4096   Aug 5 2021 11:09:26 +02:00  license_evlog
56673   drwx             4096   Aug 5 2021 11:09:09 +02:00  .prst_sync
8107    -rw-            15139   Aug 5 2021 11:08:04 +02:00  rdope_out.txt
8105    -rw-                0   Aug 5 2021 11:08:04 +02:00  dope_hist
8108    -rw-               89   Aug 5 2021 11:08:01 +02:00  rdope.log
8102    -rw-           134458   Aug 5 2021 11:05:55 +02:00  memleak.tcl
8098    -rw-             2130   Aug 5 2021 11:05:40 +02:00  boothelper.log
80984   drwx             4096   Aug 5 2021 11:05:38 +02:00  dc_profile_dir
8099    -rw-              407   Aug 5 2021 11:05:19 +02:00  bootloader_evt_handle.log
8110    drwx             4096   Aug 5 2021 11:04:08 +02:00  .rommon_sync
8101    -rw-             2130   Aug 5 2021 10:59:05 +02:00  boothelper.log.old
40491   drwx             4096  Jun 28 2021 11:57:05 +02:00  pnp-tech
48578   drwx             4096  May 24 2021 12:17:45 +02:00  .rollback_timer
40518   -rw-         40648801  May 24 2021 12:12:24 +02:00  cat9k_lite-rpboot.17.03.03.SPA.pkg
40514   -rw-             4919  May 24 2021 12:12:24 +02:00  packages.conf
40517   -rw-         11031572  May 24 2021 12:11:00 +02:00  cat9k_lite-webui.17.03.03.SPA.pkg
40516   -rw-          4133912  May 24 2021 12:11:00 +02:00  cat9k_lite-srdriver.17.03.03.SPA.pkg
40515   -rw-        426931224  May 24 2021 12:11:00 +02:00  cat9k_lite-rpbase.17.03.03.SPA.pkg
40486   drwx             4096  May 24 2021 12:05:48 +02:00  .dbpersist
40484   drwx             4096  May 24 2021 12:02:56 +02:00  core
48580   drwx             4096  May 24 2021 12:02:36 +02:00  pnp-info
40488   drwx             4096  May 24 2021 12:02:35 +02:00  onep
89057   drwx             4096  May 24 2021 12:01:21 +02:00  .USWAP
113345  drwx             4096  May 24 2021 11:56:40 +02:00  Tbot
105249  drwx             4096  May 24 2021 11:56:39 +02:00  .CRFT
80986   drwx             4096  May 24 2021 11:56:33 +02:00  sys_report
80961   drwx             4096  May 24 2021 11:56:32 +02:00  tech_support
56676   drwx             4096  May 24 2021 11:56:32 +02:00  ss_disc
8100    -rw-          5242880  May 24 2021 11:56:32 +02:00  ssd

1956904960 bytes total (1359216640 bytes free)

sw#dir flash-2:core
Directory of flash-2:/core/

40490   -rw-                1   Aug 5 2021 14:09:26 +02:00  .callhome
64769   drwx             4096  May 24 2021 11:56:29 +02:00  modules

1956904960 bytes total (1359216640 bytes free)

sw#dir crashinfo-2:
Directory of crashinfo-2:/

29313   drwx            24576   Aug 5 2021 14:19:42 +02:00  tracelogs
15      -rw-         11033250   Aug 5 2021 10:57:35 +02:00  sw_2_RP_0-system-report_2_20210805-105728-CEST.tar.gz
14      -rw-          2932907   Aug 4 2021 19:48:30 +02:00  sw_2_RP_0_trace_archive_0-20210804-194826.tar.gz
13      -rw-          9819246   Aug 4 2021 18:00:54 +02:00  sw_2_RP_0-system-report_2_20210804-180048-CEST.tar.gz
11      -rw-          2797408   Aug 4 2021 18:00:46 +02:00  sw_2_RP_0_trace_archive_0-20210804-180041.tar.gz
12      -rw-                0  Dec 12 2020 05:36:51 +01:00  koops.dat

825753600 bytes total (751304704 bytes free)

sw#sh log on switch 2 uptime detail
--------------------------------------------------------------------------------
UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 05/24/2021 12:01:55
Total uptime            :  0  years  0  weeks  1  days  0  hours  43 minutes
Total downtime          :  0  years  10 weeks  2  days  1  hours  20 minutes
Number of resets        : 8
Number of slot changes  : 1
Current reset reason    : Reload Slot Command
Current reset timestamp : 08/05/2021 11:06:41
Current slot            : 2
Chassis type            : 247
Current uptime          :  0  years  0  weeks  0  days  3  hours  0  minutes
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp          | Reset                       | Uptime
MM/DD/YYYY HH:MM:SS | Reason                      | years weeks days hours minutes
--------------------------------------------------------------------------------
05/24/2021 12:01:55   Power Failure or Unknown      0     0     0     0     0  
05/24/2021 12:20:42   Image Install                 0     0     0     0     15 
05/24/2021 12:24:27   Reload Command                0     0     0     0     0  
06/28/2021 11:54:49   Power Failure or Unknown      0     0     0     0     0  
06/28/2021 15:37:30   Power Failure or Unknown      0     0     0     1     0  
08/04/2021 15:39:30   Power Failure or Unknown      0     0     0     1     0  
08/04/2021 18:03:26   stack merge                   0     0     0     1     58 
08/05/2021 11:00:07   stack merge                   0     0     0     16    0  
08/05/2021 11:06:41   Reload Slot Command           0     0     0     0     0  
--------------------------------------------------------------------------------