9800-40 Triggered dual-active recovery

[scotv]

I have two new 9800-40 controllers running 17.06.03

One chassis set to to number 1 priority 2 and the other is number 2 priority 1.

When they are booted at the same time, eventually dual-active recovery is triggered and one chassis goes to active recovery mode. I'll add the show romvar outputs of both controllers.This is the log of the chassis that goes to recovery mode:

*Aug 30 22:37:55.442: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet0/0/0, changed state to up
*Aug 30 22:37:56.441: %LINK-3-UPDOWN: Interface Port-channel36, changed state to up
*Aug 30 22:37:57.441: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel36, changed state to up
*Aug 30 22:37:57.442: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan550, changed state to up
*Aug 30 22:37:59.603: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY created succesfully
*Aug 30 22:37:59.609: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI_LEGACY has been generated or imported by pki-sudi
*Aug 30 22:37:59.625: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY0 created succesfully
*Aug 30 22:38:01.574: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI created succesfully
*Aug 30 22:38:01.579: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI has been generated or imported by pki-sudi
*Aug 30 22:38:01.596: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI0 created succesfully
*Aug 30 22:38:12.556: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.

*Aug 30 22:38:26.646: %RIF_MGR_FSM-6-RMI_LINK_UP: Chassis 1 R0/0: rif_mgr: The RMI link is UP.
*Aug 30 22:38:26.646: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 1 R0/0: stack_mgr: Dual Active Detection link is available now
*Aug 30 22:38:26.750: %RIF_MGR_FSM-6-GW_REACHABLE_ACTIVE: Chassis 1 R0/0: rif_mgr: Gateway reachable from Active

Aug 30 22:38:48.411: %PKI-6-AUTHORITATIVE_CLOCK: The system clock has been set.

Aug 30 22:47:26.160: %RIF_MGR_FSM-6-RMI_LINK_DOWN: Chassis 1 R0/0: rif_mgr: The RMI link is DOWN.
Aug 30 22:47:26.160: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 1 R0/0: stack_mgr: Dual Active Detection links are not available anymore

WLC2#Ewlc: triggered dual-active recovery,setting hostname to
% Hostname "WLC2(RECOVERY-MO " is not a legal LAT node name, Using "CISCO_1BB52C"
Aug 30 22:54:46.237: RMI-HAINFRA-INFO: Originating event to Shut all Interfaces
Aug 30 22:54:46.238: RMI-HAINFRA-INFO: Shutting down all interfaces in ActiveRecovery
Aug 30 22:54:46.238: RMI-HAINFRA-INFO: Not shutting down the interface-rmi: Vlan550
.Aug 30 22:54:46.236: %RIF_MGR_FSM-6-RMI_LINK_UP: Chassis 1 R0/0: rif_mgr: The RMI link is UP.
.Aug 30 22:54:46.236: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 1 R0/0: stack_mgr: Dual Active Detection link is available now
.Aug 30 22:54:46.236: %RIF_MGR_FSM-6-RMI_ACTIVE_RECOVERY_MODE: Chassis 1 R0/0: rif_mgr: Going to Active(Recovery) from Active state
.Aug 30 22:54:48.238: %LINK-5-CHANGED: Interface GigabitEthernet0, changed state to administratively down

[balaji.bandi]

how is these 2 WLC connected, what code running, if the Physical link lost between WLC then they become both active (depends on the config).

[scotv]

They are connected by the fiber RP port. Running 17.06.03.

[marce1000]

 -  Review the 9800-40  configuration with the CLI command : show  tech wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.           Checkout all advisories 

 M.

[scotv]

This did not yield anything helpful. The output did take 20 minutes to run though.

[jtpntx]

On a busy controller  show  tech wireless never returns. either running it as exec cli or bundle. The backup wlc with no APs returns pretty qucik
None of this is mentioned in it.

[scotv]

I've reset both to factory and just configured them for RP. They still do not join and both show up as active. Here is the romvar settings for both:

Chassis# Role Mac Address Priority Version State IP
-------------------------------------------------------------------------------------
*1 Active 806a.001b.b520 2 V02 Ready 169.254.71.17

WLC#sh romvar
ROMMON variables:
PS1 = rommon ! >
LICENSE_BOOT_LEVEL =
THRPUT =
STACK_1_1 = 0_0
? = 1
CHASSIS_HA_PEER_TIMEOUT = 1000
CHASSIS_HA_LOCAL_IP = 169.254.71.17
CHASSIS_HA_REMOTE_IP = 169.254.71.16
CHASSIS_HA_LOCAL_MASK = 255.255.0.0
BOOT = bootflash:packages.conf,12;
RET_2_RTS =
SWITCH_NUMBER = 1
SWITCH_PRIORITY = 2
BSI = 0
RET_2_RCALTS =
RANDOM_NUM = 1650818792

Chassis# Role Mac Address Priority Version State IP
-------------------------------------------------------------------------------------
*2 Active 806a.001b.b500 1 V02 Ready 169.254.71.16

wlc2#sh romvar
ROMMON variables:
PS1 = rommon ! >
LICENSE_BOOT_LEVEL =
THRPUT =
STACK_1_1 = 0_0
? = 1
CHASSIS_HA_PEER_TIMEOUT = 1000
CHASSIS_HA_LOCAL_IP = 169.254.71.16
CHASSIS_HA_REMOTE_IP = 169.254.71.17
CHASSIS_HA_LOCAL_MASK = 255.255.0.0
BOOT = bootflash:packages.conf,12;
RET_2_RTS =
SWITCH_NUMBER = 2
SWITCH_PRIORITY = 1
BSI = 0
RET_2_RCALTS =
RANDOM_NUM = 1547926146
wlc2#

[Arshad Safrulla]

What was the RP link status when this happened?

Which controller booted up first?

Which controller went to active recovery mode?

Can you paste the CLI outputs from both WLC's during the complete process? (You can sanitize sensitive info)

Was the Gateway reachable from both RMI IP's of both WLC's?

Read the below for more info. Check what is your scenario and verify against the documented behavior. Review your HA configuration and make sure that is as per the best practices. If everything is in order reach out to Cisco TAC for further help.

High Availability SSO Deployment Guide for Cisco Catalyst 9800 Series Wireless Controllers, Cisco IOS XE Amsterdam 17.1

[scotv]

I'm not seeing anything in the logs regarding the RP link. Controller 1 set as priority 2 booted up first. The controller 2 set as priority one went to active recovery mode. The first CLI output is from controller 2:

Base Ethernet MAC Address : 80:6A:00:1B:B5:20

Installation mode is INSTALL

--------------------------------------------------
System is booted with ASCII based startup configuration
due to missing binary configuration or previous condition.
Please perform "write mem" to generate binary
configuration. System uses binary-config internally to
reduce overall bootime significantly.
--------------------------------------------------

WARNING: ** NOTICE ** The H.323 protocol is no longer supported from IOS-XE release 17.6.1. Please consider using SIP for multimedia applications.

Press RETURN to get started!

Adding registry invocations for the WLC platform

*Sep 1 15:10:38.269: %SMART_LIC-6-AGENT_ENABLED: Smart Agent for Licensing is enabled
*Sep 1 15:10:41.235: WLC-HA-Notice: RF Progression event: RF_PROG_ACTIVE_FAST, Switchover triggered
*Sep 1 15:10:41.251: mcp_pm_subsys_init : Init done sucessfullyRA Tracing tool registry return: 0
*Sep 1 15:10:44.169: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Sep 1 15:10:45.860: %TLSCLIENT-5-TLSCLIENT_BINOS: TLS Client is BINOS based
*Sep 1 15:10:46.821: %CUBE-3-LICENSING: SIP trunking (CUBE) licensing is now based on dynamic sessions counting, static license capacity configuration through 'mode border-element license capacity' would be ignored.
*Sep 1 15:10:46.821: %SIP-5-LICENSING: CUBE license reporting period has been set to the minimum value of 8 hours.
*Sep 1 15:10:46.855: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_ENFORCED: Cisco PSB security compliance is being enforced
*Sep 1 15:10:46.855: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by this platform for use of RSA Key Size
*Sep 1 15:10:46.932: %VOICE_HA-7-STATUS: CUBE HA-supported platform detected.
*Sep 1 15:10:46.933: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKEv2 for use of DES

*Sep 1 15:10:47.194: mcp_pm_init_done : Called
*Sep 1 15:10:47.201: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Sep 1 15:10:47.217: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Sep 1 15:10:47.217: %LINEPROTO-5-UPDOWN: Line protocol on Interface VoIP-Null0, changed state to up
*Sep 1 15:10:47.218: %LINEPROTO-5-UPDOWN: Line protocol on Interface LI-Null0, changed state to up
*Sep 1 15:10:47.218: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Sep 1 15:10:47.219: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Sep 1 15:10:47.316: %PNP-6-PNP_DISCOVERY_STARTED: PnP Discovery started
*Sep 1 15:08:02.832: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:08:28.034: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 1 R0/0: stack_mgr: Stack port 1 on Chassis 1 is down
*Sep 1 15:08:28.034: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 1 R0/0: stack_mgr: Stack port 2 on Chassis 1 is down
*Sep 1 15:08:28.034: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 1 R0/0: stack_mgr: Stack port 1 on Chassis 1 is up
*Sep 1 15:08:28.034: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 1 R0/0: stack_mgr: Stack port 2 on Chassis 1 is up
*Sep 1 15:08:28.526: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 1 R0/0: stack_mgr: Dual Active Detection links are not available anymore
*Sep 1 15:08:28.526: %RIF_MGR_FSM-6-RP_LINK_DOWN: Chassis 1 R0/0: rif_mgr: Setting RP link status to DOWN
*Sep 1 15:08:29.499: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:08:31.357: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:08:44.526: %CMFP-6-CRYPTO_MODULE: Chassis 1 F0/0: cman_fp: Crypto Hardware Module is present
*Sep 1 15:10:28.040: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 1 15:10:28.719: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:29.738: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 1 15:10:31.744: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 1 15:10:31.744: %STACKMGR-6-ACTIVE_ELECTED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been elected ACTIVE.
*Sep 1 15:10:31.808: %TDLDB-6-DBPERSIST_OPERATION_INFO: Chassis 1 R0/0: btman: Cancelled restore operation and is intended - DB: persistent-config.meta, Reason: Abort binary-config restore
*Sep 1 15:10:31.808: %TDLDB-3-DBPERSIST_OPERATION_FAILURE: Chassis 1 R0/0: btman: Operation: restore, DB: /tmp/rp/tdldb/0/INFRA_CONFIG_DB, Failure reason: Operation canceled
*Sep 1 15:10:38.282: %LMRP-3-RTU_UNINITIALIZED: Chassis 1 R0/0: lman: RTU not yet initialized: stack enabled 0
*Sep 1 15:10:39.163: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:41.197: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:43.214: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:47.916: %SYS-5-CONFIG_P: Configured programmatically by process MGMT VRF Process from console as vty0
*Sep 1 15:10:48.524: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-intf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Sep 1 15:10:48.574: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Sep 1 15:10:48.575: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi0, changed state to up
*Sep 1 15:10:48.575: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC0, changed state to up
*Sep 1 15:10:48.575: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Sep 1 15:10:48.575: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Sep 1 15:10:48.578: %SYS-5-CONFIG_P: Configured programmatically by process MGMT VRF Process from console as vty0
*Sep 1 15:10:49.182: %ONEP_BASE-6-SS_ENABLED: ONEP: Service set Base was enabled by Default
*Sep 1 15:10:49.393: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:52.322: %SYS-7-NVRAM_INIT_WAIT_TIME: Waited 0 seconds for NVRAM to be available
*Sep 1 15:10:52.392: %IOSXE_RP_CFG_INFO-6-MCPRP_ASCII_CFG_APPLIED: System configuration : ASCII
*Sep 1 15:10:52.444: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: TP-self-signed-1362648598 created succesfully
*Sep 1 15:10:52.483: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: SLA-TrustPoint created succesfully
*Sep 1 15:10:52.573: % Redundancy mode change to SSO

*Sep 1 15:10:52.573: %VOICE_HA-7-STATUS: NONE->SSO; SSO mode will not take effect until after a platform reload.
*Sep 1 15:10:53.003: RMI-HAINFRA-INFO: Learning Management IP: 10.19.71.16, mask: 255.255.255.224, if_number: 15l
*Sep 1 15:10:53.440: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan550, changed state to down
*Sep 1 15:10:53.550: %SYS-5-CONFIG_I: Configured from memory by console
*Sep 1 15:10:53.556: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/0, interfaces disabled
*Sep 1 15:10:53.586: %SPA_OIR-6-OFFLINECARD: SPA (BUILT-IN-4X10G/1G) offline in subslot 0/0
*Sep 1 15:10:53.598: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0
*Sep 1 15:10:53.598: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0
*Sep 1 15:10:53.635: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0
*Sep 1 15:10:53.635: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 0
*Sep 1 15:10:53.896: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/0
*Sep 1 15:10:53.789: %CFMGR_LOG-4-COUNTRY_CFG_DEPRECATED_CLI: Chassis 1 R0/0: wncd: Deprecated CLI used: 'ap country <coutry-code>' is deprecated, instead use 'wireless country <country-code>'
*Sep 1 15:10:55.396: %LINK-5-CHANGED: Interface GigabitEthernet0, changed state to administratively down
*Sep 1 15:10:55.397: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Sep 1 15:10:56.673: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Bengaluru], C9800 Software (C9800_IOSXE-K9), Version 17.6.3, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2022 by Cisco Systems, Inc.
Compiled Wed 30-Mar-22 23:12 by mcpre
*Sep 1 15:10:56.674: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel36, changed state to down
*Sep 1 15:10:59.212: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Sep 1 15:10:59.212: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Sep 1 15:11:01.679: %PNP-6-PNP_BEST_UDI_UPDATE: Best UDI [PID:C9800-40-K9,VID:V06,SN:TTM254100Q6] identified via (entity-mibs)
*Sep 1 15:11:01.679: %PNP-6-PNP_CDP_UPDATE: Device UDI [PID:C9800-40-K9,VID:V06,SN:TTM254100Q6] identified for CDP
*Sep 1 15:11:01.679: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)
*Sep 1 15:11:02.333: %SPA_OIR-6-ONLINECARD: SPA (BUILT-IN-4X10G/1G) online in subslot 0/0
*Sep 1 15:11:02.571: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to remote fault
*Sep 1 15:11:02.709: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/1, link down due to remote fault
*Sep 1 15:11:02.848: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/2, link down due to remote fault
*Sep 1 15:11:02.974: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/3, link down due to remote fault
*Sep 1 15:11:04.258: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to down
*Sep 1 15:11:04.306: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/1, changed state to down
*Sep 1 15:11:04.313: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/2, changed state to down
*Sep 1 15:11:04.314: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/3, changed state to down
*Sep 1 15:11:06.899: %PKI-2-NON_AUTHORITATIVE_CLOCK: PKI functions can not be initialized until an authoritative time source, like NTP, can be obtained.
**Sep 1 15:11:05.334: %LINK-3-UPDOWN: SIP0/0: Interface TenGigabitEthernet0/0/0, changed state to up
*Sep 1 15:11:08.450: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/1, link down due to remote fault
*Sep 1 15:11:08.488: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/2, link down due to local fault
*Sep 1 15:11:08.526: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/3, link down due to remote fault
*Sep 1 15:11:08.324: %RIF_MGR_FSM-6-GW_UNREACHABLE_ACTIVE: Chassis 1 R0/0: rif_mgr: Gateway not reachable from Active
*Sep 1 15:11:08.728: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to up
*Sep 1 15:11:09.729: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet0/0/0, changed state to up
*Sep 1 15:11:10.305: %LINK-3-UPDOWN: Interface Port-channel36, changed state to up
*Sep 1 15:11:11.305: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel36, changed state to up
*Sep 1 15:11:11.306: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan550, changed state to up
*Sep 1 15:11:15.205: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY created succesfully
*Sep 1 15:11:15.211: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI_LEGACY has been generated or imported by pki-sudi
*Sep 1 15:11:15.229: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY0 created succesfully
*Sep 1 15:11:17.191: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI created succesfully
*Sep 1 15:11:17.196: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI has been generated or imported by pki-sudi
*Sep 1 15:11:17.217: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI0 created succesfully

*Sep 1 15:11:27.423: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by SLA-KeyPair
*Sep 1 15:11:27.424: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Sep 1 15:11:27.430: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.
*Sep 1 15:11:28.281: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair.server has been generated or imported by crypto-engine
*Sep 1 15:11:41.233: %RIF_MGR_FSM-6-GW_REACHABLE_ACTIVE: Chassis 1 R0/0: rif_mgr: Gateway reachable from Active
*Sep 1 15:11:47.298: %CRYPTO_ENGINE-4-CSDL_COMPLIANCE_RSA_WEAK_KEYS: RSA keypair CISCO_IDEVID_SUDI_LEGACY is in violation of Cisco security compliance guidelines and will be rejected by future releases.Ewlc: triggered dual-active recovery,setting hostname to
% Hostname "WLC(RECOVERY-MOD " is not a legal LAT node name, Using "CISCO_1BB52C"
*Sep 1 15:11:48.555: RMI-HAINFRA-INFO: Originating event to Shut all Interfaces
*Sep 1 15:11:48.555: RMI-HAINFRA-INFO: Shutting down all interfaces in ActiveRecovery
*Sep 1 15:11:48.556: RMI-HAINFRA-INFO: Not shutting down the interface-rmi: Vlan550
*Sep 1 15:11:48.554: %RIF_MGR_FSM-6-RMI_LINK_UP: Chassis 1 R0/0: rif_mgr: The RMI link is UP.
*Sep 1 15:11:48.554: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 1 R0/0: stack_mgr: Dual Active Detection link is available now
*Sep 1 15:11:48.555: %RIF_MGR_FSM-6-RMI_ACTIVE_RECOVERY_MODE: Chassis 1 R0/0: rif_mgr: Going to Active(Recovery) from Active state

Controller 1 output:

Base Ethernet MAC Address : 80:6A:00:1B:B5:00

Installation mode is INSTALL

--------------------------------------------------
System is booted with ASCII based startup configuration
due to missing binary configuration or previous condition.
Please perform "write mem" to generate binary
configuration. System uses binary-config internally to
reduce overall bootime significantly.
--------------------------------------------------

WARNING: ** NOTICE ** The H.323 protocol is no longer supported from IOS-XE release 17.6.1. Please consider using SIP for multimedia applications.

Press RETURN to get started!

Adding registry invocations for the WLC platform

*Sep 1 15:10:42.089: %SMART_LIC-6-AGENT_ENABLED: Smart Agent for Licensing is enabled
*Sep 1 15:10:45.057: WLC-HA-Notice: RF Progression event: RF_PROG_ACTIVE_FAST, Switchover triggered
*Sep 1 15:10:45.072: mcp_pm_subsys_init : Init done sucessfullyRA Tracing tool registry return: 0
*Sep 1 15:10:47.995: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Sep 1 15:10:49.682: %TLSCLIENT-5-TLSCLIENT_BINOS: TLS Client is BINOS based
*Sep 1 15:10:50.644: %CUBE-3-LICENSING: SIP trunking (CUBE) licensing is now based on dynamic sessions counting, static license capacity configuration through 'mode border-element license capacity' would be ignored.
*Sep 1 15:10:50.644: %SIP-5-LICENSING: CUBE license reporting period has been set to the minimum value of 8 hours.
*Sep 1 15:10:50.678: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_ENFORCED: Cisco PSB security compliance is being enforced
*Sep 1 15:10:50.678: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by this platform for use of RSA Key Size
*Sep 1 15:10:50.755: %VOICE_HA-7-STATUS: CUBE HA-supported platform detected.
*Sep 1 15:10:50.757: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKEv2 for use of DES
*Sep 1 15:10:50.757: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKEv2 for use of 3DES
*Sep 1 15:10:50.757: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKEv2 for use of DH
*Sep 1 15:10:50.757: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKEv2 for use of MD5
*Sep 1 15:10:50.757: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKEv2 for use of SHA1
*Sep 1 15:10:50.759: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKMP for use of DES
*Sep 1 15:10:50.759: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKMP for use of 3DES
*Sep 1 15:10:50.759: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKMP for use of DH
*Sep 1 15:10:50.759: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKMP for use of MD5
*Sep 1 15:10:50.759: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by Crypto IKMP for use of SHA1
*Sep 1 15:10:50.760: %CRYPTO_ENGINE-5-CSDL_COMPLIANCE_EXCEPTION_ADDED: Cisco PSB security compliance exception has been added by IPSEC key engine for use of 3DES
*Sep 1 15:10:51.018: mcp_pm_init_done : Called
*Sep 1 15:10:51.025: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Sep 1 15:10:51.042: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Sep 1 15:10:51.042: %LINEPROTO-5-UPDOWN: Line protocol on Interface VoIP-Null0, changed state to up
*Sep 1 15:10:51.042: %LINEPROTO-5-UPDOWN: Line protocol on Interface LI-Null0, changed state to up
*Sep 1 15:10:51.043: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Sep 1 15:10:51.043: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Sep 1 15:10:51.142: %PNP-6-PNP_DISCOVERY_STARTED: PnP Discovery started
*Sep 1 15:08:06.708: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:08:31.917: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 2 R0/0: stack_mgr: Stack port 1 on Chassis 2 is down
*Sep 1 15:08:31.917: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 2 R0/0: stack_mgr: Stack port 2 on Chassis 2 is down
*Sep 1 15:08:31.917: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 2 R0/0: stack_mgr: Stack port 1 on Chassis 2 is up
*Sep 1 15:08:31.917: %STACKMGR-6-STACK_LINK_CHANGE: Chassis 2 R0/0: stack_mgr: Stack port 2 on Chassis 2 is up
*Sep 1 15:08:32.407: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 2 R0/0: stack_mgr: Dual Active Detection links are not available anymore
*Sep 1 15:08:32.407: %RIF_MGR_FSM-6-RP_LINK_DOWN: Chassis 2 R0/0: rif_mgr: Setting RP link status to DOWN
*Sep 1 15:08:33.364: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:08:35.214: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:08:48.411: %CMFP-6-CRYPTO_MODULE: Chassis 2 F0/0: cman_fp: Crypto Hardware Module is present
*Sep 1 15:10:31.923: %STACKMGR-6-CHASSIS_ADDED: Chassis 2 R0/0: stack_mgr: Chassis 2 has been added to the stack.
*Sep 1 15:10:32.601: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:33.594: %STACKMGR-6-CHASSIS_ADDED: Chassis 2 R0/0: stack_mgr: Chassis 2 has been added to the stack.
*Sep 1 15:10:35.601: %STACKMGR-6-CHASSIS_ADDED: Chassis 2 R0/0: stack_mgr: Chassis 2 has been added to the stack.
*Sep 1 15:10:35.601: %STACKMGR-6-ACTIVE_ELECTED: Chassis 2 R0/0: stack_mgr: Chassis 2 has been elected ACTIVE.
*Sep 1 15:10:35.665: %TDLDB-6-DBPERSIST_OPERATION_INFO: Chassis 2 R0/0: btman: Cancelled restore operation and is intended - DB: persistent-config.meta, Reason: Abort binary-config restore
*Sep 1 15:10:35.665: %TDLDB-3-DBPERSIST_OPERATION_FAILURE: Chassis 2 R0/0: btman: Operation: restore, DB: /tmp/rp/tdldb/0/INFRA_CONFIG_DB, Failure reason: Operation canceled
*Sep 1 15:10:42.101: %LMRP-3-RTU_UNINITIALIZED: Chassis 2 R0/0: lman: RTU not yet initialized: stack enabled 0
*Sep 1 15:10:42.992: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:45.059: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:47.075: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:51.746: %SYS-5-CONFIG_P: Configured programmatically by process MGMT VRF Process from console as vty0
*Sep 1 15:10:52.347: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-intf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Sep 1 15:10:52.398: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1, changed state to down
*Sep 1 15:10:52.398: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi0, changed state to up
*Sep 1 15:10:52.398: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC0, changed state to up
*Sep 1 15:10:52.398: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Sep 1 15:10:52.398: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Sep 1 15:10:52.401: %SYS-5-CONFIG_P: Configured programmatically by process MGMT VRF Process from console as vty0
*Sep 1 15:10:52.982: %ONEP_BASE-6-SS_ENABLED: ONEP: Service set Base was enabled by Default
*Sep 1 15:10:53.214: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 2 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:56.031: %SYS-7-NVRAM_INIT_WAIT_TIME: Waited 0 seconds for NVRAM to be available
*Sep 1 15:10:56.100: %IOSXE_RP_CFG_INFO-6-MCPRP_ASCII_CFG_APPLIED: System configuration : ASCII
*Sep 1 15:10:56.144: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: TP-self-signed-1858731100 created succesfully
*Sep 1 15:10:56.183: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: SLA-TrustPoint created succesfully
*Sep 1 15:10:56.233: % Redundancy mode change to SSO

*Sep 1 15:10:56.233: %VOICE_HA-7-STATUS: NONE->SSO; SSO mode will not take effect until after a platform reload.
*Sep 1 15:10:57.139: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan550, changed state to down
*Sep 1 15:10:57.271: %SYS-5-CONFIG_I: Configured from memory by console
*Sep 1 15:10:57.276: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/0, interfaces disabled
*Sep 1 15:10:57.304: %SPA_OIR-6-OFFLINECARD: SPA (BUILT-IN-4X10G/1G) offline in subslot 0/0
*Sep 1 15:10:57.317: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0
*Sep 1 15:10:57.317: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0
*Sep 1 15:10:57.355: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0
*Sep 1 15:10:57.355: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 0
*Sep 1 15:10:57.621: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/0
*Sep 1 15:10:57.470: %CFMGR_LOG-4-COUNTRY_CFG_DEPRECATED_CLI: Chassis 2 R0/0: wncd: Deprecated CLI used: 'ap country <coutry-code>' is deprecated, instead use 'wireless country <country-code>'
*Sep 1 15:10:59.155: %LINK-5-CHANGED: Interface GigabitEthernet0, changed state to administratively down
*Sep 1 15:10:59.156: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
*Sep 1 15:10:59.323: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel35, changed state to down
*Sep 1 15:11:00.467: %SYS-5-RESTART: System restarted --
Cisco IOS Software [Bengaluru], C9800 Software (C9800_IOSXE-K9), Version 17.6.3, RELEASE SOFTWARE (fc4)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2022 by Cisco Systems, Inc.
Compiled Wed 30-Mar-22 23:12 by mcpre
*Sep 1 15:11:03.331: %CRYPTO-6-ISAKMP_ON_OFF: ISAKMP is OFF
*Sep 1 15:11:03.331: %CRYPTO-6-GDOI_ON_OFF: GDOI is OFF
*Sep 1 15:11:05.475: %PNP-6-PNP_BEST_UDI_UPDATE: Best UDI [PID:C9800-40-K9,VID:V06,SN:TTM2548030F] identified via (entity-mibs)
*Sep 1 15:11:05.475: %PNP-6-PNP_CDP_UPDATE: Device UDI [PID:C9800-40-K9,VID:V06,SN:TTM2548030F] identified for CDP
*Sep 1 15:11:05.476: %PNP-6-PNP_DISCOVERY_STOPPED: PnP Discovery stopped (Startup Config Present)
*Sep 1 15:11:06.165: %SPA_OIR-6-ONLINECARD: SPA (BUILT-IN-4X10G/1G) online in subslot 0/0
*Sep 1 15:11:06.307: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/0, link down due to remote fault
*Sep 1 15:11:06.524: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/1, link down due to remote fault
*Sep 1 15:11:06.654: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/2, link down due to remote fault
*Sep 1 15:11:06.780: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/3, link down due to remote fault
*Sep 1 15:11:08.111: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to down
*Sep 1 15:11:08.143: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/1, changed state to down
*Sep 1 15:11:08.145: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/2, changed state to down
*Sep 1 15:11:08.148: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/3, changed state to down
*Sep 1 15:11:09.751: %SYS-6-BOOTTIME: Time taken to reboot after reload = 304 seconds
*Sep 1 15:11:10.721: %PKI-2-NON_AUTHORITATIVE_CLOCK: PKI functions can not be initialized until an authoritative time source, like NTP, can be obtained.
*Sep 1 15:11:09.166: %LINK-3-UPDOWN: SIP0/0: Interface TenGigabitEthernet0/0/0, changed state to down
*Sep 1 15:11:12.284: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/1, link down due to remote fault
*Sep 1 15:11:12.323: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/2, link down due to remote fault
*Sep 1 15:11:12.362: %IOSXE_SPA-6-UPDOWN: Interface TenGigabitEthernet0/0/3, link down due to remote fault
*Sep 1 15:11:12.632: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to up
*Sep 1 15:11:13.194: %RIF_MGR_FSM-6-GW_UNREACHABLE_ACTIVE: Chassis 2 R0/0: rif_mgr: Gateway not reachable from Active
*Sep 1 15:11:14.175: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet0/0/0, changed state to up
*Sep 1 15:11:15.174: %LINK-3-UPDOWN: Interface Port-channel35, changed state to up
*Sep 1 15:11:16.175: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel35, changed state to up
*Sep 1 15:11:16.175: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan550, changed state to up
*Sep 1 15:11:18.959: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY created succesfully
*Sep 1 15:11:18.965: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI_LEGACY has been generated or imported by pki-sudi
*Sep 1 15:11:18.983: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI_LEGACY0 created succesfully
*Sep 1 15:11:20.930: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI created succesfully
*Sep 1 15:11:20.936: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named CISCO_IDEVID_SUDI has been generated or imported by pki-sudi
*Sep 1 15:11:20.952: %PKI-6-TRUSTPOINT_CREATE: Trustpoint: CISCO_IDEVID_SUDI0 created succesfully
*Sep 1 15:11:31.214: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by SLA-KeyPair
*Sep 1 15:11:31.215: %SSH-5-ENABLED: SSH 1.99 has been enabled
*Sep 1 15:11:31.222: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.
*Sep 1 15:11:32.149: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair.server has been generated or imported by crypto-engine
*Sep 1 15:11:46.969: %RIF_MGR_FSM-6-RMI_LINK_UP: Chassis 2 R0/0: rif_mgr: The RMI link is UP.
*Sep 1 15:11:46.969: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 2 R0/0: stack_mgr: Dual Active Detection link is available now
*Sep 1 15:11:51.123: %CRYPTO_ENGINE-4-CSDL_COMPLIANCE_RSA_WEAK_KEYS: RSA keypair CISCO_IDEVID_SUDI_LEGACY is in violation of Cisco security compliance guidelines and will be rejected by future releases.
*Sep 1 15:11:55.731: %WSMAN-3-INVALID_TRUSTPOINT: Trustpoint

[scotv]

*Sep 1 15:10:28.040: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 1 15:10:28.719: %PMAN-3-PROC_EMPTY_EXEC_FILE: Chassis 1 R0/0: pvp: Empty executable used for process bt_logger
*Sep 1 15:10:29.738: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 1 15:10:31.744: %STACKMGR-6-CHASSIS_ADDED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been added to the stack.
*Sep 1 15:10:31.744: %STACKMGR-6-ACTIVE_ELECTED: Chassis 1 R0/0: stack_mgr: Chassis 1 has been elected ACTIVE.

Chassis 1 booted up first and went into active recovery mode - it is set to priority 2

*Sep 1 15:11:48.554: %RIF_MGR_FSM-6-RMI_LINK_UP: Chassis 1 R0/0: rif_mgr: The RMlink is UP.
*Sep 1 15:11:48.554: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Chassis 1 R0/0: stack_mgr:ual Active Detection link is available now
*Sep 1 15:11:48.555: %RIF_MGR_FSM-6-RMI_ACTIVE_RECOVERY_MODE: Chassis 1 R0/0: rimgr: Going to Active(Recovery) from Active state

[Emmanuel-Sec]

Hi Scot,

How did you end up getting this to work? I am facing the same issue on 2 C9800-CL vWLCs. It's unclear what is wrong but my chassis #2 keeps going into Active Recovery.

[balaji.bandi]

provide more logs to understand also give more information about environment.

[Emmanuel-Sec]

Hi Balaji,

Forgot to post an update here but I got it resolved, it was a problem with one of the interconnecting trunks in esxi.