Random APs won't join 9800-40 running 17.9.5

[Xibachao1]

We had upgraded 9800-40 to 17.9.5 for a year, But recently some 9115AX-I have suddenly left WLC and cannot rejoin. Produce the following errors:

[*03/13/2025 01:20:01.9000] CAPWAP State: Discovery
[*03/13/2025 01:20:01.9010] Got WLC address xx.xx.xx.xx from DHCP.
[*03/13/2025 01:20:01.9010] Got WLC address xx.xx.xx.xx from DHCP.
[*03/13/2025 01:20:01.9030] Discovery Request sent to xx.xx.xx.xx, discovery type DHCP(2)
[*03/13/2025 01:20:01.9040] Discovery Request sent to xx.xx.xx.xx, discovery type DHCP(2)
[*03/13/2025 01:20:01.9060] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*03/13/2025 01:20:01.9070] Discovery Response from xx.xx.xx.xx
[*03/13/2025 01:20:01.9130]
[*03/13/2025 01:20:01.9130] CAPWAP State: Discovery
[*03/13/2025 01:21:59.0000] Started wait dtls timer (60 sec)
[*03/13/2025 01:21:59.0070]
[*03/13/2025 01:21:59.0070] CAPWAP State: DTLS Setup
[*03/13/2025 01:21:59.0280] dtls_verify_server_cert: Controller certificate verification successful
[*03/13/2025 01:21:59.0290] 548045546368:error:14102438:lib(20):func(258):reason(1080):NA:0:SSL alert number 80
[*03/13/2025 01:21:59.0290] dtls_process_packet: Error connecting TLS context ERR: 6
[*03/13/2025 01:21:59.0290] DTLS: Error while processing DTLS packet 0x559d5ad000.
[*03/13/2025 01:22:12.8120] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*03/13/2025 01:22:56.0270] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 01:22:56.0270] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:22:56.0510]
[*03/13/2025 01:22:56.0510] CAPWAP State: DTLS Teardown
[*03/13/2025 01:22:56.1120] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:22:56.1930] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2025 01:22:56.2400] do CANCEL, part1 is active part
[*03/13/2025 01:22:56.2630] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2025 01:22:56.2890] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 01:22:56.2890] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 01:23:00.7780] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 01:23:00.7780] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:23:00.7960] No more AP manager addresses remain..
[*03/13/2025 01:23:00.7960] No valid AP manager found for controller 'WLC_DC1_01' (ip: xx.xx.xx.xx)
[*03/13/2025 01:23:00.7960] Failed to join controller WLC_DC1_01.
[*03/13/2025 01:23:00.7960] Failed to join controller.

But it can join WLC-02. Can some body help me about it?

[Xibachao1]

I tried to reset AP by hold reset button in 20s and still can not rejoin WLC01: 

[*03/13/2025 02:00:37.6920] CAPWAP State: Discovery
[*03/13/2025 02:00:37.6940] Discovery Response from WLC01
[*03/13/2025 02:00:37.6940] Found Configured MWAR 'WLC_DC1_01' (respIdx 0).
[*03/13/2025 02:02:24.0000] Started wait dtls timer (60 sec)
[*03/13/2025 02:02:24.0070]
[*03/13/2025 02:02:24.0070] CAPWAP State: DTLS Setup
[*03/13/2025 02:02:24.0110] Invalid event 2 & state 3 combination.
[*03/13/2025 02:02:24.0110] CAPWAP SM handler: Failed to process message type 2 state 3.
[*03/13/2025 02:02:24.0110] Failed to handle capwap control message from controller - status 1
[*03/13/2025 02:02:24.0110] Failed to process unencrypted capwap packet 0x55c007a000 from WLC02
[*03/13/2025 02:02:24.0110] Failed to send message to CAPWAP state machine, msgId 0
[*03/13/2025 02:02:24.0110] Failed to send capwap message 0 to the state machine. Packet already freed.
[*03/13/2025 02:02:24.0110] IPv4 wtpProcessPacketFromSocket returned 1
[*03/13/2025 02:02:24.0110] Invalid event 2 & state 3 combination.
[*03/13/2025 02:02:24.0110] CAPWAP SM handler: Failed to process message type 2 state 3.
[*03/13/2025 02:02:24.0110] Failed to handle capwap control message from controller - status 1
[*03/13/2025 02:02:24.0110] Failed to process unencrypted capwap packet 0x55c007c000 from WLC02
[*03/13/2025 02:02:24.0110] Failed to send message to CAPWAP state machine, msgId 0
[*03/13/2025 02:02:24.0110] Failed to send capwap message 0 to the state machine. Packet already freed.
[*03/13/2025 02:02:24.0110] IPv4 wtpProcessPacketFromSocket returned 1
[*03/13/2025 02:02:24.0110] Invalid event 2 & state 3 combination.
[*03/13/2025 02:02:24.0110] CAPWAP SM handler: Failed to process message type 2 state 3.
[*03/13/2025 02:02:24.0110] Failed to handle capwap control message from controller - status 1
[*03/13/2025 02:02:24.0110] Failed to process unencrypted capwap packet 0x55c007e000 from WLC01
[*03/13/2025 02:02:24.0110] Failed to send message to CAPWAP state machine, msgId 0
[*03/13/2025 02:02:24.0110] Failed to send capwap message 0 to the state machine. Packet already freed.
[*03/13/2025 02:02:24.0110] IPv4 wtpProcessPacketFromSocket returned 1
[*03/13/2025 02:02:24.0440] dtls_verify_server_cert: Controller certificate verification successful
[*03/13/2025 02:02:24.0450] 547832718208:error:14102438:lib(20):func(258):reason(1080):NA:0:SSL alert number 80
[*03/13/2025 02:02:24.0450] dtls_process_packet: Error connecting TLS context ERR: 6
[*03/13/2025 02:02:24.0460] DTLS: Error while processing DTLS packet 0x55c0080000.
[*03/13/2025 02:02:26.5110] Start: RPC thread 2346705792 created.
[03/13/2025 02:02:45.0850] audit_log_lost: 18 callbacks suppressed
[03/13/2025 02:02:45.1490] audit: audit_lost=41 audit_rate_limit=100 audit_backlog_limit=8192
[03/13/2025 02:02:45.2480] audit: rate limit exceeded
[*03/13/2025 02:02:45.9710] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*03/13/2025 02:03:20.8750] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 02:03:20.8750] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 02:03:20.8990]
[*03/13/2025 02:03:20.8990] CAPWAP State: DTLS Teardown
[*03/13/2025 02:03:20.9660] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 02:03:21.0440] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2025 02:03:21.0870] do CANCEL, part1 is active part
[*03/13/2025 02:03:21.1100] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2025 02:03:21.1350] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 02:03:21.1360] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 02:03:25.6250] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 02:03:25.6250] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 02:03:25.6430] No more AP manager addresses remain..
[*03/13/2025 02:03:25.6430] No valid AP manager found for controller 'WLC_DC1_01' (ip: WLC01)
[*03/13/2025 02:03:25.6430] Failed to join controller WLC_DC1_01.
[*03/13/2025 02:03:25.6430] Failed to join controller.

[Leo Laohoo]

---

@Xibachao1 wrote:

No more AP manager addresses remain..

---

If the AP can join WLC 2 but not WLC 1, how many APs can WLC 1 support and how many are currently joined?

[Xibachao1]

actually it joined WLC-01 before. But it got out of WLC by itself and cannot rejoin. Totally 400 APs. It's not happen with a new AP. 

[Scott Fella]

Try to connect one of the affected ap's on the same vlan as the controller management and see if it joins.  It's best if you can connect it to the same switch or a switch a hop away and see.

[Xibachao1]

It not happen with a new AP. Only with AP have that issue.

[Scott Fella]

So new ap's are able to join but not any of the 9115's? What are the new ap's that can join and how many 9115's are still joined?

[Xibachao1]

All the AP is 9115. Now i have 389 joined on WLC-01 . 0 on WLC-02

[Leo Laohoo]

Please apply APSP10.

[Xibachao1]

Thanks for your support.

I have checked our system missing this one. Could u explain to me why need to apply it, how it works before that because we still have around 400 AP joined.

[Xibachao1]

Provide more information. APSP10 seem relate to Cisco IOS XE Amsterdam 17.9.5.

WLC_DC1_01#show version | include 9800|IOSXE
Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)

WLC_DC1_01#
WLC_DC1_01#
WLC_DC1_01#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 16 weeks, 5 days, 12 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Simplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = Non-redundant
Maintenance Mode = Disabled
Communications = Down Reason: Failure

Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 16 weeks, 5 days, 12 minutes
Image Version = Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2024 by Cisco Systems, Inc.
Compiled Tue 30-Jan-24 15:36 by mcpre
BOOT = bootflash:packages.conf,1;
CONFIG_FILE =
Configuration register = 0x102
Recovery mode = Not Applicable
Fast Switchover = Enabled
Initial Garp = Enabled

Peer (slot: 0) information is not available because it is in 'DISABLED' state

[Leo Laohoo]

Is this happening to many APs or just one AP?

If this is happening to just one AP, please share the serial number of the AP.

[Xibachao1]

Now i have 2 APs automatically happen this issue. 

And 2 APs occurs when i changed the Site tag. After i changed the Site Tag it can not rejoin WLC-01 but can join WLC-02 ( WLC-01 and WLC-02 are the same config) . I tried reset many times but can not fix

[marce1000]

            - FYI : https://bst.cisco.com/bugsearch/bug/CSCvz11337?rfs=qvred

  M.

[Xibachao1]

Thank you @marce1000 . So i have two options: Upgrade APSP10 or upgrade to 17.9.6 Cupertino.

May i am right?