topic Re: Random APs won't join 9800-40 running 17.9.5 in Wireless

Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 01:29:05 GMT

We had upgraded 9800-40 to 17.9.5 for a year, But recently some 9115AX-I have suddenly left WLC and cannot rejoin. Produce the following errors:

[*03/13/2025 01:20:01.9000] CAPWAP State: Discovery
[*03/13/2025 01:20:01.9010] Got WLC address xx.xx.xx.xx from DHCP.
[*03/13/2025 01:20:01.9010] Got WLC address xx.xx.xx.xx from DHCP.
[*03/13/2025 01:20:01.9030] Discovery Request sent to xx.xx.xx.xx, discovery type DHCP(2)
[*03/13/2025 01:20:01.9040] Discovery Request sent to xx.xx.xx.xx, discovery type DHCP(2)
[*03/13/2025 01:20:01.9060] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*03/13/2025 01:20:01.9070] Discovery Response from xx.xx.xx.xx
[*03/13/2025 01:20:01.9130]
[*03/13/2025 01:20:01.9130] CAPWAP State: Discovery
[*03/13/2025 01:21:59.0000] Started wait dtls timer (60 sec)
[*03/13/2025 01:21:59.0070]
[*03/13/2025 01:21:59.0070] CAPWAP State: DTLS Setup
[*03/13/2025 01:21:59.0280] dtls_verify_server_cert: Controller certificate verification successful
[*03/13/2025 01:21:59.0290] 548045546368:error:14102438:lib(20):func(258):reason(1080):NA:0:SSL alert number 80
[*03/13/2025 01:21:59.0290] dtls_process_packet: Error connecting TLS context ERR: 6
[*03/13/2025 01:21:59.0290] DTLS: Error while processing DTLS packet 0x559d5ad000.
[*03/13/2025 01:22:12.8120] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*03/13/2025 01:22:56.0270] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 01:22:56.0270] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:22:56.0510]
[*03/13/2025 01:22:56.0510] CAPWAP State: DTLS Teardown
[*03/13/2025 01:22:56.1120] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:22:56.1930] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2025 01:22:56.2400] do CANCEL, part1 is active part
[*03/13/2025 01:22:56.2630] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2025 01:22:56.2890] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 01:22:56.2890] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 01:23:00.7780] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 01:23:00.7780] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:23:00.7960] No more AP manager addresses remain..
[*03/13/2025 01:23:00.7960] No valid AP manager found for controller 'WLC_DC1_01' (ip: xx.xx.xx.xx)
[*03/13/2025 01:23:00.7960] Failed to join controller WLC_DC1_01.
[*03/13/2025 01:23:00.7960] Failed to join controller.

But it can join WLC-02. Can some body help me about it?

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 02:05:57 GMT

I tried to reset AP by hold reset button in 20s and still can not rejoin WLC01:

[*03/13/2025 02:00:37.6920] CAPWAP State: Discovery
[*03/13/2025 02:00:37.6940] Discovery Response from WLC01
[*03/13/2025 02:00:37.6940] Found Configured MWAR 'WLC_DC1_01' (respIdx 0).
[*03/13/2025 02:02:24.0000] Started wait dtls timer (60 sec)
[*03/13/2025 02:02:24.0070]
[*03/13/2025 02:02:24.0070] CAPWAP State: DTLS Setup
[*03/13/2025 02:02:24.0110] Invalid event 2 & state 3 combination.
[*03/13/2025 02:02:24.0110] CAPWAP SM handler: Failed to process message type 2 state 3.
[*03/13/2025 02:02:24.0110] Failed to handle capwap control message from controller - status 1
[*03/13/2025 02:02:24.0110] Failed to process unencrypted capwap packet 0x55c007a000 from WLC02
[*03/13/2025 02:02:24.0110] Failed to send message to CAPWAP state machine, msgId 0
[*03/13/2025 02:02:24.0110] Failed to send capwap message 0 to the state machine. Packet already freed.
[*03/13/2025 02:02:24.0110] IPv4 wtpProcessPacketFromSocket returned 1
[*03/13/2025 02:02:24.0110] Invalid event 2 & state 3 combination.
[*03/13/2025 02:02:24.0110] CAPWAP SM handler: Failed to process message type 2 state 3.
[*03/13/2025 02:02:24.0110] Failed to handle capwap control message from controller - status 1
[*03/13/2025 02:02:24.0110] Failed to process unencrypted capwap packet 0x55c007c000 from WLC02
[*03/13/2025 02:02:24.0110] Failed to send message to CAPWAP state machine, msgId 0
[*03/13/2025 02:02:24.0110] Failed to send capwap message 0 to the state machine. Packet already freed.
[*03/13/2025 02:02:24.0110] IPv4 wtpProcessPacketFromSocket returned 1
[*03/13/2025 02:02:24.0110] Invalid event 2 & state 3 combination.
[*03/13/2025 02:02:24.0110] CAPWAP SM handler: Failed to process message type 2 state 3.
[*03/13/2025 02:02:24.0110] Failed to handle capwap control message from controller - status 1
[*03/13/2025 02:02:24.0110] Failed to process unencrypted capwap packet 0x55c007e000 from WLC01
[*03/13/2025 02:02:24.0110] Failed to send message to CAPWAP state machine, msgId 0
[*03/13/2025 02:02:24.0110] Failed to send capwap message 0 to the state machine. Packet already freed.
[*03/13/2025 02:02:24.0110] IPv4 wtpProcessPacketFromSocket returned 1
[*03/13/2025 02:02:24.0440] dtls_verify_server_cert: Controller certificate verification successful
[*03/13/2025 02:02:24.0450] 547832718208:error:14102438:lib(20):func(258):reason(1080):NA:0:SSL alert number 80
[*03/13/2025 02:02:24.0450] dtls_process_packet: Error connecting TLS context ERR: 6
[*03/13/2025 02:02:24.0460] DTLS: Error while processing DTLS packet 0x55c0080000.
[*03/13/2025 02:02:26.5110] Start: RPC thread 2346705792 created.
[03/13/2025 02:02:45.0850] audit_log_lost: 18 callbacks suppressed
[03/13/2025 02:02:45.1490] audit: audit_lost=41 audit_rate_limit=100 audit_backlog_limit=8192
[03/13/2025 02:02:45.2480] audit: rate limit exceeded
[*03/13/2025 02:02:45.9710] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*03/13/2025 02:03:20.8750] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 02:03:20.8750] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 02:03:20.8990]
[*03/13/2025 02:03:20.8990] CAPWAP State: DTLS Teardown
[*03/13/2025 02:03:20.9660] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 02:03:21.0440] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2025 02:03:21.0870] do CANCEL, part1 is active part
[*03/13/2025 02:03:21.1100] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2025 02:03:21.1350] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 02:03:21.1360] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 02:03:25.6250] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 02:03:25.6250] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 02:03:25.6430] No more AP manager addresses remain..
[*03/13/2025 02:03:25.6430] No valid AP manager found for controller 'WLC_DC1_01' (ip: WLC01)
[*03/13/2025 02:03:25.6430] Failed to join controller WLC_DC1_01.
[*03/13/2025 02:03:25.6430] Failed to join controller.

Re: Random APs won't join 9800-40 running 17.9.5

Scott Fella — Thu, 13 Mar 2025 02:31:18 GMT

Try to connect one of the affected ap's on the same vlan as the controller management and see if it joins. It's best if you can connect it to the same switch or a switch a hop away and see.

Re: Random APs won't join 9800-40 running 17.9.5

Leo Laohoo — Thu, 13 Mar 2025 02:34:10 GMT

@Xibachao1 wrote:

No more AP manager addresses remain..

If the AP can join WLC 2 but not WLC 1, how many APs can WLC 1 support and how many are currently joined?

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 02:52:04 GMT

actually it joined WLC-01 before. But it got out of WLC by itself and cannot rejoin. Totally 400 APs. It's not happen with a new AP.

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 02:53:03 GMT

It not happen with a new AP. Only with AP have that issue.

Re: Random APs won't join 9800-40 running 17.9.5

Scott Fella — Thu, 13 Mar 2025 02:59:30 GMT

So new ap's are able to join but not any of the 9115's? What are the new ap's that can join and how many 9115's are still joined?

Re: Random APs won't join 9800-40 running 17.9.5

Leo Laohoo — Thu, 13 Mar 2025 03:03:13 GMT

Is this happening to many APs or just one AP?

If this is happening to just one AP, please share the serial number of the AP.

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 03:10:33 GMT

Now i have 2 APs automatically happen this issue.

And 2 APs occurs when i changed the Site tag. After i changed the Site Tag it can not rejoin WLC-01 but can join WLC-02 ( WLC-01 and WLC-02 are the same config) . I tried reset many times but can not fix

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 03:11:43 GMT

All the AP is 9115. Now i have 389 joined on WLC-01 . 0 on WLC-02

Re: Random APs won't join 9800-40 running 17.9.5

Leo Laohoo — Thu, 13 Mar 2025 03:51:47 GMT

Please apply APSP10.

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 06:07:16 GMT

Thanks for your support.

I have checked our system missing this one. Could u explain to me why need to apply it, how it works before that because we still have around 400 AP joined.

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 07:37:20 GMT

Provide more information. APSP10 seem relate to Cisco IOS XE Amsterdam 17.9.5.

WLC_DC1_01#show version | include 9800|IOSXE
Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)

WLC_DC1_01#
WLC_DC1_01#
WLC_DC1_01#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 16 weeks, 5 days, 12 minutes
Switchovers system experienced = 0
Standby failures = 0
Last switchover reason = none

Hardware Mode = Simplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = Non-redundant
Maintenance Mode = Disabled
Communications = Down Reason: Failure

Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 16 weeks, 5 days, 12 minutes
Image Version = Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2024 by Cisco Systems, Inc.
Compiled Tue 30-Jan-24 15:36 by mcpre
BOOT = bootflash:packages.conf,1;
CONFIG_FILE =
Configuration register = 0x102
Recovery mode = Not Applicable
Fast Switchover = Enabled
Initial Garp = Enabled

Peer (slot: 0) information is not available because it is in 'DISABLED' state

Re: Random APs won't join 9800-40 running 17.9.5

Mark Elsen — Thu, 13 Mar 2025 07:52:15 GMT

- FYI : https://bst.cisco.com/bugsearch/bug/CSCvz11337?rfs=qvred

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Thu, 13 Mar 2025 08:25:41 GMT

Thank you @Mark Elsen . So i have two options: Upgrade APSP10 or upgrade to 17.9.6 Cupertino.

May i am right?

Re: Random APs won't join 9800-40 running 17.9.5

Mark Elsen — Thu, 13 Mar 2025 10:42:59 GMT

@Xibachao1 - Go directly for 17.12.4 , yet when visiting the download page for thar version choose the one behind the 'hidden URL' to have the latest bugfixes! ===> at this hidden URL,

Re: Random APs won't join 9800-40 running 17.9.5

Xibachao1 — Mon, 17 Mar 2025 04:55:01 GMT

Hi @Mark Elsen

I don't want to upgrade yet, I just want to test first because it only happened to a few APs. What will happen if I upgrade APSP for AP by USB and during the process of joining WLC, will the image be overwritten? (overwritten with image on WLC)

What will happen if I use this function on WLC?

Re: Random APs won't join 9800-40 running 17.9.5

Mark Elsen — Mon, 17 Mar 2025 07:54:12 GMT

- Not sure what will happen, I advise to go for the full upgrade as I explained ,

Re: Random APs won't join 9800-40 running 17.9.5

Rich R — Wed, 19 Mar 2025 13:51:28 GMT

That will upgrade 25% of APs at a time.

This problem can also be caused by stale CAPWAP control connections on the WLC:
Run radioactive trace for the AP MAC address on the WLC while the AP is trying to join.
Check the radioactive trace for reference to "stale session in connected state" which will mention the MAC address of a different AP. Reload the other AP which that MAC address refers to then reload the original problem AP which should now be able to join (if it is the stale session problem). You can then also see the same problem affect the other APs after you reload them so you might need to repeat the process a few times for each affected AP. This is one of those problems which gets progressively worse the longer the uptime of the WLC, and made worse when APs drop and re-join for any reason.
There are a number of bugs for this issue but the "fixes" have not fixed the issue - it remains...
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwk44459
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwi16509
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwn76129
It might finally get fixed properly when CSCwn76129 is fixed (we can only hope - maybe 3rd time lucky ...)

Re: Random APs won't join 9800-40 running 17.9.5

Scott Fella — Wed, 19 Mar 2025 14:29:39 GMT

@Xibachao1 if you are worried that a change might break your production, it's best to have TAC look at it. If you don't have a test environment it's hard to say what risk you will face. You can patch or upgrade, but everyone has experienced at one time a failure. Look at your possible risk and make sure you call that out so that others are aware. It's easy for us to tell you what to do, but majority of use know how to remediate right away or even push ap's to a different controller while working on the controller that are the issue. It's better to be safe than sorry.