Random APs won't join 9800-40 running 17.9.5

[Xibachao1]

We had upgraded 9800-40 to 17.9.5 for a year, But recently some 9115AX-I have suddenly left WLC and cannot rejoin. Produce the following errors:

[*03/13/2025 01:20:01.9000] CAPWAP State: Discovery
[*03/13/2025 01:20:01.9010] Got WLC address xx.xx.xx.xx from DHCP.
[*03/13/2025 01:20:01.9010] Got WLC address xx.xx.xx.xx from DHCP.
[*03/13/2025 01:20:01.9030] Discovery Request sent to xx.xx.xx.xx, discovery type DHCP(2)
[*03/13/2025 01:20:01.9040] Discovery Request sent to xx.xx.xx.xx, discovery type DHCP(2)
[*03/13/2025 01:20:01.9060] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*03/13/2025 01:20:01.9070] Discovery Response from xx.xx.xx.xx
[*03/13/2025 01:20:01.9130]
[*03/13/2025 01:20:01.9130] CAPWAP State: Discovery
[*03/13/2025 01:21:59.0000] Started wait dtls timer (60 sec)
[*03/13/2025 01:21:59.0070]
[*03/13/2025 01:21:59.0070] CAPWAP State: DTLS Setup
[*03/13/2025 01:21:59.0280] dtls_verify_server_cert: Controller certificate verification successful
[*03/13/2025 01:21:59.0290] 548045546368:error:14102438:lib(20):func(258):reason(1080):NA:0:SSL alert number 80
[*03/13/2025 01:21:59.0290] dtls_process_packet: Error connecting TLS context ERR: 6
[*03/13/2025 01:21:59.0290] DTLS: Error while processing DTLS packet 0x559d5ad000.
[*03/13/2025 01:22:12.8120] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*03/13/2025 01:22:56.0270] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 01:22:56.0270] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:22:56.0510]
[*03/13/2025 01:22:56.0510] CAPWAP State: DTLS Teardown
[*03/13/2025 01:22:56.1120] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:22:56.1930] status 'upgrade.sh: Script called with args:[CANCEL]'
[*03/13/2025 01:22:56.2400] do CANCEL, part1 is active part
[*03/13/2025 01:22:56.2630] status 'upgrade.sh: Cleanup tmp files ...'
[*03/13/2025 01:22:56.2890] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 01:22:56.2890] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
[*03/13/2025 01:23:00.7780] OOBImageDnld: OOBImageDownloadTimer expired for image download..
[*03/13/2025 01:23:00.7780] OOBImageDnld: Do common error handler for OOB image download..
[*03/13/2025 01:23:00.7960] No more AP manager addresses remain..
[*03/13/2025 01:23:00.7960] No valid AP manager found for controller 'WLC_DC1_01' (ip: xx.xx.xx.xx)
[*03/13/2025 01:23:00.7960] Failed to join controller WLC_DC1_01.
[*03/13/2025 01:23:00.7960] Failed to join controller.

But it can join WLC-02. Can some body help me about it?

[marce1000]

 @Xibachao1 - Go directly for 17.12.4 , yet when visiting the download page for thar version choose the one behind the 'hidden URL' to have the latest bugfixes!  ===> at this hidden URL,

 M.

[Xibachao1]

Hi @marce1000 

I don't want to upgrade yet, I just want to test first because it only happened to a few APs. What will happen if I upgrade APSP for AP by USB and during the process of joining WLC, will the image be overwritten? (overwritten with image on WLC)

What will happen if I use this function on WLC?

[marce1000]

  - Not sure what will happen, I advise to go for the full upgrade as I explained ,

 M.

[Rich R]

That will upgrade 25% of APs at a time.

This problem can also be caused by stale CAPWAP control connections on the WLC:
Run radioactive trace for the AP MAC address on the WLC while the AP is trying to join.
Check the radioactive trace for reference to "stale session in connected state" which will mention the MAC address of a different AP.  Reload the other AP which that MAC address refers to then reload the original problem AP which should now be able to join (if it is the stale session problem).  You can then also see the same problem affect the other APs after you reload them so you might need to repeat the process a few times for each affected AP.  This is one of those problems which gets progressively worse the longer the uptime of the WLC, and made worse when APs drop and re-join for any reason.
There are a number of bugs for this issue but the "fixes" have not fixed the issue - it remains...
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwk44459
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwi16509
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwn76129
It might finally get fixed properly when CSCwn76129 is fixed (we can only hope - maybe 3rd time lucky ...)

[Scott Fella]

@Xibachao1 if you are worried that a change might break your production, it's best to have TAC look at it.  If you don't have a test environment it's hard to say what risk you will face. You can patch or upgrade, but everyone has experienced at one time a failure.  Look at your possible risk and make sure you call that out so that others are aware.  It's easy for us to tell you what to do, but majority of use know how to remediate right away or even push ap's to a different controller while working on the controller that are the issue. It's better to be safe than sorry. 

[Xibachao1 · Accepted solution]

Reboot WLC resolved the problem. We will consider upgrade to new firmware later.

Thank all.

[Scott Fella]

That is one item that many folks will try.  It doesn't hurt as long as you can move ap's or have redundancy.