Random APs won't join 9800-80 running 17.9.5

[netdipace]

After upgrading 9800-80 to 17.9.5, random 9130s won't join and produce the following errors:

Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0113] CAPWAP State: DTLS Setup
Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0592] dtls_verify_server_cert: Controller certificate verification successful
Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0595] 548269699984:error:14102438:lib(20):func(258):reason(1080):NA:0:SSL alert number 80
Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0595] dtls_process_packet: Error connecting TLS context ERR: 6
Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0601] DTLS: Error while processing DTLS packet 0x558b41a000.
Jun 12 01:55:03 kernel: [*06/12/2024 01:55:03.1044] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
Jun 12 01:55:03 kernel: [*06/12/2024 01:55:03.6086] systemd[1]: Starting Lighttpd Watcher...
Jun 12 01:55:03 kernel: [*06/12/2024 01:55:03.6417] systemd[1]: Started Lighttpd Watcher.
Jun 12 01:55:32 kernel: [*06/12/2024 01:55:32.9114] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
Jun 12 01:55:34 kernel: [*06/12/2024 01:55:34.5536] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.0190] OOBImageDnld: OOBImageDownloadTimer expired for image download..
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.0190] OOBImageDnld: Do common error handler for OOB image download..
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.0524]
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.0524] CAPWAP State: DTLS Teardown
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.0833] OOBImageDnld: Do common error handler for OOB image download..
Jun 12 01:55:57 upgrade: Script called with args:[CANCEL]
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.1817] status 'upgrade.sh: Script called with args:[CANCEL]'
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.2161] do CANCEL, part2 is active part
Jun 12 01:55:57 upgrade: Cleanup tmp files ...
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.2395] status 'upgrade.sh: Cleanup tmp files ...'
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.2738] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
Jun 12 01:55:57 kernel: [*06/12/2024 01:55:57.2739] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Teardown(4).
Jun 12 01:56:01 kernel: [*06/12/2024 01:56:01.7706] OOBImageDnld: OOBImageDownloadTimer expired for image download..
Jun 12 01:56:01 kernel: [*06/12/2024 01:56:01.7706] OOBImageDnld: Do common error handler for OOB image download..
Jun 12 01:56:01 kernel: [*06/12/2024 01:56:01.7944] No more AP manager addresses remain..
Jun 12 01:56:01 kernel: [*06/12/2024 01:56:01.7944] No valid AP manager found for controller 'xxxxx' (ip: x.x.x.x)
Jun 12 01:56:01 kernel: [*06/12/2024 01:56:01.7944] Failed to join controller xxxxx.
Jun 12 01:56:01 kernel: [*06/12/2024 01:56:01.7944] Failed to join controller.

[balaji.bandi]

Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0595] dtls_process_packet: Error connecting TLS context ERR: 6
Jun 12 01:55:00 kernel: [*06/12/2024 01:55:00.0601] DTLS: Error while processing DTLS packet 0x558b41a000.

where is your AP in the same environment ? suggest to reset one of the AP to factory and test, if this was working before upgrade ?

No valid AP manager found for controller 'xxxxx' (ip: x.x.x.x)

Is the AP behind NAT ?  how many AP you have ? (all same model having same issue ?)

Can you reset one of the AP to factory and test it ?

[netdipace]

We have 3000 APs that we're trying to get on the same code. We've had a TAC case open, but not getting any traction. Initially thought it had to do with the "wireless manager trustpoint" change in 17.9.5.  The APs were working fine before on 17.9.3.  We were able to get 761 APs joined to the WLC running 17.9.5, but in the same maintenance window 22 others woudn't join. They are all the same model, and some even on the same switch: (i.e. 1 AP on switch joins, 1 on same switch doesn't). No NAT in place. 

[balaji.bandi]

then i would work with TAC as P1 since upgrade took defective of setup, until there is any caveats or bugs mentioned in the document.

check release notes :

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/release-notes/rn-17-9-9800.html

If you have configured CISCO_IDEVID_SUDI trustpoint in your configuration, you will need to replace it with CISCO_IDEVID_CMCA3_SUDI to avoid client connection and AP join issues. The reason for this change being the CISCO_IDEVID_SUDI changed from SW-SUDI certificate in previous releases to HW-SUDI certificate. The processing of HW-SUDI certificate is much slower than the SW-SUDI. Here, CISCO_IDEVID_CMCA3_SUDI is the new SW-SUDI certificate.

[netdipace]

We had: Trustpoint Name : CISCO_IDEVID_CMCA3_SUDI

TAC had us change to : Trustpoint Name : CISCO_IDEVID_CMCA2_SUDI

But same issue

[balaji.bandi]

i would strongly work with TAC (they are SME to suggest what to be done next to avoid that issue).

[Leo Laohoo]

Console into one of the affected APs and post the complete output to the command "sh version".

[marce1000]

  - Have a checkup of the 9800-80 controller's configuration with the CLI command show tech wireless
    and feed the output from that into Wireless Config Analyzer

 M.

[Rich R]

Did you try a factory default reset of any of the affected APs @netdipace ?