9800 Series WLC - Cert validation status:pki_ssl_status@pk

[FedericoAlicante]

I am having big issues with a migration in which I moved some 3802 APs from a 5500 series WLC to a 9800 series. They were happy enough to join, but now I am plagued by them dropping and rejoining over and over. Here is a snippet of examples of this:

001001: Sep 2 2022 22:17:48.950 UTC: %PKI-3-CERTIFICATE_INVALID: Certificate chain validation has failed.
001002: Sep 2 2022 22:17:48.952 UTC: %CERT_MGR_ERRMSG-3-CERT_VALIDATION_ERR: Chassis 1 R0/0: wncd: Certificate Validation Error, Cert validation status:pki_ssl_status@pki_ssl_status:PKI_SSL_VERIFY_FAILURE
001003: Sep 2 2022 22:17:48.952 UTC: %DTLS_TRACE_MSG-3-WLC_DTLS_ERR: Chassis 1 R0/0: wncd: DTLS Error, session:x.x.x.x[5264] MAC: x.x.x.x, Certificate validation failed

000621: Sep 2 2022 21:53:34.350 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 10.96.40.43[5256] Mac: x.x.x.x CAPWAP DTLS session closed for AP, cause: DTLS server session shutdown
000622: Sep 2 2022 21:53:34.357 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 2 R0/0: wncd: AP Event: AP Name: xxxxxxxxxxx, MAC: x.x.x.x Disjoined
000623: Sep 2 2022 21:53:54.101 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:1x.x.x.x[5256] CAPWAP DTLS session closed for AP, cause: DTLS handshake error
000624: Sep 2 2022 21:53:59.816 UTC: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP:x.x.x.x[5256] CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Anyone run into this before? I had a look before posting, but the errors were always specific to certificates expiring which isn't the case her.

Many thanks

[marce1000]

 -  Review the 9800   configuration with the CLI command : show  tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.               Checkout all advisories!

 M.

[FedericoAlicante]

Thanks for the reply. I did that, but I didn't turn anything up other than I am seeing what I am seeing in the logs.

[marce1000]

 - If you currently have an IRCM setup between the 5508 and the 9800 then check and execute : https://community.cisco.com/t5/wireless/inter-release-controller-mobility-ircm-with-5508-fail-control/m-p/4274202#M225325

 M.

[FedericoAlicante]

TAC found that the issue was two-fold. DTLS had expired, but I implemented the work-around yesterday and yet those errors persisted which pointed me in the wrong direction. We were able to grab logs from the APs and could see the Re-Tx Count was going up to the maximum (5) where it would then go back to Discovery. This suggested that it was a connectivity issue. I looked at the port-channel on the switch (we are running in Local mode) and saw that it was flapping. I shut down the flapping port and everything has been stable since.