Solved: Re: AP's wont connect to 5508 WLC after update to 8.3.143 - PKI initialization error

PJR_CDF · ‎08-17-2018

Topology

2 x WLC 5508 with 13 x 2602 APs and 4 x 2802 APs
No WLC SSO HA in place - just primary and secondary controllers specified in the AP's config

I have previously posted about this issue in this thread

https://community.cisco.com/t5/wireless-and-mobility/2602-ap-s-wont-connect-to-5508wlc-after-update-to-8-3-143-pki/m-p/3681836#M102238

Previously I had been upgrading our secondary controller only to 8.3.143.0. Last night I upgraded our primary WLC to 8.3.143.0 and all the AP's upgraded and connected without issue. This is the exact step that failed when I upgraded our secondary as per the previous post.

When I shut down our primary WLC to allow the (now upgraded) APs to failover to our secondary, the APs were still unable to negotiate a DTLS connection to the secondary controller.

The issue is now no longer an AP/WLC software incompatibility. It appears the issue is specific to our secondary WLC and only, when it runs 8.3.143.0 (as all APs are able to connect to it when it runs 8.0.152.0).

We have also now been able to rule out issues relating to the age of the 2602 APs, as since the update to 8.3.143.0 was successful on our primary WLC, we introduced 4 new box fresh 2802 APs which connected immediately without issue. When failed over to our secondary WLC, these exhibited the same issue as the 2602s (see below logs from secondary WLC with IPs obfuscated)

*spamApTask4: Aug 16 17:47:20.847: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask5: Aug 16 17:47:20.349: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 b0:fa:eb:56:5d:78: Failed to create DTLS connection for AP xx.xx.0.22 (26072).
*spamApTask5: Aug 16 17:47:20.348: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask4: Aug 16 17:47:20.005: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 00:06:f6:ec:e3:85: Failed to create DTLS connection for AP xx.xx.0.21 (52793).
*spamApTask4: Aug 16 17:47:20.004: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask6: Aug 16 17:47:19.992: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 00:06:f6:ec:d7:9f: Failed to create DTLS connection for AP xx.xx.0.23 (52602).
*spamApTask6: Aug 16 17:47:19.992: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask7: Aug 16 17:47:19.720: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:f9: Failed to create DTLS connection for AP xx.xx.0.20 (13505).
*spamApTask7: Aug 16 17:47:19.720: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask5: Aug 16 17:47:19.372: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 b0:fa:eb:56:5c:b8: Failed to create DTLS connection for AP xx.xx.0.19 (26061).
*spamApTask5: Aug 16 17:47:19.372: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask3: Aug 16 17:47:18.999: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:fb: Failed to create DTLS connection for AP xx.xx.0.17 (13504).
*spamApTask3: Aug 16 17:47:18.999: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask2: Aug 16 17:47:18.858: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 00:06:f6:a5:51:dc: Failed to create DTLS connection for AP xx.xx.7.49 (13657).
*spamApTask2: Aug 16 17:47:18.857: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask5: Aug 16 17:47:16.349: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 b0:fa:eb:56:5d:78: Failed to create DTLS connection for AP xx.xx.0.22 (26072).
*spamApTask5: Aug 16 17:47:16.348: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask6: Aug 16 17:47:15.992: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 00:06:f6:ec:d7:9f: Failed to create DTLS connection for AP xx.xx.0.23 (52602).
*spamApTask6: Aug 16 17:47:15.991: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask7: Aug 16 17:47:15.720: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 4c:4e:35:03:4b:f9: Failed to create DTLS connection for AP xx.xx.0.20 (13505).
*spamApTask7: Aug 16 17:47:15.720: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed
*spamApTask3: Aug 16 17:47:14.748: %CAPWAP-3-DTLS_DB_ERR: capwap_ac_sm.c:9507 b0:fa:eb:56:5d:53: Failed to create DTLS connection for AP xx.xx.0.14 (26070).
*spamApTask3: Aug 16 17:47:14.748: %DTLS-3-PKI_ERROR: openssl_dtls.c:562 PKI initialization error : Certificate initialization failed

A comparison of show certificate all from both WLCs looks good to my eyes

Any ideas on what the issue could be?

PJR_CDF · ‎08-17-2018

This root cause for this issue has now been established.

It would appear the controller in question had previously been setup to use LSC certs and configured to talk to a (now offline) CA.

Previous OS updates of the WLC didnt cause an issue but the jump from 8.0 to 8.3 caused this issue.

It wasnt possible to disable LSC from the GUI (it errored) but once disabled via the CLI, APs were able to connect.

Thanks to all those who provided input into this thread.

View solution in original post

Mikey Boy · ‎08-17-2018

Those logs look identical to the known issue of the certificate installed at manufacture having expired when the APs try to join. I did not think it affected the 2600 though, but maybe the wlc with issues has a different manutacture date?

You can verify the expiry of the wlc and ap certs

On the AP:

show crypto pki certificates

On the wlc:

WLC_CLI: show certificate all

If you disable the ap certs check on the problem wlc does it allow them to join?

For Versions 7.4.140.0 and later, use this command:

(WLC)>config ap cert-expiry-ignore {mic|ssc} enable

Info on the field notice is here:

https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html

PJR_CDF · ‎08-17-2018

Thanks for that - I had checked it previously.

The serials of our APs were not listed as affected.

Plus the same APs are able to connect to another 5508 that was purchased at the exact same time.

The issues also affects our new box fresh 2802 APs which have a much more recent manufacture date than the 2602's.

The cert expiry dates on both WLCs are identical. Which cert specifically on the WLC is critical as there are some that have expired (listed below)

Airespace Root CA cert
Old Airespace CA cert
Airespace Id cert
Old Airespace Id cert

The Cisco SHA1 device cert on both devices is valid until 2023

Mikey Boy · ‎08-17-2018

It is an odd one, did you disable the certificate check though? I am more curious to see whether that resolves the issue or whether the debugs are misleading and you have found an unreported bug.

I would say TAC case is the next stop, but personal curiosity would have me disabling the cert check.

Sandeep Choudhary · ‎08-17-2018

Looks like a bug: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf76274

you must try to upgrade the software to 8.5.x and try again.

Regards

Dont forget to rate helpful posts

PJR_CDF · ‎08-17-2018

thanks for this but 8.3.143.0 isnt listed as an affected release?

Sandeep Choudhary · ‎08-17-2018

Then best way to contact cisco TAC.