WLC 5520 Upgrade Issue

[kishen32]

Hi guys, i am having a weird issue post upgrade of HA WLC 5520 from 8.1.102.0 to 8.5.164.0. The previous active unit is standby and current active box i do see APs associated. The problem is when i failover the WLC, the actual active unit unable to get the APs associated. When i see the error logs, i see as below. Is this related to come SSL cert missing in the WLCs, since i see some certificate present on the standby unit but not in the active unit.

*spamApTask0: Sep 21 10:08:52.561: %DTLS-3-PKI_ERROR: [PA]openssl_dtls.c:547 PKI initialization error : Certificate initialization failed
*spamApTask0: Sep 21 10:08:52.561: %LOG-3-Q_IND: [PA]sshpmcert.c:897 Accessing certificate table before initialization
*spamApTask0: Sep 21 10:08:52.561: %SSHPM-3-CERT_TABLE_INVALID: [PA]sshpmcert.c:897 Accessing certificate table before initialization
*spamApTask2: Sep 21 10:08:51.608: %CAPWAP-3-DTLS_DB_ERR: [PA]capwap_ac_sm.c:9726 78:48:59:de:34:04: Failed to create DTLS connection for AP

I did some checks online and the closest i see if the below solution, but it looks like not applicable to my scenario. The WLC time looks ok to me.

https://community.cisco.com/t5/wireless-and-mobility/ap-can-t-join-dtls-connection-closed-by-controller/td-p/1871401

https://community.cisco.com/t5/wireless-and-mobility/ap-s-wont-connect-to-5508-wlc-after-update-to-8-3-143-pki/td-p/3690280

When i see the certificate, the output as below on the actual active unit.

(Cisco Controller) >show certificate all

--------------- Verification Certificates ---------------

-------------- Identification Certificates --------------

(Cisco Controller) >show certificate summary
Web Administration Certificate................... Locally Generated
Web Authentication Certificate................... Locally Generated
Certificate compatibility mode:.................. off
Lifetime Check Ignore for MIC ................... Disable
Lifetime Check Ignore for SSC ................... Disable

While this is how it looks in the standby unit.

(Cisco Controller) >show certificate all

--------------- Verification Certificates ---------------

-------------- Identification Certificates --------------

(Cisco Controller) >show certificate summary
Web Administration Certificate................... 3rd party
Web Authentication Certificate................... Locally Generated
Certificate compatibility mode:.................. off
Lifetime Check Ignore for MIC ................... Disable
Lifetime Check Ignore for SSC ................... Disable

Appreciate any lead from you guys as this point i am out of ideas and the device is out of contract, hence i can't raise a tac case.

P.S. Tried below config as well, no luck

config ap cert-expiry-ignore mic disable
config ap cert-expiry-ignore ssc disable

config auth-list ap-policy ssc disable
config certificate ssc hash validation enable

[Leo Laohoo]

The APs are not joining the 8.5.X.X controller?
What model is the AP?

[kishen32]

Combination of 2800 and 2700, it can join on the standby unit when i failover from active to standby, but not associating to the actual active unit when i make it primary

[Scott Fella]

The cert you show is only for web admin. I’m assuming you didn’t have any issues with failover before you upgraded and not units were able to associate AP’s? This is what I would do, but again, others might handle it differently. I would take the wlc that is not working out and factory default it. Then I would go through the startup wizard and configure the basic settings. Then I would follow the guide eon replacing the primary controller in SSO and see if that fixes the issue. I have done this a few times when one of the units would fail and or had an issue with one of the units. Once you enable SSO on the one you factory default, it will sync the configuration from the existing and then you can try the fail over once both show up and sync.

[kishen32]

What about the license, from the actual active unit i can't see the license file which i had previously. i added manually and can see it. Not sure if the box it self had crashed.

[Leo Laohoo]

Console into the AP and reboot the AP. 
Post the entire boot-up process.

[kishen32]

will try and let you know

[Rich R]

First - did you follow the release note recommendations (always read the release notes carefully).

https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn85mr6_ircm.html#software-rel-types-and-recommendations_85_mr56

If you were upgrading from 8.1.102.0 you should have upgraded to 8.2.166.0 or 8.2.170.0 first and then from there to 8.5.164.0 - which by the way is still only recommended/supported if you need IRCM feature otherwise you're recommended to use 8.5.161.0. 

But regardless of all that I agree with Scott to rebuild from factory default.

If public certificates are missing you'll need to re-install them - simple.

If self-signed certificates are missing/corrupted (have seen that happen on upgrade before) then you'll need to re-generate them.

[mtexter]

I have a handful of these APs with no certificates installed, and none of the guides I've found say anything about how to install or generate certificates, why they're needed, etc.  Can someone help?

[Scott Fella]

Certificates are required for the AP to trust the controller and vice versa.  You have AP's with no certificates or expired certificates?  Have you tried to search the forum for answers to your questions?  You might as well open a new thread and post exactly what AP's you have, what controller and what code.  Also show log's that can help identify the issue.

[mtexter]

No APs at all currently, just looking to upgrade the AireOS on a WLC5520, currently at 8.10.130.0, getting the age-old "failed to validate signature!" error after transferring the image.  Been looking all over old forum posts and tried everything from this one https://community.cisco.com/t5/wireless-mobility-blogs/wlc-5520-or-8540-upgrade-failing-with-failure-while-validating/ba-p/3102518 - no joy.  

Issue definitely appears to stem from lack of certificates which is why I necroed this thread.  

I can upload logs / output from any commands you'd like to see, just let me know

[Scott Fella]

What ap models? No ap's, so basically you just have a 5520 you want to test with?  The link you posted is correct, just use http not https.

[mtexter]

Correct.  We're getting these ready for a customer who I assume will install their own certificates once they get to site.  We just wanted to update AireOS and ensure full functionality before shipping.

Regarding the link I posted, I don't see any difference in the pages if I use https:// vs http://  the content is exactly the same.  In fact if I use http it just redirects to https.  Am I missing something?

[marce1000]

 @mtexter wrote : Issue definitely appears to stem from lack of certificates which is why I necroed this thread.  
                            Please start a new thread and describe the problem from scratch , include screenshot(s) from what you are seeing (e.g.)

 M.

[mtexter]

I also found the config certificate generate command which looked like it was going to create new certs for webadmin and webauth, the other options were to create certificate signing requests, which i'm not sure what to do with.  In any case, those commands appear to have done nothing, even after a system reset.