Problem getting LAP's to re-regiser with WLC

[GRANT GATHAGAN]

I'm having difficulties getting 2 of my 3602i access points to re-register with their respective WLC5508 controller.

The 5508 controllers are connected to a 3750X cluster in my server room.
Each floor of my buildings have their own 3750X cluster.
Each of those clusters are configured with a DHCP pool for the AP's, but the first 20 addresses are excluded from the pool.

Once an AP grabs an IP address from the cluster pool, it gets its configuration from the WLC, then we change its configuration to give it a static IP address.
All but a few of the AP's have been configured with static addresses.

I lost connection to the first access point when I changed its IP address from DHCP-provided to static in the AP configuration.
Prior to this, it had been in service for over 5 years, but I never got around to changing the IP address until today.
The IP address change was the only change I made. I clicked on "Apply" and was informed that the AP would lose connectivity while it rebooted with the new IP address.
That was about 2 hours ago, and it's still absent from the AP list on the WLC.
When I run an IP scanner on the subnet in question, the static IP address shows up.
I can ping that IP address successfully from the WLC

The 2nd access point problem is similar.
In this case, it's the only AP on the floor.
I had to replace the switch to which it was connected.
I see the AP grabbing an IP address from the appropriate DHCP pool, but it does not re-appear on the AP list of its WLC.
As with the 1st AP, I can ping this AP from the WLC.

Is there a way to manually add an AP to the WLC by either its MAC address or its IP address?

[ammahend]

Can you share console logs from AP as well we capwap debug logs from wlc ? 

[GRANT GATHAGAN]

Thanks for the quick response, ammahend

I just started a log for the capwap debug messages, and will share them after I see the results and whether or not any messages appear from the LAP in question.

I'm not sure how to get the AP logs if the AP isn't registering with the WLC.
Can you point to any documentation on that?

[ammahend]

I was saying if you can console into the AP, but if the console is not accessible, it’s fine, let’s start with capwap debug. 

[Rich R]

I'm willing to bet you've hit the expired certs problem.

Read this field notice through very carefully (twice if necessary) then follow all the instructions carefully in the right order:
https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html
I'll summarise it for the umpteenth time:
1. Upgrade to latest version which supports your APs and WLC - probably 8.5.182.0
https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html#anc10
2. Apply the config workaround on the WLC
3. Disable NTP and set time manually to before your certs expired
4. Allow all the APs to join, download new code, pick up the config workaround
5 Re-enable NTP

[GRANT GATHAGAN]

Aloha, ammahend and rrudling
Ammahend, the output log is attached, but it contains more that just the capwap debug output. 
Given what I've read on troubleshooting this issue, I issued additional commands.
The two AP's are on different controllers, so I was able to run the commands simultaneously.
The two logs were very similar in what messages were received.
I've attached the results of one of the logs.
The commands:
config session timeout 60
config serial timeout 60
debug mac addr <AP MAC address>
debug client <AP MAC address>
debug lwapp events enable
debug lwapp errors enable
debug pm pki enable
debug dtls all enable
debug capwap events enable

rrudling,
I had read that field notice and I tend to agree. The only thing that made me wonder is that I get no error in the log files that I collected that specifically identifies certificate failure. I realize, though, that I'm looking at the WLC logs, and that notice discusses the output of the AP, which I cannot easily access (mounted on a 12 foot high ceiling).
I *do* have a spare AP that I can use as a test and look at its output when connected to the switch.

There are a few hurdles that I need to get past in order to update the WLC controller software.
Our SMARTNET contract expired and I'm waiting on our fiscal office to issue the PO for the renewal.
You are correct on the software version.
I'm assuming that updating the WLC software to 8.5.182.0 also provides the update for the AP's to 15.3(3)JF15.
Is that correct?

Regards,
Grant

[ammahend]

if SSH was enable before you assigned static IP, you should still be able to SSH and get logs, get output for show capwap ip config.

run these commands, see if it makes any difference for the APs in question.

(Cisco Controller) >config ap cert-expiry-ignore ssc enable

(Cisco Controller) >config ap cert-expiry-ignore mic enable

[Rich R]

> I'm assuming that updating the WLC software to 8.5.182.0 also provides the update for the AP's to 15.3(3)JF15.
Is that correct?
Yes you can see the WLC version and corresponding AP version at https://www.cisco.com/c/en/us/td/docs/wireless/compatibility/matrix/compatibility-matrix.html#ctr-ap_support

Also as per FN: 

If you run AireOS Version 8.0 or later, then in order to determine when the WLC certificate expires, enter this command and look for the Cisco SHA1 device cert entry:

WLC_CLI: show certificate all

[GRANT GATHAGAN]

Aloha, all

ammahend,
Sadly, since both AP's were in use prior to these disruptions, SSH connections to both AP's are disabled.

rrudling,
The WLC5508 controllers are currently running Cisco Wireless Release 7.2.103.0, so the closest command I can use is "show local-auth certificates".
The certificate that command shows *is* expired, as of the 1st of this month.
The headscratcher for me is that everything I can see about that cert ties it to the webUI, not the AP's.
"show local-auth certificates", for instance, responds with:
Certificate issuer .............................. vendor
CA certificate: Not installed.
Device certificate: Not installed.
Certificate issuer .............................. cisco
Device certificate:
Valid: 2012 Sep 1st, 16:45:19 GMT to 2022 Sep 1st, 16:55:19 GMT

The response to "show certificate lsc summary" contains:
LSC Enabled...................................... No
LSC CA-Server.................................... None
LSC AP-Provisioning.............................. No
LSC Certs:
CA Cert...................................... Not Configured
RA Cert...................................... Not Configured

If I issue the "show local-auth statistics" command, the Certificate operations section shows:
Certificate operations:
Local device certificate load failures .......... 0
Total peer certificates checked ................. 0
Failures:
CA issuer check ............................... 0
CN name not equal to identity ................. 0
Dates not valid or expired .................... 0

Am I making wrong assumptions on what I see?
At any rate, I will be testing tomorrow with the spare AP, so I hope to have more answers from the AP's perspective.

Regards,
Grant

[Rich R · Accepted solution]

You *really* need to get that software updated is all I can say!  Otherwise you're just wasting your time and effort and putting off what you're going to have to do.
Obviously that old code is not showing you the MIC.
If you want to prove to yourself that it's cert expiry do a packet capture of the AP join process and you can look at the cert dates in the DTLS exchange in Wireshark.
To get upgraded from that old code you'll need to go through the release notes very carefully.
https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn85mr8.html#upgrade-software_85mr8
In theory you could upgrade direct to this release but watch out for a few config changes that might need to be corrected after upgrade as per the release notes.  A more cautious approach would be to upgrade to 8.0 first then 8.5 but you'll need to wait for all APs to complete download and reboot on 8.0 before starting 8.5 upgrade.

[GRANT GATHAGAN]

I just finished testing with my spare AP, and as rrudling surmised, the APs are rejecting the expired certificate.

*Mar 1 00:01:02.791: AP has SHA2 MIC certificate - Using SHA1 MIC certificate for DTLS.
*Sep 23 21:31:00.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.0.0.91 peer_port: 5246
*Sep 23 21:31:00.127: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed.
The certificate (SN: 15814C390000000B2D53) has expired.
Validity period ended on 16:55:19 UTC Sep 1 2022Peer certificate verification failed 001A

*Sep 23 21:31:00.127: DTLS_CLIENT_ERROR: ../capwap/base_capwap/capwap/base_capwap_wtp_dtls.c:496 Certificate verified failed!
*Sep 23 21:31:00.127: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 10.0.0.91:5246
*Sep 23 21:31:00.127: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 10.0.0.91:5246
*Sep 23 21:31:20.247: %CDP_PD-4-POWER_OK: All radios disabled - NEGOTIATED inline power source

My supervisor tells me that we're waiting on the fiscal office to issue the PO for the SMARTNET renewal, so there's noting I can do but wait.
Fortunately the problem only impacts 2 APs, and both of them are in areas of little use.
I thank you both for your assistance
Regards,
Grant

[Rich R]

You could turn off NTP on the WLC and set the time back to well before the cert expired (eg exactly 1 year back) to get things working meanwhile though.
That does mean all the timestamps in your logs etc will be wrong but it will allow the APs to join.
Just make sure you don't have anything else that's time dependent that could break.

[Rich R]

And as @Leo Laohoo likes to remind people - read the "Customers Without Service Contracts" section of a recent applicable Security Advisory like https://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-ap-privesc-wEVfp8Ud then EMAIL Cisco TAC referring to the URL of the advisory, referencing that paragraph, include the link to the software you need to download and your WLC serial number.  TAC should then provide a link for you to download the software.