APs not joining a new WLC

[davidrodriguez]

Hi guys!
We need help with two WLCs, one with a free license and another with 27 APs licenses.
Currently, we are trying to change these WLCs one for another so we can use the WLC Licensed.
When we disconnect the current WLC and connect the new one, the APs are not joining. On the logs, the only thing we can see is this:

CAPWAP LOG ERROR:

*spamApTask3: Jun 06 17:45:36.919: DTLS connection was closed

CAPWAP LOGS EVENTS:

*spamApTask2: Jun 06 17:44:58.092: DTLS Session established server (1.1.1.1:5246), client (1.1.1.2:5264)
*spamApTask2: Jun 06 17:44:58.092: Starting wait join timer for AP: 1.1.1.2:5264

*spamApTask2: Jun 06 17:44:58.103: acDtlsPlumbControlPlaneKeys: lrad:1.1.1.2(5264) mwar:1.1.1.1(5246)

*spamApTask2: Jun 06 17:44:58.103: DTLS keys for Control Plane deleted successfully for AP 1.1.1.2

*spamApTask2: Jun 06 17:44:58.104:  DTLS connection closed event receivedserver (1.1.1.1/5246) client (1.1.1.2/5264)
*spamApTask2: Jun 06 17:44:58.104: No entry exists for AP (1.1.1.2/5264)
*spamApTask2: Jun 06 17:44:58.105: Deleting AP entry 1.1.1.2:5264 from temporary database.

Both WLCs are Cisco 5508 and are on version: 8.3.150.0. The APs are 3802, on version 8.3.150.0.
What we tried:
We tried to configure manually the new WLC IPs on the APs under Wireless/All APs/[AP]/High Availability.
We checked the Country, and it's set to US-Canada.
We checked the date/time and Time zone.

Any help is much appreciated!

[davidrodriguez]

Sorry for the late response, i was out. Here are the outputs on the other WLC (the one with the license)

(Cisco Controller) >show sysinfo

Manufacturer's Name.............................. Cisco Systems Inc.
Product Name..................................... Cisco Controller
Product Version.................................. 8.3.150.0
Bootloader Version............................... 1.0.1
Field Recovery Image Version..................... 6.0.182.0
Firmware Version................................. FPGA 1.3, Env 1.6, USB console 1.27
OUI File Update Time............................. Sun Sep 07 10:44:07 IST 2014

Build Type....................................... DATA + WPS

System Name...................................... WLC01
System Location..................................
System Contact...................................
System ObjectID.................................. 1.3.6.1.4.1.9.1.1069
Redundancy Mode.................................. Disabled
IP Address....................................... 1.1.1.2
IPv6 Address..................................... ::
Last Reset....................................... Power on reset
System Up Time................................... 3 days 1 hrs 33 mins 13 secs
System Timezone Location......................... (GMT -5:00) Eastern Time (US and Canada)
System Stats Realtime Interval................... 5

System Stats Normal Interval..................... 180

Configured Country............................... Multiple Countries : CA,US
Operating Environment............................ Commercial (0 to 40 C)
Internal Temp Alarm Limits....................... 0 to 65 C
Internal Temperature............................. +32 C
External Temperature............................. +22 C
Fan Status....................................... OK

State of 802.11b Network......................... Enabled
State of 802.11a Network......................... Enabled
Number of WLANs.................................. 2
Number of Active Clients......................... 0

OUI Classification Failure Count................. 0

Burned-in MAC Address............................ 44:2B:03:B4:C0:20
Power Supply 1................................... Present, OK
Power Supply 2................................... Absent
Maximum number of APs supported.................. 500
System Nas-Id....................................
WLC MIC Certificate Types........................ SHA1

(Cisco Controller) >show license all

License Store: Primary License Storage
StoreIndex:  0 Feature: base                              
Version: 1.0
License Type: Permanent
License State: Active, Not in Use
License Count: Non-Counted
License Priority: Medium

License Store: Primary License Storage
StoreIndex:  0 Feature: base-ap-count                     
Version: 1.0
License Type: Permanent
License State: Inactive
License Count: 12 / 0 (Active/In-use)
License Priority: Medium

License Store: Primary License Storage
StoreIndex:  0 Feature: base-ap-count                     
Version: 1.0
License Type: Permanent
License State: Inactive
License Count: 17 / 0 (Active/In-use)
License Priority: Medium

License Store: Primary License Storage
StoreIndex:  0 Feature: base-ap-count                     
Version: 1.0
License Type: Permanent
License State: Inactive
License Count: 22 / 0 (Active/In-use)
License Priority: Medium

License Store: Primary License Storage
StoreIndex:  0 Feature: base-ap-count                     
Version: 1.0
License Type: Permanent
License State: Active, In Use
License Count: 27 / 27 (Active/In-use)
License Priority: Medium

License Store: Evaluation License Storage
StoreIndex:  1 Feature: base-ap-count                     
Version: 1.0
License Type: Evaluation
License State: Inactive
Evaluation total period: 8 weeks 4 days
Evaluation period left: 8 weeks 4 days
License Count: 500 / 0 (Active/In-use)
License Priority: None

Thanks for all the help! 

[Rich R]

Hmmmm!  One last idea (although I don't think that's it): Do you have "config ap cert-expiry-ignore mic enable" configured on both of the WLCs?  APs must pick up the config from the WLC they've joined before trying to join the other one. (FN-63942)

Otherwise you need to attach a complete console log from an AP trying to join (attach as text file), from power-on,  and debugs from the WLC at the same time.  Packet capture of the CAPWAP control packets (UDP 5246) at the same time would also be good to see.

[davidrodriguez]

Do you know which cert I should be looking for? 

I see from this post: https://www.wiresandwi.fi/blog/cisco-wlc-or-ap-device-certificate-expired-what-you-can-do

That I should check the Certificate Name: Cisco SHA1 device cert; if that is so, then I could see that the certificate is expired, and you could be in the right direction: 

Certificate Name: Cisco SHA1 device cert
     Subject Name :
         C=US, ST=California, L=San Jose, O=Cisco Systems, CN=AIR-CT5508-K9-442b03b4c020, emailAddress=support@cisco.com
     Issuer Name :
         O=Cisco Systems, CN=Cisco Manufacturing CA
     Serial Number (Hex):
         6BC13D7800000011CF23
     Validity :
         Start : May  4 02:13:03 2012 GMT
         End   : May  4 02:23:03 2022 GMT

 Then I'll try to run that command you mentioned on the new WLC to force the APs to join it. 

[Rich R · Accepted solution]

Aha then it is that!
Configure the command on the OLD WLC - then the APs will pick up the config and then they will be able to join the NEW WLC (with the expired cert).  But you must also configure it on the NEW WLC otherwise the APs will remove the config and then not be able to join again the next they reset.

In summary: configure that on new and old WLCs.

The command is harmless - it won't impact the APs at all - but if for some reason you don't want to apply it to the old WLC then on the new WLC you'll need to:
- apply config ap cert-expiry-ignore mic enable
- disable NTP, configure date to earlier than May 4 2022
- allow all APs to join and get the new config
- after all APs have joined and picked up the config fix then re-enable NTP

Warning: if you don't configure it on the old WLC and one of the APs joins the old WLC again then the AP will lose the config then not be able to join the new WLC again - so your easy path is just configure both.

[davidrodriguez]

Thank you! I'll run the command on both WLC, and then next week, we are planning a maintenance window to switch to the other WLC. 

I already apply the command config ap cert-expiry-ignore mic enable on the new WLC.

And disabled NTP, when we are ready to switch them, i'll change the date on the new WLC. 

I'll run the command on the old WLC too today. 

I'll let you know how it goes next week. 

[davidrodriguez]

Thanks, @Rich R, that worked perfectly! 

We could join the APs by only running that command to ignore the certificate and then changing the date and time to a date before the certificate's expiration. 

Something to add:

Under the Old WLC (unlicensed), we also manually changed the AP's Primary WLC to the New WLC. Under Wireless/All APs/AP#/High Availability. And then an AP reboot was enough to join the New WLC (Licensed). 

Thanks for the help!