Connect Previously Configured LAP to new WLC without access to old WLC

[jtrohm]

My educational client has approximately 50 3802i LAPs deployed across the campus. Over the weekend, they managed to blow up the only working copy of their vWLC. I was able to stand one up quickly and get new LAPs to connect, but I can't get the existing ones to connect to the new controller without resetting them at the console (ssh/telnet disabled). The problem appears to be a WLC/certificate validation error (APs are caching the old controller info).

I've tried the obvious stuff like giving the new controller a different name/IP, etc. I'm out of ideas.

I'd obviously like to avoid visiting 50 different classrooms and climbing the ladder 100 times to fix this. Is there a work around for resetting the APs if I don't have access to the old WLC or its certificates?

Thanks so much!

[jagan.chowdam]

How do the APs configured to discover the WLC? 

Do you see AP join request on the new controller from old APs? See whats the status under "Monitor --> Statistics --> AP Join". 

Decode the " Last Error Summary" message for those old APs, if you see an entry there. Console into one AP and see the boot messages.

Check "Security --> Policy Configuration --> Accept Manufactured Installed Certificate (MIC)" is checked on the new controller. Also, you can check the other options such as "Self Signed Cert, Local Significant Cert etc'

Here is the good blog post about AP - WLC join process.

http://revolutionwifi.blogspot.com/2010/11/capwap-ap-join-process.html

Make sure you keep only the required AP Policies once the old APs join the WLC.

Check NTP and Country codes as well.

*** Please rate all helpful posts ***

Jagan

[jtrohm]

Thanks Jagan.

I see the discover but never a join request (all zeros in the stats). The country codes are good (these were working up until Friday when the server admin burned the VM to the ground). The clock is good. Ironically, there were six APs that worked without intervention. They are the same model but they were installed later, so I don't know what is different.

Resetting the config fixes the problem. I'm on AP 6 of 44 now.... ugh.

[jtrohm]

I see the discover but never a join request (all zeros in the stats). The country codes are good (these were working up until Friday when the server admin burned the VM to the ground). The clock is good. Ironically, there were six APs that worked without intervention. They are the same model but they were installed later, so I don't know what is different.

Resetting the config fixes the problem. I'm on AP 6 of 44 now.... ugh.

[jagan.chowdam]

You can also try using the following command on your WLC.

config ap cert-expiry-ignore {mic|ssc} enable

With this command in effect, the WLC and AP will ignore the expiration date on the device MICs and SSCs.

[jtrohm]

Sorry, should have mentioned that I tried that too. The APs are 3802s, so the MICs don't expire until 2027 anyhow. But, gave it a go anyhow.

I think the bottom line is that I'm hosed. Lesson learned about proper change control and backups.

[jagan.chowdam]

Can you share AP boot console output.

Can you share debug output for the following on the WLC (filter it for one AP).

debug capwap events enable

debug capwap packet enable

debug pm pki enble

Jagan

[jtrohm]

[*01/28/2019 18:53:48.0000] CAPWAP State: DTLS Setup
[*01/28/2019 18:53:48.0004] dtls_connectionDB_add_connection: Number of DTLS connections exceeded two
[*01/28/2019 18:53:48.0413] dtls_load_ca_certs: LSC Root Certificate not present
[*01/28/2019 18:53:48.0413]
[*01/28/2019 18:53:48.0446] dtls_verify_con_cert: Controller certificate verification error
[*01/28/2019 18:53:48.0446] dtls_process_packet: controller cert verification failed
[*01/28/2019 18:53:48.0453] sendPacketToDtls: DTLS: Closing connection 0x26cac00.
[*01/28/2019 18:53:48.0459] Restarting CAPWAP State Machine.
[*01/28/2019 18:53:48.0459] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*01/28/2019 18:53:48.0463]
[*01/28/2019 18:53:48.0463] CAPWAP State: DTLS Teardown
[*01/28/2019 18:53:52.7607] Discovery Response from 10.1.1.71

[jtrohm]

I stand corrected. Maybe there is something going on with region.


Jan 28 11:19:38 10.1.1.71 DOWLC01: *spamApTask6: Jan 28 11:19:38.533: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:6985 6c:b2:ae:70:eb:00: DTLS connection closed forAP 10:9:6:38 (5248), Controller: 10:1:1:71 (5246) AP Message Timeout
Jan 28 11:22:23 10.1.1.71 DOWLC01: *spamApTask7: Jan 28 11:22:23.866: %LWAPP-3-RD_ERR9: spam_lrad.c:12817 APs 6c:b2:ae:70:eb:00 country code changed from (UX) to (US )
Jan 28 11:22:24 10.1.1.71 DOWLC01: *spamApTask2: Jan 28 11:22:24.011: %LOG-3-Q_IND: spam_lrad.c:12817 APs 6c:b2:ae:70:eb:00 country code changed from (UX) to (US )
Jan 28 11:24:06 10.1.1.71 DOWLC01: *spamApTask2: Jan 28 11:24:06.416: %CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c:7533 Max retransmissions reached on AP(6c:b2:ae:88:f2:60),message (CAPWAP_IMAGE_DATA_REQUEST
),number of pending messages(1)
Jan 28 11:24:06 10.1.1.71 DOWLC01: *spamApTask2: Jan 28 11:24:06.416: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:6985 6c:b2:ae:88:f2:60: DTLS connection closed forAP 10:9:6:34 (5248), Controller: 10:1:1:71 (5246) AP Message Timeout
Jan 28 11:26:56 10.1.1.71 DOWLC01: *spamApTask3: Jan 28 11:26:56.956: %LWAPP-3-RD_ERR9: spam_lrad.c:12817 APs 6c:b2:ae:88:f2:60 country code changed from (UX) to (US )
Jan 28 11:26:57 10.1.1.71 DOWLC01: *spamApTask2: Jan 28 11:26:57.336: %LOG-3-Q_IND: spam_lrad.c:12817 APs 6c:b2:ae:88:f2:60 country code changed from (UX) to (US )
[root@nettools1 10.1.1.71]# cat 20190128 | grep 'fb:40'
Jan 28 11:00:52 10.1.1.71 DOWLC01: *spamApTask3: Jan 28 11:00:52.327: %CAPWAP-3-MAX_RETRANSMISSIONS_REACHED: capwap_ac_sm.c:7533 Max retransmissions reached on AP(6c:b2:ae:88:fb:40),message (CAPWAP_IMAGE_DATA_REQUEST
Jan 28 11:00:52 10.1.1.71 DOWLC01: *spamApTask3: Jan 28 11:00:52.327: %CAPWAP-3-DTLS_CLOSED_ERR: capwap_ac_sm.c:6985 6c:b2:ae:88:fb:40: DTLS connection closed forAP 10:9:6:35 (5264), Controller: 10:1:1:71 (5246) AP Message Timeout
Jan 28 11:03:44 10.1.1.71 DOWLC01: *spamApTask2: Jan 28 11:03:44.691: %LWAPP-3-RD_ERR9: spam_lrad.c:12817 APs 6c:b2:ae:88:fb:40 country code changed from (UX) to (US )
Jan 28 11:03:44 10.1.1.71 DOWLC01: *spamApTask1: Jan 28 11:03:44.849: %LOG-3-Q_IND: spam_lrad.c:12817 APs 6c:b2:ae:88:fb:40 country code changed from (UX) to (US )
[root@nettools1 10.1.1.71]# apwap_ac_sm.c:6985 6c:b2:ae:88:fb:40: DTLS connection closed forAP 10:9:6:35 (5264), Controller: 10:1:1:71 (5246) AP Message Timeout

[vuyomasina]

if you run to such a problem. disable your NTP server. change the time manually to a year before and Ap's should join. after you can upload the new certs.