WiSM AP Join Issues

[joshkelly13]

I am having a problem with APs joining a WiSM Controller.

Upon installation of the controller (several months ago), i had no issues.  APs joined as expected with no issues.  Upon returning to add more APs i have found that APs are no longer joining.  They show up on the Monitor > Statistics > "AP Join" screen with an IP address in the correct subnet but status = Not Joined.  Also, some APs that have previously joined are now un-configured and show as "Not Joined" leaving only a few still working correctly.

I am greatly confused by this problem.

Any help would be appreciated.

Thanks.

Josh Kelly

[Scott Fella]

Can you post an output from one of the AP’s that is not able to join? Is the AP’s on a different subnet than the WLC management ip address? I suppose you were doing either option 43 or dns for AP discovery, is that still in place?

[joshkelly13]

Output from the console???  I am currently off-site and won't be able to get back till Friday.  I can maybe get a local tech to check on that for me.

And Yes, the deployment is across multiple subnets.  i have verified that my dhcp option 43 is still in place.

Thanks for such a fast response.

[Scott Fella]

From the console would be good to see the discovery process the ap is going through. Is dhcp working okay for the AP subnet? I just ran into an issue today in which we had to restart the dhcp services for some odd reason.

Thanks,

Scott Fella

Sent from my iPhone

[joshkelly13]

DHCP is working.  The AP is getting an ip address and the controller is recieving the discovery requests.

one particular AP i am am looking at has recieved 124 requests and the controller has sent 64 responses (as listed on the AP Join page).

i have a tech on-site getting ready to send me remote console access (via getconsole on iphone)

[Scott Fella]

Sounds good.

[joshkelly13]

looks like i have a certificate issue...

i do not have lsc enabled.

*Nov  9 04:00:10.042: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY     

*Nov  9 04:00:10.043: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY     

*Nov  9 04:00:10.051: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to administratively down                     

*Nov  9 04:00:10.051: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to administratively down

*Nov  9 04:00:10.066: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up

*Nov  9 04:00:10.066: %LINK-3-UPDOWN: Interface Dot11Radio1, changed state to up

*Nov  9 04:00:10.068: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset

*Nov  9 04:00:10.097: %LINK-3-UPDOWN: Interface Dot11Radio0, changed state to up

*Nov  9 04:00:10.098: %LINK-5-CHANGED: Interface Dot11Radio1, changed state to reset

*Nov  9 04:00:20.064: %CAPWAP-3-ERRORLOG: Go join a capwap controller

*Nov  9 04:00:20.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent

peer_ip: 172.16.100.245 peer_port: 5246

*Nov  9 04:00:20.001: %CAPWAP-5-CHANGED: CAPWAP changed state to

*Nov  9 04:00:20.099: %PKI-3-CERTIFICATE_INVALID_EXPIRED: Certificate chain validation has failed. 

The certificate (SN: 3D41813F00000016DADF) has expired.   

Validity period ended on 20:04:41 UTC Dec 28 2020

*Nov  9 04:00:20.100: %LWAPP-3-CLIENTERRORLOG: Peer certificate verification failed

*Nov  9 04:00:20.100: %CAPWAP-3-ERRORLOG: Certificate verification failed!

*Nov  9 04:00:20.100: DTLS_CLIENT_ERROR: ../capwap/capwap_wtp_dtls.c:326 Certificate verified failed!

*Nov  9 04:00:20.101: %DTLS-4-BAD_CERT: Certificate verification failed. Peer IP: 172.16.100.245

*Nov  9 04:00:20.101: %DTLS-5-SEND_ALERT: Send FATAL : Bad certificate Alert to 172.16.100.245:5246

*Nov  9 04:00:20.101: %DTLS-3-BAD_RECORD: Erroneous record received from 172.16.100.245: Malformed Certificate

[Stephen Rodriguez]

Update the time and date on your WLC.  Looks like you are set to Nov 9, at 4am.

Steve

[joshkelly13]

i checked the time/date earlier today.  it was off by a couple of hours, but date was correct.  would an incorrect time/date on the ap cause it not to join?

[Stephen Rodriguez · Accepted solution]

The MIC/SSC are time sensative.  if the time/date is too far off the cert will be invalid for the AP, and the WLC won't let it join.

Steve

[joshkelly13]

These are brand new LAPs out of the box.  Should they not just join and get the correct time?

[Stephen Rodriguez]

The should get a time update when they join the WLC.  Are you sure the year is correct on the WLC?

Validity period ended on 20:04:41 UTC Dec 28 2020.  would indicate to me that the WLC year is incorrect on the WLC.

Are all the WLC set to the correct time?  Are you NTP synching them?

Steve

[joshkelly13]

Got it figured out...

i had 2 ips on the option 43 (primary and backup controller, wism has 2 controllers)

i had yet to configure the time on the backup controller.  it was dec 2028.... 

i removed the backup controller ip from the dhcp opt43 earlier today in my troubleshooting.

shortly after correcting the time on the backup controller, 1 of the aps joined, but no other aps were attempting... earlier after clearing ap join stats, within 1 minute all aps would reappear, now they were not reappearing.

i began to remotely reboot switches that the aps were connected to, and they began to reappear in the ap join log, but would not join...

after a reboot of the controller, all aps immediatly joined...

i am still quite confused by this... not sure where the ap i posted the console output from got its time and why that affected it joining

uptime on the controller was 102 days, should i be rebooting it regularly?

i do recall setting the time on the primary controller, and i recall not setting it on the backup

but even if the time is incorrect on the controller, it does not know that it is wrong, the ap should just get that time and begin the join process right?

as for ntp, i'll point them to the primary domain controller.  i did set the time during installation and it was a couple hours off... reminds me of the time slip on VMs

[Stephen Rodriguez]

so the AP always gets its time from the WLC that it is joined to. I can't say why it was trying to hit the backup.

The reason time matters is the cert on the AP has a lifetime. So time and date need to be correct.

Once the time is good, you shouldn't need to reboot the WLC unless you upgrade the code.

Steve

Sent from Cisco Technical Support iPhone App

[joshkelly13]

Thanks for the prompt answers.

I really appreciate your help.

Josh