2602 APs lose connection to 5508-WLC Controller

[maerz-helpdesk]

Hi,

We have stumbled upon an issue with the Cisco 2602 Access Points. In the last 3 days APs started to seemingly randomly lose connection to the WLC and are unable to join the Controller again. As of now, about 15 of our 300 APs had the problem.

The Access Point recieves a DHCP IPv4 correctly and seems to find the controller but then somehow cannot connect to it:

*Feb 28 09:44:53.583: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 10.100.56.153, mask 255.255.252.0, hostname AP1005.ca63.f181

*Feb 28 09:44:55.507: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:44:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:45:25.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x7837228!

*Feb 28 09:45:55.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:45:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:46:25.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x7D8AD4C!

*Feb 28 09:46:55.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:46:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:47:25.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x7837228!

*Feb 28 09:47:55.999: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:48:11.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:48:11.483: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:48:11.483: %CAPWAP-5-SENDJOIN: sending Join Request to 172.20.20.9
*Feb 28 09:48:16.483: %CAPWAP-5-SENDJOIN: sending Join Request to 172.20.20.9
*Feb 28 09:48:16.487: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:48:16.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:48:27.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:48:27.000: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:224 Connection 0xA774218 is already there for this server port 5246, Deleting it. Number of connections: 1

*Feb 28 09:48:27.000: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:48:56.999: DTLS_CLIENT_ERROR: ../capwap/base_capwap/dtls/base_capwap_dtls_connection_db.c:2214 Max retransmission count reached for Connection 0x6EEF964!

*Feb 28 09:49:17.435: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:49:32.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:49:32.507: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:49:32.507: %CAPWAP-5-SENDJOIN: sending Join Request to 172.20.20.9
*Feb 28 09:49:32.523: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:49:33.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 172.20.20.9 peer_port: 5246
*Feb 28 09:49:38.503: %CAPWAP-3-DHCP_RENEW: Could not discover WLC. Either IP address is not assigned or assigned IP is wrong. Renewing DHCP IP.
*Feb 28 09:49:41.519: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 172.20.20.9:5246
*Feb 28 09:49:41.583: %DHCP-6-ADDRESS_ASSIGN: Interface BVI1 assigned DHCP address 10.100.56.153, mask 255.255.252.0, hostname AP1005.ca63.f181

Although it says "Could not discover WLC." , the WLC displays a message regarding that AP:

Mon Feb 28 11:02:41 2022 Failed to authorize AP Name AP1005.ca63.f181 with Base Radio MAC 6c:fa:89:90:c2:30. Authorization entry does not exist in AAA server.

We successfully were able to reach the WLC from the AP-CLI via a ping. All APs are on random points of the network and have not much in common. Some are connected with Power-Injector, some via PoE-switch

I have attached a screenshot of the auth-list and a part of the traps, showing the other APs with the same Problem.

  

The system worked for several years now and the last controller restart was about 2 months ago. Originally there were no seperate entries in the auth-list, but we now tried to enter the APs manually. Sadly, this did not resolve the issue.

We want to restart the WLC again, but could not find a time window yet. We suspect there being something wrong with the WLC, since it displays an error when we tried adding an entry to the auth-list via Web GUI. We had to enter MAC via the SSH connection.

[ammahend]

can you share

show auth-list (or Security > AP Policies in GUI ) as well as AP port config on switch. 
WLC version ?

[maerz-helpdesk]

(Cisco Controller) >show auth-list

Authorize MIC APs against Auth-list or AAA ...... enabled
Authorize LSC APs against Auth-List ............. disabled
APs Allowed to Join
  AP with Manufacturing Installed Certificate.... yes
  AP with Self-Signed Certificate................ no
  AP with Locally Significant Certificate........ no

Mac Addr                  Cert Type    Key Hash
-----------------------   ----------   ------------------------------------------
00:3a:9a:b7:f7:b0         MIC
1c:aa:07:c6:52:b0         MIC
1c:aa:07:c6:8c:00         MIC
6c:fa:89:90:c2:30         MIC
6c:fa:89:ac:e4:a0         MIC
6c:fa:89:ac:ea:e0         MIC
6c:fa:89:c9:86:e0         MIC
78:72:5d:2d:fe:60         MIC
84:b5:17:3c:98:30         MIC

As for the switchports, the switches in use are not Cisco, but HP. I have checked the config and it is identical between joined and not joined APs. Also there have been no changes to the affected ports according to the change history.

Also, please dont forget that we just added a few APs to the auth-list AFTER the problem occured. We tried that as a fix. Usually the APs can just join with their MICs. The initially mentioned error on the controller side also does not mention certificates as a possible cause.

[ammahend]

Can you also confirm airos version please?
If I understood you right you enabled this and added 2600 AP Mac addresses in local database for testing only to see if this fixes AP join issues ? 

Authorize MIC APs against Auth-list or AAA ...... enabled

Don’t want to jump to conclusion yet, but wondering if it’s expired MIC issue. In that case you can try 

config ap cert-expiry-ignore mic enable

this will basically allow APs with expired MIC to join as well. But let’s start with the WLC version first. 

[maerz-helpdesk]

Current airos version is 8.0.133.0

We thought about a firmware update, but also are looking for a maintenance window at the moment.

Yes, we only enabled the auth-list for troubleshooting. Before the issue appeared, no entries were in the auth-list. Also, only about half of the affected APs are in there at the moment, so we should can test with both cases.

I ran into a problem with expired MICs about half a year ago (different system, but also a 5508 WLC) and from what i recall the WLC gave out an error about the certificate. But I am also not fully sure about that.

[ammahend]

would certainly recommend to disable cert check and see if it fixes the issue, let us know. Since you are disabling MIC check, needless to say it  does have some security downside.

i remember there use to be a database link where you can validate your serial, couldn’t don’t it anymore. 

[Haydn Andrews]

You are probably hitting this, had a customer with same logs the other day:

https://www.cisco.com/c/en/us/support/docs/field-notices/639/fn63942.html 

Work arounds are dependant on the WLC software version.

How to Identify Which Software Versions Have the Fix

If you run AireOS software, the fixes required depend on the APs that you have in your network:

• Cisco IOS APs that were manufactured prior to August 2014 can be fixed via Cisco bug ID CSCuq19142 for AireOS Versions 7.0.252.0, 7.4.140.0, 8.0.120.0, 8.1.102.0, and later.
• AP-COS APs can be fixed via Cisco bug ID CSCvb93909 in AireOS 8.5 and later.
• Cisco IOS APs that were manufactured with SHA-2 certificates in August 2014 and later can be fixed via Cisco bug ID CSCvs22835 in Version 8.5.160.0 and later.

[maerz-helpdesk]

I dont think it has anything to do with the MICs, since all are active until at least 2024 and all devices I checked get the correct time via NTP.

Also, the screenshot from the Traps on the controller do not mention the certificates, but a missing AAA entry. I had the cert problem before and the messages look very different from them.

[Rich R]

If you haven't hit the cert expiration already you will - just read through the field notice carefully, deploy the latest software and config the workaround (disable checks).  Remember you might need to set WLC time back to let the APs join and pick up the new software and config.  That will involve reloading the WLC anyway so if it's some other bug then the reload clears that and at the same time you future-proof yourself against the cert expiry even if that isn't the immediate cause.

[maerz-helpdesk]

I will try the update and most likely disable the cert check in advance. It might take some days to organize everything for a worst case scenario. I will get back on this as soon as I have news.

As mentioned above, I still dont think it has anything to do with the MICs, since all are active until at least 2024 and all devices I checked get the correct time via NTP.

[maerz-helpdesk · Accepted solution]

Firmware Update and restart resolved the issue:

I want to give you an update after we could resolve the issue. Sadly, we could not make out a specific cause, but since we first noticed, that the auth list was not wotking on the web GUI we wanted to try a firmware update.

We wanted to update to the recommended 8.5.171.0 initially, but then realised on site that the network contained some old 1242 APs. That meant, the newest Firmware update we could go to was the 8.0.152.0 from Oct 2017.

After commiting to the update we restarted the Controller Cluster and all APs were able to join without any warnings in the logs. We are currently discussing disabling the MIC auth internally and are looking to get rid of the Cisco 1242 APs as soon as possible.

[Scott Fella]

I love those 1242's... they are like tanks!  I still have a few in my collection just to have.  One thing to keep in mind also, when you keep ap's for a long time, there is a point where there are no vulnerability and security patches after a certain date.  This also goes for the controller image.