Updated WLC has strange error log and AP's not joining

[Tony Dann]

Hi we recently updated all of our WLC's to 7.098 and it all went smoothly, controllers rebooted and AP's updated their firmware and rebooted OK.

One WLC (4402) which was working fine since the update now has no AP's associated. The AP's were all configured to run in HREAP mode and are on remote sites within our WAN. I have checked that all policies and ports are still open (none have changed anyway) but the AP's can not join with the contoller.

The log from an AP trying to join with the WLC.

*Mar  1 00:15:24.966: %CAPWAP-3-ERRORLOG: Did not get log server settings from DHCP.
*Mar  1 00:15:34.991: %CAPWAP-3-ERRORLOG: Go join a capwap controller
*Jan 12 02:17:56.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: 10.96.4.17 peer_port: 5246
*Jan 12 02:17:56.000: %CAPWAP-5-CHANGED: CAPWAP changed state to
*Jan 12 02:18:17.447: %CDP_PD-2-POWER_LOW: All radios disabled - NON_CISCO-NO_CDP_RECEIVED  (0000.0000.0000)
*Jan 12 02:18:25.999: DTLS_CLIENT_ERROR: ../dtls/dtls_connection_db.c:2013 Max retransmission count reached!
*Jan 12 02:18:25.999: %DTLS-3-HANDSHAKE_RETRANSMIT: Max retransmit count for 10.96.4.17 is reached.
*Jan 12 02:18:56.000: %DTLS-5-SEND_ALERT: Send WARNING : Close notify Alert to 10.96.4.17:5246
*Jan 12 02:18:56.000: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Jan 12 02:18:56.001: %CAPWAP-5-CHANGED: CAPWAP changed state to DISCOVERY
*Jan 12 02:19:06.006: %CAPWAP-3-ERRORLOG: Go join a lwapp controller
*Jan 12 02:19:06.006: %LWAPP-3-CLIENTERRORLOG: Set Transport AddressCalled
*Jan 12 02:19:06.014: %LWAPP-5-CHANGED: CAPWAP changed state to JOIN
*Jan 12 02:19:11.013: %LWAPP-3-CLIENTERRORLOG: Join Timer: did not recieve join response (controller - ceo-wlc-01)

The logs on WLC show as below.

*emWeb: Jan 12 13:14:13.629: %AAA-5-AAA_AUTH_ADMIN_USER: aaa.c:1289 Authentication succeeded for admin user 'adann'
*spamReceiveTask: Jan 12 13:14:12.919: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:21:a0:81:a4:10 supporting CAPWAP
*spamReceiveTask: Jan 12 13:14:11.543: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:21:a0:81:8f:a0 supporting CAPWAP
*spamReceiveTask: Jan 12 13:14:11.395: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:24:14:ff:f1:70 supporting CAPWAP
*emWeb: Jan 12 13:14:10.731: %AAA-5-AAA_AUTH_ADMIN_USER: aaa.c:1289 Authentication succeeded for admin user 'adann'[...It occurred 2 times/sec!.]
*emWeb: Jan 12 13:14:09.459: %AAA-5-AAA_AUTH_ADMIN_USER: aaa.c:1289 Authentication succeeded for admin user 'adann'
*spamReceiveTask: Jan 12 13:14:09.457: %LWAPP-6-CAPWAP_SUPP_VER: spam_lrad.c:1440 Discarding discovery request in LWAPP from AP 00:24:14:ff:ec:00 supporting CAPWAP

Any suggestions would be appreciated!

Tony

[Nicolas Darchis · Accepted solution]

From which version are you upgrading. Is it possible that the previous version of AP was using lwapp (before 5.2) and that your current firewall only allows lwapp ports and not capwap ports for that site ?

Nicolas

[Tony Dann]

Thanks for that Nicolas,

I was upgrading from 6.182 and had already added CAPWAP ports. The strange thing is that is was working for about 8 weeks and has only now decided to stop working correctly.

Tony

[Nicolas Darchis]

Strange. Something must have happened that you didn't notice.

From the logs, it looks like only lwapp requests are arriving at the WLC. And WLC discards them cause it knows the AP can also do capwap so it's waiting for the capwap join packet.

As next step, I'd take a look at network traffic. Mostly close to the WLC where we want to know if we are receiving capwap discovery/join from the AP or not.

Nicolas

[Tony Dann]

Thanks Nicolas,

I'm pretty sure they are both open but I will double check my ACLs and post how I go.

Tony

[Tony Dann]

Thank you again Nicolas,

Upon further checking it turned out to be an IP conflict with the 2nd AP-Manager address. We had another device installed in our core infrastructure and some one assumed that the WLC only had one IP.

All fixed and working perfectly now. Its always the simple things that break it!!!!

Tony