We have a pair of AIR-CT8540-K9 controllers running 8.5.135 with a mixture of 2702 and 3702 access points. There are 1061 APs on controller A and 984 on controller B. We are getting regular disassociation alerts generated by Prime for bunches of devices all at the same time as can be seen from this snippet of show ap uptime from one of the controllers:-
(mycontroller) >show ap uptime
Number of APs.................................... 1061
Global AP User Name.............................. myusername
Global AP Dot1x User Name........................ Not Configured
AP Name Ethernet MAC AP Up Time Association Up Time
------------------ ----------------- ----------------------- -----------------------
site1-AP-32 18:80:90:5c:a9:f4 92 days, 19 h 19 m 20 s 0 days, 02 h 16 m 29 s
site1-AP-23 18:80:90:a1:0e:7c 92 days, 18 h 02 m 45 s 0 days, 02 h 16 m 25 s
site1-AP-24 18:80:90:bb:42:d4 92 days, 17 h 41 m 12 s 0 days, 02 h 16 m 17 s
site1-AP-04 18:80:90:a2:ee:ac 92 days, 19 h 14 m 49 s 0 days, 02 h 12 m 18 s
site1-AP-06 18:80:90:bb:35:8c 92 days, 19 h 41 m 48 s 0 days, 01 h 15 m 30 s
site1-AP-09 18:80:90:a2:3e:f4 92 days, 20 h 30 m 36 s 0 days, 01 h 11 m 19 s
site1-AP-13 18:80:90:78:77:d4 92 days, 19 h 38 m 08 s 0 days, 01 h 09 m 54 s
site2-AP-12 40:ce:24:e8:0a:14 103 days, 19 h 37 m 44 s 0 days, 00 h 41 m 21 s
site1-AP-21 18:80:90:bb:44:d0 92 days, 17 h 53 m 08 s 0 days, 00 h 29 m 27 s
site1-AP-30 18:80:90:bb:42:ec 92 days, 18 h 20 m 00 s 0 days, 00 h 29 m 27 s
site1-AP-28 18:80:90:bb:42:d8 92 days, 18 h 05 m 22 s 0 days, 00 h 29 m 20 s
site1-AP-02 18:80:90:b4:f1:f8 92 days, 22 h 09 m 13 s 0 days, 00 h 29 m 09 s
site3-AP-22 40:ce:24:e8:08:4c 139 days, 07 h 33 m 45 s 0 days, 00 h 05 m 02 s
We have put the following debugging on:-
DTLS:
DTLS ERROR debugging is on
LWAPP:
LWAPP Client ERROR display debugging is on
CAPWAP:
CAPWAP Client AVC Netflow Error debugging is on
CAPWAP Client ERROR display debugging is on
And received the following output:-
*Feb 18 02:23:13.631: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Feb 18 02:23:13.647: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to X.X.X.X:5246
*Feb 18 02:23:13.655: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Feb 18 02:23:13.659: %CLEANAIR-6-STATE: Slot 0 down
*Feb 18 02:23:13.659: %CLEANAIR-6-STATE: Slot 1 down
*Feb 18 02:23:23.655: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.
*Feb 18 02:23:23.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: X.X.X.X peer_port: 5246
*Feb 18 02:23:23.263: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: X.X.X.X peer_port: 5246
*Feb 18 02:23:23.267: %CAPWAP-5-SENDJOIN: sending Join Request to X.X.X.X
*Feb 18 02:23:23.511: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Feb 18 02:23:23.959: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller mycontroller id 1, SSID mySSID, L2ACL , L2ACL AP
WLAN id 2, SSID my2ndSSID, L2ACL , L2ACL AP
WLAN id 3, SSID my3rdSSID, L2ACL , L2ACL AP
WLAN id 5, SSID my4thSSID, L2ACL , L2ACL AP
WLAN id 7, SSID mylastSSID, L2ACL , L2ACL AP
There is nothing of any interest at all before the "Switching to Standalone Mode" line and the APs re-associate immediately. We do not believe there is a WAN problem as usage graphs do not show anything out of the ordinary and not all APs at a site will disassociate at the same time.
Anyone got any ideas?
Thanks
Alan
