APs disassociating from 8540 controller

ALAN MURRAY — Mon, 05 Jul 2021 18:44:52 GMT

We have a pair of AIR-CT8540-K9 controllers running 8.5.135 with a mixture of 2702 and 3702 access points. There are 1061 APs on controller A and 984 on controller B. We are getting regular disassociation alerts generated by Prime for bunches of devices all at the same time as can be seen from this snippet of show ap uptime from one of the controllers:-

(mycontroller) >show ap uptime

Number of APs.................................... 1061
Global AP User Name.............................. myusername
Global AP Dot1x User Name........................ Not Configured

AP Name    Ethernet MAC AP    Up Time                           Association Up Time
------------------ ----------------- ----------------------- -----------------------
site1-AP-32 18:80:90:5c:a9:f4    92 days, 19 h 19 m 20 s    0 days, 02 h 16 m 29 s
site1-AP-23 18:80:90:a1:0e:7c   92 days, 18 h 02 m 45 s    0 days, 02 h 16 m 25 s
site1-AP-24 18:80:90:bb:42:d4   92 days, 17 h 41 m 12 s    0 days, 02 h 16 m 17 s
site1-AP-04 18:80:90:a2:ee:ac    92 days, 19 h 14 m 49 s 0 days, 02 h 12 m 18 s
site1-AP-06 18:80:90:bb:35:8c   92 days, 19 h 41 m 48 s    0 days, 01 h 15 m 30 s
site1-AP-09 18:80:90:a2:3e:f4    92 days, 20 h 30 m 36 s    0 days, 01 h 11 m 19 s
site1-AP-13 18:80:90:78:77:d4   92 days, 19 h 38 m 08 s    0 days, 01 h 09 m 54 s
site2-AP-12 40:ce:24:e8:0a:14   103 days, 19 h 37 m 44 s   0 days, 00 h 41 m 21 s
site1-AP-21 18:80:90:bb:44:d0    92 days, 17 h 53 m 08 s    0 days, 00 h 29 m 27 s
site1-AP-30 18:80:90:bb:42:ec    92 days, 18 h 20 m 00 s    0 days, 00 h 29 m 27 s
site1-AP-28 18:80:90:bb:42:d8   92 days, 18 h 05 m 22 s    0 days, 00 h 29 m 20 s
site1-AP-02 18:80:90:b4:f1:f8     92 days, 22 h 09 m 13 s    0 days, 00 h 29 m 09 s
site3-AP-22 40:ce:24:e8:08:4c 139 days, 07 h 33 m 45 s 0 days, 00 h 05 m 02 s

We have put the following debugging on:-

DTLS:
DTLS ERROR debugging is on
LWAPP:
LWAPP Client ERROR display debugging is on
CAPWAP:
CAPWAP Client AVC Netflow Error debugging is on
CAPWAP Client ERROR display debugging is on

And received the following output:-

*Feb 18 02:23:13.631: %LWAPP-3-CLIENTERRORLOG: Switching to Standalone mode
*Feb 18 02:23:13.647: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to X.X.X.X:5246
*Feb 18 02:23:13.655: %WIDS-6-DISABLED: IDS Signature is removed and disabled.
*Feb 18 02:23:13.659: %CLEANAIR-6-STATE: Slot 0 down
*Feb 18 02:23:13.659: %CLEANAIR-6-STATE: Slot 1 down
*Feb 18 02:23:23.655: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Feb 18 02:23:23.000: %CAPWAP-5-DTLSREQSEND: DTLS connection request sent peer_ip: X.X.X.X peer_port: 5246
*Feb 18 02:23:23.263: %CAPWAP-5-DTLSREQSUCC: DTLS connection created sucessfully peer_ip: X.X.X.X peer_port: 5246
*Feb 18 02:23:23.267: %CAPWAP-5-SENDJOIN: sending Join Request to X.X.X.X
*Feb 18 02:23:23.511: %LWAPP-4-CLIENTEVENTLOG: OfficeExtend Localssid saved in AP flash
*Feb 18 02:23:23.959: %CAPWAP-5-JOINEDCONTROLLER: AP has joined controller mycontroller id 1, SSID mySSID, L2ACL , L2ACL AP
WLAN id 2, SSID my2ndSSID, L2ACL , L2ACL AP
WLAN id 3, SSID my3rdSSID, L2ACL , L2ACL AP
WLAN id 5, SSID my4thSSID, L2ACL , L2ACL AP
WLAN id 7, SSID mylastSSID, L2ACL , L2ACL AP

There is nothing of any interest at all before the "Switching to Standalone Mode" line and the APs re-associate immediately. We do not believe there is a WAN problem as usage graphs do not show anything out of the ordinary and not all APs at a site will disassociate at the same time.

Anyone got any ideas?

Thanks

Alan

Re: APs disassociating from 8540 controller

patoberli — Fri, 21 Feb 2020 16:12:08 GMT

You might be hitting an AP bug, for example https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvf89335
For this reason I suggest you upgrade to the brand new 8.5.161.0 release (avoid 8.5.160.0!).

topic APs disassociating from 8540 controller in Wireless

APs disassociating from 8540 controller

Re: APs disassociating from 8540 controller