AP C9115AXI-A disassociates from the virtualized C9800-L WLC

[Electronic20]

Hi Community:

I have some AP C9115AXI-A that detaches for almost 2 minute and re-associates to the WLC, intermittently at any time for about 2 to 3 times a day.

I performed a debug capture and it showed me this message:

2024/02/08 15:32:42.637965694 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:42.637972049 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:42.638042630 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:42.638044673 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:42.638219294 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:42.638221430 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:42.641548325 {wncmgrd_R0-0}{1}: [capwapdata-keepalive] [16701]: (ERR): a00f.372f.3a20 Loadbalancing data keep-alive message failed
2024/02/08 15:32:42.641576915 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:42.641579003 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:45.483303126 {wncmgrd_R0-0}{1}: [capwapdata-keepalive] [16701]: (ERR): a00f.372f.3a20 Loadbalancing data keep-alive message failed
2024/02/08 15:32:45.483623666 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:45.483626730 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:48.334518369 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:48.334522061 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:51.186436900 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:51.186440732 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:51.186877051 {wncmgrd_R0-0}{1}: [capwapdata-keepalive] [16701]: (ERR): a00f.372f.3a20 Loadbalancing data keep-alive message failed
2024/02/08 15:32:54.037390675 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:54.037394695 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:56.888737132 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:56.888741140 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:32:59.901085508 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (info): Dropping DTLS pkt, not a client-hello message
2024/02/08 15:32:59.901089085 {wncmgrd_R0-0}{1}: [capwapac-srvr] [16701]: (ERR): Message dispatching failed. Source IP: 172.16.240.60[5273]
2024/02/08 15:33:14.545276469 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 IP:172.16.240.60[5273], Discovery Request received
2024/02/08 15:33:14.545295051 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 EWLC TAGS payload is not present in discovery request
2024/02/08 15:33:14.545376179 {wncmgrd_R0-0}{1}: [ewlc-infra-evq] [16701]: (debug): instance :0 port:17931MAC: 4a43.3236.3136
2024/02/08 15:33:14.545602753 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 Public IP learnt is FALSE, public IP discovery is FALSE, private IP discovery is TRUE.
2024/02/08 15:33:14.545671472 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 IP:172.16.240.60[5273], Discovery Response sent
2024/02/08 15:33:14.545995726 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 IP:172.16.240.60[5273], Discovery Request received
2024/02/08 15:33:14.546020395 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 EWLC TAGS payload is not present in discovery request
2024/02/08 15:33:14.546051899 {wncmgrd_R0-0}{1}: [ewlc-infra-evq] [16701]: (debug): instance :0 port:17931MAC: 4a43.3236.3136
2024/02/08 15:33:14.546086803 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 Public IP learnt is FALSE, public IP discovery is FALSE, private IP discovery is TRUE.
2024/02/08 15:33:14.546129565 {wncmgrd_R0-0}{1}: [capwapac-discovery] [16701]: (note): MAC: a00f.372f.3a20 IP:172.16.240.60[5273], Discovery Response sent

15:33:23.983688444 {wncd_x_R0-0}{1}: [ewlc-dtls-sessmgr] [17093]: (info): Remote Host: 172.16.240.60[5273] MAC: a00f.372f.3a20 Completed cert verification, status:CERT_VALIDATE_SUCCESS
2024/02/08 15:33:24.317792833 {wncd_x_R0-0}{1}: [ewlc-infra-evq] [17093]: (debug): DTLS record type: 22, handshake
2024/02/08 15:33:24.318134311 {wncd_x_R0-0}{1}: [ewlc-dtls-sess] [17093]: (info): Remote Host: 172.16.240.60[5273] MAC: a00f.372f.3a20 DTLS session established
2024/02/08 15:33:24.318547818 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (info): MAC: a00f.372f.3a20 PMTU 576, effective capwap size 488, DTLS overhead len: 37
2024/02/08 15:33:24.318556013 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [17093]: (info): Mac: Session-IP: 172.16.240.60[5273] 172.16.240.10[5246] DTLS session has been established for AP
2024/02/08 15:33:29.038825685 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (info): Mac: Session-IP: 172.16.240.60[5273] 172.16.240.10[5246] Capwap message received, type: join_request
2024/02/08 15:33:29.038827430 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (note): MAC: a00f.372f.3a20 Received CAPWAP join request

2024/02/08 15:33:29.862251308 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [17093]: (info): Mac: a00f.372f.3a20 Session-IP: 172.16.240.60[5273] 172.16.240.10[5246] capwap control message window size 1
2024/02/08 15:33:29.862277440 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (info): MAC: a00f.372f.3a20 Capwap IFID 9000008d allocation is successful
2024/02/08 15:33:29.862633416 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (note): MAC: a00f.372f.3a20 Create Capwap plumb send success, IFID 9000008d
2024/02/08 15:33:29.862710595 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (note): MAC: a00f.372f.3a20 CAPWAP-IDB: Tunnel create send success. IFID : 9000008d
2024/02/08 15:33:29.862719666 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [17093]: (note): MAC: a00f.372f.3a20 AP Name: PISO-05-AP12 Mac: a00f.372f.3a20 Session-IP: 172.16.240.60[5273] 172.16.240.10[5246] Ethernet MAC: a00f.3712.2744 Joined

2024/02/08 15:33:29.864433726 {wncd_x_R0-0}{1}: [errmsg] [17093]: (note): %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: R0/0: wncd: AP Event: AP Name: PISO-05-AP12 Mac: a00f.372f.3a20 Session-IP: 172.16.240.60[5273] 172.16.240.10[5246] Ethernet MAC: a00f.3712.2744 Joined

2024/02/08 15:33:29.864485346 {wncd_x_R0-0}{1}: [capwapac-smgr-sess-fsm] [17093]: (info): Mac: a00f.372f.3a20 Session-IP: 172.16.240.60[5273] 172.16.240.10[5246] CAPWAP Data session established

In the WLL log via cli I get this:

The IP of the AP is 172.16.240.60
The IP of the WLC is 172.16.240.10

The network and PoE issue on the switch is discarded, the AP does not restart.

Please give us your support to solve this problem.

[Daniel Ordóñez Flores]

Hi @Electronic20 

What do you see from the AP side?

[Electronic20]

Hi @Daniel Ordóñez Flores 

I could not capture the trunks within the AP, I only manage them from the WLC.

Based on the logs I showed, could we know what causes this problem?

[balaji.bandi]

2024/02/08 15:32:45.483303126 {wncmgrd_R0-0}{1}: [capwapdata-keepalive] [16701]: (ERR): a00f.372f.3a20 Loadbalancing data keep-alive message failed

Can you provide more information about the environment ?

is the WLC connected single port or dual ports ? (if the dual port for testing, shutdown one of the port and observe ?)

is the only 1 AP having issue or all the AP ?

[Electronic20]

Hi @balaji.bandi 

This is the topology

The WLC is virtualized and only one port is used.

This disassociation of the AP from the WLC randomly happens to any AP at any time, but it is for about 2 minutes and without doing anything the AP re-registers with the WLC after 2 minutes. It usually happens 2 or 3 times a day.

[balaji.bandi]

what IOS XE code running on vWLC. how long the WLC uptime, try 17.9.4a (upgrading and check).

Hope there is no issue in the path any network issues ? Do you see any drops on Access switch or Trunk port connected to CORE and virtual environment ?