AP C9130AXI-E JOIN THE WLC AND GO BACK TO STANDALONE MODE AND DISJOIN

[FrejusMA]

Hello Team

I have some AP C9130AXI-E which are managed by a controller and were working very well.
And just a week ago I started seeing that they are no longer managed by the WLC, what is weird is that they continue to bradcast the SSIDs.
But somehow the SSIDs with PSK are working fine but the one with web auth are not. In the mean time I found out that the APs switched into standalone mode. 
My first question, is it possible for the APs to switch mode by themselves.
Also when you restart the AP, you see it clearly join the controller and after a few minutes it is no longer there.

MY WLC is an appliance which is installed on a virtual machine, my license is OK and I have some AP that joined and are working very well.

After analysing the log from the AP, I found the below and those logs just keep coming and coming:

[*11/23/2023 20:22:18.1952] Capwap payload encoding request.
[*11/23/2023 20:22:18.1952] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: DTLS Setup(3).
[*11/23/2023 20:22:18.5151] Received Capwap watchdog update msg.
[*11/23/2023 20:22:23.2637] Received Capwap watchdog update msg.
[*11/23/2023 20:22:28.0122] Received CAPWAP_WAIT_DTLS_TIMER_EXPIRY Capwap Timer Msg.
[*11/23/2023 20:22:28.0122] Wait DTLS timer has expired
[*11/23/2023 20:22:28.0122] Dtls session establishment failed
[*11/23/2023 20:22:28.0522]

[*11/24/2023 15:20:49.7917] No more AP manager addresses remain..
[*11/24/2023 15:20:49.7917] No valid AP manager found for controller 'WLC_NAME' (ip: WLC_IP)
[*11/24/2023 15:20:49.7917] Failed to join controller WLC_NAME.
[*11/24/2023 15:20:49.7917] Failed to join controller.

Futher log 

Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8128]
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8128] Encoded length 0 for payload: .Msg Elem Type: CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION(1048)
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8134] Encoded length 0 for payload: ...TLV Type: TLV_AP_EWLC_TAGS_PAYLOAD(1113)
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8134] encodeLen = 223.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8135] Ignoring duplicate discovery request to primary/secondary/tertiary controller 0. WLC_IP. Sent as part of mobility group member.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8135] Ignoring discovery to controller 1
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8135] Ignoring discovery to controller 2
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8143] send_ipc_msg: Slot id :3 invalid, range check 3
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8143]
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8143] Encoded length 0 for payload: .Msg Elem Type: CAPWAP_DOT11_MSGELE_WTP_RADIO_INFORMATION(1048)
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8148] Encoded length 0 for payload: ...TLV Type: TLV_AP_EWLC_TAGS_PAYLOAD(1113)
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8148] encodeLen = 223.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8150] Received Capwap watchdog update msg.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8158] [CAPWAP RX] CTRL: WLC_IP[5246] -> AP_IP[5272] len 145
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8158] Received Capwap Control Msg From AC.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8158] Rx unencrypted CAPWAP packet from WLC_IP
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8158] Received Capwap Control Msg.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8158] Control message: length = 145.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8158] Msg Type = CAPWAP_DISCOVERY_RESPONSE(2) Capwap State = Discovery(2).
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159] Discovery Response from WLC_IP
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159] Decode Discovery Response: WLC WLC_IP(5246) in DiscResp[0]
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159] Total msgEleLen = 129.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159] Not found entry 661 left 252 right 251 idx 251 iter 8
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159]
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159] Discovery Response received at (1700846470)
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8159]
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8160] << TLV table entries: 2 for TLV_EXPIRE_MIC_TYPE_PAYLOAD>>
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8160] Entry( 0): level 0 type 661 size 0 value Nested
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8160] Entry( 1): level 1 type 1 size 1 value 0
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8160]
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8160] << end of the TLV entry table>>
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8160] Do not allow expired MIC/SSC
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8163] Capwap control packet processed. Freeing packet 0x559ccad000.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8418] systemd[1]: Starting dhcpv6 client watcher...
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8536] systemd[1]: Stopping DHCPv6 client...
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8582] systemd[1]: Starting DHCPv6 client...
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.8938] systemd[1]: Started DHCPv6 client.
Nov 24 17:21:10 kernel: [*11/24/2023 17:21:10.9130] systemd[1]: Started dhcpv6 client watcher.
Nov 24 17:21:15 kernel: [*11/24/2023 17:21:15.4809] Received Capwap watchdog update msg.
Nov 24 17:21:21 kernel: [*11/24/2023 17:21:20.2326] Received CAPWAP_DISCOVERY_INTERVAL_EXPIRY Capwap Timer Msg.
Nov 24 17:21:21 kernel: [*11/24/2023 17:21:21.0000] Synchronizing time with AC time (1700846471) with delta (10)
Nov 24 17:21:21 kernel: [*11/24/2023 17:21:21.0003]
Nov 24 17:21:21 kernel: [*11/24/2023 17:21:21.0003] CAPWAP State: DTLS Setup
Nov 24 17:21:21 kernel: [*11/24/2023 17:21:21.0010] CAPWAP control packet sent to WLC_IP
Nov 24 17:21:21 kernel: [*11/24/2023 17:21:21.0011] Received Capwap watchdog update msg.
Nov 24 17:21:23 kernel: [*11/24/2023 17:21:23.8407] CDP CACHE msg received from click len 535
Nov 24 17:21:23 kernel: [*11/24/2023 17:21:23.8407]
Nov 24 17:21:23 kernel: [*11/24/2023 17:21:23.8407] Capwap payload encoding request.
Nov 24 17:21:25 kernel: [*11/24/2023 17:21:25.7509] Received Capwap watchdog update msg.
Nov 24 17:21:30 kernel: [*11/24/2023 17:21:30.5024] Received Capwap watchdog update msg.
Nov 24 17:21:35 kernel: [*11/24/2023 17:21:35.2542] Received Capwap watchdog update msg.
Nov 24 17:21:40 kernel: [*11/24/2023 17:21:40.0058] Received Capwap watchdog update msg.
Nov 24 17:21:44 kernel: [*11/24/2023 17:21:44.7574] Received Capwap watchdog update msg.
Nov 24 17:21:46 kernel: [*11/24/2023 17:21:46.3256] RRM INTERFERENCE DATA msg received from click len 1284
Nov 24 17:21:46 kernel: [*11/24/2023 17:21:46.3256]
Nov 24 17:21:46 kernel: [*11/24/2023 17:21:46.3256] Capwap payload encoding request.
Nov 24 17:21:46 kernel: [*11/24/2023 17:21:46.3257] RRM INTERFERENCE DATA msg received from click len 1284
Nov 24 17:21:46 kernel: [*11/24/2023 17:21:46.3257]
Nov 24 17:21:46 kernel: [*11/24/2023 17:21:46.3257] Capwap payload encoding request.
Nov 24 17:21:49 kernel: [*11/24/2023 17:21:49.5091] Received Capwap watchdog update msg.
Nov 24 17:21:54 kernel: [*11/24/2023 17:21:54.2607] Received Capwap watchdog update msg.
Nov 24 17:21:59 kernel: [*11/24/2023 17:21:59.0124] Received Capwap watchdog update msg.
Nov 24 17:22:02 kernel: [*11/24/2023 17:22:02.2845] RRM INTERFERENCE DATA msg received from click len 1284
Nov 24 17:22:02 kernel: [*11/24/2023 17:22:02.2846]
Nov 24 17:22:02 kernel: [*11/24/2023 17:22:02.2846] Capwap payload encoding request.
Nov 24 17:22:03 kernel: [*11/24/2023 17:22:03.7641] Received Capwap watchdog update msg.
Nov 24 17:22:08 kernel: [*11/24/2023 17:22:08.5159] Received Capwap watchdog update msg.
Nov 24 17:22:13 kernel: [*11/24/2023 17:22:13.2676] Received Capwap watchdog update msg.
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.0193] Received CAPWAP_WAIT_DTLS_TIMER_EXPIRY Capwap Timer Msg.
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.0412]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.0412] CAPWAP State: DTLS Teardown
Nov 24 17:22:18 upgrade: Script called with args:[ABORT]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.1245] upgrade.sh: Script called with args:[ABORT]
Nov 24 17:22:18 upgrade: Cleanup tmp files ...
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.1591] do ABORT, part1 is active part
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.1939] upgrade.sh: Cleanup tmp files ...
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2175] Received Capwap watchdog update msg.
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2178] Received MOM TUNNEL STATUS payload from click
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2178]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2178] MOM Payload - Version: 0 Status: 0
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2178] Capwap payload encoding request.
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2179] << TLV table entries: 3 for TLV_MOM_TUNNEL_STATUS_PAYLOAD>>
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2179] Entry( 0): level 0 type 2371 size 0 value Nested
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2179] Entry( 1): level 1 type 1 size 1 value 0
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2179] Entry( 2): level 1 type 2 size 1 value 0
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2179]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2179] << end of the TLV entry table>>
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] Received MOM TUNNEL STATUS payload from click
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] MOM Payload - Version: 1 Status: 0
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] Capwap payload encoding request.
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] << TLV table entries: 3 for TLV_MOM_TUNNEL_STATUS_PAYLOAD>>
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] Entry( 0): level 0 type 2371 size 0 value Nested
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] Entry( 1): level 1 type 1 size 1 value 1
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] Entry( 2): level 1 type 2 size 1 value 0
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.2180] << end of the TLV entry table>>
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.9339] CDP CACHE msg received from click len 535
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.9339]
Nov 24 17:22:18 kernel: [*11/24/2023 17:22:18.9339] Capwap payload encoding request.
Nov 24 17:22:22 kernel: [*11/24/2023 17:22:22.7709] Received CAPWAP_DTLS_SESSION_DELETE_TIMER_EXPIRY Capwap Timer Msg.
Nov 24 17:22:32 kernel: [*11/24/2023 17:22:32.8384] systemd[1]: Starting dhcpv6 client watcher...
Nov 24 17:22:32 kernel: [*11/24/2023 17:22:32.8500] systemd[1]: Stopping DHCPv6 client...
Nov 24 17:22:32 kernel: [*11/24/2023 17:22:32.8542] systemd[1]: Starting DHCPv6 client...
Nov 24 17:22:32 kernel: [*11/24/2023 17:22:32.8905] systemd[1]: Started DHCPv6 client.
Nov 24 17:22:32 kernel: [*11/24/2023 17:22:32.9117] systemd[1]: Started dhcpv6 client watcher.

Please help

[Leo Laohoo]

What is the exact firmware of the controller?

[FrejusMA]

He dear,

Software version is 8.10.142.0

Emergency image version is 8.9.100.0

Model : So as stated in my first message, the image is deployed on a virtual machine which serve as a controller

[Leo Laohoo]

What is the serial number of that AP?

[Rich R]

I think where Leo is going with this is - read FN-72424 (link below).
And I'll repeat again what I already said in my previous reply: Upgrade your WLC to 8.10.190.0 software as per the TAC recommended versions link below because 8.10.142.0 does not support the latest hardware revisions of the APs.

[marce1000]

 - Include controller model and software version , and check the controller logs too ,

 M.

[marce1000]

 - Added reply ; EWC capable APs may revert back to EWC-standalone mode , if a controller can not be found  , albeit being for a short while or not ; in that context visit this reply https://community.cisco.com/t5/wireless/cisco-ap-c9115axe-e-cupertino-17-09-04-firmware-change-to/m-p/4949837#M262227  . follow instructions to have them stay in CAPWAP mode all the time , 

 M.

[Rich R]

And make sure your WLC software is up to date as per the TAC recommended versions link below.

[MHM Cisco World]

Show wireless ap join summary 

Share this.

[FrejusMA · Accepted solution]

Hi team

It turned out I had a spanning tree issue on my network which was blocking the AP vlan. STP issues solved, everything got back to normal. The AP joined the WLC.

Thanks you guys for your interest in my problem