9800-CL - All AP's are reconnecting every two minutes

[philipb232]

Hi, I just getting started with Cisco wireless devices and I setup a Catalyst 9800-CL VM (17.3.6) and a few AP's (AIR-CAP2702I-E-K9) in my lab. The AP's are joining my controller but after two minutes they just rejoin and I have no clue why.

The log output of one of the AP's:

*Feb 7 12:02:27.027: %CAPWAP-3-EVENTLOG: Echo Request sent to 192.168.200.2
*Feb 7 12:02:27.027: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:02:27.027: %CAPWAP-3-EVENTLOG: Queue Empty.
*Feb 7 12:02:27.027: %CAPWAP-3-EVENTLOG: Echo Response from 192.168.200.2
*Feb 7 12:02:27.000: %CAPWAP-3-EVENTLOG: Setting time to 12:02:27 UTC Feb 7 2023

*Feb 7 12:02:32.415: %DOT11-6-DFS_SCAN_COMPLETE: DFS scan complete on frequency 5680 MHz
*Feb 7 12:02:36.271: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:02:36.271: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:02:36.271: %CAPWAP-3-EVENTLOG: Queue Empty.
*Feb 7 12:02:36.271: %CAPWAP-3-EVENTLOG: Wtp Event Response from 192.168.200.2
*Feb 7 12:02:39.259: %CAPWAP-3-EVENTLOG: Unencrypted Data keep-alive: Transmitted to 192.168.200.2:5247
*Feb 7 12:02:39.259: %CAPWAP-3-EVENTLOG: Send Data keep-alive: 3 1.
*Feb 7 12:02:39.259: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for retransmission in 3 sec, sec_drop_count=0
*Feb 7 12:02:39.259: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for transmission in 30 sec.
*Feb 7 12:02:39.259: %CAPWAP-3-EVENTLOG: Data keep-alive: round-trip time=0.000 sec
*Feb 7 12:02:39.271: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:02:39.271: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:02:39.271: %CAPWAP-3-EVENTLOG: Queue Empty.
*Feb 7 12:02:39.271: %CAPWAP-3-EVENTLOG: Wtp Event Response from 192.168.200.2
*Feb 7 12:03:09.259: %CAPWAP-3-EVENTLOG: Unencrypted Data keep-alive: Transmitted to 192.168.200.2:5247
*Feb 7 12:03:09.259: %CAPWAP-3-EVENTLOG: Send Data keep-alive: 3 1.
*Feb 7 12:03:09.259: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for retransmission in 3 sec, sec_drop_count=0
*Feb 7 12:03:09.259: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for transmission in 30 sec.
*Feb 7 12:03:09.259: %CAPWAP-3-EVENTLOG: Data keep-alive: round-trip time=0.000 sec
*Feb 7 12:03:09.271: %CAPWAP-3-EVENTLOG: Echo Interval Expired.
*Feb 7 12:03:09.271: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:03:09.271: %CAPWAP-3-EVENTLOG: Echo Request sent to 192.168.200.2
*Feb 7 12:03:09.271: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:03:19.095: %CAPWAP-3-EVENTLOG: No changes in lwapp configs
*Feb 7 12:03:29.951: %CAPWAP-3-EVENTLOG: Primary Discovery Request sent to 255.255.255.255
*Feb 7 12:03:29.951: %CAPWAP-3-EVENTLOG: IPv6 source address :: for multicast primary discovery request

*Feb 7 12:03:29.951: %CAPWAP-3-EVENTLOG: Failed to send packet to destination FF01::18C
*Feb 7 12:03:29.951: %CAPWAP-3-EVENTLOG: Primary Discovery Request sent to FF01::18C
*Feb 7 12:03:29.955: %CAPWAP-3-EVENTLOG: Primary Discovery Response from 192.168.200.2
*Feb 7 12:03:29.955: %CAPWAP-3-EVENTLOG: Primary Discovery response from MWAR ''running version 17.3.6.76.
*Feb 7 12:03:29.955: %CAPWAP-3-EVENTLOG: FOUND CONFIGURED MWAR(Primary) REDISCOVER TO CONNECT WITH THAT.
*Feb 7 12:03:29.955: %DTLS-5-SEND_ALERT: Send FATAL : Close notify Alert to 192.168.200.2:5246
*Feb 7 12:03:29.955: %CAPWAP-3-EVENTLOG: CAPWAP State: DTLS Teardown.
*Feb 7 12:03:30.687: %CAPWAP-3-EVENTLOG: LRAD state down. Skip sending PHY_TX_POWER_LEVEL_PAYLOAD
*Feb 7 12:03:30.687: %CAPWAP-3-EVENTLOG: Discarding msg type 9 in CAPWAP state: 4.
*Feb 7 12:03:30.687: %CAPWAP-3-EVENTLOG: Configuration update for Power Mode sent to192.168.200.2

*Feb 7 12:03:30.687: %DOT11-5-EXPECTED_RADIO_RESET: Restarting Radio interface Dot11Radio1 due to the reason code 10
*Feb 7 12:03:30.711: %CAPWAP-3-EVENTLOG: Starting Discovery. Initializing discovery latency in discovery responses.
*Feb 7 12:03:30.711: %CAPWAP-3-EVENTLOG: CAPWAP State: Discovery.
*Feb 7 12:03:30.711: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 192.168.200.1
*Feb 7 12:03:30.711: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - gateway found 192.168.200.1
*Feb 7 12:03:30.711: %CAPWAP-3-EVENTLOG: spamResolveStaticGateway - Adding default route for gateway 192.168.200.1
*Feb 7 12:03:32.099: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Feb 7 12:03:32.107: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Feb 7 12:03:32.139: %CAPWAP-3-EVENTLOG: LRAD state down. Skip sending PHY_TX_POWER_LEVEL_PAYLOAD
*Feb 7 12:03:32.143: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Feb 7 12:03:32.151: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to down
*Feb 7 12:03:32.159: %LINK-5-CHANGED: Interface Dot11Radio0, changed state to reset
*Feb 7 12:03:33.143: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Feb 7 12:03:33.151: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Feb 7 12:03:33.175: %CAPWAP-3-EVENTLOG: LRAD state down. Skip sending PHY_TX_POWER_LEVEL_PAYLOAD
*Feb 7 12:03:33.179: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Feb 7 12:03:34.179: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: Calling wtpGetAcToJoin from timer expiry.
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: !mwarname
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: !mwarname
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: !mwarname
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: Selected MWAR '' (index 0).
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: Ap mgr count=1
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: Controller: . ApMgr count is 1 ipTransportTried 0 prefer-mode 1

*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: Adding Ipv4 AP manager 192.168.200.2 to least load
*Feb 7 12:03:40.711: %CAPWAP-3-EVENTLOG: IPv4 Pref mode. Choosing AP Mgr with index 0, IP = 192.168.200.2, load = 28 ap ip: (192.168.200.103)
*Feb 7 12:03:40.711: AP has SHA2 MIC certificate - Using SHA2 MIC certificate for DTLS.

*Feb 7 12:03:41.995: %CAPWAP-3-EVENTLOG: Send channel power and PhyTxPowerLevel message sent to192.168.200.2

*Feb 7 12:03:41.995: %CAPWAP-3-EVENTLOG: Configuration update response sent to 192.168.200.2
*Feb 7 12:03:41.995: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:03:41.995: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:03:41.995: %CAPWAP-3-EVENTLOG: Wtp Event Response from 192.168.200.2
*Feb 7 12:03:41.999: %CAPWAP-3-EVENTLOG: Configuration update request from 192.168.200.2
*Feb 7 12:03:41.999: %CAPWAP-3-EVENTLOG: Configuration update response sent to 192.168.200.2
*Feb 7 12:03:41.999: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:03:41.999: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:03:43.335: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Feb 7 12:03:43.379: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to down
*Feb 7 12:03:43.411: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:03:43.411: %CAPWAP-3-EVENTLOG: Send channel power and PhyTxPowerLevel message sent to192.168.200.2

*Feb 7 12:03:43.411: %DOT11-6-DFS_SCAN_START: DFS: Scanning frequency 5680 MHz for 60 seconds.
*Feb 7 12:03:43.411: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:03:43.411: %CAPWAP-3-EVENTLOG: Queue Empty.
*Feb 7 12:03:43.411: %CAPWAP-3-EVENTLOG: Wtp Event Response from 192.168.200.2
*Feb 7 12:03:43.415: %LINK-6-UPDOWN: Interface Dot11Radio1, changed state to up
*Feb 7 12:03:44.291: %CAPWAP-3-EVENTLOG: NOT Sending Data keep-alive first time: 6 1
*Feb 7 12:03:44.291: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for retransmission in 6 sec, sec_drop_count=0
*Feb 7 12:03:44.415: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio1, changed state to up
*Feb 7 12:03:44.423: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to down
*Feb 7 12:03:44.447: %CAPWAP-3-EVENTLOG: Sending packet to AC

*Feb 7 12:03:44.447: %CAPWAP-3-EVENTLOG: Send channel power and PhyTxPowerLevel message sent to192.168.200.2

*Feb 7 12:03:44.451: %LINK-6-UPDOWN: Interface Dot11Radio0, changed state to up
*Feb 7 12:03:44.451: %CAPWAP-3-EVENTLOG: Resetting reTransmissionCnt to 0
*Feb 7 12:03:44.451: %CAPWAP-3-EVENTLOG: Queue Empty.
*Feb 7 12:03:45.451: %LINEPROTO-5-UPDOWN: Line protocol on Interface Dot11Radio0, changed state to up
*Feb 7 12:03:50.291: %CAPWAP-3-EVENTLOG: Unencrypted Data keep-alive: Transmitted to 192.168.200.2:5247
*Feb 7 12:03:50.291: %CAPWAP-3-EVENTLOG: Send Data keep-alive: 12 1.
*Feb 7 12:03:50.291: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for retransmission in 12 sec, sec_drop_count=0
*Feb 7 12:03:50.291: %CAPWAP-3-EVENTLOG: Data keep-alive: schedule for transmission in 30 sec.
*Feb 7 12:03:50.291: %CAPWAP-3-EVENTLOG: Data keep-alive: round-trip time=0.000 sec
*Feb 7 12:03:50.291: %CAPWAP-3-EVENTLOG: Sending FT Client Info Payload for IOS controller
*Feb 7 12:03:57.011: %CAPWAP-3-EVENTLOG: Sending packet to AC

[marce1000]

 - For starters have a checkup of the current controller configuration  with the CLI command : show  tech   wireless , have the output analyzed by  https://cway.cisco.com/tools/WirelessAnalyzer/  , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer.               Checkout all advisories!

                      Also use the following commands below to get more insights
                            show wireless stats ap join summary
                            show wireless dtls connections
                            show platform hardware chassis active qfp feature wireless capwap datapath statistics drop all
                            show platform hardware chassis active qfp feature wireless dtls datapath statistics all 
                            show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> details
                           show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac>
statistics
                            show platform hardware chassis active qfp feature wireless wlclient datapath statistics drop all 

   And last but not least check the controller logs too

 M.

[Leo Laohoo]

Wait until Cisco WLBU releases 17.9.3.  Make sure to read the Release Notes. 

That is all I am going to say.

[marce1000]

  - @Leo Laohoo  2700 series are being used and only supported up to 17.3.x (no later) , 

 M.

[marce1000]

  - Additional debugging info's can be found here :
               https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin   (corrected I hope)

 M.

 M.

[Scott Fella]

Its very hard to say what is wrong without actually seeing how things are configured.  There have been a few folks that have ran into issues and it was due to not following the steps when you setup your hypervisor.  Search the forum to find more, but here is one I found during my search:

Solved: Can't join 2702i to C9800-CL WLC - Cisco Community

[Rich R]

@marce1000 what @Leo Laohoo is alluding to was actually announced at Cisco Live Amsterdam today so it's official and public now - the Wave 1 APs (x700) will be supported in 17.9 from 17.9.3 to help customers with migrations to 9800 without being stuck on old 17.3 IOS-XE!

[Scott Fella]

Hopefully they post it soon, at least I don't see the code yet.

[David Ritter]

Sometimes, after resetting the WLC clock and turning off NTP,  The AP continues to restert/power up with the current date, rather than the 'temp' controller date, thus still flagging cert expiration. fail.  a big pain even in AIRos environments.

[Rich R]

"will be" ... not out yet @Scott Fella 
They're saying tentatively March to early April but as you probably know these dates can shift

[philipb232]

I tried a fresh installation of the controller with just a dumb switch, one AP and still have this issue. So, I think it must be a hypervisor (Hyper-V) problem even I followed the instructions. Need to test it with a different one, I guess.

The only log/analyzer output that showed something special was the controller log:

Feb 7 21:59:44.392: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 192.168.88.249[59364] Mac: 00a2.ee06.d860 CAPWAP DTLS session closed for AP, cause: DTLS server session error

Feb 7 21:57:53.292: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP843d.c64e.7e34, MAC: 843d.c64e.7e34 Joined

Feb 7 21:57:33.342: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: Session-IP: 192.168.88.249[59364] Mac: 00a2.ee06.d860 CAPWAP DTLS session closed for AP, cause: DTLS server session error

Feb 7 21:55:42.249: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP Name: AP843d.c64e.7e34, MAC: 843d.c64e.7e34 Joined

[Scott Fella]

Keep us posted on what fixed the issue.

[philipb232]

Is it possible that APSP5 and APSP6 resolve the problem? I can't download them do to the fact that I only run a homelab and don't have a service contract.

[Scott Fella]

I don't know what you mean.... even if you don't have a service contract, the image for the 9800-CL is free to download.  I have run the 9800-CL on both Hyper-V and VMware with no issues, of course I was running them in FlexConnect mode.  Look at the ap console output and see if you see anything that tells you why it is disassociating.