"AIR-AP2802I-B-K9" APs Joining/disconnecting from WLC "C9800_IOSXE-K9"

[ktawfik]

Hi all,

I'm trying to connect new APs to WLC at a new location and the APs are joining and seconds later they are disconnecting from the WLC. When I consoled into one of the APs it's showing the output I have attached, below is part of what I'm seeing on the AP, there's more in the attachment, I also attached the logging from the WLC too. We have been working with TAC since Monday but they keep asking us to run some show commands on the WLC and then escalate and they haven't been much help. Any insight would be appreciated! Thank you!

AP log,

[*05/12/2023 20:04:10.0004] CAPWAP State: DTLS Setup
[*05/12/2023 20:04:10.0498] dtls_verify_server_cert: Controller certificate verification successful
[*05/12/2023 20:04:10.4871]
[*05/12/2023 20:04:10.4871] CAPWAP State: Join
[*05/12/2023 20:04:10.5086] Sending Join request to 10.65.93.4 through port 5248
[*05/12/2023 20:04:14.5272] Discarding msg CAPWAP_WTP_EVENT_REQUEST(type 9) in CAPWAP state: Join(5).
[*05/12/2023 20:04:15.1264] Sending Join request to 10.65.93.4 through port 5248
[*05/12/2023 20:04:15.1301] Join Response from 10.65.93.4
[*05/12/2023 20:04:15.1301] AC accepted join request with result code: 0
[*05/12/2023 20:04:15.1475] Received wlcType 0, timer 30
[*05/12/2023 20:04:15.2728]
[*05/12/2023 20:04:15.2728] CAPWAP State: Image Data
[*05/12/2023 20:04:15.2732] AP image version 17.6.3.33 backup 0.0.0.0, Controller 17.6.3.33
[*05/12/2023 20:04:15.2732] Version is the same, do not need update.
[*05/12/2023 20:04:15.3141] status 'upgrade.sh: Script called with args:[NO_UPGRADE]'
[*05/12/2023 20:04:15.3762] do NO_UPGRADE, part2 is active part
[*05/12/2023 20:04:15.3831]
[*05/12/2023 20:04:15.3831] CAPWAP State: Configure
[*05/12/2023 20:04:16.2726] DOT11_CFG[1]: Starting radio 1
[*05/12/2023 20:04:16.8602] DOT11_DRV[1]: Started Radio 1
[*05/12/2023 20:04:16.8608] DOT11_CFG[0]: Starting radio 0
[*05/12/2023 20:04:17.4606] DOT11_DRV[0]: Started Radio 0
[*05/12/2023 20:04:17.8058]
[*05/12/2023 20:04:17.8059] CAPWAP State: Run
[*05/12/2023 20:04:17.8453] AP has joined controller Nav-3rdBLDG-WLC
[*05/12/2023 20:04:20.6397] Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=18, NumofPendingMsgs=14
[*05/12/2023 20:04:20.6397]
[*05/12/2023 20:04:23.4905] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=19, NumofPendingMsgs=15
[*05/12/2023 20:04:23.4905]
[*05/12/2023 20:04:26.3415] Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=19, NumofPendingMsgs=15
[*05/12/2023 20:04:26.3415]
[*05/12/2023 20:04:29.1923] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=19, NumofPendingMsgs=15
[*05/12/2023 20:04:29.1923]
[*05/12/2023 20:04:32.0432] Re-Tx Count=5, Max Re-Tx Value=5, SendSeqNum=19, NumofPendingMsgs=15
[*05/12/2023 20:04:32.0432]
[*05/12/2023 20:04:34.8941] Max retransmission count exceeded, going back to DISCOVER mode.

WLC sh commands,

Nav-3rdBLDG-WLC#show wireless stats ap mac "MAC" join detailed
Discovery phase statistics
Discovery requests received : 3913
Successful discovery responses sent : 3913
Unsuccessful discovery request processing : 0
Reason for last unsuccessful discovery attempt : Non-wireless Mgmt interface
Time at last successful discovery attempt : 05/12/2023 20:25:10
Time at last unsuccessful discovery attempt : 05/12/2023 09:43:23

Control Dtls phase statistics
Dtls session requests received : 1956
Established Dtls session : 1953
Unsuccessful Dtls session : 3
Reason for last unsuccessful dtls session : DTLS Timer expired
Time at last successful dtls session : 05/12/2023 20:25:20
Time at last unsuccessful dtls session : 05/12/2023 05:31:24

Join phase statistics
Join requests received : 1952
Successful join responses sent : 1951
Unsuccessful join request processing : 1
Reason for last unsuccessful join attempt : Heart beat timer expiry
Time at last successful join attempt : 05/12/2023 20:25:25
Time at last unsuccessful join attempt : 05/11/2023 21:37:29

Configuration phase statistics
Configuration requests received : 5853
Successful configuration responses sent : 5853
Unsuccessful configuration request processing : 0
Reason for last unsuccessful configuration attempt : NA
Time at last successful configuration attempt : 05/12/2023 20:25:26
Time at last unsuccessful configuration attempt : NA

Data Dtls phase statistics
Dtls session requests received : 0
Established Dtls session : 0
Unsuccessful dtls session : 0
Reason for last unsuccessful dtls session : NA
Time at last successful dtls session : NA
Time at last unsuccessful dtls session : NA

Last AP message decryption failure details
Reason for last message decryption failure : NA

AP reported disconnect detail
Disconnect reason from AP : NA
AP reported reboot detail
Reboot reason from AP : No reboot reason

Last AP disconnect details
Last Disconnect Phase : Join
Last Disconnect Reason : Heart beat timer expiry
Last Disconnect Time : 05/12/2023 09:45:04
Current Join Status : Not Joined

[marce1000]

  - 1)  What is the controller software version ; have a go with 17.9.3 (e.g.) which now also supports the older Wave1 APs again
     2) Have a checkup of the controller configuration with the CLI command show tech wireless ; have the output analyzed with https://cway.cisco.com/wireless-config-analyzer
    3) Use these tools for analyzing AP join issues : https://logadvisor.cisco.com/logadvisor/wireless/9800/9800APJoin

  Appendix : a number of other commands related to analyzing DTLS  (AP joining problems)
show wireless stats ap join summary
show wireless dtls connections
show platform hardware chassis active qfp feature wireless capwap datapath statistics drop all
show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> details
show platform hardware chassis active qfp feature wireless capwap datapath mac-address <APradio-mac> statistics
show platform hardware chassis active qfp feature wireless dtls datapath statistics all
show platform hardware chassis active qfp statistics drop all | inc Global | Wls (Data Plane Statistics – Global Wireless Drops)
show tunnel eogre manager stats instance 0
show ap name APa80c.0dd2.1fa8 tag detail
show capwap client config
show capwap detailed
show capwap summary

 M.


 

[Rich R]

1. Why are you using 17.6.3 - should be at least 17.6.5?  See TAC recommended below.
2. Have you tried factory default reset of the AP(s)?
3. What is the EXACT model of the 9800 WLC?
4. Your logging buffer on the WLC needs to be 200K minimum, even 2M to be useful - 4K is just a waste of time - you lose crucial logs. "logging buffered 2000000 debug"
5. Did you run a radioactive trace on the AP MAC address?
6. "Max retransmission count exceeded, going back to DISCOVER mode." - are you sure there's no packet loss between the AP and WLC?  Speed/duplex mismatches or erroring port?  You didn't attach the full log (from power-on) of the AP so it's not clear if it's on the same subnet as the WLC or routed?

[ktawfik]

1. Why are you using 17.6.3 - should be at least 17.6.5?  Will do an upgrade and see if that helps.
2. Have you tried factory default reset of the AP(s)? Yes, that didn't fix it.
3. What is the EXACT model of the 9800 WLC? Cisco C9800-L-C-K9 Chassis
4. Your logging buffer on the WLC needs to be 200K minimum, even 2M to be useful - 4K is just a waste of time - you lose crucial logs. "logging buffered 2000000 debug" I adjusted the buffer.
5. Did you run a radioactive trace on the AP MAC address? Yes, Attached for one of the APs (from power on)
6. "Max retransmission count exceeded, going back to DISCOVER mode." - are you sure there's no packet loss between the AP and WLC? No packet loss Speed/duplex mismatches or erroring port?  No packet loss or speed/duplex mismatches

[ktawfik]

AP radioactive trace attached

[marce1000 · Accepted solution]

 - Check if the AP regulatory domain  and controller configured country code(s) match

           >.... Will do an upgrade and see if that helps.
                      Have go with  IOS-XE 17.9.3

 M.

[Rich R · Accepted solution]

> reg domain chk status failed
> Failed to send capwapcountry code update to AP
> AP model: AIR-AP2802E-B-K9

What country is the WLC configured for, and what country is the AP being used in?

Also worth checking the AP serial number using the Serial Number validation tool just to make sure it's not affected by:
https://www.cisco.com/c/en/us/support/docs/field-notices/704/fn70479.html
The link to the tool is right near the end of the FN below the photos showing where to check the serial number.
However I do not see the exact error message mentioned in the FN so probably not related to this manufacturing issue.

[ktawfik]

The upgrade to 17.9.3 did it! Thank you so much Marce and Rich for your help!!!