Select 3702i APs will not connect to new 9800 WLC

[JY01]

Hello all,

I'm running into a very specific issue; our team has migrated one of ours remote sites from a local 5508 WLC to a local 9800 WLC.

All is working as expected with the exception of 3 APs model 3702i (that we know of) not forming the CAPWAP DTLS connection with the new 9800 WLC. Here are some things to note;

• 9800 WLC is running IOS-XE version 17.3.6
• There are currently 9 other 3702i-Z-K9 APs connected to this new 9800 WLC.
• License configuration is set to "Airgap" as opposed to "Directly Connected" with the TokenID
• APs obtain IPs from a DHCP server. Option 43 and the correct HEX value for the new WLC has been configured on the DHCP server and confirmed.
• Country code has been properly configured on WLC for Brazil.
• APs and WLC are on the same VLAN configured for "Local" mode.
• APs are going through a rigorous cycle of 1. Discovering the WLC, 2. Successfully "joining" the WLC, 3. DTLS handshake failures with unclear logs as follows;

AP Delete processing failed ap ctxt handle is Invalid

Record already deleted and recreated. Session ID in AP context 0x0 does not match with session 0xab980009b70010c3 being deleted

CAPWAP DTLS session closed for AP, cause: DTLS handshake error

Failed to override default values inradio oper for slot 1, reg domain chk status failed

Unable to fetch wtp session to update AP name

Keep in mind, these logs are pretty identical across these 3 APs that cannot join this new 9800 WLC. Has anyone else run into this issue? I'm holding off on contacting TAC based on these symptoms, but would like to know if the community has seen this type of issue. Thank you in advance.

[Leo Laohoo]

Console into the AP and post the complete output to the command "sh version".

[marce1000]

  - Have a checkup of the 9800 WLC configuration with the CLI command show tech wireless ; feed the output into :
                                                                                   https://cway.cisco.com/wireless-config-analyzer/

 M.
                                      

[balaji.bandi]

as suggesting posting the console full output help us to identify and suggest what to be done.

in my case we have same issue some AP moving from airos to ios xe - console to AP do below and check.

# clear lwapp private-config or #clear capwap private-config

[Rich R]

Are you sure your 3702's are all the exact same model number?
> Failed to override default values inradio oper for slot 1, reg domain chk status failed
That errors suggests the 3 that are failing are the wrong regulatory domain for Brazil.

[JY01]

@Leo Laohoo  @balaji.bandi awaiting for our site contact to acquire console access to acquire those outputs. Thanks!

@marce1000will review that aspect as well, thank you sir!

@Rich RI thought the same thing, but there are 12 total 3702i APs of the same exact model for this location. 9 out of the 12 are registered to the new 9800 WLC with no issues and with the proper regulatory domain, it's the 3 outliers that are proving to be tedious. The suggestions above are what I'll go through and I will advise if we get anywhere with the outputs.

[JY01]

@All Just wanted to provide an update on this matter.

I ended up creating a TAC case for this issue, and they were able to conclude the AP was stuck in the download phase from being migrated from the previous AireOS WLC. When consoled into the AP, log messages continued to show success and failures all within minutes of the AP attempting to download the new firmware.

Also, when observing the join messages exchanged between the AP and WLC this log was also observed towards the tailend of each discovery completion and failure;

The certificate (SN: 4E78A210000000000007) has expired. Validity period ended on 21:43:46 UTC Dec 4 2022

As some of us know this is a known bug which requires removing the NTP server configuration and rolling the clock backwards prior to the date listed in order for APs to join. This is something I'm not willing to try as the 9800 WLC is currently in production and I wouldn't want to tamper with something else during that attempted test. As a workaround, we ended up replacing that AP to continue serving Wireless connections for this area but the next advisement is to manually upload the appropriate firmware to the AP itself via TFTP which appears to work each time.

I thank you all for your efforts to assist me on this matter, and I look forward to future posts that can help the entire community with issues like this!

[Leo Laohoo]

---

@JY01 wrote:
they were able to conclude the AP was stuck in the download phase

---

And I've got a fix that does not require an AP replacement.  The most important part of this fix/workaround is remote access to the AP.  

If the 9800 is on 17.3.6 then the corresponding firmware for the 2700/3700 should be the file "ap3g2-rcvk9w8-tar.153-3.JPJ9.tar".  

Remote into the AP and do the following: 

debug capwap console cli
delete /f /r flash:ap3g2*
archive tar /x tftp://<TFTP IP ADDRESS>/ap3g2-rcvk9w8-tar.153-3.JPJ9.tar flash:

After the file has been unpacked, reboot the AP.   

IF the 2700/3700 have not yet migrated off the AirOS, I have another workaround using similar process/commands.  

[JY01]

Duly noted, thank you! I will try this and provide feedback.