Wave1 APs (2700i, 3700i) not able to join 9800-L 17.9.4a

[mqontt]

Hey Guys,

i've ran into some strange issue where i was able to migrate all 2800 series APs from old AirOS WLC (5508 8.4.100) to 9800-L version 17.9.4a

However now when i tried to move the 2700 and 3700 series, they are not able to join the WLC and i have no clue what could be the reason ( from my perspective it seems like REG domain mismatch, but that doesnt make sense since the APs are in the same reg domain as the new WLC)

What i can see in RA_traces:

2024/02/19 13:52:42.367474589 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [16619]: (note): MAC: a89d.2143.xxxx Received CAPWAP join request

2024/02/19 13:52:42.368014919 {wncd_x_R0-0}{1}: [apmgr-db] [16619]: (ERR): a89d.2143.xxxx Failed to get radio oper record. No Radio record for given slot=1

2024/02/19 13:52:42.368016081 {wncd_x_R0-0}{1}: [apmgr-db] [16619]: (ERR): Failed to set external antenna gain. Band record retrieval error:22

2024/02/19 13:52:42.368705398 {wncd_x_R0-0}{1}: [rrm-client] [16619]: (ERR): a89d.2143.xxxx Failed to override default values inradio oper for slot 1, reg domain chk status failed

2024/02/19 13:52:42.369166940 {wncd_x_R0-0}{1}: [rrm-client] [16619]: (ERR): a89d.2143.xxxx Failed to override default values inradio oper for slot 0, reg domain chk status failed

2024/02/19 13:52:42.369233832 {wncd_x_R0-0}{1}: [apmgr-capwap-join] [16619]: (ERR): a89d.2143.xxxx Process join request: AP_DOMAIN payload count is 0

2024/02/19 13:52:42.369384787 {wncmgrd_R0-0}{1}: [ewlc-infra-evq] [16327]: (note): Msg type :mesg->msgtype Priority:1

2024/02/19 13:52:42.369470233 {wncd_x_R0-0}{1}: [apmgr-capwap-join] [16619]: (note): MAC: a89d.2143.xxxx Successfully processed Join request. AP name: APf4cf.e2a9.faf4, Model: AIR-CAP2702I-E-K9, radio slots: 2, rlan slots: 0, site tag name: default-site-tag, policy tag name: default-policy-tag, rf tag name: default-rf-tag

2024/02/19 13:52:42.369648938 {wncd_x_R0-0}{1}: [capwapac-smgr-srvr] [16619]: (note): MAC: a89d.2143.xxxx Join processing complete. AP in joined state

2024/02/19 13:52:42.369687398 {wncd_x_R0-0}{1}: [capwapac-smgr-sess] [16619]: (note): Mac: a89d.2143.xxxx Session-IP: 192.168.252.XX[57264] 10.224.0.XX[5246] AP Connect msg sent to loadbalancer

2024/02/19 13:53:42.512629671 {wncd_x_R0-0}{1}: [ap-join-info-db] [16619]: (note): MAC: a89d.2143.xxxx AP disconnect initiated. Reason: DTLS close alert from peer, Phase: Join

2024/02/19 13:53:42.512781561 {wncd_x_R0-0}{1}: [ewlc-dtls-sess] [16619]: (note): Remote Host: 192.168.252.58[57264] MAC: a89d.2143.xxxx DTLS session destroy

2024/02/19 13:53:42.512815658 {wncd_x_R0-0}{1}: [capwapac-smgr-sess-fsm] [16619]: (note): Mac: a89d.2143.xxxx Session-IP: 192.168.252.XX[57264] 10.224.0.XX[5246] Last Control Packet received 60 seconds ago.

Anybody had issues like this ?

I cant ssh to the AP, since i have no idea of how the previous credentials were configured, and i cant use console cable since that is a remote site. So i have no visibility from the AP side

[marce1000]

 - Can you post a show version from the AP? It is noticed that Wave 1 AP's running certain AIreOS codes cannot join XE controllers without an upgrade performed in the AP manually ; try to install the compatible/needed  capwap image manually on one AP and test again , 

  Appendix : have a checkup of the 9800-L configuration with the CLI command show tech wireless and feed the output into 
                                                                                                                           Wireless Config Analyzer

 M.

[mqontt]

Cisco IOS Software, C3700 Software (C3700-K9W8-M), Version 15.3(3)JE, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Wed 17-May-17 14:09 by prod_rel_team

ROM: Bootstrap program is C3700 boot loader
BOOTLDR: C2700 Boot Loader (AP3G2-BOOT-M) LoaderVersion 15.2(4)JB5m, RELEASE SOFTWARE (fc2)

APf4cf.e2a9.fb24 uptime is 12 minutes
System returned to ROM by power-on
System image file is "flash:/c3700-k9w8-mx.153-3.JE/c3700-k9w8-xx.153-3.JE"
Last reload reason:

cisco AIR-CAP2702I-E-K9 (PowerPC) processor (revision A0) with 376814K/134656K bytes of memory.
Processor board ID
PowerPC CPU at 800Mhz, revision number 0x2151
Last reset from power-on
LWAPP image version 8.4.100.0
1 Gigabit Ethernet interface
2 802.11 Radios

32K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address:
Part Number : 73-15824-03
PCB Serial Number :
Top Assembly Part Number : 068-100379-01
Top Assembly Serial Number :
Top Revision Number : A0
Product/Model Number : AIR-CAP2702I-E-K9

and i could see this in logs:

*Feb 20 09:15:52.287: %CAPWAP-6-AP_IMG_DWNLD: Required image not found on AP. Downloading image from Controller.
*Feb 20 09:15:52.287: capwap_image_proc: encounter flash problem, retry here

[marce1000]

                     >...C3700 Software (C3700-K9W8-M), Version 15.3(3)JE, 
 - Corresponds to 8.4.x , indeed ; go through 8.5.x (with controller) or upgrade AP manually ,

 M.

[Rich R]

That might be a different problem - either flash corruption or not enough space on flash.
For flash corruption (which was a very common problem on the x700 APs on older code versions) refer to:
https://www.cisco.com/c/en/us/support/docs/field-notices/703/fn70330.html
https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html
The flash space issue was resolved by keeping only a single image on flash in later releases (no backup image possible) but that may not be handled correctly in the very old code version you're using.  Following Leo's procedure to delete all existing images from flash (after resolving any flash corruption issues first) will resolve the flash space problem.

[mqontt]

Hey Rich,

thanks! i've already checked and the flash isnt corrupted (fortunately )

I think you might be right with the space issue.  Currently the AP has around 18 MB free space. Im not sure how large the IosXE image is , but maybe there is not enough space to decompress the image.

I will try to delete the current image and see if AP will be able to pull the new image from IOS XE WLC and use that, once the space is freed.

will update

[marce1000]

                     >....thanks! i've already checked and the flash isnt corrupted (fortunately )...
 - Personally for the time being I would take the simplest path forward first like going with 8.5.x ; leaving additional issues aside until they occur , because then problems can be induced too.

 M.

[mqontt]

im kind of afraid to do the upgrade to be honest

if the flash is full now, im afraid that when i upgrade the WLC, APs will be giving the same problem and wont be able to download and install new image and would complain the same way.

[marce1000]

 - That was my previous point translated in  another format (but of course with a due smile...!)

 M.

[Rich R · Accepted solution]

There are major changes in the CAPWAP protocol since 8.4.100 so those APs will not be able to join the 17.9.4a WLC.
I suggest you upgrade the 5508 to 8.5.182.11 (link below), move the x700 APs back to the 5508 so they can upgrade to 8.5.182.11 and then try to migrate them again.

Otherwise you'll have to manually load the 17.9.4a IOS to the APs directly.  To do that you'd have to move the APs back to the 5508, set up a TFTP server and then use Leo's instructions for image download to the APs (obviously change the filename for x700 APs).  Once they have the 17.9.4a image installed they'll be able to join the WLC without any trouble.
https://community.cisco.com/t5/wireless/cisco-1552e-problem-associating-with-wlc5508/m-p/5020189/highlight/true#M266824
https://software.cisco.com/download/home/286256843/type/280775090/release/15.3.3-JPN3

[mqontt]

Thank you guys, i wasnt aware of this kind of issue this is the first time i am trying to move APs from this old version, usually i was moving the APs from 8.5.xxx

Will try to upgrade the 5508 to 8.5.182.11 and see from there

Thanks!

[mqontt]

Just an update.

I was able to move them once i followed the procedure from Leo.

Deleted the current image, loaded new one from tftp server and rebooted the AP and then it was able to join succesfully to WLC.

I even used different version of the image (downloaded the latest one on the software page) and it was able to pull the correct version from WLC afterwards.

Thanks a lot guys every day you learn something new

[Rich R]

Glad to hear you got it working but using the wrong version means longer downtime because the AP still has to download the correct version after it joins.  If you load the correct version in advance then there's no need for download so the AP just goes straight into operation.