2800 and 3800 APs not join to WLC errors

[schulcz]

Hi Guys!

I have a problem. New site was deployed, devices are installed, and I need to make wireless network to working.

We have WLCs on the datacenter. APs got IP addresses from the local ap mgmt subnet (e.g. 172.20.0.0/24), and all traffic from this subnet can reach controller AP Manager interface (10.xx.xx.161) without any filter. We inform APs where should them go by DHCP option 43, it's working fine, I checked a couple of APs, got WLCs IP addresses fine.

The problem is that some APs can join to WLC, some not. All of these APs in the same subnet, and all of them are AIR-AP2802I-E-K9 or AIR-AP3802I-E-K9 types. WLCs also handle many hundreds of 2800s and 3800s APs currently, and we have enough free license installed also. Time is also correct on WLC.

Most of the APs are deployed so high (7-8m), so for most the console port is inaccessible. Some APs are reachable height, I tested with them. Some of them are downloading image and joining fine, some of them only downloaded image (seen by show cdp neighbors on switches) and not joined to WLC.

Example I saw one AP that can't download image correctly:

[*07/06/2022 14:59:58.3796] CAPWAP State: Join
[*07/06/2022 14:59:58.3811] Sending Join request to 10.xx.xx.161 through port 5256
[*07/06/2022 15:00:03.1523] Join Response from 10.xx.xx.161 
[*07/06/2022 15:00:03.1527] CAPWAP State: Image Data
[*07/06/2022 15:00:03.1898] do PRECHECK, part1 is active part
[*07/06/2022 15:00:03.3241] Image Data Request sent to 10.xx.xx.161
[*07/06/2022 15:00:03.4275] .....................
[*07/06/2022 15:00:15.2502] ..................................................
[*07/06/2022 15:00:42.1130] ........Re-Tx Count=1, Max Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=1
[*07/06/2022 15:01:06.0408] Re-Tx Count=2, Max Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=1
[*07/06/2022 15:01:08.8916] Re-Tx Count=3, Max Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=1
[*07/06/2022 15:01:11.7425] Re-Tx Count=4, Max Re-Tx Value=5, SendSeqNum=3, NumofPendingMsgs=1
[*07/06/2022 15:01:13.6429] Image download interrupted for 30 seconds. 
[*07/06/2022 15:01:13.6430] Restarting CAPWAP State Machine.
[*07/06/2022 15:01:13.7481] CAPWAP State: DTLS Teardown
[*07/06/2022 15:01:18.4451] do ABORT, part1 is active part
[*07/06/2022 15:01:48.5967] Discovery Response from 10.xx.xx.161
[*07/06/2022 15:01:58.0000] CAPWAP State: DTLS Setup
[*07/06/2022 15:01:58.3776] CAPWAP State: Join
[*07/06/2022 15:01:58.3791] Sending Join request to 10.xx.xx.161 through port 5256
[*07/06/2022 15:02:02.9605] Join Response from 10.xx.xx.161 
[*07/06/2022 15:02:02.9608] CAPWAP State: Image Data
[*07/06/2022 15:02:02.9978] do PRECHECK, part1 is active part
[*07/06/2022 15:02:03.1377] Image Data Request sent to 10.xx.xx.161

I made a factory reset on this AP by mode button, since then it never get any response from the WLC:

[*12/20/2019 11:56:25.9656] CAPWAP State: Init
[*12/20/2019 11:56:25.9662] Config not found, PNP is required, Starting PNP
[*12/20/2019 11:56:44.7587] PNP:Server not reachable, Start CAPWAP Discovery
[*12/20/2019 11:56:44.7590] CAPWAP State: Discovery
[*12/20/2019 11:56:44.7597] Got WLC address 10.xx.xx.161 from DHCP.
[*12/20/2019 11:56:44.7597] IP DNS query for CISCO-CAPWAP-CONTROLLER.corp.xxxxxxx.xxx
[*12/20/2019 11:57:14.7921] Discovery Request sent to 10.xx.xx.161, discovery type DHCP(2)
[*12/20/2019 11:57:14.7933] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*12/20/2019 11:57:14.7933] CAPWAP State: Discovery
[*12/20/2019 11:57:24.3816] CAPWAP State: Discovery
[*12/20/2019 11:57:24.3823] Got WLC address 10.xx.xx.161 from DHCP.
[*12/20/2019 11:57:24.3823] IP DNS query for CISCO-CAPWAP-CONTROLLER.corp.xxxxxxx.xxx
[*12/20/2019 11:57:54.4139] Discovery Request sent to 10.xx.xx.161, discovery type DHCP(2)
[*12/20/2019 11:57:54.4151] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*12/20/2019 11:58:04.1032] CAPWAP State: Discovery
[*12/20/2019 11:58:04.1040] Got WLC address 10.xx.xx.161 from DHCP.
[*12/20/2019 11:58:04.1040] IP DNS query for CISCO-CAPWAP-CONTROLLER.corp.xxxxxxx.xxx
[*12/20/2019 11:58:34.1318] Discovery Request sent to 10.xx.xx.161, discovery type DHCP(2)
[*12/20/2019 11:58:34.1329] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*12/20/2019 11:58:43.8251] CAPWAP State: Discovery
[*12/20/2019 11:58:43.8259] Got WLC address 10.xx.xx.161 from DHCP.
[*12/20/2019 11:58:43.8259] IP DNS query for CISCO-CAPWAP-CONTROLLER.corp.xxxxxxx.xxx
[*12/20/2019 11:59:13.8559] Discovery Request sent to 10.xx.xx.161, discovery type DHCP(2)
[*12/20/2019 11:59:13.8570] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*12/20/2019 11:59:23.5469] CAPWAP State: Discovery
[*12/20/2019 11:59:23.5477] Got WLC address 10.xx.xx.161 from DHCP.
[*12/20/2019 11:59:23.5477] IP DNS query for CISCO-CAPWAP-CONTROLLER.corp.xxxxxxx.xxx
[*12/20/2019 11:59:53.5767] Discovery Request sent to 10.xx.xx.161, discovery type DHCP(2)
[*12/20/2019 11:59:53.5778] Discovery Request sent to 255.255.255.255, discovery type UNKNOWN(0)
[*12/20/2019 12:00:45.0865] PNP:Server not reachable, Start CAPWAP Discovery
[*12/20/2019 12:00:45.0866] Lost connection to the controller, going to restart CAPWAP (reason : Post startCapwapDiscovery)...
[*12/20/2019 12:00:45.0867] Restarting CAPWAP State Machine.

If I made a temporary vlan, put this AP and a temporary 2504 wlc there, AP found the WLC and join to that without any problem!

I tried to debug capwap events and errors on wlc side, didn't see any entry about that APs not joining to WLC (it's so weird)

I tried to put a temporary 3800 AP on local subnet running Mobility Express, it collected all APs running in capwap mode. (2800 and 3800 types also)

Most of the APs arrived with 8.5.140.0
WLC runs 8.10.162.0 (5520)

What do you think what can be the problem?

[Rasika Nayanajith]

I would try following option

1. Once you reset the AP, it does not know where is the WLC, so it is normal it is not trying to register. You can statically pointing AP to WLC on AP CLI using the below command.

"capwap ap primary-base <WLC_Name> <WLC_Mgt_IP>"

2. Upload AP image "ap3g3-k9w8-tar.153-3.JPL.tar" directly to see if that is due to an issue with the firmware that came with the AP when ordered. Once AP got an IP address, you should be able to ssh it (even aP is not accessible via console)

HTH

Rasika

*** Pls rate all useful responses ***

[schulcz]

Correct myself, after a factory reset, the AP got WLCs IP address by option 43, after that knows where should go, but it was didn't successfull.

Yeah, I can ssh to them, but can't login to them, default Cisco/Cisco not working. I also tried that login what I set on the temporary Mobility Express (because these caps were under Mobility Express for a short period of time), but that also not works.

Uploading the image directly is a good idea, thanks!

[Leo Laohoo]

---

@schulcz wrote:

I made a factory reset on this AP by mode button

---

Console into the AP and enter the command "capwap ap erase all".  

I understand that the AP are in ceilings with heights but this is the only sure-fire way to factory-reset an AP.

[Rich R]

Agree with @Leo Laohoo and @Rasika Nayanajith 
1. Full factory reset to make sure AP config is erased.
2. Upgrade AP image manually.

What version was the ME you tested with?