topic Re: APs not able connecting to controller in Wireless

APs not able connecting to controller

inju — Wed, 07 May 2025 12:56:36 GMT

Hi!

We are a small science institut having like 35-40 Cisco access points that are connected to the near-by University and their Cisco Wireless Controller (9800-40-A?). However, some of those APs do loose their connection to the WLC or cannot find the WLC reliably. Sometimes they are connected and have clients on them, but the next morning the AP is flashing green/red. Sometimes are power cycle helps, more often it does not. It's not always the same set of APs having these issues.

Previous network setup was a VLAN 300 that was connected directly via the switch and dark fiber to the University. Than the University forced us to remove the VLAN 300, because they wanted to get rid off of that VLAN. So we set up a new VLAN 30 which is now behind our pfSense firewall (yeah, we are low on budget).

We have these types of APs:

9120
9162
9176

We have no access to the controller and the guys at the University are sometimes unresponsive. There are times when 50% of the APs are offline, causing great frustration of the users about the unstable (or: not working) Wifi.

The Firewall allows the necessary ports like 5246-5248, NTP, DNS, mDNS.

I'm running out of ideas and my debugging options are limited as I have no access to the controller. Currently 9 APs are "offline":

Please find the attached log file from console output of a CW9176I AP failing to find the controller. Any help/suggestions are appreciated!

Thanks!

Ingo

Re: APs not able connecting to controller

Kasun Bandara — Wed, 07 May 2025 13:09:41 GMT

@inju hi, according to logs, its more in to network related. seems link your AP cannot discover properly.

1. check the end to end network connection stability from AP to WLC

2. try configuring the WLC IP manually in the AP and see if that change the situation.

Re: APs not able connecting to controller

inju — Wed, 07 May 2025 13:22:17 GMT

Thanks @Kasun Bandara for your quick reply!

Sadly we have no access to the AP, at least the default Cisco/Cisco does not work. I guess, when the AP accidentially found its controller once it will get a new password from the controller, which is unknown to me.

In terms of networking we already increased the UDP timeouts for UDP First/UDP Single/UDP Multiple from 60/30/60 to 600/300/600 but without improvement. Increasing UDP timeout was a recommendation of the University controller admin, though.

However, 29 of 38 APs are indeed online, so I won't assume that it is a network problem. Basically the network is the same for all APs...

Re: APs not able connecting to controller

Kasun Bandara — Wed, 07 May 2025 13:43:19 GMT

@inju hi, mm ok. i assumed your all APs are randomly going offline which may due to some network bandwidth or instability. but if its stable as you say you can remove that fact.

according to debug log, AP have difficulty to discover and find the WLC. i suspect some issue in discovering WLC. if network is stable, next points are DNS where AP using to discover IP of WLC. this can force by manually setting WLC IP in the AP.

is this affecting to other APs at main campus site? or only for your location?

Re: APs not able connecting to controller

MHM Cisco World — Wed, 07 May 2025 13:55:06 GMT

5246-5248

Udp ports need are

5246 and 5247

Are you sure you allow ports?

MHM

Re: APs not able connecting to controller

eglinsky2012 — Wed, 07 May 2025 14:10:22 GMT

What's different about the APs that are offline vs. online? Are they a different model then the others? Are they in a different VLAN/subnet? Are they connected to a different access layer switch and/or a different distribution switch/router? Check interface statistics on all switch ports between the APs and the WLC, are there CRC/input/output errors?

Re: APs not able connecting to controller

Mark Elsen — Wed, 07 May 2025 15:11:56 GMT

- @inju There seem to be lots of parameters beyond your control; you might be better of to manage the APs yourself; you can for instance download and deploy the virtual 9800 controller for free with up to 50 APs :
https://software.cisco.com/download/home/286322605/type/282046477/release/Dublin-17.12.4

Re: APs not able connecting to controller

Saikat Nandy — Wed, 07 May 2025 16:17:00 GMT

Since you say that sometimes the APs are registered and sometimes those drop offs, I am expecting the config to be okay (but no harm in double checking the WMI configuration of WLC).

The logs says that the AP has sent the discovery request and waiting for a response. Now the things need to be checked is whether this discovery is hitting the controller or not - and the only way to confirm is to have WLC access and check join stats and take some RA trace.

Curious to know - when APs drop off and especially sending Discovery Request and not getting any response back - at that time if you ping the WLC WMI from the AP connected core switch using source vlan 30 with 100/1000 repeat, how it goes?

as @Mark Elsen said, running a centralised deployment without having access to the WLC is not worthy. Better you explore 9800-CL or CW9800M

Re: APs not able connecting to controller

Leo Laohoo — Wed, 07 May 2025 23:46:59 GMT

Are the Cisco APs connected to Cisco Catalyst 9k switches?

Re: APs not able connecting to controller

srimal99 — Thu, 08 May 2025 07:28:29 GMT

When you had vlan 300 did it work without any issues ? was it after creating the new vlan 30. Can you check the DHCP pool to see enough scope is configured ?

Re: APs not able connecting to controller

inju — Thu, 08 May 2025 08:04:05 GMT

Yes, the firewall reports no blockage, so I'm pretty sure. And at least all APs are within one subnet and right now 28 of 38 APs have no issues and are working. So for me there is no general issue in networking. That doesn't mean that there might be issues, of course... but then again it would be helpful to know what I need to look for to catch those... 🙂

Re: APs not able connecting to controller

inju — Thu, 08 May 2025 08:14:29 GMT

Nope... all APs are within the same VLAN and subnet. They are of course connected to different switches. Some are connected to old WS-C2960S-48LPS-L (hey, don't blame me! I just took over what my predecessor in this job handed me over! ;-)) or some Extreme switches like X440G2-24p-10G4.

There is no clear pattern. An AP that is running today might be offline tomorrow. Or vice versa.

The APs are powered via PoE and the switches can not provide full PoE power for all features, but again: the APs are able to run with that (without USB) until they fail eventually.

sh int | i Fast|input errors shows no interface errors, though...

Re: APs not able connecting to controller

inju — Thu, 08 May 2025 08:16:48 GMT

Nope, old stuff:

Cisco WS-C2960S-48LPS-L and Extreme X440G2-24p-10G4

Re: APs not able connecting to controller

inju — Thu, 08 May 2025 08:26:53 GMT

Being a Cisco guy (well, UC & Network engineer for >11 years) that would be my solution as well, but as stated in the OP we are short on budget (which is quite normal for the public sector), so having an own 9800 controller might not the way to go over here. It is apparently totally ok that I walk every other day through the buildings and power-cycle all the not-working APs, wasting my working time for the last 8 months this is now ongoing. But investing some money to get a working infrastructure is a big thing that everyone (except me) wants to avoid. ¯\_(ツ)_/¯

Regarding the ping to WLC I need to investigate and come back later with the results. Thanks for the pointer...

Re: APs not able connecting to controller

inju — Thu, 08 May 2025 09:43:13 GMT

Regarding prior use of VLAN 300: here the feedback differs whom you'll ask. Manager says that it worked fine before the migration, some users say that it didn't work back then as well. As the VLAN 300 to VLAN 30 migration was at my job start I can't comment much how well it was working before the migration.

Meanwhile I have an estimate from our Account Manager and let me state it this way: having me walking around the house twice a week for 30 mins each to reset the APs is more expensive than ordering a virtual controller.

But until then I still have to deal with the APs and unresponsive admin of the controller... *sigh*

Re: APs not able connecting to controller

MHM Cisco World — Thu, 08 May 2025 14:01:33 GMT

Only control port is open and flow' this work if you have AP central auth and flex data' you need both port open when you use central switching abd authc.

Open port 5247.

MHM

Re: APs not able connecting to controller

Rich R — Thu, 08 May 2025 21:05:00 GMT

Well your troubleshooting options are limited without access to the controller!

From the log it's clear that the AP is not getting any response to the discovery requests - presume those 4 WLC IPs it's trying are correct?

So either:
- the discovery requests do not reach the WLC
- the WLC does not respond to the requests
- the requests or responses are getting dropped by the network

Ideally you'd be starting with a packet capture on the WLC to make sure the requests are received and the WLC replies then you'd at least know whether the problem is the WLC or network.

What is the connectivity between you and the WLC? We did once have a problem with a customer using a 3rd party network service which had integrated IDS/IPS (which we only found out about after they had a similar complaint). It turned out the IDS/IPS was detecting the CAPWAP (UDP 5246/5247) as an "attack" and dropping the packets occasionally. They had to ask the network provider to exclude the CAPWAP traffic on the IDS/IPS - so check whether your "VLAN 300" has anything like that. If you see ping to the WLC working while CAPWAP isn't that would also point to something like this.

Otherwise you're really going to need the WLC owner to work with you ...

Re: APs not able connecting to controller

inju — Fri, 09 May 2025 08:02:21 GMT

There is enough space in the pool. We have like 40 APs and the pool is configured to use the whole /24...

VLAN 30 was created after VLAN 300, but then again: out of 38 APs in that VLAN 30, as of now 29 APs are actually working. 9 APs do not.

Re: APs not able connecting to controller

inju — Fri, 09 May 2025 08:04:29 GMT

From within the network nmap -sU -p 5246-5248 to the controller shows:

PORT STATE SERVICE
5246/udp open|filtered capwap-control
5247/udp open|filtered capwap-data
5248/udp open|filtered caacws

Re: APs not able connecting to controller

Rich R — Fri, 09 May 2025 12:19:44 GMT

> There is enough space in the pool. We have like 40 APs and the pool is configured to use the whole /24...
Well it depends what DHCP server you are using ... There is a well known issue with Microsoft DHCP server that the APs will cause the server to mark addresses as "bad" and then the server runs out of usable IPs even though there should be plenty free. Cisco provided a fix for it on AireOS but have decided not to fix it on 9800. See the details at:
https://bst.cisco.com/bugsearch/bug/CSCvj14517
Although this is only known to happen with MS DHCP server it's possible that it could affect other products too.
So rather than assuming the addresses have not run out you should make sure <wink>