topic Re: APs showing offline in Wireless

APs showing offline

JohnF5 — Fri, 10 Mar 2023 19:59:58 GMT

I had about 120 APs go offline on the dashboard yesterday. They cover about 3 of my locations. They are still delivering connectivity but I can't see them to manage them from the dashboard. I also can't go out to the web address on my LAN to see them either. We've tried power cycling and factory reset with no luck. I can see my other two locations no problem.

Re: APs showing offline

aleabrahao — Fri, 10 Mar 2023 20:01:30 GMT

Have you opened a support case?

Re: APs showing offline

JohnF5 — Fri, 10 Mar 2023 20:03:04 GMT

I have. I was instructed to factory reset a device. We did but no luck.

Re: APs showing offline

Brash — Fri, 10 Mar 2023 20:24:12 GMT

Is there anything in common with those 3 sites? Do they share the same interest connection? Do they all use the same template in the dashboard?

Did the AP's in those networks get a firmware upgrade?

Re: APs showing offline

JohnF5 — Fri, 10 Mar 2023 20:31:41 GMT

We have 5 sites. They all share the same internet connection. When you say template what do you mean? I'm not too familiar with that.

The AP's did not get a firmware upgrade.

Re: APs showing offline

JohnF5 — Fri, 10 Mar 2023 20:59:42 GMT

Under configuration templates, we have nothing setup.

Re: APs showing offline

Ryan_Miles — Fri, 10 Mar 2023 22:05:22 GMT

For that many devices to all go unreachable at the same time I'd say it has to be an upstream issue. Were any changes made on the switches or upstream layer 3 devices or firewall(s)?

Also, I took a quick look at the Support case and your dashboard. I noticed from an online AP I can ping the mgmt IP of the APs showing as offline. So, to your point they are still up & functioning, but not communicating with dashboard.

Doing factory resets seems likely pointless to me as 120 devices don't all go bad at the same time. I'd focus on upstream troubleshooting (firewall logs, etc).

Re: APs showing offline

JohnF5 — Sat, 11 Mar 2023 01:02:38 GMT

No changes were made to the firewall. I'll check on the switch on Monday. I can tell you one other site in our dashboard that is up is actually downstream from one of the sites that is down. They are on the same subnet and use the same default gateway.

Re: APs showing offline

Ryan_Miles — Sat, 11 Mar 2023 03:40:11 GMT

Something like that would make me suspicious that an upstream NAT rule or something changed preventing one subnet from working while another is ok.

Re: APs showing offline

JohnF5 — Sat, 11 Mar 2023 12:43:09 GMT

No idea. Woke up this morning and everything is magically online again. 🤔

Re: APs showing offline

Ryan_Miles — Sat, 11 Mar 2023 15:23:00 GMT

I see that. However, almost every AP that has come back now shows this.

This means UDP 7351 is not working and it's using HTTP/HTTPS. More info HERE.

Re: APs showing offline

Philip D'Ath — Sun, 12 Mar 2023 20:09:11 GMT

To add to Ryan's response, I would get someone to check the firewall for blocked traffic from one of the AP IP addresses. Anything being blocked?

Re: APs showing offline

JohnF5 — Mon, 13 Mar 2023 13:07:38 GMT

Checked the firewall and traffic is being allowed.

action="accept"

dstinetsvc="Hurricane.Electric-Hurricane.Electric.Internet.Services"

dstintfrole="wan"

dstip=64.62.142.12

dstport=7351

Re: APs showing offline

Ryan_Miles — Mon, 13 Mar 2023 14:41:52 GMT

Does the firewall rule allow UDP 7351 to these 4 IPs/subnets per the firewall info page?

64.62.142.12/32

158.115.128.0/19

209.206.48.0/20

216.157.128.0/20

Checking a location of yours that didn't have the outage shows passing 7351 test to 209.206.x.x IPs. Comparing to sites with the outage those APs still show failures for 7351 checks to various 209.206.x.x IPs.

The AP's failing the test are the ones also showing the warning "Connection to the Cisco Meraki Cloud is using the backup Cloud connection."