topic Re: High Latency on APs - bouncing the AP seems to resolve the issue in Wireless

High Latency on APs - bouncing the AP seems to resolve the issue

kelven.leverett — Thu, 16 Dec 2021 16:45:03 GMT

Hi all,

We currently have a case open with Meraki going on 6 days now. At multiple sites we've been seeing sporadic issues with our clients passing traffic through access points. When we look at the AP in question we are seeing high latency from it (100ms+). We bounce the AP and everything is seemingly good from there. Wanted to post our situation here to see if anyone else is having this experience as we wait for support to take a look at our latest packet capture.

Thanks!

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Ryan_Miles — Thu, 16 Dec 2021 17:38:34 GMT

What model(s) AP?

What firmware?

Happening on all SSIDs or just some?

Happening to all clients or just some?

Re: High Latency on APs - bouncing the AP seems to resolve the issue

kelven.leverett — Thu, 16 Dec 2021 18:55:48 GMT

Hi there.

MR45/46

Firmware MR 28.5

I have yet to confirm that it is happening on all SSIDs as our staff currently only has access to 1 SSID, but you raise a good point. We are starting to look into our radio settings now.

One Hypothesis right now is potentially looking at changing the Minimum bitrate configuration to Per Band as we have 2 sites in our organization that have not reported issues and their minimum bitrate is set per band while the sites having issues are configured per SSID

Thank you

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Philip D'Ath — Thu, 16 Dec 2021 19:29:14 GMT

I have seen this caused by Windows power saving. On a Windows 10 machine, try this (which disables power saving for the WiFi NIC):

powercfg /SETDCVALUEINDEX SCHEME_CURRENT 19cbb8fa-5279-450e-9fac-8a3d5fedd0c1 12bbebe6-58d6-4636-95bb-3217ef867c1a 0
powercfg /SETACVALUEINDEX SCHEME_CURRENT 19cbb8fa-5279-450e-9fac-8a3d5fedd0c1 12bbebe6-58d6-4636-95bb-3217ef867c1a 0

Another common cause is WiFi driver bugs. Try checking the manufacturer of the WiFi chipset for driver updates. Note that the OEM (Lenovo, Dell, etc) often don't publish the current version of the driver. Hence the need to check the WiFi manufacturers web site.

I hope this is on 5Ghz. 2.4Ghz is very problematic and issues are common. Can you disable 2.4Ghz on the WiFi to get rid of this problem area?

Re: High Latency on APs - bouncing the AP seems to resolve the issue

YC2 — Thu, 10 Mar 2022 21:35:42 GMT

Running into something similar. Clients on the ap can associate and authenticate just fine but user traffic has such high latency its essentially unusable. If your only seeing 100ms your lucky... ours reaches 4 digits. Reboot and it's peachy until the next ap over, or the one on other side of building, does the same thing. Reboot, rinse, repeat.

Seems to have started a few months back and progressively getting worse. Happens across multiple platforms. Worked with support earlier after getting another example we could replicate. Captures indicated simple pings weren't making it to wired side of AP. Same client can roam to another ap and work just fine with no other changes other then walking away from the culprit AP.

Auto upgrade to 28.6 (from 28.5) was set to happen in a week. Forced it for overnight. Time will tell.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

WB2 — Mon, 14 Mar 2022 02:29:16 GMT

There's been some chat around a bug fix noted in the 28.6 release notes:

'MR could enter a state where ADDBA requests would not be initiated, resulting in reduced throughput (Wi-Fi 5 Wave 2 MRs)'

@YC2would be good to know if 28.6 provides a permanent fix for you!

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Agus KTI — Mon, 14 Mar 2022 05:08:22 GMT

The problem happened to me to it happened with device using 2,4Ghz device, when i saw on dashboard most of client coonect/receive signal with 2,4Ghz a little device connect.receive using 5Ghz. I think 2,4Ghz already saturated even maybe on one of user still on download progress on there, i thnk that make high latency.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

YC2 — Mon, 14 Mar 2022 16:30:14 GMT

While that's a possibility that's not what was happening here. My two test clients were on the same ap, with no other clients on it, on 5ghz.

After fighting the automatic firmware update scheduler all ap's at the site are on 28.6 now. Time will tell.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Jai007 — Fri, 08 Jul 2022 11:31:57 GMT

I am curious if the upgrade to 28.6 solved this issue for you?

Re: High Latency on APs - bouncing the AP seems to resolve the issue

YC2 — Fri, 08 Jul 2022 16:30:07 GMT

Unfortunately no, not completely. Maybe made it less often. Haven't heard anything since the upgrade in March. Then in the last week or two I got another report. This time we haven't been able to replicate to allow for troubleshooting. The users tend to report the issue, then move/roam, powercycle, etc until it starts working, leaving us stranded on the troubleshooting front.

I'll take every few months vs every few weeks though.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Jai007 — Mon, 11 Jul 2022 08:12:18 GMT

For us this seems to be related to the recent upgrade from 27.5.1 to 28.5. So we are going to try rolling back to the previous version and see what the results are.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

YC2 — Fri, 15 Jul 2022 01:23:16 GMT

Got reports of this again today. Tech on site was able to replicate. Got meraki on the line and went at it for a long while. Our entire Meraki infrastructure is all L2. SVI's, routing, etc happens on an upstream device.

The client had heavy packet loss to it's svi, and on packets that did get a response time was through the roof, 500+ms to 4 digit range. Capturing on ap's wifi interface didn't show anything out of ordinary, infact it showed all normal times and no drops. Capturing on wired port and suddenly things started working normally again.

Makes me think either packet didn't make it to AP, or AP dropped it without notice.

Coincidentally - the ap client was connected to was on opposite side of building, down a floor. Auto power has it so high though that the client hears it louder then the units much closer (and on the same floor!). I don't believe this has to do with the issue at hand though because client can pass traffic through the further ap just fine when not in this failure mode.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Thomas Obbekaer Thomsen — Mon, 25 Jul 2022 10:03:31 GMT

There is a known bug with MR46 where it, overtime, does something(tm) that causes low throughput.

Reload, or a "shut / no shut" of the SSID fixes the problem.

This issue should be fixed in 29.2 (and only 29 software as far as I have heard, since it requires a Radio Firmware upgrade).

Perhaps that is what you are running into.

Re: High Latency on APs - bouncing the AP seems to resolve the issue

YC2 — Mon, 25 Jul 2022 16:30:58 GMT

For all clients on the same ssid? Nah... this is unique to individual clients... other clients on same ap and same ssid work just peachy.

Had a second incident reported a couple days after the one mentioned above. This one was 0 response from it's svi but again other clients on same ap were fine. This client, roaming to a neighboring ap was ok, then back to the problem ap failed. After packet capturing... the arp request client makes to the svi never received a response. Packet capturing on the switchport the ap is connected to though shows the request AND response. It appears the AP never transmits the response back to the client. The was receiving other random broadcast/multicast traffic from the AP though so it doesn't seem to be an rf issue. The roam/dot1x/dhcp/etc worked fine.....

Re: High Latency on APs - bouncing the AP seems to resolve the issue

NavidG — Tue, 26 Jul 2022 09:10:59 GMT

Have you found any fix yet?

Re: High Latency on APs - bouncing the AP seems to resolve the issue

NavidG — Tue, 26 Jul 2022 09:12:50 GMT

Have you found any fix yet?

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Speedbird1 — Tue, 26 Jul 2022 11:57:04 GMT

We have a very similar or exact issue on MR56's 28.6.1. AP's all slowed down no client could get more than 10mbps speed pings were >100 ms.

On a normal day we get 200mbps throughput and pings <8ms.

Full reboot of the site AP's resolved it.

The known issues on 29 firmware don't look promising either

AP performance may degrade over time (Wi-Fi 6 APs)

Re: High Latency on APs - bouncing the AP seems to resolve the issue

WB2 — Wed, 27 Jul 2022 04:27:54 GMT

Release notes for 29.2 have just come out listing 'AP performance may degrade over time (Wi-Fi 6 APs)' in the Bug Fixes section. Is not included in 28.7 however!

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Speedbird1 — Wed, 27 Jul 2022 11:55:02 GMT

Ah yes thanks ... Just seen that

Re: High Latency on APs - bouncing the AP seems to resolve the issue

Thomas Obbekaer Thomsen — Thu, 28 Jul 2022 12:55:50 GMT

Interesting. I have seen something similar I think. But it seems to be kind of a "random" problem, and to us/me it seemed to only affect certain "older" clients, and it always seems to happen when the client roams (again like you the roam , dot1x and so on seems fine, but from the client perspective the network does not forward traffic). Im really looking forward to test the 29.2 software, just to see if anything changes.