Re: Packet loss on one of our C9120 access points. - Page 2

Bernd Nies · ‎08-31-2023

We have an office with 4 floors, total 90 x C9120 access points in local mode and using Catalyst 9800-40 WLAN controller with IOS XE 17.9.3. Per access point we have two SSID (office LAN and guest LAN). Number of clients per access point is at max around 20.

For one office room users complain about network outages and ICMP ping packet loss shows 1-15% when the office is full and 10-20 clients on the same access point. When there are few clients, the connection is fine. This has been verified with different laptops showng similar packet loss when connected to that access point and no packet loss when going to the room nearby and connect to another access point.

Users are connected with 5 GHz. The channel and rx/tx utilization is way below the max (see screenshot) and the access point is always in direct line of sight, less than 5 metres between notebook and access point. Dynamic channel assignment on the coltroller is set to automatich, interval 1 hour, Avoid Foreign AP Interference is checked and Avoid Non 5 GHz Noise is checked. Relevant best practises are all fulfilled and wireless config analyzer shows no issue. Ping from a server in the wired network to the access point shows no packet loss and also PoE switch interface (Catalysst 2960x) shows no errors. Radioactive trace does not reveal why it happens.

When logged in on the access point with SSH and show the network, there are errors on some interfaces:

#show interfaces network
[...]
apr0v0    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:80
          inet6 addr: fe80::5038:34f1:3dc4:ded4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:60023 errors:0 dropped:0 overruns:0 frame:914440
          TX packets:270220 errors:1793 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11014456 (10.5 MiB)  TX bytes:219823045 (209.6 MiB)
          Interrupt:48

apr0v1    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:81
          inet6 addr: fe80::9f91:3512:f7bc:6dd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:75099 errors:0 dropped:0 overruns:0 frame:914440
          TX packets:266024 errors:2220 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:23048412 (21.9 MiB)  TX bytes:255893106 (244.0 MiB)

[...]

apr1v0    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:8F
          inet6 addr: fe80::78f7:44a9:7d2c:78bb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5385435 errors:0 dropped:0 overruns:0 frame:271403
          TX packets:6557280 errors:16845 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2705161595 (2.5 GiB)  TX bytes:8534345562 (7.9 GiB)
          Interrupt:46

apr1v1    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:8E
          inet6 addr: fe80::49ff:c5d6:80e7:e996/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6389098 errors:0 dropped:0 overruns:0 frame:271403
          TX packets:11830174 errors:18392 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3405090827 (3.1 GiB)  TX bytes:12909375977 (12.0 GiB)

[...]

I don't really know how to dig that issue any further. Access point has already been factory reset. My next suspicion is hardware fault and we replace the access point with a new one. Any ideas how to narrow down the issue?

Thanks in advance,

Bernd

Bernd Nies · ‎09-21-2023

Swapping the APs did not solve the issue. Also switching to a non DFS channels. It seems to boil down to MacBook Pro M2 2023 with Ventura 13 related. It happened when there are 10-15 clients in the same office and connected to the same AP. Ping latency has high variation (2.544/46.253/289.581/39.758 ms) and sometimes packet loss.

Pinging other notebooks (Lenovos, older MacBook models) connected to the same AP shows no packet loss.

Disabling AWDL (the MacOS ad-hoc WLAN service) might have improved this. I can imagine that scanning for AirDrop or oder ad-hoc WLANs could cause outages or latency. The more clients around the more likely.

https://apple.stackexchange.com/questions/454337/new-2023-macbook-pro-internet-connection-issues

Leo Laohoo · ‎09-21-2023

Disable 802.11ax on the SSID.

Bernd Nies · ‎09-21-2023

Hm. 75% of our WLAN clients are using 802.11ax right now ...

Leo Laohoo · ‎09-21-2023

@Bernd Nies wrote:
Hm. 75% of our WLAN clients are using 802.11ax right now ...

It might say "802.11ax" but no client can push over 802.11ac speed. Not with only one or two antennas.

JPavonM · ‎09-21-2023

Neverending issues from OS vendors

Bernd Nies · ‎09-28-2023

Disabling AWDL (the MacOS ad-hoc WLAN service) did not improve issue. I also temporarily disabled 802.11ax on the SSID and with 802.11ac packet loss and random high latency >800 ms was still there. Issue happens when the office is full (10-15 clients per AP). When alone it is good. AP is using non-DCA channel 48. Channel utilization is around 5%. The client association history lists "802.11i Slow" with latency >1200 ms.

The MacBook Pro M2 are on MacOS 13.5.2. Update to 13.6 published yesterday did not fix it. Apple seems more to be concerned about implementing fancy hipster features, than improve network stability.

adventurer · ‎12-26-2023

Something interesting to add as I have been investigating similar issue, 5Ghz WiFi unstable. At times it can be working fine ping across 2ms, then increase to 1000+ms and then some packets drops, and then returns to normal 2ms.

I have narrowed it down to Reolink cameras, some unspecific compatibility issue with Cisco over 5GHz.

Now I don't recall having this issue while on 17.3.4 so my opinion is a bug might have been introduced when it went to 17.9.4