Packet loss on one of our C9120 access points.

[Bernd Nies]

We have an office with 4 floors, total 90 x C9120 access points in local mode and using Catalyst 9800-40 WLAN controller with IOS XE 17.9.3. Per access point we have two SSID (office LAN and guest LAN). Number of clients per access point is at max around 20.

For one office room users complain about network outages and ICMP ping packet loss shows 1-15% when the office is full and 10-20 clients on the same access point. When there are few clients, the connection is fine. This has been verified with different laptops showng similar packet loss when connected to that access point and no packet loss when going to the room nearby and connect to another access point. 

Users are connected with 5 GHz. The channel and rx/tx utilization is way below the max (see screenshot) and the access point is always in direct line of sight, less than 5 metres between notebook and access point. Dynamic channel assignment on the coltroller is set to automatich, interval 1 hour, Avoid Foreign AP Interference is checked and Avoid Non 5 GHz Noise is checked. Relevant best practises are all fulfilled and wireless config analyzer shows no issue. Ping from a server in the wired network to the access point shows no packet loss and also PoE switch interface (Catalysst 2960x) shows no errors. Radioactive trace does not reveal why it happens.

When logged in on the access point with SSH and show the network, there are errors on some interfaces:

#show interfaces network
[...]
apr0v0    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:80
          inet6 addr: fe80::5038:34f1:3dc4:ded4/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:60023 errors:0 dropped:0 overruns:0 frame:914440
          TX packets:270220 errors:1793 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:11014456 (10.5 MiB)  TX bytes:219823045 (209.6 MiB)
          Interrupt:48

apr0v1    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:81
          inet6 addr: fe80::9f91:3512:f7bc:6dd0/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:75099 errors:0 dropped:0 overruns:0 frame:914440
          TX packets:266024 errors:2220 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:23048412 (21.9 MiB)  TX bytes:255893106 (244.0 MiB)

[...]

apr1v0    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:8F
          inet6 addr: fe80::78f7:44a9:7d2c:78bb/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:5385435 errors:0 dropped:0 overruns:0 frame:271403
          TX packets:6557280 errors:16845 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2705161595 (2.5 GiB)  TX bytes:8534345562 (7.9 GiB)
          Interrupt:46

apr1v1    Link encap:Ethernet  HWaddr E4:4E:2D:AA:F4:8E
          inet6 addr: fe80::49ff:c5d6:80e7:e996/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:6389098 errors:0 dropped:0 overruns:0 frame:271403
          TX packets:11830174 errors:18392 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3405090827 (3.1 GiB)  TX bytes:12909375977 (12.0 GiB)

[...]

I don't really know how to dig that issue any further. Access point has already been factory reset. My next suspicion is hardware fault and we replace the access point with a new one. Any ideas how to narrow down the issue?

Thanks in advance,

Bernd

[Leo Laohoo]

What are the wireless clients?  Are they Intel- or RealTek-based wireless NIC?

IF they are, what are the model of the wireless NIC and their drivers?

[Bernd Nies]

Mixed. Some are MacBook M2 with latest MacOS, some are Lenovos with Windows 10 onboard. As packet loss goes away when client choose another access point, I suspect an issue with the access point and not the client

[Leo Laohoo]

---

@Bernd Nies wrote:
some are Lenovos with Windows 10 onboard.

---

What are the wireless NIC(s) of the Lenovo and their driver version?

[Bernd Nies]

Don't know. But I guess the same model and version as all the other users in different locations having no issue (or not reporting it). The user who reported the issue had a MacBook Pro M2 with MacOS Ventura 13.5. He got another one for testing because I suspected an issue with the notebook. Both showed packed loss when in that office room and none when going to another room and force rejoin another access point. He asked other colleagues to test ping and one directly under the access point said 17% packet loss. I could not verify that the day later, but that day the office was less full. The access point is going to be replaced today. 

[JPavonM]

Have you managed to check if that "faulty" AP is using always the same channel? HAve you managed to check if the channel where the AP operates is in use by another AP on an OBSS?

Have you checked if any DFS events or source of narrow band interference could be impacting it?

It happened to me a couple of years ago that all users were complaining about lack of performance, disconnections and been unable to connect on the vicinity of a meeting room. After some investigation with a spectrum analyzer I noticed there was a narrow-band interferer that wasn't reported by the AP not the AP was detecting the channel utilization due to the narrow-band. After finding the offender and removing it the problem was solved (FYK it was a faulty Crestron screen's power source impacting  channel 132)

I will suspect for something regarding congestion management in that channel, even with the AP showing low utilziation, as this value seen in the AP's output maybe not current.

[Bernd Nies]

Before I did a factory reset on the AP it had 5 GHz channel 112 on slot 0 and channel 52 on slot 1. Channel 52 had the most clients. After the factory reset slot 0 was configured for 2.4 GHz.

When I was on site checking it, I could not find any issue. WIFI Explorer showed no foreign WLANs. Reception in every corner of the room was good. 

AP will be replaced today.

I reduced the DCA interval from 1h to default 10m. Maybe that helps in congestion management more dynamically. We have shared offices and every day another corner is crowded. 

[Leo Laohoo]

Channel 52?  

That's a DFS channel.

[Bernd Nies]

Maybe someone brings a radar to the office?

[JPavonM]

No but RADARs are there and AP needs to avoid them (police, militaries, weather, airport stuff, ...)

[Bernd Nies]

Hm, there are quite a lot of DFS channels (52, 56, 60, 64, 100, 104, 108, 112, 116, 120, 124, 128, 132, 136, 140, and 144). They were all enabled by default in the RF profile.

https://wifinc.net/dfs-channels-and-why-to-avoid-them-even-though-you-say-you-cannot/

When disabling them all, there are not many channels left for using. We have ca 23 access points per floor. Our windows are shielded, so the WIFI signal does not go through. I only see neighboring signals when open the window. There is no military and weather radar nearby. Airport is 10 km away.

There's not much interference shown.

[JPavonM]

As recommended in some of the previos posts, check if manually changing AP to another channel on a diferent 5GHz band makes any difference (for example UNII-3 channel 149). If problem persist try swapping AP by another one to discard a faulty AP (if you swap the AP and the issue follows the AP then this is clearly a faulty unit).

[Bernd Nies]

The AP has been replaced. Let's see if users still complain. Don't know how to change the channel for only one AP. They are assigned dynamically. I would need to disable that channelf or the whole RF group. There are no DCA events and radars for that AP. Seeps that police does not take speed measurements three floors above the street.

[JPavonM]

The commands to manually change channels/power on specific APs are:

ap name <APname> dot11 5ghz slot 1 shutdown
ap name <APname> dot11 5ghz slot 1 radio role manual client-serving
ap name <APname> dot11 5ghz slot 1 channel <Channel#>
ap name <APname> dot11 5ghz slot 1 channel width <40|20>
ap name <APname> no dot11 5ghz slot 1 shutdown
!
ap name <APname> dot11 5ghz slot 1 shutdown
ap name <APname> dot11 5ghz slot 1 radio role manual client-serving
ap name <APname> dot11 5ghz slot 1 txpower [auto|1-8]
ap name <APname> no dot11 5ghz slot 1 shutdown

And to rollback changes to automatic:

 ap name <AP-name> dot11 dual-band slot 1 shutdown
 ap name <AP-name> dot11 dual-band role auto
 ap name <AP-name> dot11 dual-band slot 1 band 24ghz
 ap name <AP-name> no dot11 dual-band slot 1 shutdown 

[Rich R]

Swap the AP with one of the other APs.  If the problem follows the AP then faulty AP.  If the problem stays in that location (on new AP) then probably something in that location causing interference.
We've had a customer ask us to disable use of channel 36 for them because they have Clickshare devices using channel 36 in all their meeting rooms so consider things like that.