Poor 5Ghz throughput - 9800 / 17.9.4a / 1800-series APs

Stuart Patton · ‎11-16-2023

Hi,

Per the title, I've got an issue with 5Ghz clients that have good signal strength because they are sat almost underneath the AP (1832 APs) yet but struggling to get more than 1Mbps throughput. I can see the channel utilisation is over 50% but zero TX/RX, which I assume is a measurement of wifi frames in the channel? We have sub 1ms latency from WLC to the AP on the wired network.

I'm not sure if this has only happened since upgrading to 17.9.4a from 17.9.3 (only done to close IOS XE HTTPS vuln). Out of curiosity, would upgrading WLC and APs cause all APs to change channel or do they remember their last channel and use it post upgrade? Reason for asking is that I have the same issue on multiple APs, which we managed to fix by manually changing the channel.

From what I can understand from the output below, there are no 5Ghz wifi interferers, RSSI is good and SNR is excellent. Does this mean it's likely to be a non-wifi interferer?

We do have Ekahau Pro and a Sidekick, so we can try to identify get to this location and scan the 5Ghz bands to see if we can see an interferer.

MA-WLC-HA#show ap name MGH.1.27.AP24 auto-rf dot11 5ghz
###################################################################

Number of Slots : 2
AP Name : MGH.1.27.AP24
MAC Address : 501c.b0b1.b3a0
Ethernet MAC Address : 501c.b0b0.5368
Slot ID : 1
Radio Type : 802.11ac
Subband Type : All

Noise Information
Noise Profile : Passed
Channel 36 : -102 dBm
Channel 40 : -102 dBm
Channel 44 : -102 dBm
Channel 48 : -102 dBm
Channel 52 : -103 dBm
Channel 56 : -102 dBm
Channel 60 : -103 dBm
Channel 64 : -102 dBm
Channel 100 : -102 dBm
Channel 104 : -102 dBm
Channel 108 : -102 dBm
Channel 112 : -102 dBm
Channel 116 : -100 dBm
Channel 132 : -100 dBm
Channel 136 : -101 dBm
Channel 140 : -101 dBm

Interference Information
Interference Profile : Passed
Channel 36 : -128 dBm @ 0% busy
Channel 40 : -128 dBm @ 0% busy
Channel 44 : -128 dBm @ 0% busy
Channel 48 : -128 dBm @ 0% busy
Channel 52 : -128 dBm @ 0% busy
Channel 56 : -128 dBm @ 0% busy
Channel 60 : -128 dBm @ 0% busy
Channel 64 : -128 dBm @ 0% busy
Channel 100 : -128 dBm @ 0% busy
Channel 104 : -128 dBm @ 0% busy
Channel 108 : -128 dBm @ 0% busy
Channel 112 : -128 dBm @ 0% busy
Channel 116 : -128 dBm @ 0% busy
Channel 132 : -128 dBm @ 0% busy
Channel 136 : -128 dBm @ 0% busy
Channel 140 : -128 dBm @ 0% busy

Rogue Histogram (20/40/80)
Channel 36 : 0/ 0/ 0
Channel 40 : 0/ 0/ 0
Channel 44 : 0/ 0/ 0
Channel 48 : 0/ 0/ 0
Channel 52 : 0/ 0/ 0
Channel 56 : 0/ 0/ 0
Channel 60 : 0/ 0/ 0
Channel 64 : 0/ 0/ 0
Channel 100 : 0/ 0/ 0
Channel 104 : 0/ 0/ 0
Channel 108 : 0/ 0/ 0
Channel 112 : 0/ 0/ 0
Channel 116 : 0/ 0/ 0
Channel 132 : 0/ 0/ 0
Channel 136 : 0/ 0/ 0
Channel 140 : 0/ 0/ 0

Load Information
Load Profile : Passed
Receive Utilization : 0%
Transmit Utilization : 0%
Channel Utilization : 55%
Attached Clients : 7 clients

Coverage Information
Coverage Profile : Passed
Failed Clients : 0 clients

Client Signal Strengths
RSSI -100 dBm : 0 clients
RSSI -92 dBm : 0 clients
RSSI -84 dBm : 0 clients
RSSI -76 dBm : 0 clients
RSSI -68 dBm : 0 clients
RSSI -60 dBm : 2 clients
RSSI -52 dBm : 5 clients

Client Signal to Noise Ratios
SNR 0 dB : 0 clients
SNR 5 dB : 0 clients
SNR 10 dB : 0 clients
SNR 15 dB : 0 clients
SNR 20 dB : 0 clients
SNR 25 dB : 0 clients
SNR 30 dB : 0 clients
SNR 35 dB : 0 clients
SNR 40 dB : 1 clients
SNR 45 dB : 6 clients

Nearby APs
AP 501c.b0b1.9daf slot 1 : -34 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.a2ef slot 1 : -53 dBm on (140, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bd2f slot 1 : -54 dBm on ( 36, 20 MHz) (10.90.160.116)
AP 501c.b0b1.9d4f slot 1 : -76 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.b78f slot 1 : -79 dBm on (100, 20 MHz) (10.90.160.116)
AP 501c.b0b1.ba0f slot 1 : -82 dBm on ( 48, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bc2f slot 1 : -82 dBm on ( 64, 20 MHz) (10.90.160.116)

Radar Information
Channel changes due to radar : 0

Channel Assignment Information via DCA
Current Channel Average Energy : -86 dBm
Previous Channel Average Energy : -86 dBm
Channel Change Count : 0
Last Channel Change Time : 11/02/2023 23:14:35
Recommended Best Channel : 132

RF Parameter Recommendations
Power Level : 6
RTS/CTS Threshold : 2347
Fragmentation Threshold : 2346
Antenna Pattern : 0

Persistent Interference Devices
Class Type Channel DC (%%) RSSI (dBm) Last Update Time
------------------------- ------- ------ --------- ----------------
All third party trademarks are the property of their respective owners.

MA-WLC-HA#show ap name MGH.1.27.AP24 neighbor summary
BSSID Channel Channel-width Slot RSSI Last-Heard SSID Neighbour
--------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------
b095.7582.a5ca 3 40 Mhz 0 -88 11/16/2023 09:32:27 TP-Link_A5CA FALSE

MA-WLC-HA#

Thanks,

Stuart

Leo Laohoo · ‎11-16-2023

What country is this?

I ask because those allowed DCA channels are right in the middle of DFS channels.

Stuart Patton · ‎11-16-2023

UK

JPavonM · ‎11-16-2023

RSSI and SNR figures from the AP point of view are good for the conected wclients so expect high data rates.

But having a high link speed is not synonymous of having high throughput as that depend on RF conditions such as the number of connected clients that increase contention, and the load of the RF medium that also increase contention.

I see 55% channel utilization under that channel which is a bit high, maybe due to high load applications or multiple transfer at the same time. Is such a high figure normal on that site?

Can you see if that channel is been used by nearby APs from neighbours?

I would advise to upgrade some devices drivers/firmwares to see how they behave.

Stuart Patton · ‎11-16-2023

You can see the neighbour APs and their channels in the output, and this one itself is on channel 132 so does not appear to be CCI from neighbours or rogues.

We only upgraded a week ago and as to be expected it takes time for tickets to trickle in but we are seeing issues across the site, which we weren't prior to 17.9.4a. With regards to channel utilisation, I am seeing the same thing (more than 50%) on APs even without any clients associated.

My question about whether APs remember channels prior to upgrade/reload was to understand if we've just been lucky in avoiding interference in the past and this has triggered a mass change of channels or not.

We do have another site on the same version with 9120AX APs but not getting any tickets there, so I'm just running the "show tech wireless" through WCAE as it does a great job of tabulating all the output, just to see if this is something specific to the 1832s.

JPavonM · ‎11-16-2023

But that neighbour APs are from your infrastructure and not from overlapped networks OBSSID (those not from your company). With Ekahau Live/Analyzer you should be able to look for them. RF is a shared medium and it's used by all APs from your comapny and neighbours so maybe the later ones impacting utilization.

On a side note, not too much time ago I got a similar issue seeing association and disconnection issues on clients. After using Ekahau I discovered a narrow interferer in the same channel the AP was operating (136) and after locating it, and removing it, clients stopped from suffering the issues. That time the interferer was also impacting RF utilization, but due to the way Cisco RRM works, it wasn't avoiding it and I couldn't get the interferer to further investigate with BU/DE to fix that.

Regarding the channel selection, yes Cisco AP (and normally all vendors) use the last good known channel after reboot if that channel is not been marked as not fair by RRM, shich might be your case. Try manually changin channel on that AP to another UNII band and see how it looks like.

Stuart Patton · ‎11-16-2023

Sure, I get wifi is a shared medium but it's not a shared building - it's a hospital and it is well away from any other buildings. We manage the wifi in the building, including that from any suppliers that are based in the hospital, so apart from people running hotspots on their phones, for example, I don't expect there to be many other wifi devices.

The first command shows the other access points managed by the WLC, and the "show ap name <ap name> neighbor summary" shows other neighbours. I know that because when I run it against APs that are close to the car park (for example), I see the vehicle hotspot SSID names, eg Audi_xxx, VW_xxx. The affected AP is only showing a single 2.4Ghz neighbour, nothing in 5Ghz.

Leo Laohoo · ‎11-16-2023

Reboot all the 9120.

There are several known bugs about 911x/912x where the AP would routinely drop packets. The common Workaround is rebooting the APs or powering them off, wait for about 5 seconds and then powering them on again.

Stuart Patton · ‎11-17-2023

We don't appear to have the issue with our campus running 9120s, only the campus running 1832. At the 9210 campus, we do see a number of APs with high-ish channel utilisation but nowhere near 50%, and there is TX/RX % that can account for it.

JPavonM · ‎11-16-2023

I would also recommend you to check for DoS-like attack patterns sucha as managment frames floods. This could be increasing the RF utilization a lot and impacting users, but that could also addressing some of your users.

I've also seen that a lot from untrained administators enabling such features in platforms that provide this feature for free like Meraki. I would recommend you to enable awips in some APs there and look for such signatures, BUT do not enable any countermeasure nor containment (https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-9/config-guide/b_wl_17_9_cg/m_awips.html).

JPavonM · ‎11-17-2023

Then check RF Utilization in the campus with Aironet APs for excessive utilization (from nearby networks, maybe hotspots or other companies' APs), and check for any broadcast or packet flood events from legitimate or malicious users.

Stuart Patton · ‎12-06-2023

An update... We've visited the affected areas and used Ekahau with the sidekick and not found any obvious signs of interference. If the client is physically under the AP and roams to it, the throughput is terrible, we have this first-hand rather than from customer tickets now, with known-good client devices.

One of my team has now found this little gem: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwh75431 which is a 100% match for the AP version, symptoms and workaround (change channel). We're just exploring the factory reset on one of the affected units to see if this resolves the issue, otherwise we're in the realms of either upgrading or downgrading the WLC.

Stuart Patton · ‎12-06-2023

An update to my update... factory resetting the AP ("capwap ap erase all") appeared to work but after a few minutes the utilisation jumped back to earlier levels. During this process the AP has changed channels too, so it's definitely not interference. Looking like an upgrade/downgrade is the only option.

casanavep · ‎12-06-2023

Rather than 17.9.4a to fix the HTTP/2 vulnerability, have you tried 17.9.4 + the HTTP/2 fix SMU? This would give you access to AP Service Pack 6 which includes dozens of 1815w related fixes. 17.9.4a also shows access to APSP 6, but it's a bad Cisco release. The release notes show AP firmware 17.9.4.206 (SP6 roll up) but it actually pushed 17.9.4.202 (SP2). Cisco's yet to fix that poorly published AP SP image and related release notes, so you'd need to roll back to 17.9.4 then add the SMU and SP6 to get all the fixes contained in AP firmware 17.9.4.206.

Stuart Patton · ‎12-07-2023

Thanks for the suggestion. I'll feed this back to my team and our partner to see what they think. I'm assuming ap-17.9.5.38 (which the bug says fixes the issue) is part of WLC 17.9.5, so I'll see if our partner can get a date out of Cisco as to when this will be released.