Poor 5Ghz throughput - 9800 / 17.9.4a / 1800-series APs

[Stuart Patton]

Hi,

Per the title, I've got an issue with 5Ghz clients that have good signal strength because they are sat almost underneath the AP (1832 APs) yet but struggling to get more than 1Mbps throughput.  I can see the channel utilisation is over 50% but zero TX/RX, which I assume is a measurement of wifi frames in the channel?  We have sub 1ms latency from WLC to the AP on the wired network.

I'm not sure if this has only happened since upgrading to 17.9.4a from 17.9.3 (only done to close IOS XE HTTPS vuln).  Out of curiosity, would upgrading WLC and APs cause all APs to change channel or do they remember their last channel and use it post upgrade?   Reason for asking is that I have the same issue on multiple APs, which we managed to fix by manually changing the channel.

From what I can understand from the output below, there are no 5Ghz wifi interferers, RSSI is good and SNR is excellent.  Does this mean it's likely to be a non-wifi interferer?  

We do have Ekahau Pro and a Sidekick, so we can try to identify get to this location and scan the 5Ghz bands to see if we can see an interferer.

MA-WLC-HA#show ap name MGH.1.27.AP24 auto-rf dot11 5ghz
###################################################################

Number of Slots : 2
AP Name : MGH.1.27.AP24
MAC Address : 501c.b0b1.b3a0
Ethernet MAC Address : 501c.b0b0.5368
Slot ID : 1
Radio Type : 802.11ac
Subband Type : All

Noise Information
Noise Profile : Passed
Channel 36 : -102 dBm
Channel 40 : -102 dBm
Channel 44 : -102 dBm
Channel 48 : -102 dBm
Channel 52 : -103 dBm
Channel 56 : -102 dBm
Channel 60 : -103 dBm
Channel 64 : -102 dBm
Channel 100 : -102 dBm
Channel 104 : -102 dBm
Channel 108 : -102 dBm
Channel 112 : -102 dBm
Channel 116 : -100 dBm
Channel 132 : -100 dBm
Channel 136 : -101 dBm
Channel 140 : -101 dBm

Interference Information
Interference Profile : Passed
Channel 36 : -128 dBm @ 0% busy
Channel 40 : -128 dBm @ 0% busy
Channel 44 : -128 dBm @ 0% busy
Channel 48 : -128 dBm @ 0% busy
Channel 52 : -128 dBm @ 0% busy
Channel 56 : -128 dBm @ 0% busy
Channel 60 : -128 dBm @ 0% busy
Channel 64 : -128 dBm @ 0% busy
Channel 100 : -128 dBm @ 0% busy
Channel 104 : -128 dBm @ 0% busy
Channel 108 : -128 dBm @ 0% busy
Channel 112 : -128 dBm @ 0% busy
Channel 116 : -128 dBm @ 0% busy
Channel 132 : -128 dBm @ 0% busy
Channel 136 : -128 dBm @ 0% busy
Channel 140 : -128 dBm @ 0% busy

Rogue Histogram (20/40/80)
Channel 36 : 0/ 0/ 0
Channel 40 : 0/ 0/ 0
Channel 44 : 0/ 0/ 0
Channel 48 : 0/ 0/ 0
Channel 52 : 0/ 0/ 0
Channel 56 : 0/ 0/ 0
Channel 60 : 0/ 0/ 0
Channel 64 : 0/ 0/ 0
Channel 100 : 0/ 0/ 0
Channel 104 : 0/ 0/ 0
Channel 108 : 0/ 0/ 0
Channel 112 : 0/ 0/ 0
Channel 116 : 0/ 0/ 0
Channel 132 : 0/ 0/ 0
Channel 136 : 0/ 0/ 0
Channel 140 : 0/ 0/ 0

Load Information
Load Profile : Passed
Receive Utilization : 0%
Transmit Utilization : 0%
Channel Utilization : 55%
Attached Clients : 7 clients

Coverage Information
Coverage Profile : Passed
Failed Clients : 0 clients

Client Signal Strengths
RSSI -100 dBm : 0 clients
RSSI -92 dBm : 0 clients
RSSI -84 dBm : 0 clients
RSSI -76 dBm : 0 clients
RSSI -68 dBm : 0 clients
RSSI -60 dBm : 2 clients
RSSI -52 dBm : 5 clients

Client Signal to Noise Ratios
SNR 0 dB : 0 clients
SNR 5 dB : 0 clients
SNR 10 dB : 0 clients
SNR 15 dB : 0 clients
SNR 20 dB : 0 clients
SNR 25 dB : 0 clients
SNR 30 dB : 0 clients
SNR 35 dB : 0 clients
SNR 40 dB : 1 clients
SNR 45 dB : 6 clients

Nearby APs
AP 501c.b0b1.9daf slot 1 : -34 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.a2ef slot 1 : -53 dBm on (140, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bd2f slot 1 : -54 dBm on ( 36, 20 MHz) (10.90.160.116)
AP 501c.b0b1.9d4f slot 1 : -76 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.b78f slot 1 : -79 dBm on (100, 20 MHz) (10.90.160.116)
AP 501c.b0b1.ba0f slot 1 : -82 dBm on ( 48, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bc2f slot 1 : -82 dBm on ( 64, 20 MHz) (10.90.160.116)

Radar Information
Channel changes due to radar : 0

Channel Assignment Information via DCA
Current Channel Average Energy : -86 dBm
Previous Channel Average Energy : -86 dBm
Channel Change Count : 0
Last Channel Change Time : 11/02/2023 23:14:35
Recommended Best Channel : 132

RF Parameter Recommendations
Power Level : 6
RTS/CTS Threshold : 2347
Fragmentation Threshold : 2346
Antenna Pattern : 0

Persistent Interference Devices
Class Type Channel DC (%%) RSSI (dBm) Last Update Time
------------------------- ------- ------ --------- ----------------
All third party trademarks are the property of their respective owners.

MA-WLC-HA#show ap name MGH.1.27.AP24 neighbor summary
BSSID Channel Channel-width Slot RSSI Last-Heard SSID Neighbour
--------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------
b095.7582.a5ca 3 40 Mhz 0 -88 11/16/2023 09:32:27 TP-Link_A5CA FALSE

MA-WLC-HA#

Thanks,

Stuart

[Leo Laohoo]

---

@Jason Salmans wrote:
They've also asked for some additional CLI and over-the-air captures so I'll have to work on that as I can.

---

Back in 2022, we had a problem with wireless voice traffic and 4800.  They asked several hundred hours of over-the-air captures and wireshark captures.  The entire case dragged on for months while the hospital suffered.  The TAC engineer slipped when recommended the 4800 be replaced with 9130 (we suspect we were hitting the MARVEL chipset hardware bug) but he never put that down in writing.  They did make a promise that "this issue would be fixed" if we migrated all our 4800 to IOS-XE.  Well, guess what.  We upgraded.  The issue remains.  

Don't put any faith in this.  The 1800 are low-end (aka low-quality) APs and approaching End-of-Life.  Unless this network belongs to a large-customer-Cisco-cannot-ignore (aka a "whale"), expect TAC to request several thousand hours of OTA and wireshark captures and nothing will come in fruition.  You might try to get a WLAN TAC engineer to come to the site and do the captures themselves so you come to an event where TAC will turn around and say, "oh, we forgot something and can you do more captures with this-and-that scenario."

Good luck.

[Rich R]

> TAC will turn around and say, "oh, we forgot something and can you do more captures with this-and-that scenario."
That made me laugh - had that more than once last year.  Ask TAC to make sure dev team have everything they need before we clear the fault - YES.  2 or 3 weeks later - dev team say they can't progress any more on this without the following additional captures .... which can no longer be done because the fault is cleared so case gets closed and we wait for it to happen again so we can get the additional captures next time around! So frustrating!

[Leo Laohoo]

Cisco will try to drag this one out until End-of-Support date or the client loses interest (whichever comes first).

[Jason Salmans]

Would it be helpful at all to post case #s here for each of us to reference with TAC?

[p.lan]

Absolutely, I agree.
our case: 1815W 5Ghz (false high utilization) - SR 696789741
Another case our SE found: SR 696090884
Another case that we heard was opened via EDUCAUSE: SR 696827014 (likely just one of many that other EDUCAUSE members opened).

[Jason Salmans]

Mine is SR 696644347: 1815w units reporting high 5ghz utilization

[Rich R]

As 17.9.5 is released now it would be interesting to see if you guys still observe the problem on 17.9.5 because that's likely what TAC will suggest next anyway ...
If you do, then at least you can go back to TAC and tell them it's not fixed in 17.9.5 either.

[Stuart Patton]

Ours was SR 696634093

We now have a separate issue where we have conducted failure testing on our core switches, which resulted in our WLCs switching over (operating in SSO) and some 1832 APs lost connectivity and are now unable to join (9105s and 9120s are also affected too).  I'm not sure if that will result in a new TAC case, but may mean we have to try 17.9.5.  If it does, I will ask our next TAC engineer to confirm the fix for CSCwh75431 has been rolled into the AP image to avoid reintroducing the last problem.

In general terms, we still see a small number of APs with high utilisation, and sometimes that can be explained with high numbers of clients or high TX/RX utilisation.  Sometimes it can not be explained.  Our TAC engineer on this case has confirmed that the channel utilisation figure is periodically updated so is not always representative of the point in time when you run the "show ap name <ap name> auto-rf dot11 5ghz" command, whereas the number of attached clients is at that point in time the show command is run.   We do not appear to have had any tickets from our users regarding wireless performance since 17.9.4a APSP8 so for us it appears to resolved the issues.

[Rich R]

CSCwh75431 says fixed in ap-17.9.5.47 which is the 17.9.5 released version so it should have the fix.

[p.lan]

It was also listed as fixed in 17.9.4a APSP8, but that was partially incorrect. What we learned is that it was fixed for most APs except the 1815W, which most of us are complaining about in this thread as of late.

I suppose its possible they included the 1815W fix in CCO version of 17.9.5, but our engineer has not told us that yet. We will ask our engineer and see if this is true.

[t-nitta]

The AP1815 issue seems to separated from CSCwh75431 to CSCwi92631.
Currently, no known fix release announcements for bug CSCwi92631.

[Rich R]

Thanks for that @t-nitta 

CSCwi92631 is showing Status: Duplicate but no link to bug which fixes it (dev teams have become so sloppy).  They might have made sure the release version of CSCwh75431 includes 1815W.

@p.lan can you ask TAC to confirm?

[p.lan]

---

@Rich R wrote:

Thanks for that @t-nitta 

CSCwi92631 is showing Status: Duplicate but no link to bug which fixes it (dev teams have become so sloppy).  They might have made sure the release version of CSCwh75431 includes 1815W.

@p.lan can you ask TAC to confirm?

---

While no mention of that specific new Bug ID, this is what our TAC engineer said to us yesterday:

It was initially believed that a previous software (APSP8) had resolved this error. However, it has come to light that the fix only applies to 1830/1850. Our developers are actively working on a new fix for AP1815s, specifically addressing the issue of high channel utilization on 5GHz. A version with a tentative fix is currently undergoing testing, and if successful, it will be integrated into 17.12.3, with an APSP planned for 17.9.5. Cisco places significant value on customer reports and feedback to continuously enhance the product.

[Jason Salmans]

I received a reply from TAC this morning on my ticket for high 5ghz utilization reported on the 1815w.  The software team indicates the behavior is a known issue and is being actively worked on.  Currently listed fixed under 17.9.6 and 17.12.3.  I've asked for estimated release times for those.

Suggested workaround is to change the channel width or reload the affected APs.  The issue with these for me is that RRM is doing these changes (technically DNA AI RRM) so even though I have it limited to a specific time of day, it's still going to be changing them automatically and I've seen in the AP RF history where the radio is fine until it changes channels and then suddenly has the high utilization issue.

[Leo Laohoo]

TAC can knock up an APSP faster than waiting for 17.9.6 &/or 17.12.3 release.  

Contact your Cisco AM/SE and push for an APSP release.