Poor 5Ghz throughput - 9800 / 17.9.4a / 1800-series APs

[Stuart Patton]

Hi,

Per the title, I've got an issue with 5Ghz clients that have good signal strength because they are sat almost underneath the AP (1832 APs) yet but struggling to get more than 1Mbps throughput.  I can see the channel utilisation is over 50% but zero TX/RX, which I assume is a measurement of wifi frames in the channel?  We have sub 1ms latency from WLC to the AP on the wired network.

I'm not sure if this has only happened since upgrading to 17.9.4a from 17.9.3 (only done to close IOS XE HTTPS vuln).  Out of curiosity, would upgrading WLC and APs cause all APs to change channel or do they remember their last channel and use it post upgrade?   Reason for asking is that I have the same issue on multiple APs, which we managed to fix by manually changing the channel.

From what I can understand from the output below, there are no 5Ghz wifi interferers, RSSI is good and SNR is excellent.  Does this mean it's likely to be a non-wifi interferer?  

We do have Ekahau Pro and a Sidekick, so we can try to identify get to this location and scan the 5Ghz bands to see if we can see an interferer.

MA-WLC-HA#show ap name MGH.1.27.AP24 auto-rf dot11 5ghz
###################################################################

Number of Slots : 2
AP Name : MGH.1.27.AP24
MAC Address : 501c.b0b1.b3a0
Ethernet MAC Address : 501c.b0b0.5368
Slot ID : 1
Radio Type : 802.11ac
Subband Type : All

Noise Information
Noise Profile : Passed
Channel 36 : -102 dBm
Channel 40 : -102 dBm
Channel 44 : -102 dBm
Channel 48 : -102 dBm
Channel 52 : -103 dBm
Channel 56 : -102 dBm
Channel 60 : -103 dBm
Channel 64 : -102 dBm
Channel 100 : -102 dBm
Channel 104 : -102 dBm
Channel 108 : -102 dBm
Channel 112 : -102 dBm
Channel 116 : -100 dBm
Channel 132 : -100 dBm
Channel 136 : -101 dBm
Channel 140 : -101 dBm

Interference Information
Interference Profile : Passed
Channel 36 : -128 dBm @ 0% busy
Channel 40 : -128 dBm @ 0% busy
Channel 44 : -128 dBm @ 0% busy
Channel 48 : -128 dBm @ 0% busy
Channel 52 : -128 dBm @ 0% busy
Channel 56 : -128 dBm @ 0% busy
Channel 60 : -128 dBm @ 0% busy
Channel 64 : -128 dBm @ 0% busy
Channel 100 : -128 dBm @ 0% busy
Channel 104 : -128 dBm @ 0% busy
Channel 108 : -128 dBm @ 0% busy
Channel 112 : -128 dBm @ 0% busy
Channel 116 : -128 dBm @ 0% busy
Channel 132 : -128 dBm @ 0% busy
Channel 136 : -128 dBm @ 0% busy
Channel 140 : -128 dBm @ 0% busy

Rogue Histogram (20/40/80)
Channel 36 : 0/ 0/ 0
Channel 40 : 0/ 0/ 0
Channel 44 : 0/ 0/ 0
Channel 48 : 0/ 0/ 0
Channel 52 : 0/ 0/ 0
Channel 56 : 0/ 0/ 0
Channel 60 : 0/ 0/ 0
Channel 64 : 0/ 0/ 0
Channel 100 : 0/ 0/ 0
Channel 104 : 0/ 0/ 0
Channel 108 : 0/ 0/ 0
Channel 112 : 0/ 0/ 0
Channel 116 : 0/ 0/ 0
Channel 132 : 0/ 0/ 0
Channel 136 : 0/ 0/ 0
Channel 140 : 0/ 0/ 0

Load Information
Load Profile : Passed
Receive Utilization : 0%
Transmit Utilization : 0%
Channel Utilization : 55%
Attached Clients : 7 clients

Coverage Information
Coverage Profile : Passed
Failed Clients : 0 clients

Client Signal Strengths
RSSI -100 dBm : 0 clients
RSSI -92 dBm : 0 clients
RSSI -84 dBm : 0 clients
RSSI -76 dBm : 0 clients
RSSI -68 dBm : 0 clients
RSSI -60 dBm : 2 clients
RSSI -52 dBm : 5 clients

Client Signal to Noise Ratios
SNR 0 dB : 0 clients
SNR 5 dB : 0 clients
SNR 10 dB : 0 clients
SNR 15 dB : 0 clients
SNR 20 dB : 0 clients
SNR 25 dB : 0 clients
SNR 30 dB : 0 clients
SNR 35 dB : 0 clients
SNR 40 dB : 1 clients
SNR 45 dB : 6 clients

Nearby APs
AP 501c.b0b1.9daf slot 1 : -34 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.a2ef slot 1 : -53 dBm on (140, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bd2f slot 1 : -54 dBm on ( 36, 20 MHz) (10.90.160.116)
AP 501c.b0b1.9d4f slot 1 : -76 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.b78f slot 1 : -79 dBm on (100, 20 MHz) (10.90.160.116)
AP 501c.b0b1.ba0f slot 1 : -82 dBm on ( 48, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bc2f slot 1 : -82 dBm on ( 64, 20 MHz) (10.90.160.116)

Radar Information
Channel changes due to radar : 0

Channel Assignment Information via DCA
Current Channel Average Energy : -86 dBm
Previous Channel Average Energy : -86 dBm
Channel Change Count : 0
Last Channel Change Time : 11/02/2023 23:14:35
Recommended Best Channel : 132

RF Parameter Recommendations
Power Level : 6
RTS/CTS Threshold : 2347
Fragmentation Threshold : 2346
Antenna Pattern : 0

Persistent Interference Devices
Class Type Channel DC (%%) RSSI (dBm) Last Update Time
------------------------- ------- ------ --------- ----------------
All third party trademarks are the property of their respective owners.

MA-WLC-HA#show ap name MGH.1.27.AP24 neighbor summary
BSSID Channel Channel-width Slot RSSI Last-Heard SSID Neighbour
--------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------
b095.7582.a5ca 3 40 Mhz 0 -88 11/16/2023 09:32:27 TP-Link_A5CA FALSE

MA-WLC-HA#

Thanks,

Stuart

[patoberli]

Cisco now updated the bug CSCwh75431 for the 1830/1950 and lists the affected/fixed releases.

Affected as of bug information now:

ap-17.9.4.27

ap-17.6.6.66

ap-17.3.7.36

ap-17.12.1.5

17.12.2

008.010(185.000)

Fixed:

ap-17.9.5.47

ap-17.9.5.38

ap-17.9.4.159

ap-17.6.7.12

ap-17.14.0.34

ap-17.12.3.4

ap-17.12.3.31

ap-17.12.2.152

Dublin-17.12.3

Cupertino-17.9.5

CG-17.9.5.108

CG-17.14.1.80

CG-17.12.2.121

AP.CI.6756

AP.CI.11.4.11.243

 They have now also updated the bug CSCwi92439  for the 1815.

Affected: 

ap-17.9.4.208 (-> APSP8)

8.10.185.0

Fixed:

ap-17.9.6.2

ap-17.9.5.152 (-> 17.9.5 APSP1)

ap-17.6.7.21

ap-17.15.0.31

ap-17.14.0.63

ap-17.12.3.31

Dublin-17.12.3

CG-17.9.6.19

CG-17.14.1.181

CG-17.12.3.44

AP.CI.6985

AP.CI.11.4.12.169

008.010(190.008)

[Leo Laohoo]

Avoid 17.9.4 + APSP 8 if there are 9136 or 916x -- CSCwj45141

[eglinsky2012]

@Leo Laohoo, thank you for posting that bug! Sounds like the cause of the issue I discussed in this thread: https://community.cisco.com/t5/wireless/9166-clients-not-connecting-at-5-ghz/td-p/5008408 Shame on TAC that I heard about it here before I heard about it from them....

Anyway, regarding CSCwi92439, I contacted TAC to obtain 8.10.190.8 and they sent me 8.10.190.9 instead since .8 was "retired". From its release notes, below is a list of the 8.10.190.x escalation builds and their resolved bugs if anyone's interested. Fixes are cumulative.

I asked if I should reach back out to them to see if there's a newer version if I'm unable to do the upgrade until summer. They said that can't hurt but to check if there's a newer publicly-available version by then first.

8.10.190.9
CSCwi92439 AP1815s are reporting high channel utilization in 5Ghz.
CSCwj04904 9300LM switch is not compatible with 1815 WAP + 7945G IP phone connected together on a port

8.10.190.7
CSCwh74663 2800/3800/4800/1560/6300 not sending QoS data frames downstream
CSCwi67013 Unable to set channels (52, 120, 124, 128) in AP2800-T domain
CSCwd46252 WLC shows AP no neighbor caused radio0/1 power level set to maximum

8.10.190.6
CSCwf53520 Cisco 1815 AP running version 17.9.2: Kernel panic crash observed
CSCwh27366 AP radio firmware crash reset code 2 with crash signature gdp
CSCwh02986 AP transmit power in dbm is not matching between WLC/AP and Ekahau survey
CSCwf65794 Cisco 1852 AP reloads unexpectedly due to radio failure (radio recovery failed)

[leo.laohoo]

CSCwi53998:  1815 APs neighbor RSSI is 0dBm

[Leo Laohoo]

CSCwj45141 has been updated with 17.9.5 added into the Known Affect Release.

[eglinsky2012]

AireOSs 8.10.196.0 was recently released publicly. Its release notes do not specify that CSCwi92439 for 1815 5 GHz is resolved nor that it's still an issue. I opened SR 697378246 to inquire of the status and was initially told that that version was not released to the public, then that the release notes did not say that the bug was not resolved and that the escalation build (8.10.190.9) should be used since it specifies the bug is resolved. Since fixes are usually cumulative, I pressed further and after internal investigation from TAC, it turns out that the bug IS resolved in the 196 version. They said since version 196 is not yet the golden recommended release, the release notes are not yet fully updated. I'll be upgrading to this version this summer. Time will tell what new issue inevitably pops up....

[Rich R]

We've also got 196 on the roadmap, particularly for CSCwh61011 which we believe we've been hit by a number of times although TAC still could not confirm that was the cause!
The general release notes never include all fixes - only BU selected sev 1, 2 and sometimes 3 bugs, so the idea that they might be updated to include all fixes is just rubbish.  On the other hand escalation build release notes usually do include (almost) all fixes (except those considered security, sensitive or internal only).  The trouble with many new junior TAC staff is they tend not to know how these things work so often you will know more about it than they do (as was clearly the case here)!  The only advantage they have is that they can make internal enquiries to someone who does know, but it can be so difficult to persuade them to do that these days!

[Stuart Patton]

I know this is not strictly related to the original issue, but given the amazing number of contributors to the thread I thought I'd make you aware of this particular bug hitting 17.9 train: CSCwj73634 

[eglinsky2012]

That's a nasty one. Haven't seen it here... yet. But, don't tell my 9800s I said that.

[Leo Laohoo]

I like to call the "repm" the "redundancy port manager" and is an IOS-XE VSS/Stack process that is responsible for "distributing" the config (and other data), from the stack master, to the entire stack.  N+1 deployment does not use the "repm" process.

[Rich R]

Thanks for that @Stuart Patton - interesting that there's a bug for this now (only opened in April and already 22 cases attached!).  We opened a TAC case for this in Feb 2022 and TAC eventually closed the case in July 2022 after concluding they had no root cause and could not reproduce in their lab (so they would not open a bug for it).  Luckily we haven't seen it since then.  What we observed is that the WLC comes back without any of the wireless config, only the base IOS-XE config.  The running-config reduced from 19,003 lines (which is still in startup) to 950 lines - it just ignores all that startup config!  In our case we have SSO pair + SSO pair (N+1) resilience in different data centres so the APs moved to the other WLC SSO pair causing minimal customer disruption but would clearly be a disaster if you relied purely on the affected SSO pair.

The bug says "Issue has only been observed on 17.9 train code." but we saw it on 17.6 code so they've only taken it seriously on 17.9 but it has definitely existed since at least 17.6.

[Stuart Patton]

I've been advised by our partner that the notes on CSCwj73634 have been updated.  Apparently the issue has only been seen if you use intermediate switching between your WLC's RP ports.  There is an SMU available that fixes the issue linked on the bug page.

[Rich R]

Thanks for the update Stuart.

Ours are all connected back-to-back and we saw it, so again not 100% true!  Probably more accurate to say switched connections increase the likelihood of encountering the issue.

[eglinsky2012]

I upgraded one AireOS controller to the 196.0 code yesterday morning and I'm still seeing the same high interference/channel utilization in APs in mostly-empty buildings, even after a subsequent reboot of one building's APs, which makes me wonder if either the fix didn't work or it wasn't actually included in the 196.0 code. I haven't received end-user feedback on whether performance has improved yet.

Has anyone confirmed a resolution in any version, or upgraded but still sees the issue, whether it's with IOS or AireOS?

The bug was reportedly resolved starting in IOS 17.9.5 APSP1. I was told by TAC that it was resolved in AireOS escalation build 8.10.190.9, and in the subsequent 8.10.196.0 (even though it's not explicitly stated in the 196.0 release notes or BST).

[casanavep]

Have you tried power cycling the APs? We found this requirement with 17.9.5 APSP 1.