Poor 5Ghz throughput - 9800 / 17.9.4a / 1800-series APs

[Stuart Patton]

Hi,

Per the title, I've got an issue with 5Ghz clients that have good signal strength because they are sat almost underneath the AP (1832 APs) yet but struggling to get more than 1Mbps throughput.  I can see the channel utilisation is over 50% but zero TX/RX, which I assume is a measurement of wifi frames in the channel?  We have sub 1ms latency from WLC to the AP on the wired network.

I'm not sure if this has only happened since upgrading to 17.9.4a from 17.9.3 (only done to close IOS XE HTTPS vuln).  Out of curiosity, would upgrading WLC and APs cause all APs to change channel or do they remember their last channel and use it post upgrade?   Reason for asking is that I have the same issue on multiple APs, which we managed to fix by manually changing the channel.

From what I can understand from the output below, there are no 5Ghz wifi interferers, RSSI is good and SNR is excellent.  Does this mean it's likely to be a non-wifi interferer?  

We do have Ekahau Pro and a Sidekick, so we can try to identify get to this location and scan the 5Ghz bands to see if we can see an interferer.

MA-WLC-HA#show ap name MGH.1.27.AP24 auto-rf dot11 5ghz
###################################################################

Number of Slots : 2
AP Name : MGH.1.27.AP24
MAC Address : 501c.b0b1.b3a0
Ethernet MAC Address : 501c.b0b0.5368
Slot ID : 1
Radio Type : 802.11ac
Subband Type : All

Noise Information
Noise Profile : Passed
Channel 36 : -102 dBm
Channel 40 : -102 dBm
Channel 44 : -102 dBm
Channel 48 : -102 dBm
Channel 52 : -103 dBm
Channel 56 : -102 dBm
Channel 60 : -103 dBm
Channel 64 : -102 dBm
Channel 100 : -102 dBm
Channel 104 : -102 dBm
Channel 108 : -102 dBm
Channel 112 : -102 dBm
Channel 116 : -100 dBm
Channel 132 : -100 dBm
Channel 136 : -101 dBm
Channel 140 : -101 dBm

Interference Information
Interference Profile : Passed
Channel 36 : -128 dBm @ 0% busy
Channel 40 : -128 dBm @ 0% busy
Channel 44 : -128 dBm @ 0% busy
Channel 48 : -128 dBm @ 0% busy
Channel 52 : -128 dBm @ 0% busy
Channel 56 : -128 dBm @ 0% busy
Channel 60 : -128 dBm @ 0% busy
Channel 64 : -128 dBm @ 0% busy
Channel 100 : -128 dBm @ 0% busy
Channel 104 : -128 dBm @ 0% busy
Channel 108 : -128 dBm @ 0% busy
Channel 112 : -128 dBm @ 0% busy
Channel 116 : -128 dBm @ 0% busy
Channel 132 : -128 dBm @ 0% busy
Channel 136 : -128 dBm @ 0% busy
Channel 140 : -128 dBm @ 0% busy

Rogue Histogram (20/40/80)
Channel 36 : 0/ 0/ 0
Channel 40 : 0/ 0/ 0
Channel 44 : 0/ 0/ 0
Channel 48 : 0/ 0/ 0
Channel 52 : 0/ 0/ 0
Channel 56 : 0/ 0/ 0
Channel 60 : 0/ 0/ 0
Channel 64 : 0/ 0/ 0
Channel 100 : 0/ 0/ 0
Channel 104 : 0/ 0/ 0
Channel 108 : 0/ 0/ 0
Channel 112 : 0/ 0/ 0
Channel 116 : 0/ 0/ 0
Channel 132 : 0/ 0/ 0
Channel 136 : 0/ 0/ 0
Channel 140 : 0/ 0/ 0

Load Information
Load Profile : Passed
Receive Utilization : 0%
Transmit Utilization : 0%
Channel Utilization : 55%
Attached Clients : 7 clients

Coverage Information
Coverage Profile : Passed
Failed Clients : 0 clients

Client Signal Strengths
RSSI -100 dBm : 0 clients
RSSI -92 dBm : 0 clients
RSSI -84 dBm : 0 clients
RSSI -76 dBm : 0 clients
RSSI -68 dBm : 0 clients
RSSI -60 dBm : 2 clients
RSSI -52 dBm : 5 clients

Client Signal to Noise Ratios
SNR 0 dB : 0 clients
SNR 5 dB : 0 clients
SNR 10 dB : 0 clients
SNR 15 dB : 0 clients
SNR 20 dB : 0 clients
SNR 25 dB : 0 clients
SNR 30 dB : 0 clients
SNR 35 dB : 0 clients
SNR 40 dB : 1 clients
SNR 45 dB : 6 clients

Nearby APs
AP 501c.b0b1.9daf slot 1 : -34 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.a2ef slot 1 : -53 dBm on (140, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bd2f slot 1 : -54 dBm on ( 36, 20 MHz) (10.90.160.116)
AP 501c.b0b1.9d4f slot 1 : -76 dBm on ( 64, 20 MHz) (10.90.160.116)
AP 501c.b0b1.b78f slot 1 : -79 dBm on (100, 20 MHz) (10.90.160.116)
AP 501c.b0b1.ba0f slot 1 : -82 dBm on ( 48, 20 MHz) (10.90.160.116)
AP 501c.b0b1.bc2f slot 1 : -82 dBm on ( 64, 20 MHz) (10.90.160.116)

Radar Information
Channel changes due to radar : 0

Channel Assignment Information via DCA
Current Channel Average Energy : -86 dBm
Previous Channel Average Energy : -86 dBm
Channel Change Count : 0
Last Channel Change Time : 11/02/2023 23:14:35
Recommended Best Channel : 132

RF Parameter Recommendations
Power Level : 6
RTS/CTS Threshold : 2347
Fragmentation Threshold : 2346
Antenna Pattern : 0

Persistent Interference Devices
Class Type Channel DC (%%) RSSI (dBm) Last Update Time
------------------------- ------- ------ --------- ----------------
All third party trademarks are the property of their respective owners.

MA-WLC-HA#show ap name MGH.1.27.AP24 neighbor summary
BSSID Channel Channel-width Slot RSSI Last-Heard SSID Neighbour
--------------------------------------------------------------------------------------------------------------------------------------- ----------------------------------------
b095.7582.a5ca 3 40 Mhz 0 -88 11/16/2023 09:32:27 TP-Link_A5CA FALSE

MA-WLC-HA#

Thanks,

Stuart

[Stuart Patton]

Well, having looked into this, it's as clear as mud.  The APSP release notes do not say that this particular bug is fixed in 202 or 206.

Are you assuming the bug is not present because .202 and .206 are not listed in known affected releases?  However nor is it listed in the known fixed releases version list either.  My experience that it being missing in known affected releases is because TAC have not had a case on that version, whereas known fixed release is because they explicitly know what version(s) the fixes are in

[Rich R]

Thanks for that info @casanavep what a shocker - yet again Cisco software release ops QA non-existent - no testing and no checks!
Edit: just had a closer look - in fact the 17.9.4a APSP6 release notes say APSP version: 17.9.4.201, and that matches what I see in the APSP file itself and in the AP images!  My guess would be whoever built it made it 201 as it's the first APSP for 17.9.4a but they forgot to change the main version number to 4a so it should probably be called 17.9.4a APSP1 <smile>

I decided on 17.9.4 + SMU + APSP (so we're now running that) rather than 17.9.4a because at that point they hadn't released the 17.9.4a SP yet - glad I made that choice <smile>

[eglinsky2012]

@Rich R  - So the next question is, how to get the following SMUs that have been released for 17.9.4a but not 17.9.4?!

17.9.4a:

17.9.4:

[Rich R]

Interesting - they've just lost the plot!  All I can say is ask TAC
Also I have C9800-universalk9_wlc.17.09.04.CSCwh31966.SPA.smu.bin (Controller crash on WNCD process during DB abort) which seems to have disappeared now!

I had started answering your previous post but seem to have lost that: Do not use 17.9.5 - it's a standard support release which will never get fixes.  << Ignore that - wasn't thinking straight! And I agree that it's not fixed in the 17.9.4 APSPs.
But I have 13 x 1832 on my 9800-80 with 17.9.4 + C9800-universalk9_wlc.17.09.04.CSCwh31966.SPA.smu.bin + C9800-universalk9_wlc.17.09.04.CSCwh87343.SPA.smu.bin + C9800-universalk9_wlc.17.09.04.CSCwh93727.SPA.apsp.bin running for just over a week now and not seeing any of them doing that:
Channel Utilization : 1%
Channel Utilization : 2%
Channel Utilization : 3%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 4%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 1%
Channel Utilization : 2%

We were running 17.9.4 without the SMUs and APSP for about 3 months before that and no problems with those APs reported although I didn't specifically check them.  So might be worth trying that combo anyway?

[eglinsky2012]

---

@casanavep wrote:

...

17.9.4a also shows access to APSP 6, but it's a bad Cisco release. The release notes show AP firmware 17.9.4.206 (SP6 roll up) but it actually pushed 17.9.4.202 (SP2). Cisco's yet to fix that poorly published AP SP image and related release notes, so you'd need to roll back to 17.9.4 then add the SMU and SP6 to get all the fixes contained in AP firmware 17.9.4.206. 

---

Per the TAC recommended software sheet that was mentioned: "17.9.4a APSP6 (AP version: 17.9.4.201). 17.9.4a APSP6 includes same fixes as 17.9.4 APSP6 even though the AP version label is different than 17.6.4 APSP6."

The release notes for 17.9.4A APSP6 accordingly list 17.9.4.201 as the AP version. Are you saying that the actual image that appears on the APs once the APSP is installed is 17.9.4.202?

I have a change window scheduled for a few weeks out and an debating which route to take since 17.9.4a would also allow me to install the SMUs I mentioned previously that are not available in 17.9.4. But I think the right service pack is more important.

On the other hand, I have a TAC case open for a separate issue and I'm told they'll be providing me a pre-release version of 17.9.5 to address CSCwe83994 (even though it says 17.9.5 is affected; that was an older iteration of 17.9.5 and it is now fixed, allegedly), and presumably all the other 17.9.4 SMU/APSP fixes would be included as well.

Whichever option, I'm afraid that a) it'll cause more problems and/or b) it won't solve the current problems. There's just no winning these days, and like others have said, clear as mud, FUBAR.

[Rich R]

Yes damned if you do, damned if you don't.  However CSCwe83994 is fixed in 17.12.2a so in your position I'd be seriously considering that as long as it addresses any other bugs you need fixing.

[Leo Laohoo]

---

@eglinsky2012 wrote:
I have a TAC case open for a separate issue and I'm told they'll be providing me a pre-release version of 17.9.5 to address CSCwe83994

---

Cisco developers no longer have the time, the resources (as well as a "care" factor) to do software testing.  TAC has provided the "pre-release" software because Cisco has pushed the "responsibility of testing to the customer. 

[Leo Laohoo]

---

@Stuart Patton wrote:
Well, having looked into this, it's as clear as mud. 

---

@Stuart Patton, @eglinsky2012 

I have just returned from Cisco Live Melbourne (2023) and I met a US-based Cisco staff and I asked about the state of the Cisco documentation.  I interpreted the response as "corporate-level BS to create, edit and update documentation has reached a certain level of FUBAR level that no one (at a technical-level) wants to do it any more."

The era of Cisco's once well-known and well-acknowledged documentation library is long gone.  `tis over.  `tis, Game Over.  

 

[casanavep]

You may follow this, but it is a great link if you have not seen it:  https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html

I find this Recommended Cisco IOS XE Releases for Catalyst 9800 Wireless LAN Controllers document much more reliable than trusting the gold star when it comes to finding the right combination of IOS-XE + SMU + APSP to arrive at the best Wi-Fi outcomes

[Rich R]

@casanavep  you'll see that's the first line in my signature links and it's the only thing I ever recommend - which I do quite frequently.

That is directly maintained by TAC engineers which is why it's more trustworthy than the rest of the documentation.

[Stuart Patton]

@Leo Laohoo @Rich R @casanavep @eglinsky2012 thanks for the useful replies.  Keep them coming...

The reason I'm interested in 17.9 rather than 17.10 / 11 / 12 is because we are using SDA.  Although we have not implemented fabric wireless, we are using DNA Center to manage our network and according to the compatibility matrix (https://www.cisco.com/c/dam/en/us/td/docs/Website/enterprise/sda_compatibility_matrix/index.html) 17.12 is not yet supported.  Our preference is to use the long-life release trains, and ideally we want release-train parity with our switching.

[Ben Crissman]

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwh75431
This might be relevant for the high channel utilization no RX TX. 

[Stuart Patton]

We're already aware of this.  See my earlier post on page 1 of this thread on 6th December.

[Stuart Patton]

Just waiting for the latest from my engineer that did the upgrade last night, but it looks like we applied C9800-universalk9_wlc.17.09.04a.CSCwh93727.SPA.apsp.bin (which I think is 17.9.4a APSP6) which took the APs from 17.9.4.27 to 17.9.4.201 and this has *NOT* fixed the issue.  Still seeing the high utilisation with zero clients.

[Ben Crissman]

@Stuart Patton 
It looks like CSCwh75431 was fixed in 17.9.4a APSP8 which just released today.
https://www.cisco.com/web/software/286325254/166630/C9800-universalk9_wlc.17.09.04a.CSCwi44524.txt