9166 XOR radio slow in 5 GHz mode

[eglinsky2012]

While attempting to work around a different issue, I realized that the XOR Slot 2 radios in Cisco 9166s provide very slow throughput when in 5GHz mode when they've been up for a time. Throughput is as low as 30Mbps up and down in an empty room with a single client, while the Slot 1 5 GHz radio will provide over 200Mbps up and down. I tried this on 3 APs in the same room with the same result.

It does not seem load related; when I made the 6 GHz to 5 GHz switch on Slot 2 on an AP here in the office, I saw the issue immediately. It occurs on different channels and both 20 and 40 MHz widths. I even tried setting Slot 1 to the same channel and width that Slot 2 was using and the speed was good. Slot 2 in 6 GHz mode provides several hundred Mbps up and down. The wireless spectrum is clean, no interference. The Slot 2 speed improves for a time when the AP reboots but slows down eventually (I rebooted the office AP yesterday afternoon, and it was slow again this morning).

I noticed this on 17.9.4a/APSP8 and it continues on 17.12.4/APSP2. I'm testing with a Dell laptop with an Intel AX211 NIC with NetSetMan to connect to specific BSSIDs. I have not tested a 9136 to see if its 8x8 5 GHz radio, when split to dual 4x4 5 GHz radios, has the same issue.

I did open a case, SR 697995892. I provided an AP show tech-support before and after a reboot. TAC asked for client debugs on the AP side with the client connected on both Slot 1 and Slot 2 as well as OTA captures. I'm waiting to hear back on their analysis of those.

Has anyone else seen this issue? If anyone is thinking of utilizing Slot 2 as secondary 5 GHz on a 9166, you might want to do some testing first.

[Leo Laohoo]

What happens if the APs are rebooted?

[eglinsky2012]

The Slot 2 speed improves (matches Slot 1) for a time when the AP reboots but slows down eventually (I rebooted the office AP one afternoon, and it was slow again the next morning).

Since I wrote this post (which I posted elsewhere a few days ago), I worked with TAC to troubleshoot the issue again. This time, they had me run the following command after testing Slot 1:

test crash radiofw 2 1 1

This is the conclusion of the gobledygook that ensued:

Oct 16 13:17:04 HBL-A66 /usr/bin/cisco_core.sh: +=== Radio Firmware version: QC_IMAGE_VERSION_STRING=WLAN.HK.2.7-04674-QCAHKSWPL_SILICONZ-1
Oct 16 13:17:04 HBL-A66 kernel: [*10/16/2024 13:17:04.8014] /usr/bin/cisco_core.sh: +=== Radio Firmware version: QC_IMAGE_VERSION_STRING=WLAN.HK.2.7-04674-QCAHKSWPL_SILICONZ-1
Oct 16 13:17:04 HBL-A66 /usr/bin/cisco_core.sh: +=== FW crashed on Radio 2
Oct 16 13:17:36 HBL-A66 kernel: [*10/16/2024 13:17:36.4467] Unit grpcd.service could not be found.

TAC mentioned there was a "known issue" and they were confirming if it is still occurring on this version (17.12.4/APSP2).

A subsequent speed test on Slot 2 performed well, as fast as Slot 1... which TAC said was because of the crash during the preceding test causing the radio to reboot.

That was yesterday; today, Slot 2 is still performing well (>200Mbps up and down on a 20 MHz channel).

Also, I saw your first post (which is deleted). The Intel AX211 driver version is 23.60.1.2 (latest available from Dell). As for other devices, it's hard to get most devices to connect to a specific radio. However, in the classroom the issue was occurring in while Slot 2 was in 5 GHz mode, I saw random MacBooks and PCs that had very slow throughput while others were fine.

[Leo Laohoo]

---

@eglinsky2012 wrote:
Also, I saw your first post (which is deleted).

---

I deleted it because it was a stupid question and has no place in this thread.  

---

@eglinsky2012 wrote:

Oct 16 13:17:04 HBL-A66 /usr/bin/cisco_core.sh: +=== FW crashed on Radio 2

---

I have completely lost count of how many Bug IDs that causes the radio(s) to crash.  However, the worse one I've seen is CSCwk55224.  This bug, in my opinion, is very bad.  Instead of crashing the radio(s), this bug will take the AP down.  

Going back to the issue, the only thing I can think of going back to my opinion of "daily AP reboot".  

[eglinsky2012]

Update on this issue - A bug has been identified: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwn03468

We were provided a debugging image on a few APs and my colleague and I were on a WebEx with several Cisco and Qualcomm employees for 3.5 hours yesterday. After lots of debugging commands and OTAs and speed tests, it was determined that the ampdu value of slot 2 gets set to 0 when the radio gets switched from 6 to 5 GHz (and in other circumstances, as determined by my above note that the issue is only temporarily resolved after rebooting the AP). When they manually changed the value to other than 0 (such as 7 or 255 if I recall), the throughput was normal (>200Mbps reliably), and every time they changed it back to 0, throughput tanked (<50Mbps). It sounds like an application issue on the Cisco side setting that value, and they now have to determine how/why that's happening and correct it.

[Leo Laohoo]

Funny that.  

I saw the email come in about CSCwm65537 & CSCwn03468 just last night.  

Thanks for the detailed update, @eglinsky2012.

[eglinsky2012 · Accepted solution]

The bug is reportedly resolved in 17.12.5 and 17.15.3. For various reasons we will not be upgrading to either release in the near future, but for now I'll mark this as the solution.