topic Re: 9800 CLI/SSH wireless show command output slow in Wireless

9800 CLI/SSH wireless show command output slow

jasonm002 — Thu, 15 Sep 2022 12:33:19 GMT

Output of wireless related show commands like 'show wireless client summary detail', 'show ap dot11 5ghz summary', or 'show ap summary' on the 9800 CLI is very slow. It takes ~42 seconds to display data on ~2700 clients on a 9800-40 if you disable paging. This has been the case in 17.3, 17.6, and 17.9.

On the 5520 controller on any AireOS version >= 8.5 'show ap summary', 'show client summary', or 'show advanced 802.11a summary' only takes a couple of seconds at most to output around the same amount of clients and APs. I'm only noticing this because we're running wireless controllers with >=1000 APs and >=2700 clients.

Something is happening on the 9800 in all IOS XE versions that is causing a big performance hit to SSH output of wireless data which makes it very annoying to verify operational parameters for example.

I don't really have time to open a TAC case on this one, but this issue should be extremely easy to reproduce in a lab environment. Someone at Cisco with a lab environment should really spin up some virtual APs and clients and test it - it's a pretty big performance decrease going from 5520 to 9800 which is not what we want to see for a new flagship product I would expect.

Re: 9800 CLI/SSH wireless show command output slow

Mark Elsen — Thu, 15 Sep 2022 16:45:21 GMT

>...Someone at Cisco with a lab environment should really spin up some virtual APs and clients and test it

This forum is not populated by Cisco employees but by customers, to get that kind of commitment from Cisco you need to create a ticket (TAC). For the time being have a health checkup of the 9800 configuration with the CLI command : show tech wireless , have the output analyzed by https://cway.cisco.com/tools/WirelessAnalyzer/ , please note do not use classical show tech-support (short version) , use the command denoted in green for Wireless Analyzer. Checkout all advisories!

Re: 9800 CLI/SSH wireless show command output slow

Rasika Nayanajith — Fri, 16 Sep 2022 05:51:22 GMT

By any chance all those 1000AP manage by a single WNCd of that controllers or are they distributed among 5x WNCds using different site tag values.
I got 300AP (9800-80) & did not see any such delays.

HTH
Rasika

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Fri, 16 Sep 2022 11:50:06 GMT

There are 1000 APs across 66 site tags, it's set up with a site tag being used for the set of APs belonging to each building.

I also have a 9800-80 that we're in the progress of migrating to with 8 site tags, 132 APs, and 302 clients (at this moment) if I do "term len 0" on that one and "show wireless client summary" it takes ~4 sec to output the list of 302 clients. On AireOS that would take something like 100ms.

It's the kind of thing that's more annoying than impeding, so that's the reason why I've just posted here for now. I'm fine with just waiting for successive IOS XE releases to fix it at some point.

Re: 9800 CLI/SSH wireless show command output slow

Mark Elsen — Fri, 16 Sep 2022 12:27:16 GMT

- Remember to use WirelessAnalyzer (as per my initial reply) , very informative

Re: 9800 CLI/SSH wireless show command output slow

Scott Fella — Fri, 16 Sep 2022 14:56:39 GMT

If this is a problem and others are not seeing this, you should open a TAC case. I don't think future versions will fix your issue if its not a reported issue/bug. Can be something else that might be causing this for you.

Re: 9800 CLI/SSH wireless show command output slow

Rich R — Sat, 17 Sep 2022 12:22:57 GMT

9800-80 with 861 APs and 1573 clients - just tested:
APs about 9-10 seconds
Clients about 6 seconds.
So agreed it may be slightly slower than AireOS but not a problem for us.
Worth checking you have TCP tuned properly in IOS? What works best will depend on your environment but look at ip tcp mss <> - sized to ensure no fragmentation will be required, ip tcp selective-ack, ip tcp window-size 65535 and ip ssh window-size 65535. Note there seems to be a bug with scp download of large files *from* the device which hang at 99% when window is 65535 - in that case reducing to 32768 works. I've never got round to testing the breakpoint or opening a TAC case for it though. Uploading files to the device is not affected though.

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Mon, 19 Sep 2022 15:48:12 GMT

I think you actually reproduced the problem if you were testing with "show ap summary" and "show wireless client summary". Our results almost the same when looking at it in terms of ratios of output time/ap or output time/client if those commands were used.

I just tested again and the output of "show ap summary" is ~14sec for 999 APs and the output of "show wireless client summary" is ~18sec for 4086 clients.

The ~40 sec at or so time I gave above was for "show wireless client summary detail", and indeed if I run that command on my WLC with the current client count of ~4086 it takes ~65sec.

"show ap dot11 5ghz summary" for 999 APs is ~27sec for me.

All non-wireless-related SSH output from the WLC is not affected. The problem looks to be a performance issue internal to the WLC, especially since wireless show commands that return more data seem to take a lot more time for the same number of APs and clients.

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Mon, 27 Mar 2023 13:21:53 GMT

This is still an issue in 17.9.3, it gets worse the more clients and APs you have on a WLC. APs are load balanced pretty well across the wncds. CPU utilization on the wncds is low ~12% on the highest wncd over 1 min. Show wireless client summary detail takes about 1m 22s to complete for 5824 clients, and show ap summary takes about 31 seconds for 1439 APs. SSH output itself is fine, I can spam show config for example and it doesn't suffer from this output latency issue - it looks like some kind of database query slowness or IPC slowness issue maybe on SSH but not sure. Anyway, the commands that fetch more data from the back end(s) are slower, so the slowness appears related to the amount of data you're asking the thing for.

Unfortunately TAC is borderline useless to me these days. In the past (Around 2016-2018) they would actually investigate issues and file bug reports but those days seem to be gone except for maybe the high profile/very important customers. I am not one of those so for me TAC is not really much more than a glorified RMA engine.

Re: 9800 CLI/SSH wireless show command output slow

Mark Elsen — Mon, 27 Mar 2023 14:10:32 GMT

- Might be related to a more general issue concerning available resources , the following bunch of commands may be useful to investigate :
show platform resources
show processes cpu platform sorted | ex 0% 0% 0%
show processes memory platform sorted
show processes memory platform accounting

show int po1 | i line protocol|put rate|drops|broadcast (Check volume of traffic received and tx by WLC , e.g.)
show platform hardware chassis active qfp statistics drop (check for packet drops)
show platform hardware chassis active qfp feature wireless punt statistics (Check for packets punted to CPU)

show buffers | i buffers|failures (Check for buffer failures)
show platform hardware chassis active qfp datapath utilization | i Load ((Check Processing Load (pct) below to see the utilization , should not exceed 92 %)

>....TAC is not really much more than a glorified RMA engine.
That 's why we are here (LOL!)

Re: 9800 CLI/SSH wireless show command output slow

Scott Fella — Mon, 27 Mar 2023 14:26:08 GMT

I will be honest, I don't have many issues with TAC at all. If I don't get traction on a case that I have because it not vary obvious, I usually would escalate to have the BE help out. From what you are seeing, TAC would need to gather the data in order to escalate/get support up to the BE.

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Mon, 27 Mar 2023 14:36:26 GMT

It's a 9800-80, everything is verrry below the limits including the wncd cpu util and overall cpu util. The QFPs are are at 2% util over 1 min and 60min. There aren't any drops at the front panel ports/PHY level going on.

On 9-17-22 Rich R noted above

"9800-80 with 861 APs and 1573 clients - just tested:
APs about 9-10 seconds
Clients about 6 seconds."

Assuming that was just for a "show ap summary" and "show wireless client summary" (which would have to retrieve less data from the wncds I'd imagine) for him it took 0.011 sec per AP for "show ap summary" and 0.003sec per client. If I repeat the test with my values I get 20.00sec/1439 APs = 0.013 sec per AP, and 29.72sec/ 7477 clients = 0.003 sec per client. The values are very close, so it appears that the same issue is happening for the both of us but it just affects me more because I have more APs and more clients.

Also the output of commands which return more data from the underlying DBs like "show ap dot11 5ghz summary" and "show wireless client summary detail" take a lot longer to output per line than the more basic commands. Feels like a database/IPC performance issue to me.

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Mon, 27 Mar 2023 15:26:44 GMT

TAC used to be able to check internally if behavior was expected or not and file bug reports if not but they seem to have lost this capability as far as I can tell.

For example I had another case open with them about a client RA trace showing the client being deleted for IP theft when we have IP theft disabled as an exclusion reason on the WLC. As far as I can tell that's a pretty obvious bug but TAC was just totally ineffective even after being shown the RA trace of it happening and the WLC config with ip theft disabled as an exclusion condition. What was happening was: Android phone, user disabled MAC randomization, reassociates, and gets excluded for IP theft because the phone kept the same IPv6 link-local addr. I have "no wireless wps client-exclusion ip-theft" configured, but WLC still deletes IPv6 thieves regardless of that. I'd rather just use dhcp-required and IPSG to handle that instead of excluding clients because even with a short timer some clients don't handle being deauth'd well and they'll do things like switch to mobile data.

Case is still open because technically it's still a problem, but the problem is rare enough that both TAC and I have given up on it now. It's probably a lot more common of a problem in environments that force users to switch off randomized MAC - which we don't, but one user happened to and claimed it wasn't working which is how I found the issue. Assume a lot of helpdesks would probably just tell people to reboot and hope the exclusion timer ran out by then 😉

Re: 9800 CLI/SSH wireless show command output slow

Mark Elsen — Mon, 27 Mar 2023 15:56:43 GMT

- Perhaps this one is related : https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc31406

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Mon, 27 Mar 2023 16:52:40 GMT

Saw that one, but that bug looked more related to the scenario where you have IP theft exclusion enabled and it's just not working the way it's intended. In my case, I have it disabled but it's only really disabled for IPv4 apparently, IPv6 devices still cause client deletes. Anyway, getting kind of off topic for this thread I guess, sorry.

Re: 9800 CLI/SSH wireless show command output slow

Scott Fella — Mon, 27 Mar 2023 17:48:31 GMT

Cisco would be able to see this during Cisco Live! They are running the 9800's there so it's best to try to get this up to the BE team. I know they have a separate group for these events, but the BE supports them. Easy way to test in a large setting I would say, especially Cisco Live US.

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Wed, 27 Sep 2023 12:16:21 GMT

I have a TAC case open on this now, it's not a network issue because the output of show config is about 80-100 microseconds per line while the output of the wireless show commands ranges between 10-17 milliseconds per line, so it's an internal issue to the 9800.

Also I tested again this morning on 17.9.4 and under basically 5% CPU load across all cores and 1% QFP load the issue still occurs.

The post in this thread on 09-17-2022 by Rich R reproduces the issue assuming he was using "show ap summary". My time per line for show ap summary is very close to his this morning, within about 2ms per line so it's within the margin of human error. For deployments with <1000 APs maybe it's acceptable to people. Our deployments are about to go over 1500 APs per WLC so it's quite annoying to me to have the CLI output from wireless show commands be so slow.

Also I will note that this was never an issue on AireOS, it only started occurring when moving to the 9800 platform.

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Wed, 27 Sep 2023 12:54:10 GMT

they should also be able to see it any time using their internal test tool that simulates APs joining the WLC. Testing CLI output performance probably just not in their automated test suite ,in any case this is an annoying regression from AireOS, which never had this issue.

Re: 9800 CLI/SSH wireless show command output slow

JPavonM — Thu, 28 Sep 2023 13:18:53 GMT

This command output delay is something I sometimes notice on every IOS-XE release with the main C9800-80 and 1700 APs.

I think it has to be with internal processes like SNMP, NetFlow, Telemetry (if you have DNA-C).

Re: 9800 CLI/SSH wireless show command output slow

jasonm002 — Thu, 28 Sep 2023 14:15:36 GMT

If you could make a TAC case about it that would be helpful, even though it's not causing downtime it makes it very annoying/unpleasant to work with the 9800 platform at times. It is causing brand damage from a marketing perspective in my opinion because we have this very expensive enterprise wireless controller with 24 CPU cores and for whatever reason it struggles to output lines of data at a speed greater than MS-DOS PCs from the 90s. It's obviously bad performing code at some level.