9800 CLI/SSH wireless show command output slow

[jasonm002]

Output of wireless related show commands like 'show wireless client summary detail', 'show ap dot11 5ghz summary', or 'show ap summary' on the 9800 CLI is very slow. It takes ~42 seconds to display data on ~2700 clients on a 9800-40 if you disable paging. This has been the case in 17.3, 17.6, and 17.9.

On the 5520 controller on any AireOS version >= 8.5 'show ap summary', 'show client summary', or 'show advanced 802.11a summary' only takes a couple of seconds at most to output around the same amount of clients and APs. I'm only noticing this because we're running wireless controllers with >=1000 APs and >=2700 clients. 

Something is happening on the 9800 in all IOS XE versions that is causing a big performance hit to SSH output of wireless data which makes it very annoying to verify operational parameters for example.

I don't really have time to open a TAC case on this one, but this issue should be extremely easy to reproduce in a lab environment. Someone at Cisco with a lab environment should really spin up some virtual APs and clients and test it - it's a pretty big performance decrease going from 5520 to 9800 which is not what we want to see for a new flagship product I would expect.

[Scott Fella]

Cisco would be able to see this during Cisco Live!  They are running the 9800's there so it's best to try to get this up to the BE team.  I know they have a separate group for these events, but the BE supports them.  Easy way to test in a large setting I would say, especially Cisco Live US.

[jasonm002]

they should also be able to see it any time using their internal test tool that simulates APs joining the WLC. Testing CLI output performance probably just not in their automated test suite ,in any case this is an annoying regression from AireOS, which never had this issue.

[jasonm002]

I have a TAC case open on this now, it's not a network issue because the output of show config is about 80-100 microseconds per line while the output of the wireless show commands ranges between 10-17 milliseconds per line, so it's an internal issue to the 9800.

Also I tested again this morning on 17.9.4 and under basically 5% CPU load across all cores and 1% QFP load the issue still occurs.

The post in this thread on 09-17-2022 by Rich R reproduces the issue assuming he was using "show ap summary". My time per line for show ap summary is very close to his this morning, within about 2ms per line so it's within the margin of human error. For deployments with <1000 APs maybe it's acceptable to people. Our deployments are about to go over 1500 APs per WLC so it's quite annoying to me to have the CLI output from wireless show commands be so slow.

Also I will note that this was never an issue on AireOS, it only started occurring when moving to the 9800 platform.  

[JPavonM]

This command output delay is something I sometimes notice on every IOS-XE release with the main C9800-80 and 1700 APs.

I think it has to be with internal processes like SNMP, NetFlow, Telemetry (if you have DNA-C).

[jasonm002]

If you could make a TAC case about it that would be helpful, even though it's not causing downtime it makes it very annoying/unpleasant to work with the 9800 platform at times. It is causing brand damage from a marketing perspective in my opinion because we have this very expensive enterprise wireless controller with 24 CPU cores and for whatever reason it struggles to output lines of data at a speed greater than MS-DOS PCs from the 90s. It's obviously bad performing code at some level.

[jasonm002]

One of the striking things here is the time difference it takes to show the wireless client summary vs the detailed summary."show wireless client summary" with 8003 clients at the moment takes ~31sec to output if paging is disabled, and "show wireless client summary detail" (same client count more data being retrieved) takes ~113 sec.  So there is quite clearly some kind of performance issue here, looks like maybe a database performance problem in the backend. The more data it has to retrieve the worse the performance.

[jasonm002]

Still have a TAC case open on this , they thought it might have been caused by having logging synchronous enabled on the VTYs but that wasn't it.

Latest results from 17.9.4a are 12ms per line (~18sec toal) for the output of show ap uptime and 0.17ms per line for show config (<1sec total).

still looks like there is an internal performance issue of some kind