WLC 9800 HA Radioactive Trace takes too long time in GUi

[Gehrig_W]

Hello Cisco WLAN experts,

when I do an radioactive Trace on our 9800-HA-WLC, I have noticed that it takes a very long time to finally present the result in Gui.

In this example I took a Radioactice Trace inclusive Conditional Debug to search back packets for a MAC for the last 2 days.

It takes nearly 1,5 hours to finish teh step "Collecting files from chassis 2". In this examle I started at around 1:30 pm. The debug file ist written already 5 minutes after start and can be examined via Cli of course.

But in GUi, the Trace-application is blocked for around 1,5 hours to finish at around 3pm.

Is it really necessary to look in chassis 2 for such a long time ?

Same happpens,when I do a radioaactice trace without Conditional debug to show Log-messages wiht Notice-Level instead of Debug-Level.

How can I accelarate Radioactive-Traces via Gui on a 9800-WLC-HA-cluster ?

Thank You for any good tipps and hints.

Kind regards

Wini

[marce1000]

  - Check controller software version,  use a latest (advisory) release , if applicable,

  M.

[marce1000]

 -  Here is  nifty trick which may provides insights as to what is going on when the Radioactive Trace is
    executed through the GUI : https://goodwi.fi/posts/2025/01/find-9800-cli-cmd/

                 But I always prefer to use the CLI  for Radioactive Tracing,

   M.

[Gehrig_W]

Hello mare1000, thank You for this trick.

Unfortunately not all commands are catched with this CLIcatcher-skript pu into my config in my case.

event manager applet CLIcatcher
 event cli pattern ".*" sync no skip no
 action 1.0 syslog priority informational msg "$_cli_msg"
 action 2.0 set _exit_status "1"

To my surprise I see tons of other commands which were definitely not executed by myself. Her a short example:

77828978: Apr 8 09:27:31.660: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77828983: Apr 8 09:27:31.986: %HA_EM-6-LOG: CLIcatcher: show license summary
77828985: Apr 8 09:27:32.121: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77828988: Apr 8 09:27:32.410: %HA_EM-6-LOG: CLIcatcher: show license tech support
77829101: Apr 8 09:27:46.398: %HA_EM-6-LOG: CLIcatcher: show clock
77829115: Apr 8 09:27:49.326: %HA_EM-6-LOG: CLIcatcher: show running-config partition common
77829117: Apr 8 09:27:49.427: %HA_EM-6-LOG: CLIcatcher: show platform software cpu alloc
77829176: Apr 8 09:27:59.987: %HA_EM-6-LOG: CLIcatcher: show running-config interface Vlan4080
77829179: Apr 8 09:28:00.075: %HA_EM-6-LOG: CLIcatcher: show wireless client device count
77829181: Apr 8 09:28:00.150: %HA_EM-6-LOG: CLIcatcher: show call-home smart-licensing
77829193: Apr 8 09:28:00.896: %HA_EM-6-LOG: CLIcatcher: show version

....

77830047: Apr 8 09:29:47.781: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77830059: Apr 8 09:29:50.826: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77830061: Apr 8 09:29:50.929: %HA_EM-6-LOG: CLIcatcher: show clock
77830091: Apr 8 09:29:57.102: %HA_EM-6-LOG: CLIcatcher: show platform software cpu alloc
77830093: Apr 8 09:29:57.155: %HA_EM-6-LOG: CLIcatcher: show wireless summary
77830095: Apr 8 09:29:57.184: %HA_EM-6-LOG: CLIcatcher: show wlan summary sort descending client-count
77830102: Apr 8 09:29:57.904: %HA_EM-6-LOG: CLIcatcher: show ap summary sort descending client-count
77830178: Apr 8 09:30:07.586: %HA_EM-6-LOG: CLIcatcher: show logging
77830245: Apr 8 09:30:16.144: %HA_EM-6-LOG: CLIcatcher: terminal length 0
77830247: Apr 8 09:30:16.177: %HA_EM-6-LOG: CLIcatcher: terminal width 0
77830272: Apr 8 09:30:19.084: %HA_EM-6-LOG: CLIcatcher: show license udi
77830304: Apr 8 09:30:22.255: %HA_EM-6-LOG: CLIcatcher: show privilege
77830307: Apr 8 09:30:22.425: %HA_EM-6-LOG: CLIcatcher: logout
77830309: Apr 8 09:30:22.479: %HA_EM-6-LOG: CLIcatcher: terminal length 0
77830398: Apr 8 09:30:33.788: %HA_EM-6-LOG: CLIcatcher: terminal width 0
77830414: Apr 8 09:30:35.697: %HA_EM-6-LOG: CLIcatcher: show ip interface brief

Is this the HA-cluster-communication talking to each other to exchange information about the system ?

Or is DNA-Catalyst-Center or Prime sucking all this information ?

Please check.

Kind regards

Wini

[Leo Laohoo]

What firmware is the controller on? 

What is the uptime?

Is the controller on HA SSO?

[Gehrig_W]

Hello Leo, the 9800-80-pair is on 17.9.5 with uptime of 3 years and 35 weeks in HA SSO with around 2200 APs and 6000-7000 WLAN Clients.

Redundant System Information :
------------------------------
Available system uptime = 3 years, 35 weeks, 1 day, 50 minutes
Switchovers system experienced = 16
Standby failures = 0
Last switchover reason = user forced

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 28 weeks, 5 days, 18 hours, 1 minute
Image Version = Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)

Kind regards

Wini

[eglinsky2012]

Long-term RA traces and show tech-support wireless (via Troubleshooting > Debug Bundle, which I was told by TAC is faster in GUI than CLI) do take a really long time for me, too. 9800-80 running 17.12.4 in SSO. That was the case back with 17.9.3 and 17.9.4 as well. A 30-minute RA trace might take a few/several minutes to complete, so I can't even imagine how long a 2-day one would take! Interested in where this topic goes, please keep us updated if you find a solution.

[sanandy@cisco.com]

To do a good comparison, you can take take the RA trace via CLI and dump it into bootflash/harddisk and perform the same task (for same amount of time) from GUI too..that will give you a comparative study - if at all is there..usually whetever you execute via GUI, a corresponding CLI gets executed and fetch the data. So if GUI takes 1 hr for a trace, I would expect around same time in CLI too.

[Gehrig_W]

Hello Sanady,

I did a comparison between the Gui using Radioactive Trace with conditional debug looking for a MAC-address during the past 2 days and the according Cli-commands:

And Your assumption is right. Whether or not I use "level debug", it takes nearly 1,5 hours to complete the job.

Started command at around 11:19

 Decoder Statistics: 

Time of File writing:  12:42

So again nearly 1,5h to complete a simple debug for a single MAC-address for the last 2 days enidng up in 0,5MByte of data.

This is really strange or ?

Greetings

Wini

[Scott Fella]

Is it strange... I would also like to the GUI and CLI to be alike, but I have also never done a trace that long, maybe the most, less than 30 minutes.  Knowing now that a 2 day trace takes a long time via the GUI, I would probably use the cli for any trace over 12 hours, I would probably alway use the CLI vs the GUI.  Maybe this will get faster in future versions, but who know.  It's something I usually would work with my Cisco SE to get in in the books for some sort of enhancement.

[sanandy@cisco.com]

Yes, it is strange tbh. 2 days RA trace which is having 0.5Mb of data should not take this much time. Looking into the screenshot, I can see it is processing total 3957 files. Can you capture the output of following commands from WLC please in the sequence - 

show bootflash:
show file systems

Run RA trace for a client for lets say for 2 hrs - 

While RA is going on, capture

show processes cpu platform sorted | ex 0%

show processes cpu sorted | ex 0.00

Once completed, again capture - 

show bootflash:
show file systems