WLC 9800 HA Radioactive Trace takes too long time in GUi

[Gehrig_W]

Hello Cisco WLAN experts,

when I do an radioactive Trace on our 9800-HA-WLC, I have noticed that it takes a very long time to finally present the result in Gui.

In this example I took a Radioactice Trace inclusive Conditional Debug to search back packets for a MAC for the last 2 days.

It takes nearly 1,5 hours to finish teh step "Collecting files from chassis 2". In this examle I started at around 1:30 pm. The debug file ist written already 5 minutes after start and can be examined via Cli of course.

But in GUi, the Trace-application is blocked for around 1,5 hours to finish at around 3pm.

Is it really necessary to look in chassis 2 for such a long time ?

Same happpens,when I do a radioaactice trace without Conditional debug to show Log-messages wiht Notice-Level instead of Debug-Level.

How can I accelarate Radioactive-Traces via Gui on a 9800-WLC-HA-cluster ?

Thank You for any good tipps and hints.

Kind regards

Wini

[marce1000]

  - Check controller software version,  use a latest (advisory) release , if applicable,

  M.

[marce1000]

 -  Here is  nifty trick which may provides insights as to what is going on when the Radioactive Trace is
    executed through the GUI : https://goodwi.fi/posts/2025/01/find-9800-cli-cmd/

                 But I always prefer to use the CLI  for Radioactive Tracing,

   M.

[Gehrig_W]

Hello mare1000, thank You for this trick.

Unfortunately not all commands are catched with this CLIcatcher-skript pu into my config in my case.

event manager applet CLIcatcher
 event cli pattern ".*" sync no skip no
 action 1.0 syslog priority informational msg "$_cli_msg"
 action 2.0 set _exit_status "1"

To my surprise I see tons of other commands which were definitely not executed by myself. Her a short example:

77828978: Apr 8 09:27:31.660: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77828983: Apr 8 09:27:31.986: %HA_EM-6-LOG: CLIcatcher: show license summary
77828985: Apr 8 09:27:32.121: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77828988: Apr 8 09:27:32.410: %HA_EM-6-LOG: CLIcatcher: show license tech support
77829101: Apr 8 09:27:46.398: %HA_EM-6-LOG: CLIcatcher: show clock
77829115: Apr 8 09:27:49.326: %HA_EM-6-LOG: CLIcatcher: show running-config partition common
77829117: Apr 8 09:27:49.427: %HA_EM-6-LOG: CLIcatcher: show platform software cpu alloc
77829176: Apr 8 09:27:59.987: %HA_EM-6-LOG: CLIcatcher: show running-config interface Vlan4080
77829179: Apr 8 09:28:00.075: %HA_EM-6-LOG: CLIcatcher: show wireless client device count
77829181: Apr 8 09:28:00.150: %HA_EM-6-LOG: CLIcatcher: show call-home smart-licensing
77829193: Apr 8 09:28:00.896: %HA_EM-6-LOG: CLIcatcher: show version

....

77830047: Apr 8 09:29:47.781: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77830059: Apr 8 09:29:50.826: %HA_EM-6-LOG: CLIcatcher: show platform software status control-processor brief
77830061: Apr 8 09:29:50.929: %HA_EM-6-LOG: CLIcatcher: show clock
77830091: Apr 8 09:29:57.102: %HA_EM-6-LOG: CLIcatcher: show platform software cpu alloc
77830093: Apr 8 09:29:57.155: %HA_EM-6-LOG: CLIcatcher: show wireless summary
77830095: Apr 8 09:29:57.184: %HA_EM-6-LOG: CLIcatcher: show wlan summary sort descending client-count
77830102: Apr 8 09:29:57.904: %HA_EM-6-LOG: CLIcatcher: show ap summary sort descending client-count
77830178: Apr 8 09:30:07.586: %HA_EM-6-LOG: CLIcatcher: show logging
77830245: Apr 8 09:30:16.144: %HA_EM-6-LOG: CLIcatcher: terminal length 0
77830247: Apr 8 09:30:16.177: %HA_EM-6-LOG: CLIcatcher: terminal width 0
77830272: Apr 8 09:30:19.084: %HA_EM-6-LOG: CLIcatcher: show license udi
77830304: Apr 8 09:30:22.255: %HA_EM-6-LOG: CLIcatcher: show privilege
77830307: Apr 8 09:30:22.425: %HA_EM-6-LOG: CLIcatcher: logout
77830309: Apr 8 09:30:22.479: %HA_EM-6-LOG: CLIcatcher: terminal length 0
77830398: Apr 8 09:30:33.788: %HA_EM-6-LOG: CLIcatcher: terminal width 0
77830414: Apr 8 09:30:35.697: %HA_EM-6-LOG: CLIcatcher: show ip interface brief

Is this the HA-cluster-communication talking to each other to exchange information about the system ?

Or is DNA-Catalyst-Center or Prime sucking all this information ?

Please check.

Kind regards

Wini

[Leo Laohoo]

What firmware is the controller on? 

What is the uptime?

Is the controller on HA SSO?

[Gehrig_W]

Hello Leo, the 9800-80-pair is on 17.9.5 with uptime of 3 years and 35 weeks in HA SSO with around 2200 APs and 6000-7000 WLAN Clients.

Redundant System Information :
------------------------------
Available system uptime = 3 years, 35 weeks, 1 day, 50 minutes
Switchovers system experienced = 16
Standby failures = 0
Last switchover reason = user forced

Hardware Mode = Duplex
Configured Redundancy Mode = sso
Operating Redundancy Mode = sso
Maintenance Mode = Disabled
Communications = Up

Current Processor Information :
-------------------------------
Active Location = slot 1
Current Software state = ACTIVE
Uptime in current state = 28 weeks, 5 days, 18 hours, 1 minute
Image Version = Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)

Kind regards

Wini

[eglinsky2012]

Long-term RA traces and show tech-support wireless (via Troubleshooting > Debug Bundle, which I was told by TAC is faster in GUI than CLI) do take a really long time for me, too. 9800-80 running 17.12.4 in SSO. That was the case back with 17.9.3 and 17.9.4 as well. A 30-minute RA trace might take a few/several minutes to complete, so I can't even imagine how long a 2-day one would take! Interested in where this topic goes, please keep us updated if you find a solution.

[Saikat Nandy]

To do a good comparison, you can take take the RA trace via CLI and dump it into bootflash/harddisk and perform the same task (for same amount of time) from GUI too..that will give you a comparative study - if at all is there..usually whetever you execute via GUI, a corresponding CLI gets executed and fetch the data. So if GUI takes 1 hr for a trace, I would expect around same time in CLI too.

[Gehrig_W]

Hello Sanady,

I did a comparison between the Gui using Radioactive Trace with conditional debug looking for a MAC-address during the past 2 days and the according Cli-commands:

And Your assumption is right. Whether or not I use "level debug", it takes nearly 1,5 hours to complete the job.

Started command at around 11:19

 Decoder Statistics: 

Time of File writing:  12:42

So again nearly 1,5h to complete a simple debug for a single MAC-address for the last 2 days enidng up in 0,5MByte of data.

This is really strange or ?

Greetings

Wini

[Scott Fella]

Is it strange... I would also like to the GUI and CLI to be alike, but I have also never done a trace that long, maybe the most, less than 30 minutes.  Knowing now that a 2 day trace takes a long time via the GUI, I would probably use the cli for any trace over 12 hours, I would probably alway use the CLI vs the GUI.  Maybe this will get faster in future versions, but who know.  It's something I usually would work with my Cisco SE to get in in the books for some sort of enhancement.

[Saikat Nandy]

Yes, it is strange tbh. 2 days RA trace which is having 0.5Mb of data should not take this much time. Looking into the screenshot, I can see it is processing total 3957 files. Can you capture the output of following commands from WLC please in the sequence - 

show bootflash:
show file systems

Run RA trace for a client for lets say for 2 hrs - 

While RA is going on, capture

show processes cpu platform sorted | ex 0%

show processes cpu sorted | ex 0.00

Once completed, again capture - 

show bootflash:
show file systems

[Rich R · Accepted solution]

I disagree - it's totally to be expected (in my opinion) and matches all my experience.  The time taken is not related to the size of the result, it's related to the size and number of always-on debug log files which it's searching through to get the result.  And the further back you go the longer it takes because the more files and data it has to search through.  It's decompressing and decoding a lot of binary data to achieve that.  I expect the process could probably be optimised and improved to speed it up but my guess would be that the CPU use is intentionally limited to avoid compromising normal WLC operation.

[Gehrig_W · Accepted solution]

Thank You Rich fpr Your helpful clarification.

So it works as designed and I have to be patient in case of troubleshooting and reduce the time span.

Kind regards

Wini

[Rich R]

Yes I agree it's frustrating but TAC have said it's normal.