topic Re: WLC 9800 CPU Utilize Issue in Wireless

WLC 9800 CPU Utilize Issue

bakaholic39 — Tue, 11 Jul 2023 09:57:38 GMT

Hi,

I have a WLC 9800 on Firmware OS : 17.9.3
19 APs and 100 Clients

Everything looks normal, but there are reports that the CPU utilization rate is too high. The problem is only 7th core. Has anyone encountered the same problem as me. What should I do to solve it?

Thank you!

Re: WLC 9800 CPU Utilize Issue

Leo Laohoo — Tue, 11 Jul 2023 10:29:30 GMT

Please share the complete output to the following commands:

sh platform resources sh platform software status con brief

Re: WLC 9800 CPU Utilize Issue

jagan.chowdam — Tue, 11 Jul 2023 13:40:36 GMT

Check CPU utilization: Identify if there is any process using lot of CPU using the following command
sh processes cpu platform sorted | ex 0% 0% 0%

Check CPU utilization per wncd process
show processes cpu platform sorted | i wncd

Check Memory utilization: Identify if device is running out of memory and if there is some process holding lot of memory.
show platform resources (basic)
show process memory platform sorted
show processes memory platform accounting (advanced)

/**Please rate all useful responses**/

Re: WLC 9800 CPU Utilize Issue

bakaholic39 — Tue, 11 Jul 2023 13:51:17 GMT

Here are the results.

Thank you for your response.

Re: WLC 9800 CPU Utilize Issue

jagan.chowdam — Tue, 11 Jul 2023 13:54:05 GMT

For CPU monitoring we need to run following command multiple times. See if there are any processes showing CPU above 80% consistently and not just a spike.

sh processes cpu platform sorted | ex 0% 0% 0%

Catalyst 9800-CL and 9800-L platforms use CPU cores for data forwarding. Therefore, it is expected to see high CPU in ucode_pkt_PPE0. For those platforms to evaluate data plane performance use command:

show platform hardware chassis active qfp datapath utilization | i Load

Refer the following Cisco 9800 KPI blog for detailed explanation as which commands to run.

https://blogs.cisco.com/networking/wireless-catalyst-9800-wlc-kpis-part-1

Re: WLC 9800 CPU Utilize Issue

bakaholic39 — Tue, 11 Jul 2023 14:09:17 GMT

Here are the results

Is there any problem with process ucode_pkt_PPE0?

Thank you for your response.

Re: WLC 9800 CPU Utilize Issue

Leo Laohoo — Wed, 12 Jul 2023 00:53:11 GMT

NOTE: The two commands are a "real-time snapshot". It is not historical.

The first one looks at the data plane. The value are normal.

The second one looks at the control plane. And the memory utilization of 30% is not normal.

One of the CPU, #7, is grinding.

Please post the complete output to the next command: sh processes cpu platform sorted location 0

NOTE: This output will produce a very long "page". I just want to see the "first" page.

Re: WLC 9800 CPU Utilize Issue

bakaholic39 — Wed, 12 Jul 2023 06:55:22 GMT

With this command, all process seems very normal except #7 CPU

After that I tried this command. There seems to be something wrong with process " ucode_pkt_PPE0 "
My question is why doesn't the system share load with other cores.

Thank you for your response.

Re: WLC 9800 CPU Utilize Issue

Leo Laohoo — Wed, 12 Jul 2023 07:43:59 GMT

Raise a TAC Case with EUMA or NAM/LATAM. I am out of answers to what this process does.

Re: WLC 9800 CPU Utilize Issue

calverthall — Mon, 17 Jul 2023 15:37:16 GMT

Hello, Did you get any clarity on this issue? We just upgraded to 17.9.3 and noticed the exact same thing. I just opened a TAC, but figured I would ask.

Thanks, Austin

Re: WLC 9800 CPU Utilize Issue

bakaholic39 — Mon, 17 Jul 2023 16:00:45 GMT

Still no solution. I already opened TAC but he replied me that it's normal state without giving any further information.

Re: WLC 9800 CPU Utilize Issue

edmonroy — Mon, 17 Jul 2023 16:10:59 GMT

Catalyst 9800-CL and 9800-L platforms use CPU cores for data forwarding. Therefore, it is expected to see high CPU in ucode_pkt_PPE0. For those platforms to evaluate data plane performance use command: “show platform hardware chassis active qfp datapath utilization | i Load”

https://blogs.cisco.com/networking/wireless-catalyst-9800-wlc-kpis-part-1

Re: WLC 9800 CPU Utilize Issue

calverthall — Mon, 17 Jul 2023 17:09:36 GMT

Thanks. I received the "it is expected to see high CPU in one of the cores, basically this is a normal behavior and it could be on any core, you always will see one of the cores on high utilization" response as well.

I am not 100% convinced I would be getting alerts like this during the summer (we are a school that is out of session) and that this was normal prior to the upgrade. My other thought would be why have 10 cores if only one runs at 99.7%...

Austin

Re: WLC 9800 CPU Utilize Issue

calverthall — Mon, 17 Jul 2023 17:26:33 GMT

Here is the full response I received. Still not sure why an interface has a RED Alert indicating high utilization, but I asked them to share that with the UI team. Hopefully we can blissfully ignore... - Austin

The global figures reflect the actual overall resources, in this case you are in green numbers.

The readings are normal across the board and Cisco documentation indicates that it is expected that IOS-XE platforms show high CPU in the ucode_pkt_PPE0 process even when sitting idle.

Please refer to this article: https://bst.cisco.com/bugsearch/bug/CSCve73211

To improve packet throughput and reduce packet latency and jitter, the data plane more aggressively polls for new packets in later releases which raises CPU utilization even at low packet rates.

Re: WLC 9800 CPU Utilize Issue

bakaholic39 — Tue, 18 Jul 2023 08:12:00 GMT

Here is the same response from Cisco wireless team

Please note that the high CPU 7 Utilization in "ucode_pkt_PPE0" for 9800 IOS-XE 16.4 and higher is expected behavior to improve packet throughput and reduce packet latency and jitter, this increases the number of CPU cycles that "ucode_pkt_PPE0" process consumes at baseline even without any traffic.

The "ucode_pkt_PPE0" is often at 100% even if the box is not passing that much traffic. It is normal for that process to report high usage, it's not an appropriate metric of how much traffic is passing through.

Reference: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCve73211

Re: WLC 9800 CPU Utilize Issue

bakaholic39 — Tue, 18 Jul 2023 08:32:22 GMT

It's not only in the UI, if you get resources utilization from the CLI you will see the CPU usage is above threshold.
If the engineer said this is a normal, then just please remove this RED Alert from the UI. The dashboard should not display any RED Alerts if the system is functioning properly. Because it's a matter of a monthly report.

Re: WLC 9800 CPU Utilize Issue

Leo Laohoo — Wed, 19 Jul 2023 00:57:22 GMT

@edmonroy wrote:
For those platforms to evaluate data plane performance use command: “show platform hardware chassis active qfp datapath utilization | i Load”

Thanks for this info, @edmonroy.

Question: What is the difference between the command you recommended against "sh platform resources"?

I have compared the values between the two on four 9800-80 (two standalone and two in VSS) and I do not see any difference.

Re: WLC 9800 CPU Utilize Issue

Leo Laohoo — Wed, 19 Jul 2023 01:04:27 GMT

@bakaholic39, @bakaholic39, thanks for the Bug ID.

At 115 cases (since 06 June 2017) picture below sums it all up.

Re: WLC 9800 CPU Utilize Issue

Rich R — Sat, 29 Jul 2023 17:06:08 GMT

High ucode_pkt_PPE0 is "normal" - I seem to recall there's a recent change to how it's displayed (I think CSCve73211 mentioned below) but you can see mention of it here https://blogs.cisco.com/networking/wireless-catalyst-9800-wlc-kpis-part-1

As to the high CPU on that core - look at your wncd's: show process cpu platform sorted | incl wncd

If you confirm wncd causing it then look at why.
19 APs and 100 clients is very small so that's surprising but check through best practise guide on AP tagging for distribution across wncd's.
Also, if you're using web-auth - turn OFF https redirects - they are very CPU hungry and will cause high CPU and client issues.

Re: WLC 9800 CPU Utilize Issue

Rich R — Sat, 29 Jul 2023 17:09:53 GMT

@edmonroy let's ignore ucode_pkt_PPE0 which we all now agree is "normal".

Let's concentrate on the CPU core which is at 100%. Are you suggesting that is also normal?
When we had that it WAS definitely customer affecting (caused silent packet drops) - we had customer complaints and reproduced the problems ourselves. Problems resolved once CPU was restored to normal ~30% by turning off https redirects.