cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1342
Views
0
Helpful
10
Replies

High CPU usage on 3750x Stack

jkay18041
Level 3
Level 3

I've got 6 switches in a stack and the CPU usage is quite high.

CPU utilization for five seconds: 97%/2%; one minute: 94%; five minutes: 82%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
257 302010 7779 38823 66.26% 54.67% 43.82% 0 HRPC dai request
453 30494 1437 21220 5.37% 5.00% 4.13% 0 Syslog
88 31126 5122 6076 4.17% 3.94% 3.55% 0 RedEarth Tx Mana
182 25175 8381 3003 3.28% 3.67% 3.22% 0 Hulc LED Process
87 20400 8644 2360 2.38% 2.69% 2.45% 0 RedEarth I2C dri

 

I'm running IOS 15.2(4)E10

 

Any advice?

 

I've rebooted a few times with no luck

 

Thank you

 

10 Replies 10

Reza Sharifi
Hall of Fame
Hall of Fame

257 302010 7779 38823 66.26% 54.67% 43.82% 0 HRPC dai request

HRPC is running high.

Do you have any scrip running on the switch to make any remote procedure call?

Take a look at this document.

https://www.cisco.com/c/en/us/td/docs/ios/sw_upgrades/interlink/r2_0/rpc_pr/rprdesc.html

 

HTH

Not that I'm aware of, I am not the only one who has config access to it the switches. If there isn't anything in the show-running config how else could I check?

 

Thank you

Scripts usually show in the "sh run" output. So, if you don't see any config then there is none.

Can you post the output of "sh process cpu sort | ex 0.00?

 

HTH

CPU utilization for five seconds: 56%/4%; one minute: 70%; five minutes: 70%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
182 699001 124201 5627 19.03% 19.09% 19.12% 0 Hulc LED Process
258 256262 357190 717 7.19% 6.65% 6.66% 0 HULC DAI Process
453 458167 29816 15366 5.43% 5.12% 5.14% 0 Syslog
88 396051 65782 6020 4.31% 4.71% 4.58% 0 RedEarth Tx Mana
44 130422 343523 379 3.19% 3.23% 3.38% 0 ARP Snoop
257 3940165 176474 22327 2.39% 1.87% 1.80% 0 HRPC dai request
87 260878 110327 2364 2.39% 2.52% 2.63% 0 RedEarth I2C dri
199 62990 17792 3540 0.63% 0.63% 0.69% 0 HRPC qos request
59 31018 9079 3416 0.63% 0.41% 0.39% 0 Per-Second Jobs
124 57747 111548 517 0.47% 0.94% 0.96% 0 HLFM address lea
198 42701 1749 24414 0.47% 0.46% 0.47% 0 HQM Stack Proces
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
89 8884 54596 162 0.31% 0.10% 0.10% 0 RedEarth Rx Mana
234 12283 24252 506 0.31% 0.07% 0.07% 0 IP Input
233 4756 111875 42 0.15% 0.06% 0.05% 0 IP ARP Retry Age
57 17712 2050 8640 0.15% 0.13% 0.15% 0 Compute load avg
109 36223 2851 12705 0.15% 6.57% 5.84% 1 SSH Process

Have a look at this link, table-3 since the "Hulc LED" Process is running close to 20%. https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

HTH

So really we should be ok? This is just a cosmetic bug?

 

Thanks for all your help

I would keep an eye on the CPU to make sure it does not go higher. Traditionally, most of smaller Cisco switches (3750, 3850, 2960, etc..) run high on CPU and there is really not much you can do about it except upgrading to a different version, and that sometimes help but not always. As long as it is not affecting your production environment, you should be fine.

HTH

"So really we should be ok?"

Maybe.

"This is just a cosmetic bug?"

Hmm, unsure it's just cosmetic.

From what you posted you had high process stats on "HRPC dai request", "Hulc LED Process" and " HULC DAI Process" (and some others).

For "normal" packet forwarding, it's all generally done in hardware, so even if CPU was 100%, or your first posted 97%, it shouldn't be an issue for that traffic.

Yet, there is "real" dependency on sometimes having available CPU.  The issue is, CPU allocation is often prioritized such that "important" processes get it before some others, but prioritized processes can "starve" lower priority processes.

A process like "Hulc LED Process", I suspect, has both a low priority and is non-critical.  I.e. likely it might use all available (left over) CPU, without being adverse to other processes, and probably doesn't much matter if it obtains insufficient CPU.

A process like "HRPC dai request", might have a high priority and might be critical.

So, difficult to say, what high CPU usage processes are "bad", although if one of Cisco's troubleshooting documents a particular process for remediation, that's a clue it's important.  Conversely, if they don't document remediation for a high usage CPU process, it's likely unimportant, even it drives CPU to 100%.

What sometimes can have an impact on overall CPU usage, is upgrading to a "newer" IOS release.  I also suspect, sometimes they aren't fixing any bug, just having a "background" process run less often so it doesn't spike the CPU getting the owner of the equipment concerned.

 

I am not finding much on fixing the HULC DAI Process that I can figure out. Any suggestions?

 

Thank you

"Any suggestions?"

Assuming the high CPU might be "freaking out" a monitoring application, is there any perceived adverse impact to traffic or functions on that switch?  If not, as you asked earlier, are you ok, perhaps you are.

If you have maintenance on the switch, you might discuss with TAC and/or upgrade the IOS.  (For the latter, I just looked, and I see your IOS version is the latest and what's Cisco suggested.  Regardless, assuming you don't need the features of 15.2.4, you might try the tail end of an earlier release, which also is MD.)

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: