cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2321
Views
0
Helpful
16
Replies

3560X high CPU/Interrupts because of ICMP queue?

wowzie
Level 1
Level 1

Hello,

 

We've a 3560X switch which is using a lot of CPU (around 80-90% CPU) and 25-30% interrupts.

 

Top 3 processes:

175 1060646641 3649393632 0 16.13% 14.93% 14.93% 0 Hulc LED Process
224 3956630454 3320689710 0 10.86% 10.97% 11.01% 0 IP Input
13 2497812332 3353714124 0 3.35% 3.77% 3.72% 0 ARP Input

 

I think it's caused by ICMP traffic flooding the CPU because "show controllers cpu-interface" shows me this output (I monitored it for 24 hours):

 

cpu-queue-frames, icmp: increasing with 5800 packets/sec

cpu-queue-frames, sw forwarding: increasing with 485 packets/sec

cpu-queue-frames, routing: increasing with 297 packets/sec

 

show buffers output:

RxQ11 buffers, 2040 bytes (total 16, permanent 16):
1 in free list (0 min, 16 max allowed)
2225720443 hits, 2500324642 misses

 

How can I find what is causing this? I found that I can use "debug platform cpu-queues icmp-q" but this will probably crash the switch which is not an option because the amount of traffic going through this switch.

 

Thanks,


Erik

16 Replies 16


@Jan Rolny wrote:

Hi,

I think it's still the same old well-known bug related to that "Hulc LED Process". There are tens of discussions on cisco forum about same problem for this models of switches 3560X/3750X. That time when I was working with 3750X I had exactly same issue. 

https://bst.cisco.com/bugsearch/bug/CSCtn42790?rfs=qvlogin

Best regards,

Jan


Yes @Jan Rolny, the "Hulc LED Process" has been considered, by many, on-going "issue" for years, but even your reference describes it as NOT a bug, and as "expected behavior".

For @DarrelR it's not just the under 20% that process is consuming, but what else is driving his overall CPU to 90%+.

BTW, the "Hulc LED Process" issue is also explicitly mentioned (see table 3), as up 30% utilization, is considered normal in the Cisco troubleshooting article I referenced.  (You reference has its CPU listed between 15-30%.)

Also BTW, something I "discovered" long ago, Cisco switches seem to have a relatively weak CPU, compared to similar, or even more expensive software based routers.  At first, I was surprised because such switches have multi-gig capacity while software based routers might struggle at well under a gig of throughput.  But, if you think about it, all "heavy lifting" of data forwarding, on a switch, is handled by dedicated hardware so their CPU, normally, doesn't have to be a real powerhouse.

Reason I mention the forgoing, so if Cisco decides to add some system process on a switch, like "Hulc LED Process", it might consume a fair amount of CPU possibly because the original design didn't expect and allow for the extra CPU usage demand.  If this is a background level priority, again, even if such a process consumed all available CPU cycles, it would have very minimal impact to normal switch operations.

"Hulc LED Process", "RedEarth" are commonly attributed to STP.

Check for any port(s) flapping.