High CPU Utilisation on 2960S Switch Stack

[Tim Hamblin]

Hi all,

We are seeing high CPU utilisation (70-80% and higher!) on a switch stack and I have done a little digging!!  I have found the following which seems a little high to me:

stac11-chil-gbr#show processes cpu sorted

CPU utilization for five seconds: 42%/11%; one minute: 43%; five minutes: 51%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

136   196791584  15214848      12934 23.12% 22.04% 21.67%   0 Hulc LED Process

105     8756419   1720365       5089  0.79%  0.84%  0.84%   0 hpm counter proc

  67     1173880  30393238         38  0.39%  0.22%  0.22%   0 RedEarth Rx Mana

145     2310118   1341716       1721  0.29%  0.25%  0.25%   0 HRPC qos request

174      557971   3464034        161  0.29%  0.19%  0.15%   0 IP Input

  10     2118725   4273238        495  0.29%  0.41%  0.40%   0 ARP Input

144     2201315    167169      13168  0.19%  0.18%  0.19%   0 HQM Stack Proces

229      198355   1843793        107  0.19%  0.06%  0.02%   0 DHCPD Receive

  43      508028    167273       3037  0.09%  0.05%  0.04%   0 Compute load avg

153       59975    278776        215  0.09%  0.00%  0.00%   0 Power RPS Proces

  33      852798    177792       4796  0.09%  0.06%  0.06%   0 Net Background

  85      324408  22153272         14  0.09%  0.05%  0.05%   0 HLFM address lea

  61      364675  29391199         12  0.09%  0.05%  0.05%   0 Draught link sta

106      963951   5327174        180  0.09%  0.09%  0.09%   0 HRPC pm-counters

189     1191532   3393887        351  0.09%  0.12%  0.10%   0 Spanning Tree

        Is this my issue?  What coudl cause it?

All help appreciated

Tim

[david_casey]

What software version are you using?  I know with some of the 15.x code on our 3750-X's this was a bug there for a while but it looks like Cisco closed it out:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtn42790

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti61764

There is more available with a Google search.  You're not the only one seeing this.

Dave

[hdevarajan]

It is basically used to detect link status. Are you using the SFP ports? We find that it goes down if we shut the SFP ports on the ones we dont use.

But overall the 2960S seems to work fine even though the average CPU utilization varies quite a bit.

Our average is anywhere from 25-60% with Hulc moving anywhere between 10-15%.

Some of the 3750X's we run actually have a memory leak and so does the 15.0.1 code base. For the 2960S it looks more like

CPU utilization for five seconds: 36%/10%; one minute: 30%; five minutes: 29%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

153   540711233   112277424       4815 10.19% 11.14% 11.32%   0 Hulc LED Process

157         583          77       7571  9.49%  0.76%  0.15%   1 SSH Process     

208    28552035   150748312        189  0.89%  0.72%  0.63%   0 Spanning Tree   

[Leo Laohoo]

Please specify what IOS version you are using.

[Richard Primm]

Hi Tim,

This is most likely expected behavior and nothing to worry about. Part of it depends on how many switches you have in your stack?   As this is expected CPU utilization, specific to what CPU does under Hulc LED process, there is no workaround like rebooting or upgrading IOS.  It was also identified above - but checkout bug CSCtn42790 (link below)

Every 40 ms the "Hulc LED" process does following tasks:

- Check Link status on every port

- If the switch supports POE, it checks to see if there is a Power Device (PD) detected, this check is bypassed if the switch does not support POE

- Check the status of the transceiver

- Update Fan status

- Set Main LED and ports LEDs

- Update both Power Supplies and RPS

- Check on system temperature status

On similar 2K and 3K platforms (e.g.  3750X, 2960, 2960S, CBS3130X) it is expected to see 10-30% of CPU consumption on the Hulc LED process, depending on the specific platform, and this is normal behavior without any impact on the switch performance.

Reference:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtn42790

[Tim Hamblin]

Thanks for all the replies!!

The IOS version running on the stack is c2960s-universalk9-mz.122-55.SE.bin and there are 4 switches in the stack.

We have several stacks on this site and thought we may have had a CPU utilisation issue/bug but from the response above it would appear the stats are quite normal so I will stop worrying about it!!

Thanks for the help/peace of mind!!!

Tim

[Leo Laohoo · Accepted solution]

Base version of IOS 12.2(55)SE (what you are using now) has some bugs like that.  I recommend you upgrade to the latest 12.2(55)SE8.  It's more stable than what you are using now.

[hdevarajan]

Agreed. 12.2.55 SE 8 is quite stable on the 3750x's that we run.

For us 2960S both 12.2.(55) SE 8 and 15.0.(2) EX are stable. We are also running on 15.0.(2) SE2 and SE4.

( you need to use the EX version, if you want to stack with the newer 2960x).

If there is no immediate need to stack with the newer 2960x, then you have a lot of options.

** I mentioned SE3 which was a typo

[Leo Laohoo]

We are also running on 15.0.(2) SE3

OMG, how come you are still using 15.0(2)SE3???  

Stay away from this version.  Cisco has pulled this version from being downloaded because of a known TACACS bug.

Read this thread.

[hdevarajan]

GOOD CATCH. That would be SE2 and SE4.

Thanks Leo.

[Tim Hamblin]

Thanks guys, will get hold of 12.2(55)SE8 from our support company and upgrade the stacks.

Tim