"Hulc LED Process" uses 22% CPU on new WS-C2960X-48LPS-L switch, hints to solve the problem!

[Bernhard Roth]

Hello!

We recently bought one of the new "The Greenest Catalyst Switches Ever" to replace old 3750 series.

We noticed that the "Hulc LED Process" is consuming more than 22% CPU all the time.

IOS: 15.2(2)E3

I have several remarks:

1. When pressing the "mode" button on the front of the switch and setting it to e.g. "PoE", the CPU load of the process drops to 15%

Energy consumption goes lower as well as sensless blinking LEDs is stopped (in the cabinet noone cares). 

Unfortunately the mode goes back to "activity" some seconds later.

Why isn't it possible to set the LED mode permanently to something else than "activity"? This would easily save energy by no sensless blinking LEDs and CPU wakeups by the "Hulc LED Process". A timeout would be fine. Once the mode button is pressed, the LEDs go on, timeout starts again counting down.

2. That problem was signaled already several times for older switches, now I discovered the same issue on the new 2960X series as well.

Please Cisco: If energy effiency is an issue and mayor selling point of these "The Greenest Catalyst Switches Ever", please include a simple fix to disable these LEDs maybe after a timeout of n minutes (like competitors like HP do on their ProCurve switches)

Let me know if you have questions - anyhow that issue should be clear.

Best regards,

Bernhard

[DJX995]

I agree with your remarks that the mode button should be sticky or at least configurable for this exact reason. Better yet, include the option to disable the status lights all together. That should reduce the CPU consumption even more.

I have had this problem on my 3750X ever since I upgraded to 15.X code.

[Arjun Sarkar]

Hello ,

High CPU due “Hulc LED” process is an expected behavior.  Please see bug details.

CSCtn42790 :  3560X/3750X: Elevated

CPU usage due to Hulc LED process

Externally found moderate (Sev3) bug: C-Closed

Symptom:

Hulc LED Process uses 15-30% CPU on Catalyst 3560X/3750X platforms.

Conditions:

The is seen in 12.2(53)SE releases or later.

Workaround:

This is an expected behavior and there is no workaround.

Further Problem Description:

The "Hulc LED" process does following tasks:

-          Check Link status on every port

-          If the switch supports POE, it checks to see if there is a Power Device (PD)

detected

-          Check the status of the transceiver

-          Update Fan status

-          Set Main LED and ports LEDs

-          Update both Power Supplies and RPS

-          Check on system temperature status

Hope this helps.

Regards

Arjun

[DJX995]

No, I know that.

I was just agreeing with the OP that other mitigation steps to reduce the CPU usage could be implemented.

[Arjun Sarkar]

Hello ,

 Apart from "Hulc LED" Process do you see any other process spiking up? If want to block only "Hulc LED" process then there is no option as this process may go high or low depending on overall system state.

Please refer link below for high CPU troubleshooting guide.

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/68461-high-cpu-utilization-cat3750.html

Regards

Arjun

[Bernhard Roth]

Hello!

@Arjun:

Hulc LED process is the farmost CPU intensive process in my system:

sw1#sh proc cpu sor
CPU utilization for five seconds: 38%/0%; one minute: 38%; five minutes: 38%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
160 90357911 8659727 10434 22.50% 22.40% 22.38% 0 Hulc LED Process
130 1265674 346444 3653 0.29% 0.43% 0.40% 0 hpm counter proc
367 33 128 257 0.29% 0.03% 0.00% 1 Virtual Exec

...

I knew as well that this is some kind of expected behaviour and I have seen the bug ID.

Anyhow... on green IT devices there should be no process keeping the CPU occupied by *blinking LEDs* which are most of the time not seen by anyone.

@FratianiD:

I totally agree to what you said: Mode button should be sticky or configurable. Lights should be able to be disabled or activated for n seconds after link change or mode button press.

I am aware that Cisco switches never have been really power efficient and I really like the approach to make the devices more "green". I ask myself why nobody thought about these simple fixes. Seems more a management or sales decision...

Best regards,

Bernhard

[Arjun Sarkar]

Hello Bernhard,

 Let me you 22%of CPU for "Huld LED" process is not considered as high CPU in first place. We have Cisco 6880 platform where you will see  "Slcp Process" is always high and most of the customer considers this process is  CPU intensive. This is incorrect. "Slcp Process" is mangement process  which is designed in the code to be high to take care the inter process communications.

The high CPU state we consider when you have issues related to switching of data traffic. None of these processes will not impact the performance of the switch.  So Cisco Business Unit will not fix this process where there is no problem in the box.

Best Regards

Arjun

[Bernhard Roth]

Hello Arjun,

please understand, I am not complaining about switching performance of the device.

It's all about the "Greenest Catalyst Switch ever" wasting my power for useless LED flashing.

That statement was the major reason for me to buy that switch and now I discover some absurd habits which really annoy me.

I have to do some power measurements to find out the impact of "48 LEDs flashing and Hulc process using 22% CPU" vs. "3 LEDs lit and Hulc process using 15% CPU" by just pressing the mode button.

As it is not sticky, power consumption goes up again after some seconds.

Embarrassing, isn't it?

I am sure that I am not the only one having that kind of "problem"

Best regards,

Bernhard

[Joseph W. Doherty]

I have to do some power measurements to find out the impact of "48 LEDs flashing and Hulc process using 22% CPU" vs. "3 LEDs lit and Hulc process using 15% CPU" by just pressing the mode button.

Yes, if you can, please do.  I would be very interested in the results.

If I were to guess, I would expect you might find an almost immeasurable difference.

Why would I guess this?

Well, for starters, LEDs tend to draw very little power, especially those just used for a status indicator light.  Consider some LED flashlights can run 40 hours or so on a pair of AA batteries.

Regarding higher CPU usage drawing more power, it may, or it may not.  Much depends on the design of the processor and how the processor functions in an idle state.

Of the two though, I would expect the processor to draw much more power than the LEDs, although how much an additional 7% "usage" will draw, is a good question.

Modern processors, especially those designed for mobile devices that run on a battery, have been designed to minimize power draw, especially when idle.  A 2960X's processor might not be so designed.

If not, processor power draw might not change much, if at all, with CPU utilization.  If so, this might seem a glaring issue for a "green" switch, but it's possible there are "bigger" power hogs, within the switch, and Cisco might obtain more "bang for the buck" working to minimize their power consumption.  A possible example might be Cisco noting the 2960X supports IEEE 802.3az EEE.  If you consider the amount of power than can be drawn from just one PoE+ port, that would seem a reasonable area to address for power efficiency (versus worrying about LED status lights).

So, I'm unsure how embarrassed Cisco should be, but if you really want to embarrass Cisco, design and market your own switch, showing Cisco how it should be done.  ;)

BTW, I'm not a Cisco booster, nor do I have an axe to grind with Cisco.  As initially noted, if you can do your power consumption test, I would be curious about the results.  That said, I also replied on this thread because I don't agree Cisco should be embarrassed.  The 2960X is what it is.  If you don't like it, or annoyed by it, that's fine too, but to say Cisco should be embarrassed, maybe that's just a little bit over the top?

[Bernhard Roth]

Power consumption test will follow.

What I find embarrassing: When you design a product that shall be very power efficient, you have to take every component of your system into consideration. It starts by enabling/disabling pullup resistors to save some µA here and there and ends up in the actual implementation in software or operating system.

Software is crucial as a badly designed task may end up keeping your CPU busy and thus get in the way when you do clock scaling in idle times etc.

Visual indications are nice and everyone likes the view of a blinking cabinet. But when a "green" product is properly designed, there should be a possibility to disable the LEDs - even if the savings are minimal. The effort to implement a timeout counter or sticky mode button is marginal.

I am not an environmentalist but I am convinced that these kind of small changes are summing up when a product goes into mass production.

Networking is just my secondary business, I do mostly hard- and software development so probably I know what I am talking about :)

BTW: The 2960X is a great product, I like it very much! It helped recently to save around 109W power draw in the office.

[Bernhard Roth]

Just for a laugh: 

This power saving approach works as well but even I think this is overkill ;)

See: 

http://www.cisco.com/c/en/us/support/docs/field-notices/640/fn64063.html

[Joseph W. Doherty]

Well, it does save some power.  ;)

If you want to talk about Cisco being embarrassed, I would think this issue might qualify.

(BTW, nice find!)

[Joseph W. Doherty]

Laugh - our philosophies on what's a proper design, might be just a tad different, mine run more along the lines of:

https://en.wikipedia.org/wiki/Perfect_is_the_enemy_of_good

https://en.wikipedia.org/wiki/Nirvana_fallacy

https://en.wikipedia.org/wiki/Worse_is_better

[Bernhard Roth]

Here comes the power measurement:

It was a bit hard to get good results as the "mode" is going back to default after some seconds and the average power consumption varies. Anyhow the result is about 0.25W less power draw when 20 LEDs are not flashing which results in 0.65W for a switch with all ports connected. ~5,5 kWh could be saved per year under these conditions.

This is not much, I agree. As well I understand that other methods of power saving are more effective. But as said I would have really appreciated to see a function to switch LEDs off by default or after a timeout.

[Joseph W. Doherty]

Yep, not a lot, but actually even more than I expected.