[2960 Series] High CPU with Hulc LED Process

[Tai Nguyen]

Hello experts,

 I have 2 types of switch 2960, they face a problem with CPU:

Switch Ports Model              SW Version            SW Image                
------ ----- -----              ----------            ----------              
*    1 28    WS-C2960X-24PS-L   15.0(2a)EX5           C2960X-UNIVERSALK9-M    

Switch Ports Model                     SW Version            SW Image                
------ ----- -----                     ----------            ----------              
*    1 52    WS-C2960X-48FPS-L         15.2(2)E3             C2960X-UNIVERSALK9-M

SW-2960#sh proc cpu sort | e 0.0
CPU utilization for five seconds: 37%/0%; one minute: 37%; five minutes: 37%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 160    89656467     8594277      10432 22.44% 22.41% 22.41%   0 Hulc LED Process
 171     1082506       68557      15789  0.23%  0.25%  0.23%   0 HQM Stack Proces

Anybody can tell me what is "Hulc LED Process" and how can I fix it.

I really appreciate any help.

Thanks so much.

[Mark Malone]

Hi
HULC is a known bug that runs across several platform of 2/3ks and effects a lot of different versions of ios been around a while
Theres a lot of posts on it already for 2960x and other platforms below in support link

You need to find an image its not effected by it 12.2(55)SE8 I think was one of them , there are other options that have worked for some like shutting unused ports and reloading it.

https://supportforums.cisco.com/search/site/hulc?f[0]=sm_og_group_ref%3Anode%3A6016&f[1]=is_top_level%3A5411

This is one of the bug ids if you put 2960 and hulc into google you will see a lot more

Hulc LED Process uses 6-23% CPU on Catalyst 2960/2960S/2960X switch
CSCtg86211
Description
Symptom:
Hulc LED Process uses 6-23% CPU on Catalyst 2960/2960S/2960X 24 or 48-port switch.

Conditions:
The CPU utilization for Hulc LED Process will be in the 6-23% range for the
Catalyst 2960 series like 2960, 2960S, 2960X switch models.

For WS-C2960X-48LPS-L which is 48ports PoE 2960X switch Hulc LED Process
Could reach about 22%-23%.

The is seen in 12.2(50)SE03 or later releases.

Workaround:
This is an expected behavior and there is no workaround.

Further Problem Description:

The "Hulc LED" process does following tasks:

- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status

[Tai Nguyen]

Hi Mark,

Thanks for your comment. I tried reload it but not help. I think I will ask for permission to change the images.

[Tai Nguyen]

Hi Mark,

I upgraded IOS from 15.0(2a)EX5 to 15.2(5)E. But the CPU is still high. Do you have any idea?

[Leo Laohoo]

Upgrade the IOS to 15.2(4)E train (or later).

[Tai Nguyen]

Thanks. I will try and feedback later.

[Tai Nguyen]

Hi Leo,

I upgraded IOS from 15.0(2a)EX5 to 15.2(5)E. But the CPU is still high. Do you have any idea?

[Leo Laohoo]

Post the complete output to the following commands: 

1.  sh version; 

2.  sh proc cpu sort | ex 0.00

[Tai Nguyen]

Firstly, I upgraded to 15.2(5)E, but it's not help. Then, I tried other image (15.2(4)E2).

SW-2960#sh ver
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(4)E2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Mon 27-Jun-16 08:49 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(3r)E1, RELEASE SOFTWARE (fc1)

cisco WS-C2960X-24PS-L (APM86XXX) processor (revision K0) with 524288K bytes of memory.
Processor board ID FCW1951B1SC
Last reset from reload command
2 Virtual Ethernet interfaces
1 FastEthernet interface
28 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address       : 0C:11:67:61:65:80
Motherboard assembly number     : 73-16694-04
Power supply part number        : 341-0528-01
Motherboard serial number       : FOC19506HCT
Power supply serial number      : LIT194704KU
Model revision number           : K0
Motherboard revision number     : B0
Model number                    : WS-C2960X-24PS-L
Daughterboard assembly number   : 73-14200-03
Daughterboard serial number     : FOC19500NX6
System serial number            : FCW1951B1SC
Top Assembly Part Number        : 68-100472-01
Top Assembly Revision Number    : B0
Version ID                      : V03
CLEI Code Number                : CMMLR00ARC
Daughterboard revision number   : A0
Hardware Board Revision Number  : 0x18

Switch Ports Model                     SW Version            SW Image                
------ ----- -----                     ----------            ----------              
*    1 28    WS-C2960X-24PS-L          15.2(4)E2             C2960X-UNIVERSALK9-M    

Configuration register is 0xF
         
SW-2960#sh proc cpu sort | e 0.0
CPU utilization for five seconds: 30%/0%; one minute: 30%; five minutes: 30%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 163     9993925     1428082       6998 14.94% 15.00% 15.00%   0 Hulc LED Process
 175       89399       11422       7826  0.11%  0.13%  0.11%   0 HQM Stack Proces

[Leo Laohoo]

CPU utilization for five seconds: 30%/0%; one minute: 30%; five minutes: 30%

30% is perfect.  

I don't see anything wrong with this value. I would get worried if the CPU value goes higher than 60%.  Anything this low is just right.

[Tai Nguyen]

I read on Cisco page, it said that the "Hulc LED Process" from 0 to 2% is normal. But my devices, it always take more than 15% http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

[Mark Malone]

Yes I agree with Leo if switch is running at that low cpu theres nothing to worry about , I have seen it take up to 25% on devices , if you cant find an ios its fixed in you could ask TAC to provide one , the v15 ios are not as stable for bugs as the v12 , you should be able to check the release notes to see if the caveat is there before downloading another ios either

[Tai Nguyen]

Thanks so much, Leo and Mark.