Hulc LED Process - Average 20%

[Rui Figueiredo]

Dear all,

  I have a stack of 4 WS-C3750X-48P-S running 15.0.2 SE10, which on average has a 20% CPU usage for the Hulc LED Process.

  I have been reading some discussions regarding this process, and also some Cisco documentation, and I have the following doubts :

  1 - Since this is a stack, does this process only runs at the master switch level?

  2 - I've discovered that there are bug reports for this issue, but I'm assuming they were already corrected on the version I'm running, since SE6 it's known to solve this issue.

  (Please refer to bugs CSCtn42790 and CSCsi78581 https://bst.cloudapps.cisco.com/bugsearch/bug/CSCsi78581/?referring_site=bugquickviewredir)

  3 - Cisco refers that for a 48P switch, the average should not exceed 8%. Since this is stack with 4x48P should we consider a max of 4x8% ?

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

  I must say that there is no network complaints / service degradation, I'm just trying to be on the safe and preventive side.

  Thank you in advance for your help.

Best regards

Rui Figueiredo

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

#1 I believe that's how its supposed to work for the higher level processes.

#2 Yes, although it's also possible they reintroduced the problem in a later release, or a variant of it.

#3 Considering the history of issue with the HULC process, I would take that documentation with a vary large grain of salt.

BTW, regarding lack of impact, ideally, a 3K switch would be doing forwarding in hardware, so even at 100% CPU usage, you might not see any service issues.  Additionally, as different software processes might have different task priorities, a high HULC might not be adverse to more important processes.  Again, even if system was running at 100%, other processes might get the CPU attention they need.

[Rui Figueiredo]

leolaohoo  Thank you for your interest in this matter. Unfortunately we have a restrict policy regarding passing information, therefore some private output was removed.

CPU utilization for five seconds: 44%/2%; one minute: 54%; five minutes: 53%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 180   832031208    97761730       8510 20.31% 19.84% 19.34%   0 Hulc LED Proces                                                                                                                      s
  90   202969114    43293759       4688  6.23%  4.99%  4.87%   0 RedEarth Tx Man                                                                                                                      a
  89   141281289    69148985       2043  2.71%  3.39%  3.41%   0 RedEarth I2C dr                                                                                                                      i
 135    66880063     3934273      16999  1.59%  1.59%  1.59%   0 hpm counter pro                                                                                                                      c
 197    19062267     6221810       3063  0.63%  0.45%  0.46%   0 HRPC qos reques                                                                                                                      t
  58     2386886       67513      35354  0.63%  0.07%  0.05%   0 Per-minute Jobs                                                                                                                      
  13    17529312    24174759        725  0.63%  1.57%  1.63%   0 ARP Input                                                                                                                            
  99     7930418    28638028        276  0.47%  0.23%  0.34%   0 hrpc <- respons                                                                                                                      e
  46     4921330      872852       5638  0.47%  0.14%  0.12%   0 Net Background                                                                                                                       
 196    15971305      782536      20409  0.31%  0.37%  0.37%   0 HQM Stack Proce                                                                                                                      s
 131     3594781    63934370         56  0.31%  0.16%  0.15%   0 hpm main proces                                                                                                                      s
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 428    16062263    37449591        428  0.31%  0.34%  0.38%   0 LLDP Protocol                                                                                                                        
 205       89086      131769        676  0.31%  0.05%  0.01%   0 HL2MCM                                                                                                                               
 152         840         184       4565  0.31%  0.61%  0.22%   1 SSH Process                                                                                                                          
 260     5148694    35370838        145  0.31%  0.84%  0.66%   0 HRPC ip device                                                                                                                       t
 248    15166142    19208780        789  0.31%  0.37%  0.39%   0 Spanning Tree                                                                                                                        
 229     2474381    97163088         25  0.15%  0.05%  0.04%   0 IP ARP Retry Ag                                                                                                                      e
 208      174250      107283       1624  0.15%  0.05%  0.01%   0 AAA Server                                                                                                                           
 266     1540696     3893970        395  0.15%  0.06%  0.04%   0 PI MATM Aging P                                                                                                                      r
 136    10748228    13689333        785  0.15%  0.24%  0.23%   0 HRPC pm-counter                                                                                                                      s
 112     6533372    97167048         67  0.15%  0.28%  0.27%   0 HLFM address le                                                                                                                      a
 365     1819972   122287271         14  0.15%  0.07%  0.02%   0 MMON MENG                                                                                                                            
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 284     2468465     2229545       1107  0.15%  0.08%  0.07%   0 XDR mcast rcv  


Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(2)SE10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2016 by Cisco Systems, Inc.
Compiled Mon 18-Jul-16 01:28 by prod_rel_team

ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE2, RELEASE SOFTWARE (fc1)

XXXXXX uptime is 6 weeks, 3 days, 13 hours, 0 minutes
System returned to ROM by power-on
System restarted at 19:12:20 WET Sun Aug 7 2016
System image file is "flash:/c3750e-universalk9-mz.150-2.SE10/c3750e-universalk9-mz.150-2.SE10.bin"


This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: ipbase
License Type: Permanent
Next reload license Level: ipbase

cisco WS-C3750X-48P (PowerPC405) processor (revision F0) with 262144K bytes of memory.
Processor board ID XXXXXXXXXX
Last reset from power-on
3 Virtual Ethernet interfaces
1 FastEthernet interface
208 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Model revision number           : F0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48P-S
Top Assembly Revision Number    : B0
Version ID                      : V03
CLEI Code Number                : COMJZ00ARC
Hardware Board Revision Number  : 0x04


Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
     1 54    WS-C3750X-48P      15.0(2)SE10           C3750E-UNIVERSALK9-M
     2 54    WS-C3750X-48P      15.0(2)SE10           C3750E-UNIVERSALK9-M
*    3 54    WS-C3750X-48P      15.0(2)SE10           C3750E-UNIVERSALK9-M
     4 54    WS-C3750X-48P      15.0(2)SE10           C3750E-UNIVERSALK9-M


Switch 01
---------
Switch Uptime                   : 6 weeks, 3 days, 12 hours, 56 minutes
Model revision number           : F0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48P-S
Top assembly revision number    : B0
Version ID                      : V03
CLEI Code Number                : COMJZ00ARC
License Level                   : ipbase
License Type                    : Permanent
Next reboot licensing Level     : ipbase


Switch 02
---------
Switch Uptime                   : 6 weeks, 3 days, 12 hours, 56 minutes
Model revision number           : F0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48P-S
Top assembly revision number    : B0
Version ID                      : V03
CLEI Code Number                : COMJZ00ARC
License Level                   : ipbase
License Type                    : Permanent
Next reboot licensing Level     : ipbase


Switch 04
---------
Switch Uptime                   : 6 weeks, 3 days, 12 hours, 56 minutes
Model revision number           : F0
Motherboard revision number     : A0
Model number                    : WS-C3750X-48P-S
Top assembly revision number    : B0
Version ID                      : V03
CLEI Code Number                : COMJZ00ARC
License Level                   : ipbase
License Type                    : Permanent
Next reboot licensing Level     : ipbase


Configuration register is 0xF

@Joseph W. Doherty - Thank you also for your interested in this matter.

#2 Taking in consideration your comment, what would you recommend? To open a TAC Case just to be sure

Regarding your remaining notes, you're absolutely right, so it makes sense

[Joseph W. Doherty · Accepted solution]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Regarding point #2 and contacting TAC, you might, but if it is a bug, and it doesn't really have any actual adverse impact, i.e. it might be considered a cosmetic bug.  If so, you might just continue to run with the version you have if everything else is working as you desire.

[Rui Figueiredo]

Thank you both for your inputs.

I agree with your point of view Joseph W. Doherty. I'll just keep this in mind and not worry too much.

I wish you all the best.

[Leo Laohoo]

20% HULC Led is nothing.  I'm not concerned at all.

If HULC LED goes above 60% then I'm concerned.  

[Leo Laohoo]

Post the complete output to the command "sh proc cpu sort | ex 0.00" & "sh version".