cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
20592
Views
5
Helpful
8
Replies

High CPU rates due to Hulc LED Process on latest 2960X switch

lanvernier
Level 1
Level 1

Hi all,

I have planned to change all oldest access switches with 2960X. On a test one, monitored by PRTG, CPU is always around 40% as the old 3560's are 8% CPU usage.

Running the EX4 release. Does anyone have the same values ? This problem seems to be recurrent on some Cisco switch a since years....

Nothing in EX5 release notes about that.

Thanks

FBL

8 Replies 8

Leo Laohoo
Hall of Fame
Hall of Fame

Let me brutally frank with you, the current IOS for 2960X/XR is, for a lack of a better word, risky.  

 

EX3 is considered by TAC to be "stable" (suuuuuuuuuuuuuuuure).  EX4 has known CPU bugs.  

 

In my book, you'll need to wait before the main line release of the 15.X comes out.  I seriously wouldn't recommend anyone using EX-series IOS for production if the main line gets release.   

Running the latest & greatest with realistic expectations. This IOS release date is 17-FEB-2016 and the CPU utilization is still pretty high. This switch has nothing connected except for console and power. All interfaces are shutdown and all copper interfaces are configured "power inline never."

Before I made changes, the switch was showing 37% CPU utilization, mostly from the Hulc LED Process. This switch has no stack modules and is running the LAN base image.

Switch#show version
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.2(2)E4, RELEASE SOFTWARE (fc2)
~output omitted~
ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(3r)E1, RELEASE SOFTWARE (fc1)

Switch uptime is 45 minutes
System returned to ROM by power-on
System restarted at 12:36:58 UTC Tue Jun 21 2016
System image file is "flash:/c2960x-universalk9-mz.152-2.E4/c2960x-universalk9-mz.152-2.E4.bin"
Last reload reason: Reload command

~output omitted~

cisco WS-C2960X-48LPS-L (APM86XXX) processor (revision K0) with 524288K bytes of memory.
~output omitted~
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
52 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
~output omitted~
Motherboard assembly number     : 73-16692-04
Power supply part number        : 341-0528-01
~output omitted~
Model revision number           : K0
Motherboard revision number     : A0
Model number                    : WS-C2960X-48LPS-L
Daughterboard assembly number   : 73-14200-03
~output omitted~
Top Assembly Part Number        : 68-100470-01
Top Assembly Revision Number    : A0
Version ID                      : V03
CLEI Code Number                : CMMLP00ARC
Daughterboard revision number   : A0
Hardware Board Revision Number  : 0x18

Switch Ports Model                     SW Version            SW Image
------ ----- -----                     ----------            ----------
*    1 52    WS-C2960X-48LPS-L         15.2(2)E4             C2960X-UNIVERSALK9-M

Configuration register is 0xF

Switch#show processes cpu sorted 5min
CPU utilization for five seconds: 29%/0%; one minute: 29%; five minutes: 30%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 160      604129       67145       8997 15.06% 14.99% 15.14%   0 Hulc LED Process
 171        8252         552      14949  0.23%  0.25%  0.23%   0 HQM Stack Proces
   6        4247         462       9192  0.00%  0.10%  0.11%   0 Check heaps
 140        3414        1132       3015  0.00%  0.00%  0.04%   0 Exec
 386        2145         106      20235  0.00%  0.00%  0.01%   0 hulc running con
  21          24         545         44  0.05%  0.00%  0.00%   0 IPC Event Notifi
 117          51        2685         18  0.00%  0.00%  0.00%   0 Hulc ILP Alchemy
 172        1274        1088       1170  0.00%  0.00%  0.00%   0 HRPC qos request
   8           0           1          0  0.00%  0.00%  0.00%   0 DiscardQ Backgro
   7           0           1          0  0.00%  0.00%  0.00%   0 Pool Manager
  10           7         280         25  0.00%  0.00%  0.00%   0 WATCH_AFS

Peter, 

Are there any PoE &/or Dot1X ports enabled?

I also revisited the Cisco Catalyst 2960-X Series Switches Data Sheet (http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-2960-x-series-switches/data_sheet_c78-728232.html) and the CPU is a APM86392 600MHz dual core. So, I googled that and got the data sheet on the CPU: http://c1170156.r56.cf3.rackcdn.com/UK_AMC_APM86392-SGA600T_DS.pdf. According to the CPU datasheet, it could be faster.

To answer your question on that switch, all interfaces are shutdown and all copper interfaces are configured "power inline never."

I have another switch running the same code part & it's in a pod I'm configuring for a remote site. So, that one has dot1x, QoS, but again, no PoE and all SFP ports are disabled. Here's the interesting parts of the configuration:

no ip domain-lookup
ip domain-name [mycompany].com
vtp mode transparent

udld aggressive

mls qos map cos-dscp 0 8 16 24 32 46 48 56
mls qos

spanning-tree mode rapid-pvst
spanning-tree loopguard default
spanning-tree portfast bpduguard default
spanning-tree extend system-id
auto qos srnd4
errdisable recovery cause udld
errdisable recovery cause bpduguard
errdisable recovery interval 60

no cdp run

interface Port-channel1
 description Uplink_SW1
 switchport trunk allowed vlan [3 VLANs]
 switchport mode trunk
 switchport nonegotiate

#Uplink switchport configuration:

interface GigabitEthernet1/0/1
 description Uplink_SW1_Gi1/0/46
 switchport trunk allowed vlan [3 VLANs]
 switchport mode trunk
 switchport nonegotiate
 power inline never
 channel-group 1 mode active

#Access switchport configuration:

interface GigabitEthernet1/0/2
 switchport access vlan [workstation VLAN]
 switchport mode access
 switchport nonegotiate
 switchport port-security maximum 2
 switchport port-security violation  restrict
 switchport port-security aging time 5
 switchport port-security aging type inactivity
 switchport port-security
 power inline never
 speed auto 100
 storm-control broadcast level 10.00
 storm-control multicast level 10.00

And here's the CPU utilization (didn't make much of a difference):

SW3#sho proc cpu sort 5min
CPU utilization for five seconds: 29%/0%; one minute: 29%; five minutes: 29%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 160    11096203     1587145       6991 15.00% 14.98% 14.99%   0 Hulc LED Process
 166        2697         576       4682  0.00%  0.00%  0.12%   0 Exec
 171       90008       12697       7088  0.11%  0.11%  0.11%   0 HQM Stack Proces
   6      101067       10982       9202  0.00%  0.09%  0.10%   0 Check heaps
 172       37016       25386       1458  0.05%  0.04%  0.05%   0 HRPC qos request
 389        1479          66      22409  0.00%  0.00%  0.01%   0 hulc running con

However, this pod of equipment (2 x 1921 routers & 3 2960X switches) are not under load. I will be monitoring this equipment after it's deployed and plan to post the results.

errdisable recovery cause bpduguard

It won't help the issue, but I'd take this off.  BPDU Guard is a network admin's friend.  If someone's plugged an illegal switch in the network I sure would like to know about it.  Enabling a "recovery" is like sweeping the issue under the carpet.

We played with some of the errdisable settings which will be removed when this switch and others move to production. Most remote sites won't need fiber, so we're also getting rid of all udld commands. I have bpduguard enabled on pretty much all my production switch access ports.

Hi,

This is an expected behavior on the 2k and 3k switches where the Hulc led process will be running between 15-30% even on a unit with no load on it.

Regards,

Jose Solano
Level 4
Level 4

Hi,

 

This is an expected behavior and there is no workaround. 

The "Hulc LED" process does following tasks:

- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status

Hope this helps.

Review Cisco Networking for a $25 gift card