2960 - HULC LED PROCESS

CyberSecLead · ‎08-23-2012

Dear All,

I have cisco WS-C2960S-48FPS-L stacked. Weekly twice, my PoE connections are dropped and when the device is restarted, everything starts working normal. This issue happens weekly once or twice.

Verified the power input and it's absolutely fine.

show power inline

Module Available Used Remaining

(Watts) (Watts) (Watts)

------ --------- -------- ---------

1 740.0 91.5 648.5

2 740.0 60.2 679.8

3 740.0 144.3 595.7

4 740.0 157.1 582.9

Upon taking a look to the CPU Process, device is punted with 75 - 85 % at normal usage, with "Hulc LED Process" as 18% utilisation. CPU utilisation also peaks to 90 - 100% sometimes.

CPU utilization for five seconds: 78%/48%; one minute: 58%; five minutes: 50%

100 * *

90 * ** * * *

80 *###############

70 ################

60 ################

50 ################

40 ################

30 ################

20 ################

10 ################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

I can see that there is a bug id : CSCtg86211 and no work around for it.

Does anyone faced similar issue and any updates received from Cisco TAC ?

Any recommendations to overcome this issue !!

BR / Vimal.

CyberSecLead · ‎08-24-2012

Awaiting Cisco Experts to reply to the above subject.

Thanks / Vimal.

lcubero · ‎12-19-2014

Hi Vimal,

Did you have any answer from Cisco ? I am working on a project with 2960 and facing same issue. I would really appreciate if it has already been solved.

Regards,

Luis.

Mark Pottage · ‎12-31-2014

I am also seeing an issue with two stacks of 5 x WS-C2960X-48FP-S switches similar to this case.

Code in use: c2960x-universalk9-mz.150-2.EX5.bin

At the moment there is nothing connected to the copper 48 ports in any of the switches.

Each stack has a LACP port-channel to a 2 x WS-C3850-48F-E core stack utilising ports x/0/51 and x/0/52 on each 2960X - total 10 ports in each port-channel (8 in use 2 Hot-standby)

The uplink ports are using GLC-Ts

There are 5 vlans (including Vlan1) on the switch.

sw02#show proc cpu | excl 0.00
CPU utilization for five seconds: 42%/0%; one minute: 42%; five minutes: 42%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
91 1355993 26717170 50 0.29% 0.22% 0.19% 0 RedEarth Tx Mana
92 886939 26742049 33 0.11% 0.12% 0.16% 0 RedEarth Rx Mana
121 618726 15414683 40 0.11% 0.06% 0.06% 0 HLFM address lea
127 2828875 11788935 239 0.41% 0.41% 0.42% 0 hpm main process
132 637843 4108756 155 0.05% 0.07% 0.05% 0 HRPC pm-counters
162 176218586 15422705 11425 25.07% 25.30% 25.31% 0 Hulc LED Process
173 2274099 122941 18497 0.35% 0.31% 0.30% 0 HQM Stack Proces
174 1720412 1228633 1400 0.23% 0.23% 0.23% 0 HRPC qos request
183 844538 205684 4105 0.05% 0.10% 0.11% 0 Power RPS Proces
207 2879 22868 125 0.11% 0.10% 0.06% 0 IP Input
220 288284 1554997 185 0.11% 0.04% 0.05% 0 Spanning Tree
237 1356330 6170406 219 0.11% 0.07% 0.14% 0 UDLD
293 324727 1359786 238 0.05% 0.04% 0.05% 0 Marvell wk-a Pow
372 120 123 975 0.05% 0.07% 0.01% 1 Virtual Exec

sw03#sh proc cpu | exc 0.00
CPU utilization for five seconds: 42%/0%; one minute: 46%; five minutes: 45%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
63 507 1733 292 0.05% 0.06% 0.03% 0 Per-Second Jobs
91 5409 72997 74 0.35% 0.23% 0.23% 0 RedEarth Tx Mana
92 4526 73169 61 0.11% 0.19% 0.23% 0 RedEarth Rx Mana
129 11881 29115 408 0.54% 0.42% 0.42% 0 hpm main process
133 1259 1674 752 0.23% 0.16% 0.12% 0 hpm counter proc
163 494386 41535 11902 25.38% 25.35% 25.30% 0 Hulc LED Process
174 6130 374 16390 0.29% 0.31% 0.29% 0 HQM Stack Proces
175 4446 3249 1368 0.29% 0.25% 0.24% 0 HRPC qos request
183 2237 559 4001 0.11% 0.16% 0.13% 0 Power RPS Proces
206 894 41778 21 0.05% 0.03% 0.03% 0 IP ARP Retry Age
237 5298 16703 317 0.11% 0.14% 0.26% 0 UDLD
251 4045 452 8949 0.29% 2.68% 0.98% 1 Virtual Exec
291 1183 3706 319 0.05% 0.05% 0.05% 0 Marvell wk-a Pow

Any advice from Cisco would be much appreciated.

Mark

chris phillips · ‎09-03-2015

Seeing the same issue here with 2960x 48 port poe's running 15.02a no devices connected to the stack.

Cisco, are you looking to address this at any stage?

-If I helped you somehow, please, rate it as useful.-

Sam Almeida · ‎09-03-2015

Do you have cisco stackwise stack power cables connected between the stacks? If so ensure they are connected correctly.

DATA Stack Cable:
Sw1 Stack 1 >>>> SW2 Stack 2
Sw1 Stack 2 >>>> SW2 Stack 1

Stackwise Power cable
Sw1 Stack 1 >>>> SW2 Stack 1
Sw1 Stack 2 >>>> SW2 Stack 2

chris phillips · ‎09-03-2015

sadly nothing as exotic or costly as that, just a stack of 3 2960-x poe switches.

further research shows that Cisco don't think this is an issue. HULC process will be low priority and even if running 100% other interrupts will take precedence. Seems HULC is an amalgamation of other processes in one.

The biggest issue here is environmental as excess power and therefore heat is generated when not needed.

-If I helped you somehow, please, rate it as useful.-

Sam Almeida · ‎09-03-2015

You mention no devices are connected. Admin shut all the ports on the switch (except management) and verify if cpu goes down

Leo Laohoo · ‎08-25-2012

Weekly twice, my PoE connections are dropped and when the device is restarted,

If your switch stacks rebooted without any intervention, then check each switch stack for the crashinfo file(s) or subdirectories and post the three LATEST files.

Also, please state the IOS version.

CyberSecLead · ‎08-25-2012

Hi,

IOS code running is c2960s-universalk9-mz.122-53.SE2. I haven't mentioned that the device got rebooted, instead all the PoE connections dropped. Example, Ip phones and AP's were in registering phase for a long time, until a reboot is needed to re-establish the connections back.

Any suggestions.

Regards / Vimal.

tmomani · ‎08-25-2012

Hi,

Its is totally expected fir the HULC LED to have up to 30% of the cpu utilization, as thi process handles ports changes (state, poe..etc).

The high cpu u r facing is under interrups (X%/Y%) where X is the total utilization and Y is the interrupt value.

The interrupt means that the switch is forwarding alot of traffic. Would u plz send a full outpu of "sh proc cpu sorted | ex 0.00".

Based on the output, if no "ip input or stp or any otger special process is in place" then it would be a HW limitation for this platform, then u would rather offload this switch(load balance traffic to multiple devices) or maybe use a more powerful switch.

Cheers,

Tareq

Sent from Cisco Technical Support iPhone App

CyberSecLead · ‎08-25-2012

Hi, please find attached to the "show tech". There is not much process involved in "input" "stp" etc.

Regards / Vimal.

tmomani · ‎08-25-2012

Hello,

TCAM utilization is OK, lets identify the fastest growing CPU Queue, and then we will capture traffic punted to that Q.

when the CPU is High,

take "show controllers cpu-interface" command 3 - 4 times seperated by 2 seconds interval

identify the fastest growing counter, and based on that , capture traffic using "debug platform cpu-queuequeue", but please notice that the output is very large, so enable safe debugging before

+ increase the buffer log by "SW(config)#logging buffer 100000"

+ SW(config)#no logging console

+ SW#terminal no monitor

refer to below link as a reference,

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

let me know on how this goes,

Tareq

Leo Laohoo · ‎08-25-2012

IOS code running is  c2960s-universalk9-mz.122-53.SE2.

I read your reply post and didn't bother to read any further..

Do NOT use this IOS, 12.2(53)SE. I know I wouldn't.

IOS I would personally recommend are the following:

12.2(55)SE5 or 12.2(55)SE6; or
15.0(1)SE3; or
15.0(2)SE

Read the Release Notes first before making a decision.