3750X stack CPU increasing from 40 to 65% (Hulc Led process from 20 to over 40%)

[davidjohnredfern]

Hi,

Our client has a stack of four WS-C3750X-48PF-S switches running 122-58.SE2.

CPU normally sits at 40% with Hulc Led Process using 20%.

Whenever the client uses Symantec Ghost to multicast images to a few pc's, the CPU goes to 60-65%.(low interrupts)

Hulc Led Process increases from 20% to over 40% and we begin to see performance issues. (ip phones unregistering, voice quality issues)

The client has always used Ghost at other sites with switches other than the 3570X without issues.

IGMP snooping is applied to all switches which should limit flooding.

Logs just show constant QOS trust state flapping of many ports which I"m guessing is the  phones resetting.

%SWITCH_QOS_TB-5-TRUST_DEVICE_LOST

%SWITCH_QOS_TB-5-TRUST_DEVICE_DETECTED

sh proc cpu sorted | e 0.00
CPU utilization for five seconds: 62%/1%; one minute: 63%; five minutes: 64%
 PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process
 169  2217217478  1221756240       1814 41.11% 40.29% 39.63%   0 Hulc LED Process
  84  1172730971   710491500       1650  4.79%  4.51%  4.38%   0 RedEarth I2C dri
  85  2347135764   473185300       4960  3.83%  3.79%  3.85%   0 RedEarth Tx Mana
 129   900160394   102656819       8768  1.75%  1.98%  2.01%   0 hpm counter proc
 232   147346287   303244735        485  1.27%  0.87%  0.79%   0 Spanning Tree    
  12    81720858   110076589        742  0.63%  0.29%  0.28%   0 ARP Input        
 181   182012633     9973080      18250  0.47%  0.46%  0.47%   0 HQM Stack Proces
 259    92723044    74624511       1242  0.31%  0.15%  0.14%   0 XDR mcast        
 214    74069998   286681859        258  0.31%  0.12%  0.12%   0 IP Input         
 182   186201884    78177514       2381  0.31%  0.48%  0.48%   0 HRPC qos request
 130    81857170   311631399        262  0.15%  0.20%  0.19%   0 HRPC pm-counters
 107    19751840  1217497713         16  0.15%  0.15%  0.12%   0 HLFM address lea
  86    41554684   334517325        124  0.15%  0.14%  0.13%   0 RedEarth Rx Mana
 371    14803318  1052044058         14  0.15%  0.09%  0.10%   0 PIM Process      
 288    43349962     4914171       8821  0.15%  0.09%  0.10%   0 QoS stats proces
 292    14095075   107528293        131  0.15%  0.09%  0.09%   0 Marvell wk-a Pow
  43    36398731     8754807       4157  0.15%  0.08%  0.09%   0 Net Background   
 320    11719806  1527234091          7  0.15%  0.05%  0.05%   0 MMON MENG        
 279     5230640    99514899         52  0.15%  0.05%  0.02%   0 DHCPD Receive   

I have a TAC case open but any suggestions would be appreciated.

Thanks

[Leo Laohoo]

Downgrade to IOS version 12.2(55)SE9 for better stability.

"Hulc LED Process" is a process which detects if interfaces are active or down.  12.2(58)SE-series train is one of the few IOS versions/train which have CPU hog issues caused by this process.  I would recommend IOS version 12.2(55)SE9 for 3560- and/or 3750-series family of switches.

[Parvesh Paliwal]

Dear Friend,

As the process is related to syslog the link failures, what I can suspect is - there can be multiple failures at L1.

If you still sure, there are no physical link flaps, try shutting the ports one by one and see the changes in process...

--

Parvesh