High CPU usage - 3750 X stack - Page 2

johnramz · ‎06-20-2012

Support Community

We recently configured a stack of four 48 port 3750-x switches . We are noticing high CPU usage. "Hulc LED process" seems pretty high.

This has coincided with VMware servers getting slow and non-responsive at times, perhaps a coincidence, not sure.

Below I provided some outputs that might help to diagnose it

Thanks

John

System image file is "flash:/c3750e-ipbasek9-mz.122-58.SE2/c3750e-ipbasek9-mz.122-58.SE2.bin"

Show inventory output

NAME: "1", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02 ,

NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02L ,

NAME: "2", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02

NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02D ,

NAME: "3", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02

NAME: "Switch 3 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02L ,

NAME: "4", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02

NAME: "Switch 4 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02L ,

SWITCH#sh processes cpu sorted

CPU utilization for five seconds: 61%/5%; one minute: 50%; five minutes: 49%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

168 260466386 44948517 5794 14.53% 13.98% 13.70% 0 Hulc LED Process

231 97586088 27253906 3580 4.95% 4.73% 4.64% 0 Spanning Tree

213 63106121 154928892 407 4.15% 3.89% 3.91% 0 IP Input

284 70113217 34537588 2030 3.51% 3.98% 4.17% 0 RARP Input

4 6663412 421278 15817 3.03% 0.43% 0.32% 0 Check heaps

374 9872291 10805181 913 3.03% 0.77% 0.62% 0 IP SNMP

376 11142951 5370604 2074 3.03% 0.73% 0.66% 0 SNMP ENGINE

12 35389011 32152175 1100 2.87% 2.08% 2.20% 0 ARP Input

128 34962407 3622140 9652 2.07% 1.69% 1.63% 0 hpm counter proc

85 49034286 8536062 5744 1.91% 2.44% 2.44% 0 RedEarth Tx Mana

107 25127806 46459053 540 1.27% 1.10% 0.93% 0 HLFM address lea

174 2412 1714 1407 0.95% 0.39% 0.25% 1 SSH Process

220 6423643 12634764 508 0.79% 0.70% 0.56% 0 ADJ resolve proc

181 6913179 2890070 2392 0.63% 0.31% 0.36% 0 HRPC qos request

375 1681949 5000777 336 0.47% 0.08% 0.07% 0 PDU DISPATCHER

84 10180707 12623537 806 0.47% 0.30% 0.37% 0 RedEarth I2C dri

1

666666096996666666666666659666667666666666666666666766676666666656666666

249363098992351145264823289455360612252332233522344115537230141392553343

100 ** ** *

90 ** ** *

80 ** ** *

70 * * ***** * * * * * ** *** * * * **** **

60 **********************************************************************

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

455555555554444444444555554444455555555555555555555444444444

922222111118888866666000009999911111555554444422222444448888

100

90

80

70

60 *****

50 *************************************************** **

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per second (last 60 seconds)

565756555555555555555555555555555556555555555555565555565556

518841757869248569271526666733778330496833777819929379701861

100

90

80 *

70 *

60 **** ******* **** * * ***** *** * *** **** **** **** *

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

DAVE GENTON · ‎09-09-2013

I replaced the universal image with my 12.2(55)SE8 non universal image or ip base in most instances with this customer and a few others. In both customer networks the high cpu issue immediately went away by applying the new image with no loss in features whatsoever. As stated above its a known issue and documented and its likely the links he posted will take you there to review them I'm thinking. I had done that but already had my image replaced and watched cpu fall and never go above 11% since both alone and in stacks with up to 7 switches thus far. Haven't had an issue since I replaced the images the very night I posted that about 2 months ago, I suggest this image and version which is very version used for 3850 IOS-XE images compiled for FCS as 15.x

Anup Sasikumar · ‎06-21-2012

Hi ,

I dont know if it's relevant in this context but just want to share the info that we had a 2950 in the infrastructure which we removed due to High CPU Utilization . Later only we came to know that around 30 %to 50% at idle state was acceptable CPU Utilization for 2950 Switches.

So I guess its same with your 3750 X stack showing High CPU Utilization due to the LED process .

Anup Sasikumar

Sent from Cisco Technical Support Android App

Regards,
Anup

johnramz · ‎06-22-2012

Thanks to all for your replies.

Jeff

I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.

Nikolay,

I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.

This switch also serves as a gateway(L3 role) for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?

Thanks

Johnny

show controllers cpu-interface

ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain

-------------------------------------------------------------------------

ASIC0 0 0 0 0 0 0

ASIC1 0 0 0 0 0 0

ASIC2 0 0 0 0 0 0

HOL Fix Counts

--------------

No Fixes: 0 Added: 0 In Use: 0 Both: 0

CPU Heartbeat Statistics

Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

37139562 0 0 0 0 1

Rx Delay

0 1 2 3 4

---------- ---------- ---------- ---------- ----------

37139562 0 0 0 0

AddlDelay AdvanceCnt

---------- ----------

0 0

Rx Retries by RetryCount

0 1 2 3 4 5 6

---------- ---------- ---------- ---------- ---------- ---------- ----------

37139562 0 0 0 0 0 0

7 8 9

---------- ---------- ----------

0 0 0

AddlRetry

----------

0

cpu-queue-frames retrieved dropped invalid hol-block stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc 104077409 0 0 0 0

stp 19189469 0 0 0 0

ipc 11093838 0 0 0 0

routing protocol 141021559 0 0 0 0

L2 protocol 230347 0 0 0 0

remote console 17 0 0 0 0

sw forwarding 257436702 0 0 0 0

host 21146276 0 0 0 0

broadcast 332154608 0 0 0 0

cbt-to-spt 0 0 0 0 0

igmp snooping 2796987 0 0 0 0

icmp 90752156 0 0 0 0

logging 0 0 0 0 0

rpf-fail 0 0 0 0 0

dstats 0 0 0 0 0

cpu heartbeat 37139562 0 0 0 0

cpu-queue static inuse static added

----------------- ------------ ------------

rpc 0 0

stp 0 0

ipc 0 0

routing protocol 0 0

L2 protocol 0 0

remote console 0 0

sw forwarding 0 0

host 0 0

broadcast 0 0

cbt-to-spt 0 0

igmp snooping 0 0

icmp 0 0

logging 0 0

rpf-fail 0 0

dstats 0 0

cpu heartbeat 0 0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8

queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54

queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C

queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC

queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0

queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80

queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4

queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784

queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574

queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C

queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8

queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4

queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8

queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC

Supervisor ASIC exception status

--------------------------------

Receive overrun 00000000 Transmit overrun 00000000

FrameSignatureErr 00000000 MicInitialize 00000002

BadFrameErr 00000000 LenExceededErr 00000000

BadJumboSegments 00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo 80000200

MicIndicationsReceived 00000000

MicInterruptsReceived 00000009

MicPcsInfo 0000001F

MicPlbMasterConfiguration 00000000

MicRxFifosAvailable 00000000

MicRxFifosReady 0000BFFF

MicTimeOutPeriod: FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied 00000003

MicTxFifosAvailable 0000000E

MicConfiguration: Conf flag: 00000110 Interrupt Flag: 00000008

MicReceiveFifoAssignmen Queue 0 - 7: 33333333 Queue 8 - 15:33333333

MicReceiveFramesReady: FrameAvailable: 00000181 frameAvaiMask: 00000000

MicException:

Exception_flag 00000000

Message-1 00000000

Message-2 00000000

Message-3 00000000

MicIntRxFifo:

ReadPtr 000005C0 WritePtr 000005C0

WHeadPtr 000005C0 TxFifoDepth C0000800

MicIntTxFifo:

ReadPtr 00000728 WritePtr 00000728

WHeadPtr 00000728 TxFifoDepth C0000800

MicDecodeInfo:

Fifo0: address: 03FF4000 asic_num: 00000100

Fifo1: address: 03FF4400 asic_num: 00000101

MicTransmitFifoInfo:

Fifo0: StartPtrs: 0E2CE800 ReadPtr: 0E2CEBE8

WritePtrs: 0E2CEBE8 Fifo_Flag: 8A800800

Weights: 001E001E

Fifo1: StartPtrs: 0E02D000 ReadPtr: 0E02D138

WritePtrs: 0E02D138 Fifo_Flag: 89800400

Weights: 000A000A

MicReceiveFifoInfo:

Fifo0: StartPtr: 0E4AF000 ReadPtr: 0E4AF2A8

WritePtrs: 0E4AF308 Fifo_Flag: 8B000FA0

writeHeaderPtr: 0E4AF308

Fifo1: StartPtr: 0E78C000 ReadPtr: 0E78C2E8

WritePtrs: 0E78C2E8 Fifo_Flag: 89800400

writeHeaderPtr: 0E78C2E8

Fifo2: StartPtr: 0E744800 ReadPtr: 0E744A70

WritePtrs: 0E744A70 Fifo_Flag: 89800400

writeHeaderPtr: 0E744A70

Fifo3: StartPtr: 0EBD1000 ReadPtr: 0EBD13B8

WritePtrs: 0EBD13B8 Fifo_Flag: 89800400

writeHeaderPtr: 0EBD13B8

Fifo4: StartPtr: 0E7D3800 ReadPtr: 0E7D3A58

WritePtrs: 0E7D3A58 Fifo_Flag: 89800400

writeHeaderPtr: 0E7D3A58

Fifo5: StartPtr: 0EB40600 ReadPtr: 0EB40688

WritePtrs: 0EB40688 Fifo_Flag: 88800200

writeHeaderPtr: 0EB40688

Fifo6: StartPtr: 0EB87400 ReadPtr: 0EB874F0

WritePtrs: 0EB874F0 Fifo_Flag: 89800400

writeHeaderPtr: 0EB874F0

Fifo7: StartPtr: 0E880000 ReadPtr: 0E880E20

WritePtrs: 0E881520 Fifo_Flag: 8C001900

writeHeaderPtr: 0E881520

Fifo8: StartPtr: 0EB1A600 ReadPtr: 0EB1A770

WritePtrs: 0EB1A780 Fifo_Flag: 880001F0

writeHeaderPtr: 0EB1A780

Fifo9: StartPtr: 0E2E0CD8 ReadPtr: 0E2E0CD8

WritePtrs: 0E2E0CD8 Fifo_Flag: 82800008

writeHeaderPtr: 0E2E0CD8

Fifo10: StartPtr: 0E81D000 ReadPtr: 0E81D1D8

WritePtrs: 0E81D1D8 Fifo_Flag: 88800200

writeHeaderPtr: 0E81D1D8

Fifo11: StartPtr: 0E4AEF00 ReadPtr: 0E4AEF60

WritePtrs: 0E4AEF60 Fifo_Flag: 86800080

writeHeaderPtr: 0E4AEF60

Fifo12: StartPtr: 0E84A000 ReadPtr: 0E84A300

WritePtrs: 0E84A000 Fifo_Flag: 89000100

writeHeaderPtr: 0E84A000

Fifo13: StartPtr: 0E4AEE00 ReadPtr: 0E4AEE00

WritePtrs: 0E4AEE00 Fifo_Flag: 86800080

writeHeaderPtr: 0E4AEE00

Fifo14: StartPtr: 00000000 ReadPtr: 00000000

WritePtrs: 00000000 Fifo_Flag: 00800000

writeHeaderPtr: 00000000

Fifo15: StartPtr: 0E02CEC0 ReadPtr: 0E02CED0

WritePtrs: 0E02CED0 Fifo_Flag: 84800020

writeHeaderPtr: 0E02CED0

===========================================================

Complete Board Id:0x00B2

===========================================================

johnnylingo · ‎07-26-2013

I think your ACL issue is a red herring. You're not the only one seeing the 30% CPU and higher.

Now the part that really drives me nuts is I have a stack that's only at 14%. Check this out:

CPU utilization for five seconds: 21%/0%; one minute: 14%; five minutes: 14%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

162 302 77 3922 4.15% 0.33% 0.06% 1 SSH Process

74 243161037 39366660 6176 3.03% 2.85% 2.84% 0 RedEarth Tx Mana

4 96582240 4319109 22361 2.07% 1.40% 1.16% 0 Check heaps

156 118927698 191802387 620 1.91% 1.35% 1.35% 0 Hulc LED Process

116 143864159 16784938 8571 1.75% 1.68% 1.66% 0 hpm counter proc

186 9436800 8137819 1159 0.79% 0.13% 0.11% 0 CDP Protocol

Its configuration and software version is the same as a stack right next to it that's at 29% CPU. Go figure. I'm rolling out more 3750Xs at a new data center next week and it will be interesting to see how they behave.

CSCO11754706 · ‎09-08-2013

Hello !

The WS-C3750X-48T-E "12.2(58)SE2 C3750E-UNIVERSALK9-M" Today I met the same question, high cpu usage, but I just ran EIGRP,VLAN, the following information is output:

3750#show proc c

CPU utilization for five seconds: 27%/1%; one minute: 28%; five minutes: 30%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

1 0 20 0 0.00% 0.00% 0.00% 0 Chunk Manager

2 0 669 0 0.00% 0.00% 0.00% 0 Load Meter

3 0 28 0 0.00% 0.00% 0.00% 0 DHCPD Timer

4 9437 693 13617 0.00% 0.21% 0.23% 0 Check heaps

5 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager

85 84718 15847 5345 2.40% 2.37% 2.36% 0 RedEarth Tx Mana

107 48443 94138 514 3.20% 2.19% 2.43% 0 HLFM address lea

169 126798 83456 1519 13.27% 10.35% 7.59% 0 Hulc LED Process "This is highset"

Ross Phillips · ‎09-09-2013

Hi, we had the same issue on these switches where ther HULC process went through the roof,

This was down to there being unused interfaces on the switches where left in a not connect state.

Shutdown all unused interfaces on the switch and if its like what we experienced it should solve your problem.

Ross

CSCO11754706 · ‎09-09-2013

Thank you very much, I will try!

johnnylingo · ‎09-09-2013

Ross - what version of IOS are you running? I've already tried shutting down all unused ports and rebooting, but to no avail.

jawad-mukhtar · ‎09-09-2013

R u using route map if u r then u will also b using acls in that just remove deny statement from route map ace

Sent from Cisco Technical Support Android App

Jawad

insccisco · ‎10-28-2013

Guys,

What is the bottom line with regards to the software you will put in a 3750X switch? I got 5 cisco 3750X switches and I am about to put them into production (no stack config, just stand-alone) and I'd like to know if I am better safe with the

12.2(55)SE5 version Dave Genton suggested?

My switches came with IOS version (C3750E-UNIVERSALK9-M), Version 15.0(2)SE. I quickly looked this version up and it appears to have few big bugs already plus, I also see the CPU utilization at 22-26% with nothing connected in the ports. I also have couple of the switches with ALL the ports manually SHUT and I still the this CPU %.

Please advise.

Thank you in advance

Leo Laohoo · ‎10-28-2013

If all of your 3750X can run 12.2(55)SE8, then use this version as it's considered to be the most stable.

If you need features only found in 15.0 then go with 15.0(2)SE4. Do not even attempt to use 15.2(1)E.

insccisco · ‎10-28-2013

I just tried following this exact advice (I loaded IOS version c3750e-ipbasek9-mz.150-2.SE4.bin) but the switch went into an infinite loop.

Did I use the wrong image file (instead of c3750e-universalk9-mz.150-2.SE.bin) ??

All my switches are C3750X platform with license level IP Base.

Leo Laohoo · ‎10-28-2013

Post the entire sequence. If the switch crashed, I want to see the error messages.

insccisco · ‎10-28-2013

I wasnt recording the output and it is all gone by now.... I quickly performed a recovery procedure and went back to the version that the switch came with --- IOS version (C3750E-UNIVERSALK9-M), Version 15.0(2)SE

For some reason I am suspecting is the the IP Base version I tried. But before I try the Universal version of that image (c3750e-universalk9-mz.150-2.SE.bin), I wanted to ask you guys for help.

I did read the pre-requisites of the c3750e-ipbasek9-mz.150-2.SE4.bin and it stated that I needed 256/64 in the switch which I do have but for some reason I think I might not be seeing something else that perhaps states that I can't use the ip-base flavor of the image....

What is your take on this?

Leo Laohoo · ‎10-28-2013

I'm not totally sure what's going on.

Try to install the IOS on a switch without any config and any network module.