cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
Join Customer Connection to register!
51892
Views
20
Helpful
37
Replies
johnramz
Beginner

High CPU usage - 3750 X stack

Support Community

We recently configured a stack of four 48 port 3750-x switches . We are noticing high CPU usage. "Hulc LED process" seems pretty high.

This has coincided with VMware servers getting slow and non-responsive at times, perhaps a coincidence, not sure.

Below I provided some outputs that might help to diagnose it

Thanks

John

System image file is "flash:/c3750e-ipbasek9-mz.122-58.SE2/c3750e-ipbasek9-mz.122-58.SE2.bin"

Show inventory output

NAME: "1", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02  ,

NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

NAME: "2", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02D ,

NAME: "3", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 3 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

NAME: "4", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 4 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

SWITCH#sh processes cpu sorted

CPU utilization for five seconds: 61%/5%; one minute: 50%; five minutes: 49%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

168   260466386    44948517       5794 14.53% 13.98% 13.70%   0 Hulc LED Process

231    97586088    27253906       3580  4.95%  4.73%  4.64%   0 Spanning Tree

213    63106121   154928892        407  4.15%  3.89%  3.91%   0 IP Input

284    70113217    34537588       2030  3.51%  3.98%  4.17%   0 RARP Input

   4     6663412      421278      15817  3.03%  0.43%  0.32%   0 Check heaps

374     9872291    10805181        913  3.03%  0.77%  0.62%   0 IP SNMP

376    11142951     5370604       2074  3.03%  0.73%  0.66%   0 SNMP ENGINE

  12    35389011    32152175       1100  2.87%  2.08%  2.20%   0 ARP Input

128    34962407     3622140       9652  2.07%  1.69%  1.63%   0 hpm counter proc

  85    49034286     8536062       5744  1.91%  2.44%  2.44%   0 RedEarth Tx Mana

107    25127806    46459053        540  1.27%  1.10%  0.93%   0 HLFM address lea

174        2412        1714       1407  0.95%  0.39%  0.25%   1 SSH Process

220     6423643    12634764        508  0.79%  0.70%  0.56%   0 ADJ resolve proc

181     6913179     2890070       2392  0.63%  0.31%  0.36%   0 HRPC qos request

375     1681949     5000777        336  0.47%  0.08%  0.07%   0 PDU DISPATCHER

  84    10180707    12623537        806  0.47%  0.30%  0.37%   0 RedEarth I2C dri

        1

      666666096996666666666666659666667666666666666666666766676666666656666666

      249363098992351145264823289455360612252332233522344115537230141392553343

  100       ** **               *

   90       ** **               *

   80       ** **               *

   70   * * *****  *   * * *    * ** ***   *       *     * ****         **

   60 **********************************************************************

   50 ######################################################################

   40 ######################################################################

   30 ######################################################################

   20 ######################################################################

   10 ######################################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

               0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

      455555555554444444444555554444455555555555555555555444444444

      922222111118888866666000009999911111555554444422222444448888

  100

   90

   80

   70

   60                                     *****

   50 ***************************************************     **

   40 **********************************************************

   30 **********************************************************

   20 **********************************************************

   10 **********************************************************

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per second (last 60 seconds)

      565756555555555555555555555555555556555555555555565555565556

      518841757869248569271526666733778330496833777819929379701861

  100

   90

   80    *

   70    *

   60 **** *******  **** * * *****  ***  * ***  **** **** **** *

   50 ##########################################################

   40 ##########################################################

   30 ##########################################################

   20 ##########################################################

   10 ##########################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

37 REPLIES 37

  I replaced the universal image with my 12.2(55)SE8 non universal image or ip base in most instances with this customer and a few others.   In both customer networks the high cpu issue immediately went away by applying the new image with no loss in features whatsoever.    As stated above its a known issue and documented and its likely the links he posted will take you there to review them I'm thinking.  I had done that but already had my image replaced and watched cpu fall and never go above 11% since both alone and in stacks with up to 7 switches thus far.   Haven't had an issue since I replaced the images the very night I posted that about 2 months ago, I suggest this image and version which is very version used for 3850 IOS-XE images compiled for FCS as 15.x

Anup Sasikumar
Beginner

Hi ,

I dont know if it's relevant in this context but just want to share the info that we had a 2950 in the infrastructure which we removed due to High CPU Utilization . Later only we came to know that around 30 %to 50% at idle state was acceptable CPU Utilization for 2950 Switches.

So I guess its same with your 3750 X stack showing High CPU Utilization due to the LED process .

Anup Sasikumar

Sent from Cisco Technical Support Android App

Regards,
Anup

Thanks to all for your replies.

Jeff

I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.

Nikolay,

I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.

This switch also serves as a gateway(L3 role)  for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?

Thanks

Johnny

show controllers cpu-interface

ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain

-------------------------------------------------------------------------

ASIC0     0          0          0          0          0          0

ASIC1     0          0          0          0          0          0

ASIC2     0          0          0          0          0          0

HOL Fix Counts

--------------

No Fixes:          0 Added:          0 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          1

Rx Delay

         0          1          2          3          4

---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0

AddlDelay AdvanceCnt

---------- ----------

         0          0

Rx Retries by RetryCount

         0          1          2          3          4          5          6

---------- ---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          0          0

         7          8          9

---------- ---------- ----------

         0          0          0

AddlRetry

----------

         0

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc               104077409  0          0          0          0

stp               19189469   0          0          0          0

ipc               11093838   0          0          0          0

routing protocol  141021559  0          0          0          0

L2 protocol       230347     0          0          0          0

remote console    17         0          0          0          0

sw forwarding     257436702  0          0          0          0

host              21146276   0          0          0          0

broadcast         332154608  0          0          0          0

cbt-to-spt        0          0          0          0          0

igmp snooping     2796987    0          0          0          0

icmp              90752156   0          0          0          0

logging           0          0          0          0          0

rpf-fail          0          0          0          0          0

dstats            0          0          0          0          0

cpu heartbeat     37139562   0          0          0          0

cpu-queue         static inuse static added

----------------- ------------ ------------

rpc               0            0

stp               0            0

ipc               0            0

routing protocol  0            0

L2 protocol       0            0

remote console    0            0

sw forwarding     0            0

host              0            0

broadcast         0            0

cbt-to-spt        0            0

igmp snooping     0            0

icmp              0            0

logging           0            0

rpf-fail          0            0

dstats            0            0

cpu heartbeat     0            0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8

queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54

queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C

queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC

queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0

queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80

queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4

queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784

queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574

queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C

queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8

queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4

queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8

queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC

Supervisor ASIC exception status

--------------------------------

Receive overrun    00000000   Transmit overrun 00000000

FrameSignatureErr  00000000   MicInitialize    00000002

BadFrameErr        00000000   LenExceededErr   00000000

BadJumboSegments   00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo               80000200

MicIndicationsReceived          00000000

MicInterruptsReceived           00000009

MicPcsInfo                      0000001F

MicPlbMasterConfiguration       00000000

MicRxFifosAvailable             00000000

MicRxFifosReady                 0000BFFF

MicTimeOutPeriod:       FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied           00000003

MicTxFifosAvailable             0000000E

MicConfiguration:       Conf flag: 00000110     Interrupt Flag: 00000008

MicReceiveFifoAssignmen Queue 0 - 7: 33333333   Queue 8 - 15:33333333

MicReceiveFramesReady:  FrameAvailable: 00000181        frameAvaiMask: 00000000

MicException:

        Exception_flag  00000000

        Message-1       00000000

        Message-2       00000000

        Message-3       00000000

MicIntRxFifo:

        ReadPtr         000005C0        WritePtr        000005C0

        WHeadPtr        000005C0        TxFifoDepth     C0000800

MicIntTxFifo:

        ReadPtr         00000728        WritePtr        00000728

        WHeadPtr        00000728        TxFifoDepth     C0000800

MicDecodeInfo:

Fifo0:  address:        03FF4000 asic_num:      00000100

Fifo1:  address:        03FF4400 asic_num:      00000101

MicTransmitFifoInfo:

Fifo0:   StartPtrs:     0E2CE800        ReadPtr:        0E2CEBE8

        WritePtrs:      0E2CEBE8        Fifo_Flag:      8A800800

        Weights:        001E001E

Fifo1:   StartPtrs:     0E02D000        ReadPtr:        0E02D138

        WritePtrs:      0E02D138        Fifo_Flag:      89800400

        Weights:        000A000A

MicReceiveFifoInfo:

Fifo0:  StartPtr:       0E4AF000        ReadPtr:        0E4AF2A8

        WritePtrs:      0E4AF308        Fifo_Flag:      8B000FA0

        writeHeaderPtr: 0E4AF308

Fifo1:  StartPtr:       0E78C000        ReadPtr:        0E78C2E8

        WritePtrs:      0E78C2E8        Fifo_Flag:      89800400

        writeHeaderPtr: 0E78C2E8

Fifo2:  StartPtr:       0E744800        ReadPtr:        0E744A70

        WritePtrs:      0E744A70        Fifo_Flag:      89800400

        writeHeaderPtr: 0E744A70

Fifo3:  StartPtr:       0EBD1000        ReadPtr:        0EBD13B8

        WritePtrs:      0EBD13B8        Fifo_Flag:      89800400

        writeHeaderPtr: 0EBD13B8

Fifo4:  StartPtr:       0E7D3800        ReadPtr:        0E7D3A58

        WritePtrs:      0E7D3A58        Fifo_Flag:      89800400

        writeHeaderPtr: 0E7D3A58

Fifo5:  StartPtr:       0EB40600        ReadPtr:        0EB40688

        WritePtrs:      0EB40688        Fifo_Flag:      88800200

        writeHeaderPtr: 0EB40688

Fifo6:  StartPtr:       0EB87400        ReadPtr:        0EB874F0

        WritePtrs:      0EB874F0        Fifo_Flag:      89800400

        writeHeaderPtr: 0EB874F0

Fifo7:  StartPtr:       0E880000        ReadPtr:        0E880E20

        WritePtrs:      0E881520        Fifo_Flag:      8C001900

        writeHeaderPtr: 0E881520

Fifo8:  StartPtr:       0EB1A600        ReadPtr:        0EB1A770

        WritePtrs:      0EB1A780        Fifo_Flag:      880001F0

        writeHeaderPtr: 0EB1A780

Fifo9:  StartPtr:       0E2E0CD8        ReadPtr:        0E2E0CD8

        WritePtrs:      0E2E0CD8        Fifo_Flag:      82800008

        writeHeaderPtr: 0E2E0CD8

Fifo10: StartPtr:       0E81D000        ReadPtr:        0E81D1D8

        WritePtrs:      0E81D1D8        Fifo_Flag:      88800200

        writeHeaderPtr: 0E81D1D8

Fifo11: StartPtr:       0E4AEF00        ReadPtr:        0E4AEF60

        WritePtrs:      0E4AEF60        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEF60

Fifo12: StartPtr:       0E84A000        ReadPtr:        0E84A300

        WritePtrs:      0E84A000        Fifo_Flag:      89000100

        writeHeaderPtr: 0E84A000

Fifo13: StartPtr:       0E4AEE00        ReadPtr:        0E4AEE00

        WritePtrs:      0E4AEE00        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEE00

Fifo14: StartPtr:       00000000        ReadPtr:        00000000

        WritePtrs:      00000000        Fifo_Flag:      00800000

        writeHeaderPtr: 00000000

Fifo15: StartPtr:       0E02CEC0        ReadPtr:        0E02CED0

        WritePtrs:      0E02CED0        Fifo_Flag:      84800020

        writeHeaderPtr: 0E02CED0

===========================================================

Complete Board Id:0x00B2

===========================================================

I think your ACL issue is a red herring.  You're not the only one seeing the 30% CPU and higher.

Now the part that really drives me nuts is I have a stack that's only at 14%.  Check this out:

CPU utilization for five seconds: 21%/0%; one minute: 14%; five minutes: 14%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

162         302        77       3922  4.15%  0.33%  0.06%   1 SSH Process     

  74   243161037  39366660       6176  3.03%  2.85%  2.84%   0 RedEarth Tx Mana

   4    96582240   4319109      22361  2.07%  1.40%  1.16%   0 Check heaps     

156   118927698 191802387        620  1.91%  1.35%  1.35%   0 Hulc LED Process

116   143864159  16784938       8571  1.75%  1.68%  1.66%   0 hpm counter proc

186     9436800   8137819       1159  0.79%  0.13%  0.11%   0 CDP Protocol    

Its configuration and software version is the same as a stack right next to it that's at 29% CPU.  Go figure.  I'm rolling out more 3750Xs at a new data center next week and it will be interesting to see how they behave.

Hello !

The WS-C3750X-48T-E "12.2(58)SE2 C3750E-UNIVERSALK9-M" Today I met the same question, high cpu usage, but I just ran EIGRP,VLAN, the following information is output:

3750#show proc c

CPU utilization for five seconds: 27%/1%; one minute: 28%; five minutes: 30%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

   1           0          20          0  0.00%  0.00%  0.00%   0 Chunk Manager   

   2           0         669          0  0.00%  0.00%  0.00%   0 Load Meter      

   3           0          28          0  0.00%  0.00%  0.00%   0 DHCPD Timer     

   4        9437         693      13617  0.00%  0.21%  0.23%   0 Check heaps     

   5           0           1          0  0.00%  0.00%  0.00%   0 Pool Manager  

  85       84718       15847       5345  2.40%  2.37%  2.36%   0 RedEarth Tx Mana

107       48443       94138        514  3.20%  2.19%  2.43%   0 HLFM address lea

169      126798       83456       1519 13.27% 10.35%  7.59%   0 Hulc LED Process  "This is highset"

Ross Phillips
Beginner

Hi, we had the same issue on these switches where ther HULC process went through the roof,

This was down to there being unused interfaces on the switches where left in a not connect state.

Shutdown all unused interfaces on the switch and if its like what we experienced it should solve your problem.

Ross

Thank you very much, I will try!

Ross - what version of IOS are you running?  I've already tried shutting down all unused ports and rebooting, but to no avail. 

jawad-mukhtar
Enthusiast

R u using route map if u r then u will also b using acls in that just remove deny statement from route map ace

Sent from Cisco Technical Support Android App

Jawad
insccisco
Beginner

Guys,

What is the bottom line with regards to the software you will put in a 3750X switch? I got 5 cisco 3750X switches and I am about to put them into production (no stack config, just stand-alone) and I'd like to know if I am better safe with the

12.2(55)SE5 version Dave Genton suggested?

My switches came with IOS version (C3750E-UNIVERSALK9-M), Version 15.0(2)SE. I quickly looked this version up and it appears to have few big bugs already plus, I also see the CPU utilization at 22-26% with nothing connected in the ports. I also have couple of the switches with ALL the ports manually SHUT and I still the this CPU %.

Please advise.

Thank you in advance

Leo Laohoo
VIP Community Legend

If all of your 3750X can run 12.2(55)SE8, then use this version as it's considered to be the most stable.

If you need features only found in 15.0 then go with 15.0(2)SE4.  Do not even attempt to use 15.2(1)E.

I just tried following this exact advice (I loaded IOS version c3750e-ipbasek9-mz.150-2.SE4.bin) but the switch went into an infinite loop.

Did I use the wrong image file (instead of c3750e-universalk9-mz.150-2.SE.bin) ??

All my switches are C3750X platform with license level IP Base.

Leo Laohoo
VIP Community Legend

Post the entire sequence.  If the switch crashed, I want to see the error messages.

I wasnt recording the output and it is all gone by now.... I quickly performed a recovery procedure and went back to the version that the switch came with --- IOS version (C3750E-UNIVERSALK9-M), Version 15.0(2)SE

For some reason I am suspecting is the the IP Base version I tried. But before I try the Universal version of that image (c3750e-universalk9-mz.150-2.SE.bin), I wanted to ask you guys for help.

I did read the pre-requisites of the c3750e-ipbasek9-mz.150-2.SE4.bin and it stated that I needed 256/64 in the switch which I do have but for some reason I think I might not be seeing something else that perhaps states that I can't use the ip-base flavor of the image....

What is your take on this?

Leo Laohoo
VIP Community Legend

I'm not totally sure what's going on.

Try to install the IOS on a switch without any config and any network module.