cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

48179
Views
20
Helpful
37
Replies
Highlighted
Beginner

High CPU usage - 3750 X stack

Support Community

We recently configured a stack of four 48 port 3750-x switches . We are noticing high CPU usage. "Hulc LED process" seems pretty high.

This has coincided with VMware servers getting slow and non-responsive at times, perhaps a coincidence, not sure.

Below I provided some outputs that might help to diagnose it

Thanks

John

System image file is "flash:/c3750e-ipbasek9-mz.122-58.SE2/c3750e-ipbasek9-mz.122-58.SE2.bin"

Show inventory output

NAME: "1", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02  ,

NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

NAME: "2", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02D ,

NAME: "3", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 3 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

NAME: "4", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S   , VID: V02 

NAME: "Switch 4 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC   , VID: V02L ,

SWITCH#sh processes cpu sorted

CPU utilization for five seconds: 61%/5%; one minute: 50%; five minutes: 49%

PID Runtime(ms)     Invoked      uSecs   5Sec   1Min   5Min TTY Process

168   260466386    44948517       5794 14.53% 13.98% 13.70%   0 Hulc LED Process

231    97586088    27253906       3580  4.95%  4.73%  4.64%   0 Spanning Tree

213    63106121   154928892        407  4.15%  3.89%  3.91%   0 IP Input

284    70113217    34537588       2030  3.51%  3.98%  4.17%   0 RARP Input

   4     6663412      421278      15817  3.03%  0.43%  0.32%   0 Check heaps

374     9872291    10805181        913  3.03%  0.77%  0.62%   0 IP SNMP

376    11142951     5370604       2074  3.03%  0.73%  0.66%   0 SNMP ENGINE

  12    35389011    32152175       1100  2.87%  2.08%  2.20%   0 ARP Input

128    34962407     3622140       9652  2.07%  1.69%  1.63%   0 hpm counter proc

  85    49034286     8536062       5744  1.91%  2.44%  2.44%   0 RedEarth Tx Mana

107    25127806    46459053        540  1.27%  1.10%  0.93%   0 HLFM address lea

174        2412        1714       1407  0.95%  0.39%  0.25%   1 SSH Process

220     6423643    12634764        508  0.79%  0.70%  0.56%   0 ADJ resolve proc

181     6913179     2890070       2392  0.63%  0.31%  0.36%   0 HRPC qos request

375     1681949     5000777        336  0.47%  0.08%  0.07%   0 PDU DISPATCHER

  84    10180707    12623537        806  0.47%  0.30%  0.37%   0 RedEarth I2C dri

        1

      666666096996666666666666659666667666666666666666666766676666666656666666

      249363098992351145264823289455360612252332233522344115537230141392553343

  100       ** **               *

   90       ** **               *

   80       ** **               *

   70   * * *****  *   * * *    * ** ***   *       *     * ****         **

   60 **********************************************************************

   50 ######################################################################

   40 ######################################################################

   30 ######################################################################

   20 ######################################################################

   10 ######################################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

               0    5    0    5    0    5    0    5    0    5    0    5    0

                   CPU% per hour (last 72 hours)

                  * = maximum CPU%   # = average CPU%

      455555555554444444444555554444455555555555555555555444444444

      922222111118888866666000009999911111555554444422222444448888

  100

   90

   80

   70

   60                                     *****

   50 ***************************************************     **

   40 **********************************************************

   30 **********************************************************

   20 **********************************************************

   10 **********************************************************

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per second (last 60 seconds)

      565756555555555555555555555555555556555555555555565555565556

      518841757869248569271526666733778330496833777819929379701861

  100

   90

   80    *

   70    *

   60 **** *******  **** * * *****  ***  * ***  **** **** **** *

   50 ##########################################################

   40 ##########################################################

   30 ##########################################################

   20 ##########################################################

   10 ##########################################################

     0....5....1....1....2....2....3....3....4....4....5....5....6

               0    5    0    5    0    5    0    5    0    5    0

               CPU% per minute (last 60 minutes)

              * = maximum CPU%   # = average CPU%

37 REPLIES 37
Contributor

Re: High CPU usage - 3750 X stack

What about the configs?

Sent from Cisco Technical Support iPad App

Advisor

Re: High CPU usage - 3750 X stack

Beginner

High CPU usage - 3750 X stack

Thanks for your replies.

Here is another output and an excerpt from the config:

switch# show platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used

                                             Masks/Values    Masks/values

Unicast mac addresses:                       6364/6364       2029/2029

IPv4 IGMP groups + multicast routes:         1120/1120          1/1

IPv4 unicast directly-connected routes:      6144/6144       1232/1232

IPv4 unicast indirectly-connected routes:    2048/2048        104/104

IPv4 policy based routing aces:               452/452          12/12

IPv4 qos aces:                                512/512          21/21

IPv4 security aces:                           964/964         956/956

clock timezone CST -6 0

clock summer-time CDT recurring

switch 1 provision ws-c3750x-48

switch 2 provision ws-c3750x-48

switch 3 provision ws-c3750x-48

switch 4 provision ws-c3750x-48

system mtu routing 1500

ip routing

!

!

ip domain-name domain.com

vtp domain vtpdomain

vtp mode transparent

spanning-tree mode pvst

spanning-tree extend system-id

!

!

!

!

vlan internal allocation policy ascending

!

vlan 2

name Inside

!

vlan 3

name extranet

!

vlan 4

name DMZ

!

vlan 5

name Internet

!

vlan 60,169,210,230-231,240-242,300-301,500-501,503-506,777

!

ip ssh version 2

!

!

!

!

!

interface Port-channel2

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface FastEthernet0

no ip address

no ip route-cache cef

no ip route-cache

interface Vlan2

description Inside

no ip address

!

interface Vlan3

description Extranet

no ip address

!

interface Vlan4

description DMZ

no ip address

!

interface Vlan5

description Internet

no ip address

!

interface Vlan46

no ip address

!

interface Vlan60

description vLAN

no ip address

!

interface Vlan90

no ip address

!

interface Vlan169

ip address 169.254.254.254 255.255.255.0

ip access-group ip169 in

ip access-group ip169out out

!

interface Vlan210

description Management Network

ip address 10.210.254.254 255.255.0.0

ip access-group ip210 in

ip access-group ip210out out

!

interface Vlan230

description Infrastructure Network

ip address 10.230.254.254 255.255.0.0

ip access-group ip230 in

ip access-group ip230out out

!

interface Vlan231

description Services Network

ip address 10.231.254.254 255.255.0.0

ip access-group ip231 in

ip access-group ip231out out

!

interface Vlan240

description VLAN

ip address 10.240.254.254 255.255.0.0

!

interface Vlan241

ip address 10.241.254.254 255.255.0.0

ip access-group ip241 in

ip access-group ip241out out

!

interface Vlan242

description Test VLAN

ip address 10.242.254.254 255.255.0.0

ip access-group ip242 in

ip access-group ip242out out

!

interface Vlan300

description Isolation Network

ip address 192.168.3.151 255.255.255.0

ip access-group ip300 in

ip access-group ip300out out

!

interface Vlan301

description Customer Network

ip address 192.168.4.254 255.255.255.0

ip access-group ip301 in

ip access-group ip301out out

!

interface Vlan500

description Contractor Network

ip address 172.16.1.254 255.255.255.0

ip access-group ip500 in

ip access-group ip500out out

!

interface Vlan501

description Customer

ip address 172.16.2.254 255.255.255.0

ip access-group ip501 in

ip access-group ip501out out

!

interface Vlan503

description vlan

ip address 172.16.3.254 255.255.255.0

ip access-group ip503 in

!

interface Vlan504

description vlan

ip address 172.16.4.254 255.255.255.0

ip access-group ip504 in

ip access-group ip504out out

!

interface Vlan505

description vlan

ip address 172.16.5.254 255.255.255.0

ip access-group ip505 in

ip access-group ip505out out

!

interface Vlan506

description vlan

ip address 172.16.6.254 255.255.255.0

ip access-group ip506 in

ip access-group ip506out out

!

interface Vlan777

description TestVLAN

ip address 172.16.177.254 255.255.255.0

ip access-group ip777 in

ip access-group ip777out out

!

ip http server

******THERE ARE ABOUT 950 LINES OF ACLs ENTRIES LIKE THE ONE BELOW FOR DIFFERENT VLANS******

ip access-list extended ip501

permit tcp any eq 3389 any

permit tcp any host 10.250.80.13

permit udp any host 10.250.80.13

permit tcp any host 10.250.11.1

permit tcp any host 10.250.11.6

permit udp any host 10.250.11.1

permit udp any host 10.250.11.6

permit tcp any host 10.250.80.21

permit udp any host 10.250.80.21

permit tcp any host 10.250.100.60

permit udp any host 10.250.100.60

permit icmp any host 10.250.100.60

permit udp any host 10.250.72.2

permit tcp any host 10.250.72.2

permit tcp any host 10.250.72.3

permit udp any host 10.250.72.3

permit icmp any host 10.250.11.1

permit icmp any host 10.250.11.6

permit tcp host 172.16.2.252 eq www 192.168.2.0 0.0.0.255

permit icmp host 172.16.2.252 192.168.2.0 0.0.0.255

permit icmp host 172.16.2.251 192.168.2.0 0.0.0.255

permit tcp host 172.16.2.251 192.168.2.0 0.0.0.255

permit tcp any host 10.250.80.25

permit udp any host 10.250.80.25

permit tcp any host 10.250.80.32

permit tcp 172.16.2.0 0.0.0.255 199.7.0.0 0.0.255.255 eq www

permit icmp any 10.210.0.0 0.0.255.255

permit tcp any 10.210.0.0 0.0.255.255

permit udp any 10.210.0.0 0.0.255.255

permit tcp any host 192.168.3.40

ip access-list extended ip501out

deny   tcp any eq 3389 any

permit tcp any any

permit icmp any any

permit udp any host 10.250.100.60

permit udp any host 10.250.11.1

permit udp any host 10.250.11.6

permit udp any any

ip sla enable reaction-alerts

logging esm config

logging 10.1.1.32

snmp-server community ERT99

snmp-server community FTR88 RW

!

!

line con 0

logging synchronous

line vty 0 4

access-class restrict-vty in

exec-timeout 30 0

logging synchronous

login local

transport input ssh

line vty 5 15

access-class restrict-vty in

exec-timeout 30 0

logging synchronous

login local

transport input ssh

!

ntp server 10.1.1.25

end

Contributor

Re: High CPU usage - 3750 X stack

Well I guess we know why you have high CPU now. 1,000 aces looks like a lot.

I would start by making sure each acl is optimized. Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts. Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.

I don't understand the requirements, but I have to believe some of those aces are unnecessary. Why put in 3 allows (tcp, udp, icmp) for all inbound to a specific host? Would private Vlans be a better fit?

Sent from Cisco Technical Support iPad App

Cisco Employee

Re: High CPU usage - 3750 X stack

Hello,

The "Hulc LED" process does following tasks:

- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status

The Catalyst 3750-X switches have a CPU utilization level that is higher than the previous models of the Catalyst 3750 switches. This is normal behavior. One can expect that the overall CPU utilization will be between 25 – 35% with Hulc Led process taking about 15%. Plugging PoE devices into all ports on a 3750-X switch will cause the CPU utilization to increase. Since almost all of the packet forwarding is done in hardware and not by the CPU a CPU utilization of 35 – 45% should not be cause for concern.

In your case that is Higher - but Hulc Led is not the problem as I said above. Seeing this process taking about 15% in production is normal. You need to concentrate to other small processes which together add to performance.

Jeff shared a good idea - your QoS and ACL table are almost full so it is worth to optimize those.

Also the level of interrupts is about 10 percent:

CPU utilization for five seconds: 61%/5%  <<<<<<<<<<< 5 % here are the interrupts due to traffic sent to switch itself

213    63106121   154928892        407  4.15%  3.89%  3.91%   0 IP Input <<<<<< IP Input is also due to interrupts

So 10% of CPU is taken by traffic coming to the switch. You can use the tools below to understnad what packets are sent to CPU and locate the sources doing it and stop eventually:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1000983

Let us know if it helps.

*Please also rate replies if were helpful.

Nik

HTH,
Niko
Enthusiast

Re: High CPU usage - 3750 X stack

Beginner

High CPU usage - 3750 X stack

Thanks to all for your replies.

Jeff

I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.

Nikolay,

I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.

This switch also serves as a gateway(L3 role)  for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?

Thanks

Johnny

show controllers cpu-interface

ASIC    Rxbiterr   Rxunder    Fwdctfix   Txbuflos   Rxbufloc   Rxbufdrain

-------------------------------------------------------------------------

ASIC0     0          0          0          0          0          0

ASIC1     0          0          0          0          0          0

ASIC2     0          0          0          0          0          0

HOL Fix Counts

--------------

No Fixes:          0 Added:          0 In Use:          0 Both:          0

CPU Heartbeat Statistics

Tx Success Tx Fail    1st Thr    2nd Thr    Unthr      RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          1

Rx Delay

         0          1          2          3          4

---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0

AddlDelay AdvanceCnt

---------- ----------

         0          0

Rx Retries by RetryCount

         0          1          2          3          4          5          6

---------- ---------- ---------- ---------- ---------- ---------- ----------

  37139562          0          0          0          0          0          0

         7          8          9

---------- ---------- ----------

         0          0          0

AddlRetry

----------

         0

cpu-queue-frames  retrieved  dropped    invalid    hol-block  stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc               104077409  0          0          0          0

stp               19189469   0          0          0          0

ipc               11093838   0          0          0          0

routing protocol  141021559  0          0          0          0

L2 protocol       230347     0          0          0          0

remote console    17         0          0          0          0

sw forwarding     257436702  0          0          0          0

host              21146276   0          0          0          0

broadcast         332154608  0          0          0          0

cbt-to-spt        0          0          0          0          0

igmp snooping     2796987    0          0          0          0

icmp              90752156   0          0          0          0

logging           0          0          0          0          0

rpf-fail          0          0          0          0          0

dstats            0          0          0          0          0

cpu heartbeat     37139562   0          0          0          0

cpu-queue         static inuse static added

----------------- ------------ ------------

rpc               0            0

stp               0            0

ipc               0            0

routing protocol  0            0

L2 protocol       0            0

remote console    0            0

sw forwarding     0            0

host              0            0

broadcast         0            0

cbt-to-spt        0            0

igmp snooping     0            0

icmp              0            0

logging           0            0

rpf-fail          0            0

dstats            0            0

cpu heartbeat     0            0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8

queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54

queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C

queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC

queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0

queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80

queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4

queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784

queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574

queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C

queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8

queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4

queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8

queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC

Supervisor ASIC exception status

--------------------------------

Receive overrun    00000000   Transmit overrun 00000000

FrameSignatureErr  00000000   MicInitialize    00000002

BadFrameErr        00000000   LenExceededErr   00000000

BadJumboSegments   00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo               80000200

MicIndicationsReceived          00000000

MicInterruptsReceived           00000009

MicPcsInfo                      0000001F

MicPlbMasterConfiguration       00000000

MicRxFifosAvailable             00000000

MicRxFifosReady                 0000BFFF

MicTimeOutPeriod:       FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied           00000003

MicTxFifosAvailable             0000000E

MicConfiguration:       Conf flag: 00000110     Interrupt Flag: 00000008

MicReceiveFifoAssignmen Queue 0 - 7: 33333333   Queue 8 - 15:33333333

MicReceiveFramesReady:  FrameAvailable: 00000181        frameAvaiMask: 00000000

MicException:

        Exception_flag  00000000

        Message-1       00000000

        Message-2       00000000

        Message-3       00000000

MicIntRxFifo:

        ReadPtr         000005C0        WritePtr        000005C0

        WHeadPtr        000005C0        TxFifoDepth     C0000800

MicIntTxFifo:

        ReadPtr         00000728        WritePtr        00000728

        WHeadPtr        00000728        TxFifoDepth     C0000800

MicDecodeInfo:

Fifo0:  address:        03FF4000 asic_num:      00000100

Fifo1:  address:        03FF4400 asic_num:      00000101

MicTransmitFifoInfo:

Fifo0:   StartPtrs:     0E2CE800        ReadPtr:        0E2CEBE8

        WritePtrs:      0E2CEBE8        Fifo_Flag:      8A800800

        Weights:        001E001E

Fifo1:   StartPtrs:     0E02D000        ReadPtr:        0E02D138

        WritePtrs:      0E02D138        Fifo_Flag:      89800400

        Weights:        000A000A

MicReceiveFifoInfo:

Fifo0:  StartPtr:       0E4AF000        ReadPtr:        0E4AF2A8

        WritePtrs:      0E4AF308        Fifo_Flag:      8B000FA0

        writeHeaderPtr: 0E4AF308

Fifo1:  StartPtr:       0E78C000        ReadPtr:        0E78C2E8

        WritePtrs:      0E78C2E8        Fifo_Flag:      89800400

        writeHeaderPtr: 0E78C2E8

Fifo2:  StartPtr:       0E744800        ReadPtr:        0E744A70

        WritePtrs:      0E744A70        Fifo_Flag:      89800400

        writeHeaderPtr: 0E744A70

Fifo3:  StartPtr:       0EBD1000        ReadPtr:        0EBD13B8

        WritePtrs:      0EBD13B8        Fifo_Flag:      89800400

        writeHeaderPtr: 0EBD13B8

Fifo4:  StartPtr:       0E7D3800        ReadPtr:        0E7D3A58

        WritePtrs:      0E7D3A58        Fifo_Flag:      89800400

        writeHeaderPtr: 0E7D3A58

Fifo5:  StartPtr:       0EB40600        ReadPtr:        0EB40688

        WritePtrs:      0EB40688        Fifo_Flag:      88800200

        writeHeaderPtr: 0EB40688

Fifo6:  StartPtr:       0EB87400        ReadPtr:        0EB874F0

        WritePtrs:      0EB874F0        Fifo_Flag:      89800400

        writeHeaderPtr: 0EB874F0

Fifo7:  StartPtr:       0E880000        ReadPtr:        0E880E20

        WritePtrs:      0E881520        Fifo_Flag:      8C001900

        writeHeaderPtr: 0E881520

Fifo8:  StartPtr:       0EB1A600        ReadPtr:        0EB1A770

        WritePtrs:      0EB1A780        Fifo_Flag:      880001F0

        writeHeaderPtr: 0EB1A780

Fifo9:  StartPtr:       0E2E0CD8        ReadPtr:        0E2E0CD8

        WritePtrs:      0E2E0CD8        Fifo_Flag:      82800008

        writeHeaderPtr: 0E2E0CD8

Fifo10: StartPtr:       0E81D000        ReadPtr:        0E81D1D8

        WritePtrs:      0E81D1D8        Fifo_Flag:      88800200

        writeHeaderPtr: 0E81D1D8

Fifo11: StartPtr:       0E4AEF00        ReadPtr:        0E4AEF60

        WritePtrs:      0E4AEF60        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEF60

Fifo12: StartPtr:       0E84A000        ReadPtr:        0E84A300

        WritePtrs:      0E84A000        Fifo_Flag:      89000100

        writeHeaderPtr: 0E84A000

Fifo13: StartPtr:       0E4AEE00        ReadPtr:        0E4AEE00

        WritePtrs:      0E4AEE00        Fifo_Flag:      86800080

        writeHeaderPtr: 0E4AEE00

Fifo14: StartPtr:       00000000        ReadPtr:        00000000

        WritePtrs:      00000000        Fifo_Flag:      00800000

        writeHeaderPtr: 00000000

Fifo15: StartPtr:       0E02CEC0        ReadPtr:        0E02CED0

        WritePtrs:      0E02CED0        Fifo_Flag:      84800020

        writeHeaderPtr: 0E02CED0

===========================================================

Complete Board Id:0x00B2

===========================================================

VIP Expert

Re: High CPU usage - 3750 X stack

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

I would start by making sure each acl is optimized.  Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts.  Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.

Platforms like the 3750-X should be processing ACLs in hardware, so the "software" processed ACL optimizations don't normally apply.  However, 3750 series have limited TCAM resources and it's possible to overflow the TCAM allocation for them when you have lots of ACLs.  When you do, CPU will rise and performance will slow.  TCAM resources can be somewhat "tuned" by your selection of the SDM template.

Hall of Fame Community Legend

High CPU usage - 3750 X stack

Unless you really, really, really have a need to run 12.2(58)SE2, I'd recommend you go down to 12.2(55)SE5. 

Starting next week, I'm going to be testing the 15.0(1)SE3 IOS.

Contributor

Re: High CPU usage - 3750 X stack

Tried 12.2(55)SE7, 12.2(58)SE2, and 15.0(2)SE2 with universal image, and all gave 30-40% CPU, even when all nonconnected ports are disabled.  This is just on a single WS-C3750X-48T-S V02, no routing, no ACLs, Rapid-PVST with 2 VLANs.

Interesting thing is I have a pair of WS-C3750X-48T-S, stacked, running 12.2(55)SE3.  They are at 15% CPU.  I've gone through the configs and can't spot any difference.  This is a real mystery.

Hall of Fame Community Legend

Re: High CPU usage - 3750 X stack

Thanks Johnny.

Just found a small bug in 3750E/X running 15.0(2)SE2.  CPU didn't spike, but boy did it went nuts!  I had tracebacks galore and the response to the switch was slow.  In the end, it all boils down to the usual culprit:  SNMP traps.

I removed ALL SNMP traps and the tracebacks stops and I was able to gain control of the switch.

Using 12.2(55)SE7 in all my 3750/G/E/X fleet.  No issue and CPU is not that high either.  Wierd. 

Contributor

Re: High CPU usage - 3750 X stack

Yeah I've been running 12.2(55)SE7 on the 3560s with great success.  Even with PoE and ports bouncing, cpu never gets above 10% and SSH/SNMP never hang.  I avoid SNMP traps at all costs and rely only on syslogs. 

There is definitely something "unique" to the 3750X and 2960S platforms pertaining to the Hulc LED Process. Just wish there was more work being done on Cisco's side to investigate.  

Explorer

High CPU usage - 3750 X stack

Glad to see this post in the thread.  I just flew over a thousand miles to help a consistent long time customer whom is "all of a sudden" having issues with performance.   3750X devices with universal images running at 40% CPU utilization and higher with NOTHING plugged in or configured.  I erased config to kill it and have vlan 1 to tftp my preferred 12.2(55)SE5 "safe harbor" image and I cannot even get it on there, well I can but cannot surpass 4 packets per second !!  I've never seen anything like this in 21 years of being a Cisco Engineer.  

  No clue what triggered it but no seen enough today to push my "safe" release version to all devices being left behind after I forklift network and upgrade with Nexus 6009's and more 3750X's just orderd.

  Please post if you find results, I'm taking last device (the bad one mentioned above) down to 12.2(55)SE5 and buring all with universal images tonight.

Dave

Hall of Fame Community Legend

Re: High CPU usage - 3750 X stack

Sorry, Dave.  I just saw this.

In regards to 3560- and 3750-series, I have made the following conclusion:

1.  If you are running Layer 2 and no 802.1X, you can choose either 12.2(55)SE8 or 15.0(2)SE4; and

2.  If you are running Layer 3 and/or 802.1X, then use 12.2(55)SE8

In regards to 2960-series switches (2960/G/S), I am currently running 15.0(2)SE4 and I don't see any problems.  My 2960-series switches have PoE and 802.1X but very limited SNMP traps enabled.

If anyone reading this is experiencing high or abnormal CPU on 2960-series (2960/G/S) and running 15.0(2)SE4, then check your SNMP enabled traps.  DO NOT enable ALL SNMP traps.

CreatePlease to create content
Content for Community-Ad
July's Community Spotlight Awards