Re: High CPU usage - 3750 X stack

johnramz · ‎06-20-2012

Support Community

We recently configured a stack of four 48 port 3750-x switches . We are noticing high CPU usage. "Hulc LED process" seems pretty high.

This has coincided with VMware servers getting slow and non-responsive at times, perhaps a coincidence, not sure.

Below I provided some outputs that might help to diagnose it

Thanks

John

System image file is "flash:/c3750e-ipbasek9-mz.122-58.SE2/c3750e-ipbasek9-mz.122-58.SE2.bin"

Show inventory output

NAME: "1", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02 ,

NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02L ,

NAME: "2", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02

NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02D ,

NAME: "3", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02

NAME: "Switch 3 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02L ,

NAME: "4", DESCR: "WS-C3750X-48"

PID: WS-C3750X-48T-S , VID: V02

NAME: "Switch 4 - Power Supply 0", DESCR: "FRU Power Supply"

PID: C3KX-PWR-350WAC , VID: V02L ,

SWITCH#sh processes cpu sorted

CPU utilization for five seconds: 61%/5%; one minute: 50%; five minutes: 49%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

168 260466386 44948517 5794 14.53% 13.98% 13.70% 0 Hulc LED Process

231 97586088 27253906 3580 4.95% 4.73% 4.64% 0 Spanning Tree

213 63106121 154928892 407 4.15% 3.89% 3.91% 0 IP Input

284 70113217 34537588 2030 3.51% 3.98% 4.17% 0 RARP Input

4 6663412 421278 15817 3.03% 0.43% 0.32% 0 Check heaps

374 9872291 10805181 913 3.03% 0.77% 0.62% 0 IP SNMP

376 11142951 5370604 2074 3.03% 0.73% 0.66% 0 SNMP ENGINE

12 35389011 32152175 1100 2.87% 2.08% 2.20% 0 ARP Input

128 34962407 3622140 9652 2.07% 1.69% 1.63% 0 hpm counter proc

85 49034286 8536062 5744 1.91% 2.44% 2.44% 0 RedEarth Tx Mana

107 25127806 46459053 540 1.27% 1.10% 0.93% 0 HLFM address lea

174 2412 1714 1407 0.95% 0.39% 0.25% 1 SSH Process

220 6423643 12634764 508 0.79% 0.70% 0.56% 0 ADJ resolve proc

181 6913179 2890070 2392 0.63% 0.31% 0.36% 0 HRPC qos request

375 1681949 5000777 336 0.47% 0.08% 0.07% 0 PDU DISPATCHER

84 10180707 12623537 806 0.47% 0.30% 0.37% 0 RedEarth I2C dri

1

666666096996666666666666659666667666666666666666666766676666666656666666

249363098992351145264823289455360612252332233522344115537230141392553343

100 ** ** *

90 ** ** *

80 ** ** *

70 * * ***** * * * * * ** *** * * * **** **

60 **********************************************************************

50 ######################################################################

40 ######################################################################

30 ######################################################################

20 ######################################################################

10 ######################################################################

0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..

0 5 0 5 0 5 0 5 0 5 0 5 0

CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%

455555555554444444444555554444455555555555555555555444444444

922222111118888866666000009999911111555554444422222444448888

100

90

80

70

60 *****

50 *************************************************** **

40 **********************************************************

30 **********************************************************

20 **********************************************************

10 **********************************************************

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per second (last 60 seconds)

565756555555555555555555555555555556555555555555565555565556

518841757869248569271526666733778330496833777819929379701861

100

90

80 *

70 *

60 **** ******* **** * * ***** *** * *** **** **** **** *

50 ##########################################################

40 ##########################################################

30 ##########################################################

20 ##########################################################

10 ##########################################################

0....5....1....1....2....2....3....3....4....4....5....5....6

0 5 0 5 0 5 0 5 0 5 0

CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%

Jeff Van Houten · ‎06-20-2012

What about the configs?

Sent from Cisco Technical Support iPad App

glen.grant · ‎06-20-2012

Aise from looking for IOS bugs for your version this doc might shed some info you can use.

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

http://www.cisco.com/en/US/partner/products/hw/switches/ps5023/products_tech_note09186a00807213f5.shtml#topic5

johnramz · ‎06-20-2012

Thanks for your replies.

Here is another output and an excerpt from the config:

switch# show platform tcam utilization

CAM Utilization for ASIC# 0 Max Used

Masks/Values Masks/values

Unicast mac addresses: 6364/6364 2029/2029

IPv4 IGMP groups + multicast routes: 1120/1120 1/1

IPv4 unicast directly-connected routes: 6144/6144 1232/1232

IPv4 unicast indirectly-connected routes: 2048/2048 104/104

IPv4 policy based routing aces: 452/452 12/12

IPv4 qos aces: 512/512 21/21

IPv4 security aces: 964/964 956/956

clock timezone CST -6 0

clock summer-time CDT recurring

switch 1 provision ws-c3750x-48

switch 2 provision ws-c3750x-48

switch 3 provision ws-c3750x-48

switch 4 provision ws-c3750x-48

system mtu routing 1500

ip routing

!

ip domain-name domain.com

vtp domain vtpdomain

vtp mode transparent

spanning-tree mode pvst

spanning-tree extend system-id

!

vlan internal allocation policy ascending

!

vlan 2

name Inside

!

vlan 3

name extranet

!

vlan 4

name DMZ

!

vlan 5

name Internet

!

vlan 60,169,210,230-231,240-242,300-301,500-501,503-506,777

!

ip ssh version 2

!

interface Port-channel2

switchport trunk encapsulation dot1q

switchport mode trunk

!

interface FastEthernet0

no ip address

no ip route-cache cef

no ip route-cache

interface Vlan2

description Inside

no ip address

!

interface Vlan3

description Extranet

no ip address

!

interface Vlan4

description DMZ

no ip address

!

interface Vlan5

description Internet

no ip address

!

interface Vlan46

no ip address

!

interface Vlan60

description vLAN

no ip address

!

interface Vlan90

no ip address

!

interface Vlan169

ip address 169.254.254.254 255.255.255.0

ip access-group ip169 in

ip access-group ip169out out

!

interface Vlan210

description Management Network

ip address 10.210.254.254 255.255.0.0

ip access-group ip210 in

ip access-group ip210out out

!

interface Vlan230

description Infrastructure Network

ip address 10.230.254.254 255.255.0.0

ip access-group ip230 in

ip access-group ip230out out

!

interface Vlan231

description Services Network

ip address 10.231.254.254 255.255.0.0

ip access-group ip231 in

ip access-group ip231out out

!

interface Vlan240

description VLAN

ip address 10.240.254.254 255.255.0.0

!

interface Vlan241

ip address 10.241.254.254 255.255.0.0

ip access-group ip241 in

ip access-group ip241out out

!

interface Vlan242

description Test VLAN

ip address 10.242.254.254 255.255.0.0

ip access-group ip242 in

ip access-group ip242out out

!

interface Vlan300

description Isolation Network

ip address 192.168.3.151 255.255.255.0

ip access-group ip300 in

ip access-group ip300out out

!

interface Vlan301

description Customer Network

ip address 192.168.4.254 255.255.255.0

ip access-group ip301 in

ip access-group ip301out out

!

interface Vlan500

description Contractor Network

ip address 172.16.1.254 255.255.255.0

ip access-group ip500 in

ip access-group ip500out out

!

interface Vlan501

description Customer

ip address 172.16.2.254 255.255.255.0

ip access-group ip501 in

ip access-group ip501out out

!

interface Vlan503

description vlan

ip address 172.16.3.254 255.255.255.0

ip access-group ip503 in

!

interface Vlan504

description vlan

ip address 172.16.4.254 255.255.255.0

ip access-group ip504 in

ip access-group ip504out out

!

interface Vlan505

description vlan

ip address 172.16.5.254 255.255.255.0

ip access-group ip505 in

ip access-group ip505out out

!

interface Vlan506

description vlan

ip address 172.16.6.254 255.255.255.0

ip access-group ip506 in

ip access-group ip506out out

!

interface Vlan777

description TestVLAN

ip address 172.16.177.254 255.255.255.0

ip access-group ip777 in

ip access-group ip777out out

!

ip http server

******THERE ARE ABOUT 950 LINES OF ACLs ENTRIES LIKE THE ONE BELOW FOR DIFFERENT VLANS******

ip access-list extended ip501

permit tcp any eq 3389 any

permit tcp any host 10.250.80.13

permit udp any host 10.250.80.13

permit tcp any host 10.250.11.1

permit tcp any host 10.250.11.6

permit udp any host 10.250.11.1

permit udp any host 10.250.11.6

permit tcp any host 10.250.80.21

permit udp any host 10.250.80.21

permit tcp any host 10.250.100.60

permit udp any host 10.250.100.60

permit icmp any host 10.250.100.60

permit udp any host 10.250.72.2

permit tcp any host 10.250.72.2

permit tcp any host 10.250.72.3

permit udp any host 10.250.72.3

permit icmp any host 10.250.11.1

permit icmp any host 10.250.11.6

permit tcp host 172.16.2.252 eq www 192.168.2.0 0.0.0.255

permit icmp host 172.16.2.252 192.168.2.0 0.0.0.255

permit icmp host 172.16.2.251 192.168.2.0 0.0.0.255

permit tcp host 172.16.2.251 192.168.2.0 0.0.0.255

permit tcp any host 10.250.80.25

permit udp any host 10.250.80.25

permit tcp any host 10.250.80.32

permit tcp 172.16.2.0 0.0.0.255 199.7.0.0 0.0.255.255 eq www

permit icmp any 10.210.0.0 0.0.255.255

permit tcp any 10.210.0.0 0.0.255.255

permit udp any 10.210.0.0 0.0.255.255

permit tcp any host 192.168.3.40

ip access-list extended ip501out

deny tcp any eq 3389 any

permit tcp any any

permit icmp any any

permit udp any host 10.250.100.60

permit udp any host 10.250.11.1

permit udp any host 10.250.11.6

permit udp any any

ip sla enable reaction-alerts

logging esm config

logging 10.1.1.32

snmp-server community ERT99

snmp-server community FTR88 RW

!

line con 0

logging synchronous

line vty 0 4

access-class restrict-vty in

exec-timeout 30 0

logging synchronous

login local

transport input ssh

line vty 5 15

access-class restrict-vty in

exec-timeout 30 0

logging synchronous

login local

transport input ssh

!

ntp server 10.1.1.25

end

Jeff Van Houten · ‎06-20-2012

Well I guess we know why you have high CPU now. 1,000 aces looks like a lot.

I would start by making sure each acl is optimized. Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts. Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.

I don't understand the requirements, but I have to believe some of those aces are unnecessary. Why put in 3 allows (tcp, udp, icmp) for all inbound to a specific host? Would private Vlans be a better fit?

Sent from Cisco Technical Support iPad App

nkarpysh · ‎06-20-2012

Hello,

The "Hulc LED" process does following tasks:

- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status

The Catalyst 3750-X switches have a CPU utilization level that is higher than the previous models of the Catalyst 3750 switches. This is normal behavior. One can expect that the overall CPU utilization will be between 25 – 35% with Hulc Led process taking about 15%. Plugging PoE devices into all ports on a 3750-X switch will cause the CPU utilization to increase. Since almost all of the packet forwarding is done in hardware and not by the CPU a CPU utilization of 35 – 45% should not be cause for concern.

In your case that is Higher - but Hulc Led is not the problem as I said above. Seeing this process taking about 15% in production is normal. You need to concentrate to other small processes which together add to performance.

Jeff shared a good idea - your QoS and ACL table are almost full so it is worth to optimize those.

Also the level of interrupts is about 10 percent:

CPU utilization for five seconds: 61%/5% <<<<<<<<<<< 5 % here are the interrupts due to traffic sent to switch itself

213 63106121 154928892 407 4.15% 3.89% 3.91% 0 IP Input <<<<<< IP Input is also due to interrupts

So 10% of CPU is taken by traffic coming to the switch. You can use the tools below to understnad what packets are sent to CPU and locate the sources doing it and stop eventually:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp1000983

Let us know if it helps.

*Please also rate replies if were helpful.

Nik

HTH,
Niko

Darren Ramsey · ‎06-20-2012

We noticed increased CPU when 3750X were stacked with 3750V2. High hulc CPU is a known issue.

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtn42790

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCti61764

johnramz · ‎06-21-2012

Thanks to all for your replies.

Jeff

I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.

Nikolay,

I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.

This switch also serves as a gateway(L3 role) for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?

Thanks

Johnny

show controllers cpu-interface

ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain

-------------------------------------------------------------------------

ASIC0 0 0 0 0 0 0

ASIC1 0 0 0 0 0 0

ASIC2 0 0 0 0 0 0

HOL Fix Counts

--------------

No Fixes: 0 Added: 0 In Use: 0 Both: 0

CPU Heartbeat Statistics

Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax

---------- ---------- ---------- ---------- ---------- ----------

37139562 0 0 0 0 1

Rx Delay

0 1 2 3 4

---------- ---------- ---------- ---------- ----------

37139562 0 0 0 0

AddlDelay AdvanceCnt

---------- ----------

0 0

Rx Retries by RetryCount

0 1 2 3 4 5 6

---------- ---------- ---------- ---------- ---------- ---------- ----------

37139562 0 0 0 0 0 0

7 8 9

---------- ---------- ----------

0 0 0

AddlRetry

----------

0

cpu-queue-frames retrieved dropped invalid hol-block stray

----------------- ---------- ---------- ---------- ---------- ----------

rpc 104077409 0 0 0 0

stp 19189469 0 0 0 0

ipc 11093838 0 0 0 0

routing protocol 141021559 0 0 0 0

L2 protocol 230347 0 0 0 0

remote console 17 0 0 0 0

sw forwarding 257436702 0 0 0 0

host 21146276 0 0 0 0

broadcast 332154608 0 0 0 0

cbt-to-spt 0 0 0 0 0

igmp snooping 2796987 0 0 0 0

icmp 90752156 0 0 0 0

logging 0 0 0 0 0

rpf-fail 0 0 0 0 0

dstats 0 0 0 0 0

cpu heartbeat 37139562 0 0 0 0

cpu-queue static inuse static added

----------------- ------------ ------------

rpc 0 0

stp 0 0

ipc 0 0

routing protocol 0 0

L2 protocol 0 0

remote console 0 0

sw forwarding 0 0

host 0 0

broadcast 0 0

cbt-to-spt 0 0

igmp snooping 0 0

icmp 0 0

logging 0 0

rpf-fail 0 0

dstats 0 0

cpu heartbeat 0 0

Supervisor ASIC receive-queue parameters

----------------------------------------

queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8

queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54

queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C

queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC

queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0

queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80

queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4

queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784

queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574

queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C

queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8

queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4

queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8

queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C

queue E maxrecevsize 0 pakhead 0 paktail 0

queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC

Supervisor ASIC exception status

--------------------------------

Receive overrun 00000000 Transmit overrun 00000000

FrameSignatureErr 00000000 MicInitialize 00000002

BadFrameErr 00000000 LenExceededErr 00000000

BadJumboSegments 00000000

Supervisor ASIC Mic Registers

------------------------------

MicDirectPollInfo 80000200

MicIndicationsReceived 00000000

MicInterruptsReceived 00000009

MicPcsInfo 0000001F

MicPlbMasterConfiguration 00000000

MicRxFifosAvailable 00000000

MicRxFifosReady 0000BFFF

MicTimeOutPeriod: FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000

MicTransmFramesCopied 00000003

MicTxFifosAvailable 0000000E

MicConfiguration: Conf flag: 00000110 Interrupt Flag: 00000008

MicReceiveFifoAssignmen Queue 0 - 7: 33333333 Queue 8 - 15:33333333

MicReceiveFramesReady: FrameAvailable: 00000181 frameAvaiMask: 00000000

MicException:

Exception_flag 00000000

Message-1 00000000

Message-2 00000000

Message-3 00000000

MicIntRxFifo:

ReadPtr 000005C0 WritePtr 000005C0

WHeadPtr 000005C0 TxFifoDepth C0000800

MicIntTxFifo:

ReadPtr 00000728 WritePtr 00000728

WHeadPtr 00000728 TxFifoDepth C0000800

MicDecodeInfo:

Fifo0: address: 03FF4000 asic_num: 00000100

Fifo1: address: 03FF4400 asic_num: 00000101

MicTransmitFifoInfo:

Fifo0: StartPtrs: 0E2CE800 ReadPtr: 0E2CEBE8

WritePtrs: 0E2CEBE8 Fifo_Flag: 8A800800

Weights: 001E001E

Fifo1: StartPtrs: 0E02D000 ReadPtr: 0E02D138

WritePtrs: 0E02D138 Fifo_Flag: 89800400

Weights: 000A000A

MicReceiveFifoInfo:

Fifo0: StartPtr: 0E4AF000 ReadPtr: 0E4AF2A8

WritePtrs: 0E4AF308 Fifo_Flag: 8B000FA0

writeHeaderPtr: 0E4AF308

Fifo1: StartPtr: 0E78C000 ReadPtr: 0E78C2E8

WritePtrs: 0E78C2E8 Fifo_Flag: 89800400

writeHeaderPtr: 0E78C2E8

Fifo2: StartPtr: 0E744800 ReadPtr: 0E744A70

WritePtrs: 0E744A70 Fifo_Flag: 89800400

writeHeaderPtr: 0E744A70

Fifo3: StartPtr: 0EBD1000 ReadPtr: 0EBD13B8

WritePtrs: 0EBD13B8 Fifo_Flag: 89800400

writeHeaderPtr: 0EBD13B8

Fifo4: StartPtr: 0E7D3800 ReadPtr: 0E7D3A58

WritePtrs: 0E7D3A58 Fifo_Flag: 89800400

writeHeaderPtr: 0E7D3A58

Fifo5: StartPtr: 0EB40600 ReadPtr: 0EB40688

WritePtrs: 0EB40688 Fifo_Flag: 88800200

writeHeaderPtr: 0EB40688

Fifo6: StartPtr: 0EB87400 ReadPtr: 0EB874F0

WritePtrs: 0EB874F0 Fifo_Flag: 89800400

writeHeaderPtr: 0EB874F0

Fifo7: StartPtr: 0E880000 ReadPtr: 0E880E20

WritePtrs: 0E881520 Fifo_Flag: 8C001900

writeHeaderPtr: 0E881520

Fifo8: StartPtr: 0EB1A600 ReadPtr: 0EB1A770

WritePtrs: 0EB1A780 Fifo_Flag: 880001F0

writeHeaderPtr: 0EB1A780

Fifo9: StartPtr: 0E2E0CD8 ReadPtr: 0E2E0CD8

WritePtrs: 0E2E0CD8 Fifo_Flag: 82800008

writeHeaderPtr: 0E2E0CD8

Fifo10: StartPtr: 0E81D000 ReadPtr: 0E81D1D8

WritePtrs: 0E81D1D8 Fifo_Flag: 88800200

writeHeaderPtr: 0E81D1D8

Fifo11: StartPtr: 0E4AEF00 ReadPtr: 0E4AEF60

WritePtrs: 0E4AEF60 Fifo_Flag: 86800080

writeHeaderPtr: 0E4AEF60

Fifo12: StartPtr: 0E84A000 ReadPtr: 0E84A300

WritePtrs: 0E84A000 Fifo_Flag: 89000100

writeHeaderPtr: 0E84A000

Fifo13: StartPtr: 0E4AEE00 ReadPtr: 0E4AEE00

WritePtrs: 0E4AEE00 Fifo_Flag: 86800080

writeHeaderPtr: 0E4AEE00

Fifo14: StartPtr: 00000000 ReadPtr: 00000000

WritePtrs: 00000000 Fifo_Flag: 00800000

writeHeaderPtr: 00000000

Fifo15: StartPtr: 0E02CEC0 ReadPtr: 0E02CED0

WritePtrs: 0E02CED0 Fifo_Flag: 84800020

writeHeaderPtr: 0E02CED0

===========================================================

Complete Board Id:0x00B2

===========================================================

Joseph W. Doherty · ‎07-08-2013

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I would start by making sure each acl is optimized.  Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts.  Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.

Platforms like the 3750-X should be processing ACLs in hardware, so the "software" processed ACL optimizations don't normally apply. However, 3750 series have limited TCAM resources and it's possible to overflow the TCAM allocation for them when you have lots of ACLs. When you do, CPU will rise and performance will slow. TCAM resources can be somewhat "tuned" by your selection of the SDM template.

Leo Laohoo · ‎06-20-2012

Unless you really, really, really have a need to run 12.2(58)SE2, I'd recommend you go down to 12.2(55)SE5.

Starting next week, I'm going to be testing the 15.0(1)SE3 IOS.

johnnylingo · ‎05-25-2013

Tried 12.2(55)SE7, 12.2(58)SE2, and 15.0(2)SE2 with universal image, and all gave 30-40% CPU, even when all nonconnected ports are disabled. This is just on a single WS-C3750X-48T-S V02, no routing, no ACLs, Rapid-PVST with 2 VLANs.

Interesting thing is I have a pair of WS-C3750X-48T-S, stacked, running 12.2(55)SE3. They are at 15% CPU. I've gone through the configs and can't spot any difference. This is a real mystery.

Leo Laohoo · ‎05-25-2013

Thanks Johnny.

Just found a small bug in 3750E/X running 15.0(2)SE2. CPU didn't spike, but boy did it went nuts! I had tracebacks galore and the response to the switch was slow. In the end, it all boils down to the usual culprit: SNMP traps.

I removed ALL SNMP traps and the tracebacks stops and I was able to gain control of the switch.

Using 12.2(55)SE7 in all my 3750/G/E/X fleet. No issue and CPU is not that high either. Wierd.

johnnylingo · ‎05-25-2013

Yeah I've been running 12.2(55)SE7 on the 3560s with great success. Even with PoE and ports bouncing, cpu never gets above 10% and SSH/SNMP never hang. I avoid SNMP traps at all costs and rely only on syslogs.

There is definitely something "unique" to the 3750X and 2960S platforms pertaining to the Hulc LED Process. Just wish there was more work being done on Cisco's side to investigate.

DAVE GENTON · ‎07-08-2013

Glad to see this post in the thread. I just flew over a thousand miles to help a consistent long time customer whom is "all of a sudden" having issues with performance. 3750X devices with universal images running at 40% CPU utilization and higher with NOTHING plugged in or configured. I erased config to kill it and have vlan 1 to tftp my preferred 12.2(55)SE5 "safe harbor" image and I cannot even get it on there, well I can but cannot surpass 4 packets per second !! I've never seen anything like this in 21 years of being a Cisco Engineer.

No clue what triggered it but no seen enough today to push my "safe" release version to all devices being left behind after I forklift network and upgrade with Nexus 6009's and more 3750X's just orderd.

Please post if you find results, I'm taking last device (the bad one mentioned above) down to 12.2(55)SE5 and buring all with universal images tonight.

Dave

Leo Laohoo · ‎07-26-2013

Sorry, Dave. I just saw this.

In regards to 3560- and 3750-series, I have made the following conclusion:

1. If you are running Layer 2 and no 802.1X, you can choose either 12.2(55)SE8 or 15.0(2)SE4; and

2. If you are running Layer 3 and/or 802.1X, then use 12.2(55)SE8

In regards to 2960-series switches (2960/G/S), I am currently running 15.0(2)SE4 and I don't see any problems. My 2960-series switches have PoE and 802.1X but very limited SNMP traps enabled.

If anyone reading this is experiencing high or abnormal CPU on 2960-series (2960/G/S) and running 15.0(2)SE4, then check your SNMP enabled traps. DO NOT enable ALL SNMP traps.