06-20-2012 03:13 PM - edited 03-07-2019 07:22 AM
Support Community
We recently configured a stack of four 48 port 3750-x switches . We are noticing high CPU usage. "Hulc LED process" seems pretty high.
This has coincided with VMware servers getting slow and non-responsive at times, perhaps a coincidence, not sure.
Below I provided some outputs that might help to diagnose it
Thanks
John
System image file is "flash:/c3750e-ipbasek9-mz.122-58.SE2/c3750e-ipbasek9-mz.122-58.SE2.bin"
Show inventory output
NAME: "1", DESCR: "WS-C3750X-48"
PID: WS-C3750X-48T-S , VID: V02 ,
NAME: "Switch 1 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L ,
NAME: "2", DESCR: "WS-C3750X-48"
PID: WS-C3750X-48T-S , VID: V02
NAME: "Switch 2 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02D ,
NAME: "3", DESCR: "WS-C3750X-48"
PID: WS-C3750X-48T-S , VID: V02
NAME: "Switch 3 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L ,
NAME: "4", DESCR: "WS-C3750X-48"
PID: WS-C3750X-48T-S , VID: V02
NAME: "Switch 4 - Power Supply 0", DESCR: "FRU Power Supply"
PID: C3KX-PWR-350WAC , VID: V02L ,
SWITCH#sh processes cpu sorted
CPU utilization for five seconds: 61%/5%; one minute: 50%; five minutes: 49%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
168 260466386 44948517 5794 14.53% 13.98% 13.70% 0 Hulc LED Process
231 97586088 27253906 3580 4.95% 4.73% 4.64% 0 Spanning Tree
213 63106121 154928892 407 4.15% 3.89% 3.91% 0 IP Input
284 70113217 34537588 2030 3.51% 3.98% 4.17% 0 RARP Input
4 6663412 421278 15817 3.03% 0.43% 0.32% 0 Check heaps
374 9872291 10805181 913 3.03% 0.77% 0.62% 0 IP SNMP
376 11142951 5370604 2074 3.03% 0.73% 0.66% 0 SNMP ENGINE
12 35389011 32152175 1100 2.87% 2.08% 2.20% 0 ARP Input
128 34962407 3622140 9652 2.07% 1.69% 1.63% 0 hpm counter proc
85 49034286 8536062 5744 1.91% 2.44% 2.44% 0 RedEarth Tx Mana
107 25127806 46459053 540 1.27% 1.10% 0.93% 0 HLFM address lea
174 2412 1714 1407 0.95% 0.39% 0.25% 1 SSH Process
220 6423643 12634764 508 0.79% 0.70% 0.56% 0 ADJ resolve proc
181 6913179 2890070 2392 0.63% 0.31% 0.36% 0 HRPC qos request
375 1681949 5000777 336 0.47% 0.08% 0.07% 0 PDU DISPATCHER
84 10180707 12623537 806 0.47% 0.30% 0.37% 0 RedEarth I2C dri
1
666666096996666666666666659666667666666666666666666766676666666656666666
249363098992351145264823289455360612252332233522344115537230141392553343
100 ** ** *
90 ** ** *
80 ** ** *
70 * * ***** * * * * * ** *** * * * **** **
60 **********************************************************************
50 ######################################################################
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
455555555554444444444555554444455555555555555555555444444444
922222111118888866666000009999911111555554444422222444448888
100
90
80
70
60 *****
50 *************************************************** **
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)
565756555555555555555555555555555556555555555555565555565556
518841757869248569271526666733778330496833777819929379701861
100
90
80 *
70 *
60 **** ******* **** * * ***** *** * *** **** **** **** *
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
06-20-2012 03:46 PM
What about the configs?
Sent from Cisco Technical Support iPad App
06-20-2012 05:44 PM
Aise from looking for IOS bugs for your version this doc might shed some info you can use.
http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html
06-20-2012 08:34 PM
Thanks for your replies.
Here is another output and an excerpt from the config:
switch# show platform tcam utilization
CAM Utilization for ASIC# 0 Max Used
Masks/Values Masks/values
Unicast mac addresses: 6364/6364 2029/2029
IPv4 IGMP groups + multicast routes: 1120/1120 1/1
IPv4 unicast directly-connected routes: 6144/6144 1232/1232
IPv4 unicast indirectly-connected routes: 2048/2048 104/104
IPv4 policy based routing aces: 452/452 12/12
IPv4 qos aces: 512/512 21/21
IPv4 security aces: 964/964 956/956
clock timezone CST -6 0
clock summer-time CDT recurring
switch 1 provision ws-c3750x-48
switch 2 provision ws-c3750x-48
switch 3 provision ws-c3750x-48
switch 4 provision ws-c3750x-48
system mtu routing 1500
ip routing
!
!
ip domain-name domain.com
vtp domain vtpdomain
vtp mode transparent
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
vlan 2
name Inside
!
vlan 3
name extranet
!
vlan 4
name DMZ
!
vlan 5
name Internet
!
vlan 60,169,210,230-231,240-242,300-301,500-501,503-506,777
!
ip ssh version 2
!
!
!
!
!
interface Port-channel2
switchport trunk encapsulation dot1q
switchport mode trunk
!
interface FastEthernet0
no ip address
no ip route-cache cef
no ip route-cache
interface Vlan2
description Inside
no ip address
!
interface Vlan3
description Extranet
no ip address
!
interface Vlan4
description DMZ
no ip address
!
interface Vlan5
description Internet
no ip address
!
interface Vlan46
no ip address
!
interface Vlan60
description vLAN
no ip address
!
interface Vlan90
no ip address
!
interface Vlan169
ip address 169.254.254.254 255.255.255.0
ip access-group ip169 in
ip access-group ip169out out
!
interface Vlan210
description Management Network
ip address 10.210.254.254 255.255.0.0
ip access-group ip210 in
ip access-group ip210out out
!
interface Vlan230
description Infrastructure Network
ip address 10.230.254.254 255.255.0.0
ip access-group ip230 in
ip access-group ip230out out
!
interface Vlan231
description Services Network
ip address 10.231.254.254 255.255.0.0
ip access-group ip231 in
ip access-group ip231out out
!
interface Vlan240
description VLAN
ip address 10.240.254.254 255.255.0.0
!
interface Vlan241
ip address 10.241.254.254 255.255.0.0
ip access-group ip241 in
ip access-group ip241out out
!
interface Vlan242
description Test VLAN
ip address 10.242.254.254 255.255.0.0
ip access-group ip242 in
ip access-group ip242out out
!
interface Vlan300
description Isolation Network
ip address 192.168.3.151 255.255.255.0
ip access-group ip300 in
ip access-group ip300out out
!
interface Vlan301
description Customer Network
ip address 192.168.4.254 255.255.255.0
ip access-group ip301 in
ip access-group ip301out out
!
interface Vlan500
description Contractor Network
ip address 172.16.1.254 255.255.255.0
ip access-group ip500 in
ip access-group ip500out out
!
interface Vlan501
description Customer
ip address 172.16.2.254 255.255.255.0
ip access-group ip501 in
ip access-group ip501out out
!
interface Vlan503
description vlan
ip address 172.16.3.254 255.255.255.0
ip access-group ip503 in
!
interface Vlan504
description vlan
ip address 172.16.4.254 255.255.255.0
ip access-group ip504 in
ip access-group ip504out out
!
interface Vlan505
description vlan
ip address 172.16.5.254 255.255.255.0
ip access-group ip505 in
ip access-group ip505out out
!
interface Vlan506
description vlan
ip address 172.16.6.254 255.255.255.0
ip access-group ip506 in
ip access-group ip506out out
!
interface Vlan777
description TestVLAN
ip address 172.16.177.254 255.255.255.0
ip access-group ip777 in
ip access-group ip777out out
!
ip http server
******THERE ARE ABOUT 950 LINES OF ACLs ENTRIES LIKE THE ONE BELOW FOR DIFFERENT VLANS******
ip access-list extended ip501
permit tcp any eq 3389 any
permit tcp any host 10.250.80.13
permit udp any host 10.250.80.13
permit tcp any host 10.250.11.1
permit tcp any host 10.250.11.6
permit udp any host 10.250.11.1
permit udp any host 10.250.11.6
permit tcp any host 10.250.80.21
permit udp any host 10.250.80.21
permit tcp any host 10.250.100.60
permit udp any host 10.250.100.60
permit icmp any host 10.250.100.60
permit udp any host 10.250.72.2
permit tcp any host 10.250.72.2
permit tcp any host 10.250.72.3
permit udp any host 10.250.72.3
permit icmp any host 10.250.11.1
permit icmp any host 10.250.11.6
permit tcp host 172.16.2.252 eq www 192.168.2.0 0.0.0.255
permit icmp host 172.16.2.252 192.168.2.0 0.0.0.255
permit icmp host 172.16.2.251 192.168.2.0 0.0.0.255
permit tcp host 172.16.2.251 192.168.2.0 0.0.0.255
permit tcp any host 10.250.80.25
permit udp any host 10.250.80.25
permit tcp any host 10.250.80.32
permit tcp 172.16.2.0 0.0.0.255 199.7.0.0 0.0.255.255 eq www
permit icmp any 10.210.0.0 0.0.255.255
permit tcp any 10.210.0.0 0.0.255.255
permit udp any 10.210.0.0 0.0.255.255
permit tcp any host 192.168.3.40
ip access-list extended ip501out
deny tcp any eq 3389 any
permit tcp any any
permit icmp any any
permit udp any host 10.250.100.60
permit udp any host 10.250.11.1
permit udp any host 10.250.11.6
permit udp any any
ip sla enable reaction-alerts
logging esm config
logging 10.1.1.32
snmp-server community ERT99
snmp-server community FTR88 RW
!
!
line con 0
logging synchronous
line vty 0 4
access-class restrict-vty in
exec-timeout 30 0
logging synchronous
login local
transport input ssh
line vty 5 15
access-class restrict-vty in
exec-timeout 30 0
logging synchronous
login local
transport input ssh
!
ntp server 10.1.1.25
end
06-20-2012 08:59 PM
Well I guess we know why you have high CPU now. 1,000 aces looks like a lot.
I would start by making sure each acl is optimized. Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts. Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.
I don't understand the requirements, but I have to believe some of those aces are unnecessary. Why put in 3 allows (tcp, udp, icmp) for all inbound to a specific host? Would private Vlans be a better fit?
Sent from Cisco Technical Support iPad App
06-20-2012 09:16 PM
Hello,
The "Hulc LED" process does following tasks:
- Check Link status on every port
- If the switch supports POE, it checks to see if there is a Power Device (PD)
detected
- Check the status of the transceiver
- Update Fan status
- Set Main LED and ports LEDs
- Update both Power Supplies and RPS
- Check on system temperature status
The Catalyst 3750-X switches have a CPU utilization level that is higher than the previous models of the Catalyst 3750 switches. This is normal behavior. One can expect that the overall CPU utilization will be between 25 – 35% with Hulc Led process taking about 15%. Plugging PoE devices into all ports on a 3750-X switch will cause the CPU utilization to increase. Since almost all of the packet forwarding is done in hardware and not by the CPU a CPU utilization of 35 – 45% should not be cause for concern.
In your case that is Higher - but Hulc Led is not the problem as I said above. Seeing this process taking about 15% in production is normal. You need to concentrate to other small processes which together add to performance.
Jeff shared a good idea - your QoS and ACL table are almost full so it is worth to optimize those.
Also the level of interrupts is about 10 percent:
CPU utilization for five seconds: 61%/5% <<<<<<<<<<< 5 % here are the interrupts due to traffic sent to switch itself
213 63106121 154928892 407 4.15% 3.89% 3.91% 0 IP Input <<<<<< IP Input is also due to interrupts
So 10% of CPU is taken by traffic coming to the switch. You can use the tools below to understnad what packets are sent to CPU and locate the sources doing it and stop eventually:
Let us know if it helps.
*Please also rate replies if were helpful.
Nik
06-20-2012 10:08 PM
We noticed increased CPU when 3750X were stacked with 3750V2. High hulc CPU is a known issue.
06-21-2012 08:07 AM
Thanks to all for your replies.
Jeff
I was aware of the many ACLs however we used to have the same ACLs in a previous 3750G stack about 2 weeks ago and we never had this issue. I agree I need to optimize them and do somehing because it is reaching its max before the CPU starts processing them but I am not certain this is what is causing the issue.
Nikolay,
I am trying to understand "interrupts" with the analysis of the outputs I posted. Here is another output deom the link you provided. Please post your thoughts if you can.
This switch also serves as a gateway(L3 role) for many systems. Would it make sense to offload that responsability from this switch and let an actual router do it?
Thanks
Johnny
show controllers cpu-interface
ASIC Rxbiterr Rxunder Fwdctfix Txbuflos Rxbufloc Rxbufdrain
-------------------------------------------------------------------------
ASIC0 0 0 0 0 0 0
ASIC1 0 0 0 0 0 0
ASIC2 0 0 0 0 0 0
HOL Fix Counts
--------------
No Fixes: 0 Added: 0 In Use: 0 Both: 0
CPU Heartbeat Statistics
Tx Success Tx Fail 1st Thr 2nd Thr Unthr RetryCtMax
---------- ---------- ---------- ---------- ---------- ----------
37139562 0 0 0 0 1
Rx Delay
0 1 2 3 4
---------- ---------- ---------- ---------- ----------
37139562 0 0 0 0
AddlDelay AdvanceCnt
---------- ----------
0 0
Rx Retries by RetryCount
0 1 2 3 4 5 6
---------- ---------- ---------- ---------- ---------- ---------- ----------
37139562 0 0 0 0 0 0
7 8 9
---------- ---------- ----------
0 0 0
AddlRetry
----------
0
cpu-queue-frames retrieved dropped invalid hol-block stray
----------------- ---------- ---------- ---------- ---------- ----------
rpc 104077409 0 0 0 0
stp 19189469 0 0 0 0
ipc 11093838 0 0 0 0
routing protocol 141021559 0 0 0 0
L2 protocol 230347 0 0 0 0
remote console 17 0 0 0 0
sw forwarding 257436702 0 0 0 0
host 21146276 0 0 0 0
broadcast 332154608 0 0 0 0
cbt-to-spt 0 0 0 0 0
igmp snooping 2796987 0 0 0 0
icmp 90752156 0 0 0 0
logging 0 0 0 0 0
rpf-fail 0 0 0 0 0
dstats 0 0 0 0 0
cpu heartbeat 37139562 0 0 0 0
cpu-queue static inuse static added
----------------- ------------ ------------
rpc 0 0
stp 0 0
ipc 0 0
routing protocol 0 0
L2 protocol 0 0
remote console 0 0
sw forwarding 0 0
host 0 0
broadcast 0 0
cbt-to-spt 0 0
igmp snooping 0 0
icmp 0 0
logging 0 0
rpf-fail 0 0
dstats 0 0
cpu heartbeat 0 0
Supervisor ASIC receive-queue parameters
----------------------------------------
queue 0 maxrecevsize 7E0 pakhead 5505A88 paktail 54655A8
queue 1 maxrecevsize 7E0 pakhead 5689164 paktail 5687F54
queue 2 maxrecevsize 7E0 pakhead 5547AA4 paktail 554719C
queue 3 maxrecevsize 7E0 pakhead 5DC233C paktail 5DBA4CC
queue 4 maxrecevsize 7E0 pakhead 56A7198 paktail 56A7AA0
queue 5 maxrecevsize 7E0 pakhead 5D61304 paktail 5D72F80
queue 6 maxrecevsize 7E0 pakhead 5D856D4 paktail 5D989E4
queue 7 maxrecevsize 7E0 pakhead 5BDE29C paktail 5BDC784
queue 8 maxrecevsize 7E0 pakhead 5CC00A8 paktail 5CB3574
queue 9 maxrecevsize 7E0 pakhead 59DD86C paktail 59DD86C
queue A maxrecevsize 7E0 pakhead 59BF43C paktail 59C13D8
queue B maxrecevsize 7E0 pakhead 5DD18A0 paktail 5DCE6F4
queue C maxrecevsize 7E0 pakhead 59E9CBC paktail 5A049B8
queue D maxrecevsize 7E0 pakhead 59D8EA0 paktail 59DD25C
queue E maxrecevsize 0 pakhead 0 paktail 0
queue F maxrecevsize 7E0 pakhead 59A7080 paktail 59A6BFC
Supervisor ASIC exception status
--------------------------------
Receive overrun 00000000 Transmit overrun 00000000
FrameSignatureErr 00000000 MicInitialize 00000002
BadFrameErr 00000000 LenExceededErr 00000000
BadJumboSegments 00000000
Supervisor ASIC Mic Registers
------------------------------
MicDirectPollInfo 80000200
MicIndicationsReceived 00000000
MicInterruptsReceived 00000009
MicPcsInfo 0000001F
MicPlbMasterConfiguration 00000000
MicRxFifosAvailable 00000000
MicRxFifosReady 0000BFFF
MicTimeOutPeriod: FrameTOPeriod: 00000EA6 DirectTOPeriod: 00004000
MicTransmFramesCopied 00000003
MicTxFifosAvailable 0000000E
MicConfiguration: Conf flag: 00000110 Interrupt Flag: 00000008
MicReceiveFifoAssignmen Queue 0 - 7: 33333333 Queue 8 - 15:33333333
MicReceiveFramesReady: FrameAvailable: 00000181 frameAvaiMask: 00000000
MicException:
Exception_flag 00000000
Message-1 00000000
Message-2 00000000
Message-3 00000000
MicIntRxFifo:
ReadPtr 000005C0 WritePtr 000005C0
WHeadPtr 000005C0 TxFifoDepth C0000800
MicIntTxFifo:
ReadPtr 00000728 WritePtr 00000728
WHeadPtr 00000728 TxFifoDepth C0000800
MicDecodeInfo:
Fifo0: address: 03FF4000 asic_num: 00000100
Fifo1: address: 03FF4400 asic_num: 00000101
MicTransmitFifoInfo:
Fifo0: StartPtrs: 0E2CE800 ReadPtr: 0E2CEBE8
WritePtrs: 0E2CEBE8 Fifo_Flag: 8A800800
Weights: 001E001E
Fifo1: StartPtrs: 0E02D000 ReadPtr: 0E02D138
WritePtrs: 0E02D138 Fifo_Flag: 89800400
Weights: 000A000A
MicReceiveFifoInfo:
Fifo0: StartPtr: 0E4AF000 ReadPtr: 0E4AF2A8
WritePtrs: 0E4AF308 Fifo_Flag: 8B000FA0
writeHeaderPtr: 0E4AF308
Fifo1: StartPtr: 0E78C000 ReadPtr: 0E78C2E8
WritePtrs: 0E78C2E8 Fifo_Flag: 89800400
writeHeaderPtr: 0E78C2E8
Fifo2: StartPtr: 0E744800 ReadPtr: 0E744A70
WritePtrs: 0E744A70 Fifo_Flag: 89800400
writeHeaderPtr: 0E744A70
Fifo3: StartPtr: 0EBD1000 ReadPtr: 0EBD13B8
WritePtrs: 0EBD13B8 Fifo_Flag: 89800400
writeHeaderPtr: 0EBD13B8
Fifo4: StartPtr: 0E7D3800 ReadPtr: 0E7D3A58
WritePtrs: 0E7D3A58 Fifo_Flag: 89800400
writeHeaderPtr: 0E7D3A58
Fifo5: StartPtr: 0EB40600 ReadPtr: 0EB40688
WritePtrs: 0EB40688 Fifo_Flag: 88800200
writeHeaderPtr: 0EB40688
Fifo6: StartPtr: 0EB87400 ReadPtr: 0EB874F0
WritePtrs: 0EB874F0 Fifo_Flag: 89800400
writeHeaderPtr: 0EB874F0
Fifo7: StartPtr: 0E880000 ReadPtr: 0E880E20
WritePtrs: 0E881520 Fifo_Flag: 8C001900
writeHeaderPtr: 0E881520
Fifo8: StartPtr: 0EB1A600 ReadPtr: 0EB1A770
WritePtrs: 0EB1A780 Fifo_Flag: 880001F0
writeHeaderPtr: 0EB1A780
Fifo9: StartPtr: 0E2E0CD8 ReadPtr: 0E2E0CD8
WritePtrs: 0E2E0CD8 Fifo_Flag: 82800008
writeHeaderPtr: 0E2E0CD8
Fifo10: StartPtr: 0E81D000 ReadPtr: 0E81D1D8
WritePtrs: 0E81D1D8 Fifo_Flag: 88800200
writeHeaderPtr: 0E81D1D8
Fifo11: StartPtr: 0E4AEF00 ReadPtr: 0E4AEF60
WritePtrs: 0E4AEF60 Fifo_Flag: 86800080
writeHeaderPtr: 0E4AEF60
Fifo12: StartPtr: 0E84A000 ReadPtr: 0E84A300
WritePtrs: 0E84A000 Fifo_Flag: 89000100
writeHeaderPtr: 0E84A000
Fifo13: StartPtr: 0E4AEE00 ReadPtr: 0E4AEE00
WritePtrs: 0E4AEE00 Fifo_Flag: 86800080
writeHeaderPtr: 0E4AEE00
Fifo14: StartPtr: 00000000 ReadPtr: 00000000
WritePtrs: 00000000 Fifo_Flag: 00800000
writeHeaderPtr: 00000000
Fifo15: StartPtr: 0E02CEC0 ReadPtr: 0E02CED0
WritePtrs: 0E02CED0 Fifo_Flag: 84800020
writeHeaderPtr: 0E02CED0
===========================================================
Complete Board Id:0x00B2
===========================================================
07-08-2013 04:53 PM
Disclaimer
The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.
Liability Disclaimer
In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.
Posting
I would start by making sure each acl is optimized. Highest used generic entries first (e.g., permit tcp any 172.10.0.0/16) followed by the more specific entries for specific hosts. Acls are processed top down, first match, so having the most frequently used ace first lowers processing requirements.
Platforms like the 3750-X should be processing ACLs in hardware, so the "software" processed ACL optimizations don't normally apply. However, 3750 series have limited TCAM resources and it's possible to overflow the TCAM allocation for them when you have lots of ACLs. When you do, CPU will rise and performance will slow. TCAM resources can be somewhat "tuned" by your selection of the SDM template.
06-20-2012 10:16 PM
Unless you really, really, really have a need to run 12.2(58)SE2, I'd recommend you go down to 12.2(55)SE5.
Starting next week, I'm going to be testing the 15.0(1)SE3 IOS.
05-25-2013 12:08 AM
Tried 12.2(55)SE7, 12.2(58)SE2, and 15.0(2)SE2 with universal image, and all gave 30-40% CPU, even when all nonconnected ports are disabled. This is just on a single WS-C3750X-48T-S V02, no routing, no ACLs, Rapid-PVST with 2 VLANs.
Interesting thing is I have a pair of WS-C3750X-48T-S, stacked, running 12.2(55)SE3. They are at 15% CPU. I've gone through the configs and can't spot any difference. This is a real mystery.
05-25-2013 12:33 AM
Thanks Johnny.
Just found a small bug in 3750E/X running 15.0(2)SE2. CPU didn't spike, but boy did it went nuts! I had tracebacks galore and the response to the switch was slow. In the end, it all boils down to the usual culprit: SNMP traps.
I removed ALL SNMP traps and the tracebacks stops and I was able to gain control of the switch.
Using 12.2(55)SE7 in all my 3750/G/E/X fleet. No issue and CPU is not that high either. Wierd.
05-25-2013 12:49 AM
Yeah I've been running 12.2(55)SE7 on the 3560s with great success. Even with PoE and ports bouncing, cpu never gets above 10% and SSH/SNMP never hang. I avoid SNMP traps at all costs and rely only on syslogs.
There is definitely something "unique" to the 3750X and 2960S platforms pertaining to the Hulc LED Process. Just wish there was more work being done on Cisco's side to investigate.
07-08-2013 04:04 PM
Glad to see this post in the thread. I just flew over a thousand miles to help a consistent long time customer whom is "all of a sudden" having issues with performance. 3750X devices with universal images running at 40% CPU utilization and higher with NOTHING plugged in or configured. I erased config to kill it and have vlan 1 to tftp my preferred 12.2(55)SE5 "safe harbor" image and I cannot even get it on there, well I can but cannot surpass 4 packets per second !! I've never seen anything like this in 21 years of being a Cisco Engineer.
No clue what triggered it but no seen enough today to push my "safe" release version to all devices being left behind after I forklift network and upgrade with Nexus 6009's and more 3750X's just orderd.
Please post if you find results, I'm taking last device (the bad one mentioned above) down to 12.2(55)SE5 and buring all with universal images tonight.
Dave
07-26-2013 07:13 PM
Sorry, Dave. I just saw this.
In regards to 3560- and 3750-series, I have made the following conclusion:
1. If you are running Layer 2 and no 802.1X, you can choose either 12.2(55)SE8 or 15.0(2)SE4; and
2. If you are running Layer 3 and/or 802.1X, then use 12.2(55)SE8
In regards to 2960-series switches (2960/G/S), I am currently running 15.0(2)SE4 and I don't see any problems. My 2960-series switches have PoE and 802.1X but very limited SNMP traps enabled.
If anyone reading this is experiencing high or abnormal CPU on 2960-series (2960/G/S) and running 15.0(2)SE4, then check your SNMP enabled traps. DO NOT enable ALL SNMP traps.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide