Currently, my Cisco 3750x (2 switches stacking) is having very hight CPU. Below are some of the output :
3750-ANA#sh processes cpu sorted | ex 0.00
CPU utilization for five seconds: 55%/30%; one minute: 55%; five minutes: 55%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
156 279337896 40092867 6967 12.15% 11.93% 11.78% 0 Hulc LED Process
74 56876760 9703900 5861 1.91% 2.07% 2.11% 0 RedEarth Tx Mana
230 74520206 1971456 37799 1.75% 1.67% 1.65% 0 PI MATM Aging Pr
116 41350338 4122688 10029 1.27% 1.74% 1.75% 0 hpm counter proc
341 204 154 1324 0.95% 0.19% 0.04% 2 Virtual Exec
217 123972805 87319023 1419 0.63% 0.77% 0.71% 0 Spanning Tree
73 9434496 14768667 638 0.47% 0.67% 0.66% 0 RedEarth I2C dri
167 8388382 419994 19972 0.31% 0.37% 0.39% 0 HQM Stack Proces
168 3327583 1675745 1985 0.31% 0.18% 0.20% 0 HRPC qos request
83 7428418 17580596 422 0.31% 0.07% 0.05% 0 hrpc <- response
34 1073084 2649238 405 0.31% 0.05% 0.04% 0 Net Background
117 1108809 4101450 270 0.15% 0.04% 0.03% 0 HRPC pm-counters
38 8759587 1973863 4437 0.15% 0.28% 0.30% 0 Per-Second Jobs
3750-ANA#
3750-ANA#sh processes cpu history
6666666655555555555555555555555555555555555555555555555555
0003333333333333333333555554444444444444443333322222999922
100
90
80
70
60 ******** ***** ****
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)
1 1 1 1
6666666666766606666660877766799986688686077066696666689966
4553766176008604786430036679519973648503002045395773835555
100 * * ** * * * **
90 * * **** * * * * **
80 * ** ** ***** ** * * * * ***
70 ** *** *** *** *** **********#* ***** #*** * **** ******
60 *****##**###*##**#**#################*#*#*##*#*#*****###**
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%
1 11 111111111 1111111111111111 11 1 111 11 111111 11 1
0999999999999009000000000900000000000000009009090009900990000009009099
0555555456545008000000000900000000000000005009090009900990000009009089
100 ******* *** ******#***************************#***********************
90 *#*##########*****#######################*#######################*****
80 *#############*#*########################*#######################*****
70 *#############*##################################################*****
60 ###################################################################***
50 ####################################################################*#
40 ######################################################################
30 ######################################################################
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%
3750-ANA#
3750-ANA#sh ver
Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 12.2(55)SE3, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2011 by Cisco Systems, Inc.
Compiled Thu 05-May-11 15:40 by prod_rel_team
Image text-base: 0x00003000, data-base: 0x02800000
ROM: Bootstrap program is C3750E boot loader
BOOTLDR: C3750E Boot Loader (C3750X-HBOOT-M) Version 12.2(53r)SE2, RELEASE SOFTWARE (fc1)
3750-ANA uptime is 3 weeks, 3 days, 12 hours, 19 minutes
System returned to ROM by power-on
System restarted at 23:58:00 GMT+7 Sat Oct 15 2011
System image file is "flash:/c3750e-universalk9-mz.122-55.SE3/c3750e-universalk9-mz.122-55.SE3.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export@cisco.com.
License Level: ipbase
License Type: Permanent
Next reload license Level: ipbase
cisco WS-C3750X-48 (PowerPC405) processor (revision A0) with 262144K bytes of memory.
Processor board ID FDO1529K0PC
Last reset from power-on
2 Virtual Ethernet interfaces
1 FastEthernet interface
104 Gigabit Ethernet interfaces
4 Ten Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 44:D3:CA:D9:7E:80
Motherboard assembly number : 73-12552-05
Motherboard serial number : FDO153003TU
Model revision number : A0
Motherboard revision number : B0
Model number : WS-C3750X-48T-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15291XP3
System serial number : FDO1529K0PC
Top Assembly Part Number : 800-31326-02
Top Assembly Revision Number : C0
Version ID : V02
CLEI Code Number : COMJT00ARB
Hardware Board Revision Number : 0x03
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 54 WS-C3750X-48 12.2(55)SE3 C3750E-UNIVERSALK9-M
2 54 WS-C3750X-48 12.2(55)SE3 C3750E-UNIVERSALK9-M
Switch 02
---------
Switch Uptime : 3 weeks, 3 days, 11 hours, 44 minutes
Base ethernet MAC Address : 44:D3:CA:DE:8F:00
Motherboard assembly number : 73-12552-05
Motherboard serial number : FDO153003TZ
Model revision number : A0
Motherboard revision number : B0
Model number : WS-C3750X-48T-S
Daughterboard assembly number : 800-32727-01
Daughterboard serial number : FDO15291XMS
System serial number : FDO1529P0UP
Top assembly part number : 800-31326-02
Top assembly revision number : C0
Version ID : V02
CLEI Code Number : COMJT00ARB
License Level : ipbase
License Type : Permanent
Next reboot licensing Level : ipbase
Configuration register is 0xF
3750-ANA#
3750-ANA#sh ip route summary
IP routing table name is Default-IP-Routing-Table(0)
IP routing table maximum-paths is 32
Route Source Networks Subnets Overhead Memory (bytes)
connected 0 1 64 152
static 1 0 64 152
internal 1 1172
Total 2 1 128 1476
3750-ANA#
3750-ANA#sh spanning-tree summary
Switch is in mst mode (IEEE Standard)
Root bridge for: none
EtherChannel misconfig guard is enabled
Extended system ID is enabled
Portfast Default is disabled
PortFast BPDU Guard Default is disabled
Portfast BPDU Filter Default is disabled
Loopguard Default is disabled
UplinkFast is disabled
Stack port is StackPort1
BackboneFast is disabled
Configured Pathcost method used is short (Operational value is long)
Name Blocking Listening Learning Forwarding STP Active
---------------------- -------- --------- -------- ---------- ----------
MST0 0 0 0 59 59
MST1 0 0 0 59 59
---------------------- -------- --------- -------- ---------- ----------
2 msts 0 0 0 118 118
3750-ANA#
I am not sure what is using CPU to go up like this. I had tried some troubleshooting guide on Cisco web site like "Troubleshooting high CPU Utilization" but it does not help me to find out the problem yet. Does anyone have any idea how to troubleshoot ?
