Catalyst 3750 Series Switch stackable High CPU Usage Troubleshoot

[John Eze]

Hi,

   i'm facing High CPU utilization in our stack switches,here I 've posted the out put of the switches.

Please advice how to reduce the CPU utilization and what need to be done to normalize.

Thanks in advance

#sh processes cpu history

555544444444444444455555555554444477777888889999999999888889
999955555888887777700000555557777700000777772222200000999990
100
90 *******************
80 *******************
70 ************************
60 **** ***** ************************
50 **********************************************************
40 **********************************************************
30 **********************************************************
20 **********************************************************
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per second (last 60 seconds)

989999999999999998988999999999999999999999999999999999999999
--More-- 201111221111000119099110220222131012133201101222301021311231
100
90 * ##################*####################*################
80 #*########################################################
70 #*########################################################
60 #*########################################################
50 ##########################################################
40 ##########################################################
30 ##########################################################
20 ##########################################################
10 ##########################################################
0....5....1....1....2....2....3....3....4....4....5....5....6
0 5 0 5 0 5 0 5 0 5 0
CPU% per minute (last 60 minutes)
* = maximum CPU% # = average CPU%

999999999744373444844444499995544455949999544945999779654444445444556899
443322232011579343223936793324435602879978037868979370243206014866194933
100 * * **** * ***
90 #******** **** * **** * *** * *
--More-- 80 ####***#* * * **** * **** * *** ** *
70 ########** * * *##* * **** * ****** *
60 #########* * * *##* * **** * ******** **#
50 #########* * * * **###*** *********** ********##** * *******#
40 #########****************###********************#######**************#
30 ##########***************#####****###*###****#*#########*********#####
20 ######################################################################
10 ######################################################################
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7..
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)
* = maximum CPU% # = average CPU%

01#sh cdp p processes cpu history
CPU utilization for five seconds: 50%/32%; one minute: 60%; five minutes: 76%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 42 332 126 0.00% 0.00% 0.00% 0 Chunk Manager
2 100675 989701 101 0.00% 0.00% 0.00% 0 Load Meter
3 184 1786 103 0.00% 0.00% 0.00% 0 hulc_entropy_thr
4 1517 16537 91 0.00% 0.00% 0.00% 0 OBFL TEMP obfl0
5 117085800 4927913 23759 0.00% 2.12% 1.97% 0 Check heaps
6 4239 55589 76 0.00% 0.00% 0.00% 0 Pool Manager
7 0 1 0 0.00% 0.00% 0.00% 0 DiscardQ Backgro
8 0 2 0 0.00% 0.00% 0.00% 0 Timers
9 1304 69839 18 0.00% 0.00% 0.00% 0 WATCH_AFS
10 0 3 0 0.00% 0.00% 0.00% 0 License Client N
11 16029700 82201 195008 0.00% 0.26% 0.24% 0 Licensing Auto U
--More-- 12 110 6 18333 0.00% 0.00% 0.00% 0 Image License br
--More-- 13 1461543 2716217 538 0.15% 0.02% 0.00% 0 ARP Input
--More-- 14 270847 5095521 53 0.00% 0.00% 0.00% 0 ARP Background
--More-- 15 0 1 0 0.00% 0.00% 0.00% 0 CEF MIB API
--More-- 16 0 7 0 0.00% 0.00% 0.00% 0 AAA_SERVER_DEADT
--More-- 17 0 1 0 0.00% 0.00% 0.00% 0 Policy Manager
--More-- 18 85 22 3863 0.00% 0.00% 0.00% 0 Entity MIB API
--More-- 19 0 1 0 0.00% 0.00% 0.00% 0 IFS Agent Manage
--More-- 20 60582 983781 61 0.00% 0.00% 0.00% 0 IPC Event Notifi
--More-- 21 105649 4792417 22 0.00% 0.00% 0.00% 0 IPC Mcast Pendin
--More-- 22 3676 82209 44 0.00% 0.00% 0.00% 0 IPC Dynamic Cach
--More-- 23 0 1 0 0.00% 0.00% 0.00% 0 IPC Session Serv
--More-- 24 0 1 0 0.00% 0.00% 0.00% 0 IPC Zone Manager
--More-- 25 116262 4792415 24 0.00% 0.00% 0.00% 0 IPC Periodic Tim
--More-- 26 106341 4792417 22 0.00% 0.00% 0.00% 0 IPC Deferred Por
--More-- 27 0 1 0 0.00% 0.00% 0.00% 0 IPC Process leve
--More-- 28 77025 656969 117 0.00% 0.00% 0.00% 0 IPC Seat Manager
--More-- 29 9108 282614 32 0.00% 0.00% 0.00% 0 IPC Check Queue
--More-- 30 8 27 296 0.00% 0.00% 0.00% 0 IPC Seat RX Cont
--More-- 31 0 1 0 0.00% 0.00% 0.00% 0 IPC Seat TX Cont
--More-- 32 16668 494863 33 0.00% 0.00% 0.00% 0 IPC Keep Alive M
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
33 62968 986470 63 0.00% 0.00% 0.00% 0 IPC Loadometer
34 42 4 10500 0.00% 0.00% 0.00% 0 PrstVbl
35 0 1 0 0.00% 0.00% 0.00% 0 Crash writer
36 0 1 0 0.00% 0.00% 0.00% 0 Exception contro
37 0 3 0 0.00% 0.00% 0.00% 0 License IPC stat
38 0 1 0 0.00% 0.00% 0.00% 0 License IPC serv
39 141659 4948501 28 0.00% 0.00% 0.00% 0 GraphIt
40 1718605 104226 16489 0.00% 0.00% 0.00% 0 crypto sw pk pro
41 0 2 0 0.00% 0.00% 0.00% 0 SMART
42 0 2 0 0.00% 0.00% 0.00% 0 XML Proxy Client
43 0 1 0 0.00% 0.00% 0.00% 0 ARP Snoop
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
44 81540 4902020 16 0.00% 0.00% 0.00% 0 Dynamic ARP Insp
45 0 1 0 0.00% 0.00% 0.00% 0 Critical Bkgnd
46 2197649 3465778 634 0.00% 0.05% 0.04% 0 Net Background
47 8 24 333 0.00% 0.00% 0.00% 0 IDB Work
48 22309 305497 73 0.00% 0.00% 0.00% 0 Logger
49 80810 4901941 16 0.00% 0.00% 0.00% 0 TTY Background
50 0 3 0 0.00% 0.00% 0.00% 0 CTS CORE
51 94 201 467 0.00% 0.00% 0.00% 0 SXP CORE
52 42 76 552 0.00% 0.00% 0.00% 0 IF-MGR control p
53 35 372 94 0.00% 0.00% 0.00% 0 IF-MGR event pro
54 0 1 0 0.00% 0.00% 0.00% 0 Inode Table Dest
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
55 0 4 0 0.00% 0.00% 0.00% 0 IP Admission HA
56 587 9772 60 0.00% 0.00% 0.00% 0 Net Input
57 985439 989738 995 0.00% 0.03% 0.00% 0 Compute load avg
58 3788817 89635 42269 0.00% 0.03% 0.04% 0 Per-minute Jobs
59 5199986 4948806 1050 0.00% 0.11% 0.12% 0 Per-Second Jobs
60 0 1 0 0.00% 0.00% 0.00% 0 AggMgr Process
61 2883 79484 36 0.00% 0.00% 0.00% 0 Transport Port A
62 509829 1449639 351 0.00% 0.01% 0.00% 0 HC Counter Timer
63 2387401 33627 70996 0.00% 0.00% 0.00% 0 SFF8472
64 8 69 115 0.00% 0.00% 0.00% 0 EEM ED Identity
65 0 137 0 0.00% 0.00% 0.00% 0 EEM ED MAT
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
66 84604 451609 187 0.00% 0.00% 0.00% 0 EEM ED ND
67 33 65 507 0.00% 0.00% 0.00% 0 USB Startup
68 3286716 95383712 34 0.00% 0.01% 0.00% 0 DownWhenLooped
69 0 1 0 0.00% 0.00% 0.00% 0 HULC PBR Tcam Me
70 0 1 0 0.00% 0.00% 0.00% 0 HULC ACL Tcam Me
71 0 1 0 0.00% 0.00% 0.00% 0 Hulc Warm Upgrad
72 0 1 0 0.00% 0.00% 0.00% 0 HRPC hdwl reques
73 0 1 0 0.00% 0.00% 0.00% 0 Hulc EEM Process
74 0 1 0 0.00% 0.00% 0.00% 0 HRPC lpip reques
75 0 2 0 0.00% 0.00% 0.00% 0 HLPIP Sync Proce
76 0 1 0 0.00% 0.00% 0.00% 0 HRPC Multi-FS Sy
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
77 16 30 533 0.00% 0.00% 0.00% 0 HULC multifs pro
78 0 1 0 0.00% 0.00% 0.00% 0 HRPC hnetwpol re
79 0 1 0 0.00% 0.00% 0.00% 0 HRPC EnergyWise
80 0 1 0 0.00% 0.00% 0.00% 0 HRPC actual powe
81 691958 19781319 34 0.00% 0.00% 0.00% 0 HUSB Console
82 0 1 0 0.00% 0.00% 0.00% 0 HRPC ipadm reque
83 0 2 0 0.00% 0.00% 0.00% 0 SM RBCP Monitor
84 11211 164407 68 0.00% 0.00% 0.00% 0 PSP Timer
85 0 1 0 0.00% 0.00% 0.00% 0 HULC QM Tcam Mem
86 4229 34 124382 0.00% 0.00% 0.00% 0 X2 HAMM detect p
87 0 1 0 0.00% 0.00% 0.00% 0 Madmax Process
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
88 0 1 0 0.00% 0.00% 0.00% 0 HRPC asic-stats
89 148453571 60994246 2433 3.03% 2.50% 2.55% 0 RedEarth I2C dri
90 243729799 40826320 5969 4.00% 4.07% 4.00% 0 RedEarth Tx Mana
91 5856558 30316490 193 0.00% 0.11% 0.17% 0 RedEarth Rx Mana
92 66948 989704 67 0.00% 0.00% 0.00% 0 HULC Thermal Pro
93 0 1 0 0.00% 0.00% 0.00% 0 HRPC hsm request
94 0 16 0 0.00% 0.00% 0.00% 0 Stack Mgr
95 696 73 9534 0.00% 0.00% 0.00% 0 Stack Mgr Notifi
96 703567 1967130 357 0.00% 0.01% 0.00% 0 Adjust Regions
97 416781 2464408 169 0.00% 0.00% 0.00% 0 hrpc -> response
98 67933 2047805 33 0.00% 0.00% 0.00% 0 hrpc -> request
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
99 888478 2046792 434 0.00% 0.00% 0.00% 0 hrpc <- response
100 0 3 0 0.00% 0.00% 0.00% 0 HRPC hcomp reque
101 787968 24693549 31 0.00% 0.00% 0.00% 0 yeti2_emac_proce
102 0 11 0 0.00% 0.00% 0.00% 0 HULC Device Mana
103 35855 95031 377 0.00% 0.00% 0.00% 0 HRPC hdm non blo
104 0 5 0 0.00% 0.00% 0.00% 0 HRPC hdm blockin
105 99061 983764 100 0.00% 0.00% 0.00% 0 HIPC bkgrd proce
106 0 1 0 0.00% 0.00% 0.00% 0 RTTYS Process
107 183 349 524 0.00% 0.00% 0.00% 0 Hulc Port-Securi
108 50 572 87 0.00% 0.00% 0.00% 0 HRPC hpsecure re
109 0 3 0 0.00% 0.00% 0.00% 0 HRPC hfbm reques
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
110 868 9634 90 0.00% 0.00% 0.00% 0 TACACS+
111 651 3255 200 0.00% 0.00% 0.00% 0 HRPC hlfm reques
112 84308104 132025889 638 0.47% 0.99% 1.95% 0 HLFM address lea
113 84111 4901945 17 0.00% 0.00% 0.00% 0 HLFM aging proce
114 4246678 132019888 32 0.15% 0.09% 0.10% 0 HLFM address ret
115 0 1 0 0.00% 0.00% 0.00% 0 HRPC hrcmd reque
116 52 24 2166 0.00% 0.00% 0.00% 0 HRPC emac reques
117 9 17 529 0.00% 0.00% 0.00% 0 HRPC isis reques
118 2278 5775 394 0.00% 0.00% 0.00% 0 HRPC hulc misc r
119 61446 1643999 37 0.00% 0.00% 0.00% 0 HVLAN main bkgrd
120 0 2 0 0.00% 0.00% 0.00% 0 HVLAN Mapped Vla
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
121 0 2 0 0.00% 0.00% 0.00% 0 Vlan shutdown Pr
122 27 225 120 0.00% 0.00% 0.00% 0 HRPC vlan reques
123 0 48 0 0.00% 0.00% 0.00% 0 HULC VLAN REF Ba
124 4157 82209 50 0.00% 0.00% 0.00% 0 hulc_xps_process
125 16550 132861 124 0.00% 0.00% 0.00% 0 HRPC ilp request
126 158228 4902002 32 0.00% 0.00% 0.00% 0 Hulc ILP Alchemy
127 2095384 328800 6372 0.00% 0.01% 0.00% 0 Strider Tcam Mem
128 0 1 0 0.00% 0.00% 0.00% 0 HULC PM Vector P
129 0 1 0 0.00% 0.00% 0.00% 0 HPM Msg Retry Pr
130 0 1 0 0.00% 0.00% 0.00% 0 DiagCard2/-1
131 14094274 93168444 151 0.15% 0.42% 0.47% 0 hpm main process
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
132 285 590 483 0.00% 0.00% 0.00% 0 HPM Stack Sync P
133 1401 2734 512 0.00% 0.00% 0.00% 0 HRPC pm request
134 0 5 0 0.00% 0.00% 0.00% 0 HPM if_num mappi
135 6734388 4948504 1360 0.00% 0.06% 0.14% 0 hpm counter proc
136 5874130 6173431 951 0.00% 0.07% 0.10% 0 HRPC pm-counters
137 75 121 619 0.00% 0.00% 0.00% 0 hpm vp events ca
138 101232 4902035 20 0.00% 0.00% 0.00% 0 Hulc CTS Process
139 0 1 0 0.00% 0.00% 0.00% 0 HRPC cts request
140 0 1 0 0.00% 0.00% 0.00% 0 HULC Msg Retry P
141 0 1 0 0.00% 0.00% 0.00% 0 HRPC hcmp reques
142 12790 2370 5396 0.00% 0.00% 0.00% 0 HCEF ADJ Refresh
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
143 0 1 0 0.00% 0.00% 0.00% 0 hl3md_rpfq_thrl_
144 0 1 0 0.00% 0.00% 0.00% 0 HRPC hl3mm ipv6
145 132222 3413765 38 0.00% 0.00% 0.00% 0 hl3mm_mfib_ipv6
146 38192 2464402 15 0.00% 0.00% 0.00% 0 hl3mm_mrib_rp
147 48620 2464399 19 0.00% 0.00% 0.00% 0 hl3mm_rp
148 0 1 0 0.00% 0.00% 0.00% 0 HACL Queue Proce
149 0 1 0 0.00% 0.00% 0.00% 0 HRPC acl request
150 108 382 282 0.00% 0.00% 0.00% 0 HACL Acl Manager
151 0 1 0 0.00% 0.00% 0.00% 0 HRPC aim request
152 0 5 0 0.00% 0.00% 0.00% 0 HRPC axiom-mgmt
153 46363 2464395 18 0.00% 0.00% 0.00% 0 Hulc Axiom Mgmt
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
154 0 1 0 0.00% 0.00% 0.00% 0 HRPC backup inte
155 1697 557 3046 1.43% 0.71% 0.32% 1 SSH Process
156 0 1 0 0.00% 0.00% 0.00% 0 HRPC cdp request
157 0 65 0 0.00% 0.00% 0.00% 0 HULC CISP Proces
158 100 964 103 0.00% 0.00% 0.00% 0 HRPC dot1x reque
159 0 1 0 0.00% 0.00% 0.00% 0 Dot1X Msg Retry
160 3759 3962 948 0.00% 0.00% 0.00% 0 HULC DOT1X Proce
161 0 1 0 0.00% 0.00% 0.00% 0 HRPC foxhound re
162 0 2 0 0.00% 0.00% 0.00% 0 Hulc Foxhound Pr
163 0 1 0 0.00% 0.00% 0.00% 0 HRPC lldp reques
164 0 3 0 0.00% 0.00% 0.00% 0 HRPC system mtu
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
165 0 5 0 0.00% 0.00% 0.00% 0 HRPC rep request
166 0 4 0 0.00% 0.00% 0.00% 0 REP Helper Proc
167 0 1 0 0.00% 0.00% 0.00% 0 HRPC sdm request
168 0 1 0 0.00% 0.00% 0.00% 0 SMI MSG Retry Pr
169 0 1 0 0.00% 0.00% 0.00% 0 HRPC Smart Insta
170 1629130 24304596 67 0.15% 0.03% 0.03% 0 Hulc Storm Contr
171 0 2 0 0.00% 0.00% 0.00% 0 HSTP Sync Proces
172 0 1 0 0.00% 0.00% 0.00% 0 HRPC stp_cli req
173 9 13 692 0.00% 0.00% 0.00% 0 HRPC stp_state_s
174 0 2 0 0.00% 0.00% 0.00% 0 S/W Bridge Proce
175 0 29 0 0.00% 0.00% 0.00% 0 HRPC hudld reque
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
176 0 1 0 0.00% 0.00% 0.00% 0 HRPC vqpc reques
177 0 1 0 0.00% 0.00% 0.00% 0 HRPC iec_load_ba
178 72021 4901960 14 0.00% 0.00% 0.00% 0 IEC Load balance
179 0 1 0 0.00% 0.00% 0.00% 0 HRPC l2pt qnq rp
180 0 16 0 0.00% 0.00% 0.00% 0 HRPC hled reques
181 520533755 122841039 4237 3.20% 2.90% 3.13% 0 Hulc LED Process
182 0 1 0 0.00% 0.00% 0.00% 0 HRPC hl3mm reque
183 133285 3413816 39 0.00% 0.02% 0.00% 0 hl3mm
184 22223012 3638602 6107 0.79% 0.52% 0.54% 0 HL3U bkgrd proce
185 231003 1979413 116 0.00% 0.00% 0.00% 0 HRPC hl3u reques
186 145776 1129186 129 0.00% 0.00% 0.00% 0 HIPV6 bkgrd proc
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
187 216482 1979413 109 0.00% 0.00% 0.00% 0 HRPC IPv6 Unicas
188 43346 1129241 38 0.00% 0.00% 0.00% 0 HL3U PBR bkgrd p
189 1596 41215 38 0.00% 0.00% 0.00% 0 HL3U PBR n-h res
190 0 1 0 0.00% 0.00% 0.00% 0 HRPC obfl reques
192 0 1 0 0.00% 0.00% 0.00% 0 HRPC dtp request
193 0 1 0 0.00% 0.00% 0.00% 0 HRPC hmedianet p
194 0 1 0 0.00% 0.00% 0.00% 0 HRPC show_forwar
195 13013 54015 240 0.00% 0.00% 0.00% 0 HRPC snmp reques
196 588354 27020 21774 0.00% 0.00% 0.00% 0 HULC SNMP Proces
197 24019027 983669 24417 0.31% 0.41% 0.42% 0 HQM Stack Proces
198 12881142 3925301 3281 0.00% 0.18% 0.20% 0 HRPC qos request
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
199 0 1 0 0.00% 0.00% 0.00% 0 HRPC span reques
200 0 8 0 0.00% 0.00% 0.00% 0 HRPC system post
201 0 9 0 0.00% 0.00% 0.00% 0 Hulc Reload Mana
202 26723 54143 493 0.00% 0.00% 0.00% 0 HRPC hrcli-event
203 5512 27492 200 0.00% 0.00% 0.00% 0 HCMP sync proces
204 0 2 0 0.00% 0.00% 0.00% 0 image mgr
205 0 1 0 0.00% 0.00% 0.00% 0 HRFS OIR Proc
206 667 147 4537 0.00% 0.00% 0.00% 0 HL2MCM
207 941 130 7238 0.00% 0.00% 0.00% 0 HL2MCM
208 0 6 0 0.00% 0.00% 0.00% 0 Hulc SM10G Infra
209 177 1049 168 0.00% 0.00% 0.00% 0 AAA Server
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
210 239 2051 116 0.00% 0.00% 0.00% 0 AAA ACCT Proc
211 0 1 0 0.00% 0.00% 0.00% 0 ACCT Periodic Pr
212 0 6 0 0.00% 0.00% 0.00% 0 AAA System Acct
213 0 1 0 0.00% 0.00% 0.00% 0 AUTH POLICY Fram
214 95507 4795974 19 0.00% 0.00% 0.00% 0 Auth Manager
215 0 1 0 0.00% 0.00% 0.00% 0 Auth-proxy AAA B
216 539 16495 32 0.00% 0.00% 0.00% 0 IP Admin SM Proc
217 2335374 1055859 2211 0.00% 0.03% 0.04% 0 CDP Protocol
218 0 3 0 0.00% 0.00% 0.00% 0 OBFL Cfg Dispatc
219 809140 7600070 106 0.00% 0.01% 0.00% 0 CEF: IPv4 proces
220 0 2 0 0.00% 0.00% 0.00% 0 CMD HANDLER
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
221 0 2 0 0.00% 0.00% 0.00% 0 AAA Dictionary R
222 1592 41238 38 0.00% 0.00% 0.00% 0 DHCP Snooping
223 0 1 0 0.00% 0.00% 0.00% 0 DHCP Snooping db
224 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Mgr Proces
225 0 1 0 0.00% 0.00% 0.00% 0 EAP Framework
226 0 1 0 0.00% 0.00% 0.00% 0 EAP Test
227 0 2 0 0.00% 0.00% 0.00% 0 EAPoUDP Process
228 8 2 4000 0.00% 0.00% 0.00% 0 CEF switching ba
229 2506 793 3160 0.00% 0.00% 0.00% 0 IP ARP Adjacency
230 4769085 132018989 36 0.15% 0.08% 0.12% 0 IP ARP Retry Age
231 37538153 82924474 452 1.91% 0.91% 0.76% 0 IP Input
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
232 0 1 0 0.00% 0.00% 0.00% 0 ICMP event handl
233 227476 9512543 23 0.00% 0.00% 0.00% 0 IP ARP Track
234 42719 4910373 8 0.00% 0.00% 0.00% 0 fnvRootHubPortSt
235 0 1 0 0.00% 0.00% 0.00% 0 tHUB
236 1201057 48563421 24 0.00% 0.00% 0.00% 0 MDFS MFIB Proces
237 0 1 0 0.00% 0.00% 0.00% 0 ADJ NSF process
238 0 1 0 0.00% 0.00% 0.00% 0 IPv6 ping proces
239 0 2 0 0.00% 0.00% 0.00% 0 KRB5 AAA
240 69322 328798 210 0.00% 0.00% 0.00% 0 MDFS LC Download
241 0 2 0 0.00% 0.00% 0.00% 0 REP Topology cha
242 17 44 386 0.00% 0.00% 0.00% 0 SMI Director DB
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
243 83449 417296 199 0.00% 0.00% 0.00% 0 SMI CDP Update H
244 0 1 0 0.00% 0.00% 0.00% 0 SMI Backup Proce
245 0 2 0 0.00% 0.00% 0.00% 0 SMI IBC server p
246 0 1 0 0.00% 0.00% 0.00% 0 SMI IBC client p
247 0 2 0 0.00% 0.00% 0.00% 0 SMI IBC Download
248 8060549 37796310 213 0.15% 0.13% 0.15% 0 Spanning Tree
249 11216 82471 135 0.00% 0.00% 0.00% 0 Spanning Tree St
250 0 2 0 0.00% 0.00% 0.00% 0 CDP Forward Proc
251 1654 435 3802 0.00% 0.00% 0.00% 0 802.1x switch
252 0 1 0 0.00% 0.00% 0.00% 0 802.1x Webauth F
253 91695 332103 276 0.00% 0.00% 0.00% 0 DTP Protocol
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
254 0 3 0 0.00% 0.00% 0.00% 0 HRPC IPv6 Host r
255 0 24 0 0.00% 0.00% 0.00% 0 IPv6 Platform Ho
256 0 1 0 0.00% 0.00% 0.00% 0 axiom-module cra
257 0 1 0 0.00% 0.00% 0.00% 0 HRPC dai request
258 0 1 0 0.00% 0.00% 0.00% 0 HULC DAI Process
259 0 1 0 0.00% 0.00% 0.00% 0 HRPC power down
260 0 1 0 0.00% 0.00% 0.00% 0 HRPC ip device t
261 0 1 0 0.00% 0.00% 0.00% 0 HRPC ip source g
262 0 1 0 0.00% 0.00% 0.00% 0 HULC IP Source g
263 0 1 0 0.00% 0.00% 0.00% 0 HRPC sisf reques
264 0 3 0 0.00% 0.00% 0.00% 0 HULC SISF Proces
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
265 0 1 0 0.00% 0.00% 0.00% 0 HULC SISF Source
266 634808 4901931 129 0.00% 0.03% 0.01% 0 PI MATM Aging Pr
267 3008219 49372259 60 0.00% 0.02% 0.04% 0 UDLD
268 2834 164952 17 0.00% 0.00% 0.00% 0 Port-Security
269 0 2 0 0.00% 0.00% 0.00% 0 Switch Backup In
270 0 2 0 0.00% 0.00% 0.00% 0 IP Host Track Pr
271 0 1 0 0.00% 0.00% 0.00% 0 Link State Group
272 2791 82208 33 0.00% 0.00% 0.00% 0 MMN bkgrd proces
273 18767 493220 38 0.00% 0.00% 0.00% 0 Ethchnl
274 746 436 1711 0.00% 0.00% 0.00% 0 VMATM Callback
275 0 1 0 0.00% 0.00% 0.00% 0 IPC UDP Input
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
276 20490 98580 207 0.00% 0.00% 0.00% 0 CEF background p
277 0 1 0 0.00% 0.00% 0.00% 0 fib_fib_bfd_sb e
278 0 1 0 0.00% 0.00% 0.00% 0 IP IRDP
279 0 1 0 0.00% 0.00% 0.00% 0 CEF RF HULC Conv
280 0 1 0 0.00% 0.00% 0.00% 0 XDR background p
281 296549 605575 489 0.00% 0.01% 0.00% 0 XDR mcast
282 14680 493200 29 0.00% 0.00% 0.00% 0 IP ACL XDR LC Ba
283 0 3 0 0.00% 0.00% 0.00% 0 XDR RP Ping Back
284 217344 445618 487 0.00% 0.00% 0.00% 0 XDR mcast rcv
285 7155 82208 87 0.00% 0.00% 0.00% 0 XDR RP backgroun
286 17 4 4250 0.00% 0.00% 0.00% 0 IPC LC Message H
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
287 9 1 9000 0.00% 0.00% 0.00% 0 XDR RP Test Back
288 83 20 4150 0.00% 0.00% 0.00% 0 ADJ background
289 0 16 0 0.00% 0.00% 0.00% 0 MDFS LC Process
290 0 1 0 0.00% 0.00% 0.00% 0 Critical Auth
291 0 2 0 0.00% 0.00% 0.00% 0 Routing Topology
292 3837827 390278 9833 0.79% 0.18% 0.12% 0 IP RIB Update
293 0 2 0 0.00% 0.00% 0.00% 0 MRIB Process
294 242219 136189 1778 0.15% 0.01% 0.00% 0 ADJ resolve proc
295 1184 10346 114 0.00% 0.00% 0.00% 0 TCP Timer
296 20347 63358 321 0.00% 0.00% 0.00% 0 TCP Protocols
297 23467 4911659 4 0.00% 0.00% 0.00% 0 Socket Timers
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
298 375 16507 22 0.00% 0.00% 0.00% 0 HTTP CORE
299 1864 61851 30 0.00% 0.00% 0.00% 0 Cluster L2
300 17906 493202 36 0.00% 0.00% 0.00% 0 Cluster RARP
301 79092 753506 104 0.00% 0.00% 0.00% 0 Cluster Base
302 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
303 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
304 0 2 0 0.00% 0.00% 0.00% 0 Dot1x Supplicant
305 9 3 3000 0.00% 0.00% 0.00% 0 Flow Exporter Ti
306 620 16228 38 0.00% 0.00% 0.00% 0 MFIB Master back
307 0 43 0 0.00% 0.00% 0.00% 0 HRPC hl2mcm igmp
308 24 143 167 0.00% 0.00% 0.00% 0 HRPC hl2mcm mlds
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
309 0 1 0 0.00% 0.00% 0.00% 0 MRIB RP Proxy
310 0 2 0 0.00% 0.00% 0.00% 0 Multicast Offloa
311 0 1 0 0.00% 0.00% 0.00% 0 RARP Input
312 101 62 1629 0.00% 0.00% 0.00% 0 static
313 0 1 0 0.00% 0.00% 0.00% 0 IPv6 RIB Event H
314 0 1 0 0.00% 0.00% 0.00% 0 MAB Framework
315 57374 493202 116 0.00% 0.00% 0.00% 0 QoS stats proces
316 0 1 0 0.00% 0.00% 0.00% 0 Connection Mgr
317 0 2 0 0.00% 0.00% 0.00% 0 REP LSL Proc
318 0 2 0 0.00% 0.00% 0.00% 0 REP BPA/EPA Proc
319 0 18 0 0.00% 0.00% 0.00% 0 SNMP Timers
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
320 0 1 0 0.00% 0.00% 0.00% 0 HRPC hwccp reque
321 0 3 0 0.00% 0.00% 0.00% 0 HULC_WCCP_BACKGR
322 0 1 0 0.00% 0.00% 0.00% 0 HRPC dhcp snoopi
323 0 5 0 0.00% 0.00% 0.00% 0 HULC DHCP Snoopi
324 1135668 6122867 185 0.00% 0.07% 0.05% 0 Inline Power
325 4612270 10835752 425 0.15% 0.09% 0.14% 0 Marvell wk-a Pow
326 40539 246609 164 0.00% 0.00% 0.00% 0 HULC stack power
327 11832 164577 71 0.00% 0.00% 0.00% 0 HRPC stack power
328 117 298 392 0.00% 0.00% 0.00% 0 IGMPSN L2MCM
329 0 1 0 0.00% 0.00% 0.00% 0 IGMPSN MRD
330 9 28 321 0.00% 0.00% 0.00% 0 IGMPSN
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
331 0 1 0 0.00% 0.00% 0.00% 0 IGMPQR
332 0 3 0 0.00% 0.00% 0.00% 0 L2TRACE SERVER
333 133 297 447 0.00% 0.00% 0.00% 0 MLDSN L2MCM
334 0 1 0 0.00% 0.00% 0.00% 0 MRD
335 0 1 0 0.00% 0.00% 0.00% 0 MLD_SNOOP
336 18370 383 47963 0.00% 0.00% 0.00% 0 Collection proce
337 1722899 48542046 35 0.00% 0.03% 0.00% 0 MDFS RP process
338 0 1 0 0.00% 0.00% 0.00% 0 EM Background Pr
339 0 2 0 0.00% 0.00% 0.00% 0 AAA Cached Serve
340 1144349 2906 393788 0.00% 0.00% 0.00% 0 NIST rng proc
341 239444 9512537 25 0.00% 0.00% 0.00% 0 Timer Library
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
342 0 2 0 0.00% 0.00% 0.00% 0 ENABLE AAA
343 0 3 0 0.00% 0.00% 0.00% 0 LDAP process
344 0 2 0 0.00% 0.00% 0.00% 0 Crypto Support
345 0 1 0 0.00% 0.00% 0.00% 0 IPSECv6 PS Proc
346 9 50 180 0.00% 0.00% 0.00% 0 LINE AAA
347 0 2 0 0.00% 0.00% 0.00% 0 LOCAL AAA
348 0 2 0 0.00% 0.00% 0.00% 0 EPM MAIN PROCESS
349 169 122 1385 0.00% 0.00% 0.00% 0 SpanTree Helper
350 15410 17019 905 0.00% 0.01% 0.00% 0 TPLUS
351 689757 246602 2797 0.00% 0.00% 0.00% 0 HULC ISM FNF
352 428 185 2313 0.00% 0.00% 0.00% 0 hulc cfg mgr mas
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
353 427 31 13774 0.00% 0.00% 0.00% 0 crypto engine pr
354 285504 4928579 57 0.00% 0.00% 0.00% 0 Crypto IKEv2
355 0 1 0 0.00% 0.00% 0.00% 0 IKEv2 AAA handle
356 0 1 0 0.00% 0.00% 0.00% 0 encrypt proc
357 0 3 0 0.00% 0.00% 0.00% 0 CRYPTO MAP FREE
358 0 1 0 0.00% 0.00% 0.00% 0 Crypto INT
359 0 2 0 0.00% 0.00% 0.00% 0 Crypto IKE Dispa
360 0 3 0 0.00% 0.00% 0.00% 0 Crypto IKMP
361 0 1 0 0.00% 0.00% 0.00% 0 IPSEC manual key
362 15124 246603 61 0.00% 0.00% 0.00% 0 IPSEC key engine
363 0 1 0 0.00% 0.00% 0.00% 0 Crypto ACL
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
364 0 1 0 0.00% 0.00% 0.00% 0 Crypto PAS Proc
365 0 1 0 0.00% 0.00% 0.00% 0 IPSec background
366 501 5502 91 0.00% 0.00% 0.00% 0 Crypto CA
367 0 1 0 0.00% 0.00% 0.00% 0 Crypto PKI-CRL
368 0 2 0 0.00% 0.00% 0.00% 0 MMON PROCESS
369 3691134 153648968 24 0.00% 0.02% 0.03% 0 MMON MENG
370 0 1 0 0.00% 0.00% 0.00% 0 HRPC x_setup req
371 0 2 0 0.00% 0.00% 0.00% 0 REP Switch Helpe
372 0 1 0 0.00% 0.00% 0.00% 0 Licensing MIB pr
373 0 15 0 0.00% 0.00% 0.00% 0 VTP Trap Process
374 0 1 0 0.00% 0.00% 0.00% 0 DATA Transfer Pr
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
375 0 1 0 0.00% 0.00% 0.00% 0 DATA Collector
376 184 1987 92 0.00% 0.00% 0.00% 0 ASP Process Crea
377 1518 4053 374 0.00% 0.00% 0.00% 0 AAA SEND STOP EV
378 0 1 0 0.00% 0.00% 0.00% 0 Test AAA Client
379 0 2 0 0.00% 0.00% 0.00% 0 DHCP Security He
380 0 2 0 0.00% 0.00% 0.00% 0 EEM ED Routing
381 17927 182167 98 0.00% 0.00% 0.00% 0 EEM ED Syslog
382 3242490 162204 19990 0.00% 0.02% 0.01% 0 Syslog Traps
383 108 5 21600 0.00% 0.00% 0.00% 0 OBFL MSG obfl0
384 3472 82208 42 0.00% 0.00% 0.00% 0 vlmap logger
385 0 1 0 0.00% 0.00% 0.00% 0 HCD Process
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
386 0 1 0 0.00% 0.00% 0.00% 0 HRPC cable diagn
387 0 1 0 0.00% 0.00% 0.00% 0 HRPC rsu request
388 58 7 8285 0.00% 0.00% 0.00% 0 OBFL ENV obfl0
389 0 2 0 0.00% 0.00% 0.00% 0 DiagCard1/-1
390 0 1 0 0.00% 0.00% 0.00% 0 Online Diag EEM
391 1372874 21007452 65 0.00% 0.00% 0.00% 0 PM Callback
392 658979 36314 18146 0.00% 0.00% 0.00% 0 hulc running con
393 16869 395559 42 0.00% 0.00% 0.00% 0 dhcp snooping sw
394 3003480 82395 36452 0.00% 0.03% 0.00% 0 OBFL VOLT obfl0
395 26 260 100 0.00% 0.00% 0.00% 0 EEM Server
396 25 2 12500 0.00% 0.00% 0.00% 0 Call Home proces
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
397 0 4 0 0.00% 0.00% 0.00% 0 CTS credentials
398 0 2 0 0.00% 0.00% 0.00% 0 EEM Policy Direc
400 0 2 0 0.00% 0.00% 0.00% 0 EEM ED CLI
401 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Config
402 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Counter
403 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Env
404 0 3 0 0.00% 0.00% 0.00% 0 EM ED GOLD
405 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Interface
406 0 3 0 0.00% 0.00% 0.00% 0 EEM ED IOSWD
407 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Ipsla
408 8 3 2666 0.00% 0.00% 0.00% 0 EEM ED None
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
409 0 3 0 0.00% 0.00% 0.00% 0 EEM ED OIR
410 6902 82208 83 0.00% 0.00% 0.00% 0 Call Home Timer
411 0 3 0 0.00% 0.00% 0.00% 0 EEM ED RPC
412 0 3 0 0.00% 0.00% 0.00% 0 EEM ED SNMP
413 0 3 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Obje
414 0 2 0 0.00% 0.00% 0.00% 0 EEM ED SNMP Noti
415 0 3 0 0.00% 0.00% 0.00% 0 EEM ED Test
416 6086 170633 35 0.00% 0.00% 0.00% 0 EEM ED Timer
417 274130 134392 2039 0.00% 0.00% 0.00% 0 Syslog
418 51028 191973 265 0.00% 0.00% 0.00% 0 RBM CORE
419 0 1 0 0.00% 0.00% 0.00% 0 HRPC eee request
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
420 0 2 0 0.00% 0.00% 0.00% 0 STP FAST TRANSIT
421 0 2 0 0.00% 0.00% 0.00% 0 CSRT RAPID TRANS
422 33 6 5500 0.00% 0.00% 0.00% 0 VLAN Manager
423 0 3 0 0.00% 0.00% 0.00% 0 CEF RP IPC Backg
424 183 1984 92 0.00% 0.00% 0.00% 0 SSH Event handle
425 9070 82719 109 0.00% 0.00% 0.00% 0 IP Background
426 902 936 963 0.00% 0.00% 0.00% 0 IP Connected Rou
427 138097 1042190 132 0.00% 0.00% 0.00% 0 LACP Protocol
428 731983 9830427 74 0.00% 0.00% 0.00% 0 DHCPD Receive
429 21712 328779 66 0.00% 0.00% 0.00% 0 OBFL I/O Buffer
431 6839590 15886952 430 0.00% 0.00% 0.00% 0 IP SNMP
--More-- PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
432 4070230 7728826 526 0.00% 0.00% 0.00% 0 PDU DISPATCHER
433 49037101 8282499 5920 0.00% 0.03% 0.00% 0 SNMP ENGINE
434 0 2 0 0.00% 0.00% 0.00% 0 IP SNMPV6
435 0 1 0 0.00% 0.00% 0.00% 0 SNMP ConfCopyPro
436 786719 90746 8669 0.00% 0.00% 0.00% 0 SNMP Traps
437 386955 4963289 77 0.00% 0.00% 0.00% 0 NTP
438 0 1 0 0.00% 0.00% 0.00% 0 LICENSE AGENT
439 1866 42606 43 0.00% 0.00% 0.00% 0 DHCPD Timer
440 3525 82208 42 0.00% 0.00% 0.00% 0 DHCPD Database
442 3209787 6619143 484 0.31% 0.14% 0.10% 0 EIGRP-IPv4
443 1355921 9186299 147 0.00% 0.02% 0.00% 0 EIGRP-IPv4 Hello
NAGLAG_LEKKI_MAIN1_F01_3750X-WAN-STACK_S01# sh processes cpu ?

#sh sdm prefer default
"desktop default" template:
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1024 VLANs.

number of unicast mac addresses: 6K
number of IPv4 IGMP groups + multicast routes: 1K
number of IPv4 unicast routes: 8K
number of directly-connected IPv4 hosts: 6K
number of indirect IPv4 routes: 2K
number of IPv6 multicast groups: 64
number of directly-connected IPv6 addresses: 74
number of indirect IPv6 unicast routes: 32
number of IPv4 policy based routing aces: 0
number of IPv4/MAC qos aces: 0.5K
number of IPv4/MAC security aces: 0.875k
number of IPv6 policy based routing aces: 0
number of IPv6 qos aces: 0
number of IPv6 security aces: 60

Is this normal or abnormal?

How much  percentage of CPU usage  will really affect the switch performance?

[Leo Laohoo]

1.  What is the exact IOS version?

2.  How many switches in the stack? 

3.  What is the uptime of the switches & the stack?

[John Eze]

Version - Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9NPE-M), Version 15.0(2)SE4, RELEASE SOFTWARE (fc1)

Stack uptime – 8 weeks, 4 days, 23 hours,

Switch 2 uptime - 8 weeks, 4 days, 22 hours, 56 minutes

Regards,

[Leo Laohoo]

Try using 12.2(55)SE10.

[ashishr]

Hi John,

As per the data shared,

CPU utilization for five seconds: 50%/32%; one minute: 60%; five minutes: 76%

The problem is interrupt traffic, which is 32% here. Troubleshooting this can be little tricky as interrupts are not CPU processes but any traffic that is sent to CPU to for being processed, these could be multicast packets, ICMP redirects etc. It is acceptable to have about 10% interrupt level but anything above that should be investigated.

On 3750 platform there are some specific queues where these packets are sent. Do the following-

1. When the CPU is running high do 'clear controllers cpu-interface' and check for queue getting maximum hit using 'show controllers cpu-interface'.  You need run this command repeatedly 2-3 times.

You can use the steps mentioned in the following document to troubleshoot, it contains commands and their analysis. Check for section 'Identifying Network Packets Received by the CPU'
http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html

Debugs on this platform are CPU intrusive so I would suggest running them on console.

Apart from this you can also check for excessive broadcasts, clear counters on all interfaces and run 'sh interface | i line|rate|broadcast' after 1-2 second and check for any interface having excessive broadcast counters.

Regards,

Ashish