cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
5616
Views
0
Helpful
7
Replies

High cpu spike in cisco 3750X switch

vijay kumar
Level 2
Level 2

Hi All,

We are using three 3750X switches in stack. we are seeing the cpu spike is reaching 100% once in a 2 hours or three hours . But the average CPU is upto 40% only .

we are using this switch as L2 and L3 also with PBR. IS this spikes are expected ? Kindly suggest me

BR//

Vijay.

7 Replies 7

Leo Laohoo
Hall of Fame
Hall of Fame

Vijay,

What IOS version are you using?

For a 3750X, use IOS version 12.2(55)SE8. If you need to use 15.X, then use 15.0(2)SE4.

Sent from Cisco Technical Support Nintendo App

Hi All,

I'm hoping you can help.

Having the same issue too.  Daily CPU spikes on with pair of stacked 3750X switches.

I upgraded the IOS to c3750e-universalk9-mz.152-3.E2.bin and the issue went away for a week.

It then returned, though not as often, this time two or three times a week.  Always at the same time period though, between roughly 08:19 and 9:00 am.

There are no active sessions on the stack at the time running commands.

I suspected the HULC led process but the cpu % for it only dropped from 21% to 19% once the cpu usage returned to ok level.  See during and after output below...

DURING high cpu alert à

 

3750STACK#sh processes cpu

CPU utilization for five seconds: 94%/36%; one minute: 93%; five minutes: 93%

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

   1       11102     132583         83 0.00% 0.00% 0.00%   0 Chunk Manager                                                

   2       73211     1927461         37 0.00% 0.01% 0.00%   0 Load Meter                                                    

   3       2106         130     16200 0.32% 1.97% 0.45%   1 SSH Process                                                  

   4   240689544   10262242     23453 7.53% 2.17% 2.40%   0 Check heaps                                                  

   5       3240       46595         69 0.00% 0.00% 0.00%   0 Pool Manager                                                  

   6           0           1         0 0.00% 0.00% 0.00%   0 DiscardQ Backgr                                               o

   7           0           2         0 0.00% 0.00% 0.00%   0 Timers                                                        

   8         664       31373         21 0.00% 0.00% 0.00%   0 WATCH_AFS                                                    

   9     598995   38548259         15 0.00% 0.01% 0.00%   0 HUSB Console                                                  

10           8           5       1600 0.00% 0.00% 0.00%   0 License Client                                                 N

11   26812910     160161     167413 0.00% 0.21% 0.25%   0 Licensing Auto                                                U

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

12         227         12     18916 0.00% 0.00% 0.00%   0 Image License br

13   16873425   27059329       623 0.00% 0.16% 0.21%   0 ARP Input

14     598746     9952520         60 0.00% 0.00% 0.02%   0 ARP Background

15           0           1         0 0.00% 0.00% 0.00%   0 CEF MIB API

16           0           2         0 0.00% 0.00% 0.00%   0 AAA_SERVER_DEADT

17           0           1        0 0.00% 0.00% 0.00%   0 Policy Manager

18         75         19       3947 0.00% 0.00% 0.00%   0 Entity MIB API

19           0           1         0 0.00% 0.00% 0.00%   0 IFS Agent Manage

20       68751     1915768         35 0.00% 0.00% 0.00%   0 IPC Event Notifi

21     106701     9330161         11 0.00% 0.00% 0.00%   0 IPC Mcast Pendin

22       5991     160169         37 0.00% 0.00% 0.00%   0 IPC Dynamic Cach

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min  5Min TTY Process

23           0           1         0 0.00% 0.00% 0.00%   0 IPC Session Serv

24           0           1         0 0.00% 0.00% 0.00%   0 IPC Zone Manager

25     112321     9330200         12 0.00% 0.02% 0.00%   0 IPC Periodic Tim

26     108497     9330168         11 0.00% 0.00% 0.00%   0 IPC Deferred Por

27           0           1         0 0.00% 0.00% 0.00%   0 IPC Process leve

28     993291     4947039       200 0.00% 0.00% 0.00%   0 IPC Seat Manager

29       5093     550382         9 0.00% 0.00% 0.00%   0 IPC Check Queue

30           8         25       320 0.00% 0.00% 0.00%   0 IPC Seat RX Cont

31           0           1         0 0.00% 0.00% 0.00%   0 IPC Seat TX Cont

32       9944     963755         10 0.00% 0.00% 0.00%   0 IPC Keep Alive M

33       53369     1921951         27 0.00% 0.00% 0.00%   0 IPC Loadometer

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

34           0           1        0 0.00% 0.00% 0.00%   0 IPC Session Deta

35         133           6     22166 0.00% 0.00% 0.00%   0 PrstVbl

36           0           1         0 0.00% 0.00% 0.00%   0 Crash writer

37           0           1         0 0.00% 0.00% 0.00%   0 Exception contro

38           0           3         0 0.00% 0.00% 0.00%   0 License IPC stat

39           0           1         0 0.00% 0.00% 0.00%   0 License IPC serv

40     170000     9637314         17 0.00% 0.00% 0.00%   0 GraphIt

41       3055         454       6729 0.00% 0.00% 0.00%   0 crypto sw pk pro

42           0           2         0 0.00% 0.00% 0.00%   0 SMART

43           0           2         0 0.00% 0.00% 0.00%   0 XML Proxy Client

44          0           1         0 0.00% 0.00% 0.00%   0 ARP Snoop

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

45       82537     9545454         8 0.00% 0.00% 0.00%   0 Dynamic ARP Insp

46           0           1         0 0.00% 0.00% 0.00%   0 Critical Bkgnd

47     8751222   11437155       765 0.31% 0.11% 0.11%   0 Net Background

48           0           6         0 0.00% 0.00% 0.00%   0 IDB Work

49       23237     427516         54 0.00% 0.00% 0.00%  0 Logger

50       91735     9545699         9 0.00% 0.00% 0.00%   0 TTY Background

51           0           4         0 0.00% 0.00% 0.00%   0 CTS CORE

52         41         66       621 0.00% 0.00% 0.00%   0 SXP CORE

53         16          21       761 0.00% 0.00% 0.00%   0 IF-MGR control p

54         236       10363         22 0.00% 0.00% 0.00%   0 IF-MGR event pro

55           0           4         0 0.00% 0.00% 0.00%   0 IP Admission HA

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

56     2127160   18496529       115 1.43% 2.59% 2.81%   0 Net Input

57     2871545     1957373       1467 0.00% 0.04% 0.05%   0 Compute load avg

58     6837616     171753     39810 0.00% 0.05% 0.05%   0 Per-minute Jobs

59     6492717     9722264       667 0.00% 0.14% 0.18%   0 Per-Second Jobs

60           0           1         0 0.00% 0.00% 0.00%   0 Inode Table Dest

61           0           1         0 0.00% 0.00% 0.00%   0 AggMgr Process

62         241       11495         20 0.00% 0.00% 0.00%   0 Transport Port A

63     1303619     2731833       477 0.00% 0.02% 0.00%   0 HC Counter Timer

64           0           1         0 0.00% 0.00% 0.00%   0 SFF8472

65           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED Identity

66           0           4         0 0.00% 0.00% 0.00%   0 EEM ED MAT

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

67     610244     2038417       299 0.00% 0.00% 0.00%   0 EEM ED ND

68         42         79       531 0.00% 0.00% 0.00%   0 USB Startup

69     508478   183350237         2 0.15% 0.14% 0.14%   0 DownWhenLooped

70           0           1         0 0.00% 0.00% 0.00%   0 Hulc Warm Upgrad

71           0           1         0 0.00% 0.00% 0.00%   0 HRPC hdwl reques

72           0           1         0 0.00% 0.00% 0.00%   0 Hulc EEM Process

73           0           1         0 0.00% 0.00% 0.00%   0 HRPC lpip reques

74           0           2         0 0.00% 0.00% 0.00%   0 HLPIP Sync Proce

75           0           1         0 0.00% 0.00% 0.00%   0 HRPC Multi-FS Sy

76           8         58       137 0.00% 0.00% 0.00%   0 HULC multifs pro

77           0           1         0 0.00% 0.00% 0.00%   0 HRPC hnetwpol re

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

78           0           1         0 0.00% 0.00% 0.00%   0 HRPC EnergyWise

79           0           1         0 0.00% 0.00% 0.00%   0 HRPC actual powe

80           0           1         0 0.00% 0.00% 0.00%   0 HRPC ipadm reque

81       17000     320399         53 0.00% 0.00% 0.00%   0 PSP Timer

82       3122         12     260166 0.00% 0.00% 0.00%   0 X2 HAMM detect p

83           0           1         0 0.00% 0.00% 0.00%   0 Madmax Process

84           0           1         0 0.00% 0.00% 0.00%   0 HRPC asic-stats

85   270958294   193479043       1400 3.35% 3.50% 3.89%   0 RedEarth I2C dri

86   622823829   119951017       5192 4.15% 5.07% 5.12%   0 RedEarth Tx Mana

87   16915003   90168868       187 0.63% 0.32% 0.28%   0 RedEarth Rx Mana

88       26654     1927466         13 0.00% 0.00% 0.00%   0 HULC Thermal Pro

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

89           0           1         0 0.00% 0.00% 0.00%   0 HRPC hsm request

90         126         135       933 0.00% 0.00% 0.00%   0 Stack Mgr

91         797         157       5076 0.00% 0.00% 0.00%   0 Stack Mgr Notifi

92     4110021     1097969       3743 0.00% 0.02% 0.03%   0 Adjust Regions

93     492842     4798973       102 0.00% 0.02% 0.00%  0 hrpc -> response

94     431621   12969555         33 0.00% 0.00% 0.00%   0 hrpc -> request

95     4413600   12503347       352 0.15% 0.04% 0.03%   0 hrpc <- response

96           0           3         0 0.00% 0.00% 0.00%   0 HRPC hcomp reque

97     2229885   48184196         46 0.00% 0.15% 0.14%   0 yeti2_emac_proce

98           0         22         0 0.00% 0.00% 0.00%   0 HULC Device Mana

99     116818     380967       306 0.00% 0.00% 0.00%   0 HRPC hdm non blo

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

100           0           4         0 0.00% 0.00% 0.00%   0 HRPC hdm blockin

101       99913     1915737         52 0.00% 0.00% 0.00%   0 HIPC bkgrd proce

102           0          1         0 0.00% 0.00% 0.00%   0 RTTYS Process

103           0           1         0 0.00% 0.00% 0.00%   0 HRPC hautosecure

104     1044452     3738380       279 0.00% 0.00% 0.00%   0 Hulc Port-Securi

105     649202     3087315      210 0.00% 0.00% 0.00%   0 HRPC hpsecure re

106           0           3         0 0.00% 0.00% 0.00%   0 HRPC hfbm reques

107         241         83       2903 0.00% 0.00% 0.00%   0 hulc cfg mgr mas

108           0           1         0 0.00% 0.00% 0.00%   0 HRPC hrcmd reque

109           0         30         0 0.00% 0.00% 0.00%   0 HRPC emac reques

110         16         10       1600 0.00% 0.00% 0.00%   0 HULC EMAC Proces

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

111           9           7       1285 0.00% 0.00% 0.00%   0 HRPC hulc misc r

112       61631     3203191         19 0.00% 0.00% 0.00%   0 HVLAN main bkgrd

113           0          2         0 0.00% 0.00% 0.00%   0 HVLAN Mapped Vla

114           0           2         0 0.00% 0.00% 0.00%   0 Vlan shutdown Pr

115           8         70       114 0.00% 0.00% 0.00%   0 HRPC vlan reques

116           0           1          0 0.00% 0.00% 0.00%   0 HULC VLAN REF Ba

117       6131     160168         38 0.00% 0.00% 0.00%   0 hulc_xps_process

118     207923     1589715       130 0.00% 0.00% 0.00%   0 HRPC ilp request

119     145404     9545317         15 0.00% 0.00% 0.00%   0 Hulc ILP Alchemy

120     791346     640640       1235 0.00% 0.00% 0.00%   0 Strider Tcam Mem

121       44906     232546       193 0.00% 0.00% 0.00%   0 HRPC hlfm reques

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

122     9188176   238896899         38 0.30% 0.27% 0.32%   0 HLFM address lea

123     131756     9545273         13 0.00% 0.00% 0.00%   0 HLFM aging proce

124       48715     2442975         19 0.00% 0.00% 0.00%   0 HLFM address ret

125       84422       84295       1001 0.00% 0.00% 0.00%   0 HULC PM Vector P

126           0           1         0 0.00% 0.00% 0.00%   0 HPM Msg Retry Pr

127           0           1         0 0.00% 0.00% 0.00%   0 Connection Mgr

128   19244246   188797206       101 0.46% 0.90% 0.95%   0 hpm main process

129       3528       12813       275 0.00% 0.00% 0.00%   0 HPM Stack Sync P

130       9947       30178       329 0.00% 0.00% 0.00%   0 HRPC pm request

131          0           5         0 0.00% 0.00% 0.00%   0 HPM if_num mappi

132   145074491     9637313     15053 1.08% 1.18% 1.27%   0 hpm counter proc

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

133   10254698   11963237       857 0.00% 0.08% 0.08%   0 HRPC pm-counters

134           0           4         0 0.00% 0.00% 0.00%   0 hpm vp events ca

135           0           1         0 0.00% 0.00% 0.00%   0 HRPC hcmp reques

136       20119       4718       4264 0.00% 0.00% 0.00%   0 HCEF ADJ Refresh

137           0           1         0 0.00% 0.00% 0.00%   0 hl3md_rpfq_thrl_

138       48393     4798973         10 0.00% 0.00% 0.00%   0 hl3mm_rp

139           0          1         0 0.00% 0.00% 0.00%   0 HACL Queue Proce

140           0           1         0 0.00% 0.00% 0.00%   0 HRPC acl request

141           9         81       111 0.00% 0.00% 0.00%   0 HACL Acl Manager

142           0           1         0 0.00% 0.00% 0.00%   0 HRPC aim request

143           0           5         0 0.00% 0.00% 0.00%   0 HRPC axiom-mgmt

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

144       52999     4798962         11 0.00% 0.00% 0.00%   0 Hulc Axiom Mgmt

145           0           1         0 0.00% 0.00% 0.00%   0 HRPC backup inte

146       1885       32173         58 0.00% 0.00% 0.00%   0 OBFL TEMP obfl0

147           0           1         0 0.00% 0.00% 0.00%   0 HRPC cdp request

148           0           2         0 0.00% 0.00% 0.00%   0 HULC CISP Proces

149     114619     9552706         11 0.00% 0.00% 0.00%   0 Hulc CTS Process

150           0           1         0 0.00% 0.00% 0.00%   0 HRPC cts request

151           0           1         0 0.00% 0.00% 0.00%   0 HULC Msg Retry P

152       2898       18737       154 0.00% 0.00% 0.00%   0 HRPC dot1x reque

153           0           1         0 0.00% 0.00% 0.00%   0 Dot1X Msg Retry

154       8778       29486       297 0.00% 0.00% 0.00%   0 HULC DOT1X Proce

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

155           0           1         0 0.00% 0.00% 0.00%   0 HRPC epm vlan pl

156           0          3         0 0.00% 0.00% 0.00%   0 HRPC fmd request

157           0           2         0 0.00% 0.00% 0.00%   0 HULC FMD Config

158           0           1         0 0.00% 0.00% 0.00%   0 HRPC foxhound re

159           0           2          0 0.00% 0.00% 0.00%   0 Hulc Foxhound Pr

160           0           1         0 0.00% 0.00% 0.00%   0 HRPC lldp reques

161           0           2         0 0.00% 0.00% 0.00%   0 MACsec Process

162           0           1         0 0.00% 0.00% 0.00%   0 HRPC macsec requ

163           0           3         0 0.00% 0.00% 0.00%   0 HRPC msp request

164           0           1         0 0.00% 0.00% 0.00%   0 HULC MSP Snoop P

165          0           3         0 0.00% 0.00% 0.00%   0 HRPC system mtu

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

166           0           5         0 0.00% 0.00% 0.00%   0 HRPC rep request

167           9           4      2250 0.00% 0.00% 0.00%   0 REP Helper Proc

168           0           1         0 0.00% 0.00% 0.00%   0 HRPC sdm request

169           0           1         0 0.00% 0.00% 0.00%   0 SMI MSG Retry Pr

170           8           1       8000 0.00% 0.00% 0.00%   0 HRPC Smart Insta

171     2022536   47406800         42 0.15% 0.06% 0.09%   0 Hulc Storm Contr

172           0           2         0 0.00% 0.00% 0.00%   0 HSTP Sync Proces

173           0           1         0 0.00% 0.00% 0.00%   0 HRPC stp_cli req

174           0         29         0 0.00% 0.00% 0.00%   0 HRPC stp_state_s

175           0           2         0 0.00% 0.00% 0.00%   0 S/W Bridge Proce

176           0         17         0 0.00% 0.00% 0.00%  0 HRPC hudld reque

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

177           0           1         0 0.00% 0.00% 0.00%   0 HRPC vqpc reques

178           0           1         0 0.00% 0.00% 0.00%   0 HRPC iec_load_ba

179       68128     9545293         7 0.00% 0.00% 0.00%   0 IEC Load balance

180           0           1         0 0.00% 0.00% 0.00%   0 HRPC l2pt qnq rp

181           0         18         0 0.00% 0.00% 0.00%   0 HRPC hled reques

 

182 2039427150   239961593       8498 21.91% 22.87% 21.92%   0 Hulc LED Process

 

183           0           1         0 0.00% 0.00% 0.00%   0 HRPC hl3mm reque

184     139280     6640865         20 0.15% 0.01% 0.00%   0 hl3mm

185     817818     7037532        116 0.15% 0.03% 0.00%   0 HL3U bkgrd proce

186     368792     3854799         95 0.00% 0.01% 0.00%   0 HRPC hl3u reques

187     195389     2199130         88 0.00% 0.00% 0.00%   0 HIPV6 bkgrd proc

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

188     376255     3854795         97 0.00% 0.01% 0.00%   0 HRPC IPv6 Unicas

189       39611     2199054         18 0.00% 0.00% 0.00%   0 HL3U PBR bkgrd p

190       1651       80303         20 0.00% 0.00% 0.00%   0 HL3U PBR n-h res

191           0           1         0 0.00% 0.00% 0.00%   0 HRPC obfl reques

192           0           3         0 0.00% 0.00% 0.00%   0 OBFL Cfg Dispatc

193          0           1         0 0.00% 0.00% 0.00%   0 HRPC dtp request

194           0           1         0 0.00% 0.00% 0.00%   0 HRPC hmedianet p

195           0           1         0 0.00% 0.00% 0.00%   0 HRPC show_forwar

196           0          1         0 0.00% 0.00% 0.00%   0 HRPC snmp reques

197           0           1         0 0.00% 0.00% 0.00%   0 HULC SNMP Proces

198   48175636     1915545     25149 0.30% 0.45% 0.47%   0 HQM Stack Proces

PID Runtime(ms)     Invoked    uSecs   5Sec   1Min   5Min TTY Process

199   26974889     7651862       3525 0.15% 0.19% 0.19%   0 HRPC qos request

200           0           1         0 0.00% 0.00% 0.00%   0 HRPC span reques

201           0           8         0 0.00% 0.00% 0.00%   0 HRPC system post

202           8           9       888 0.00% 0.00% 0.00%   0 Hulc Reload Mana

203         186         443       419 0.00% 0.00% 0.00%   0 HRPC hrcli-event

204       4308       53541         80 0.00% 0.00% 0.00%   0 HCMP sync proces

205           0           2         0 0.00% 0.00% 0.00%   0 image mgr

206           0           1         0 0.00% 0.00% 0.00%   0 HRFS OIR Proc

207         77         48       1604 0.00% 0.00% 0.00%   0 HL2MCM

208          66         24       2750 0.00% 0.00% 0.00%   0 HL2MCM

209           0           6         0 0.00% 0.00% 0.00%   0 Hulc SM10G Infra

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

210           0           2        0 0.00% 0.00% 0.00%   0 SM RBCP Monitor

211         60         395       151 0.00% 0.00% 0.00%   0 AAA Server

212         60         468       128 0.00% 0.00% 0.00%   0 AAA ACCT Proc

213           0           1         0 0.00% 0.00% 0.00%   0 ACCT Periodic Pr

214           0           1         0 0.00% 0.00% 0.00%   0 AAA System Acct

215           0           1         0 0.00% 0.00% 0.00%   0 Webauth client

216           0           1         0 0.00% 0.00% 0.00%   0 Auth-proxy HTTP

217           0           1         0 0.00% 0.00% 0.00%   0 IP Admin SM Proc

218   19698787   20594265       956 0.15% 0.25% 0.26%   0 CDP Protocol

219           0           1         0 0.00% 0.00% 0.00%   0 IPv6 Access Cont

220     331601     589927       562 0.00% 0.00% 0.00%   0 mdns Timer Proce

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

221           0           2        0 0.00% 0.00% 0.00%   0 AAA Dictionary R

222       4078       80312         50 0.00% 0.00% 0.00%   0 DHCP Snooping

223           0           1         0 0.00% 0.00% 0.00%   0 DHCP Snooping db

224           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Mgr Proces

225           0           1         0 0.00% 0.00% 0.00%   0 EAP Framework

226           0           1         0 0.00% 0.00% 0.00%   0 EAP Test

227     212322     9545286         22 0.00% 0.00% 0.00%   0 FHRP Main thread

228           0           1         0 0.00% 0.00% 0.00%   0 TRACK Main threa

229           0           1         0 0.00% 0.00% 0.00%   0 TRACK Client thr

230           0           1         0 0.00% 0.00% 0.00%   0 VRRP Main thread

231     4454206   142408722         31 0.00% 0.15% 0.22%   0 VRRS Main thread

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

232           0           2         0 0.00% 0.00% 0.00%   0 CEF switching ba

233       80137       81663       981 0.00% 0.00% 0.00%   0 IP ARP Adjacency

234     5962724   238894645         24 0.15% 0.27% 0.27%   0 IP ARP Retry Age

235     8080754   19460065       415 0.00% 0.17% 0.18%   0 IP Input

236           0           1          0 0.00% 0.00% 0.00%   0 ICMP event handl

237     345866   18511291         18 0.00% 0.00% 0.00%   0 IP ARP Track

238       17301     9567777         1 0.00% 0.00% 0.00%   0 fnvRootHubPortSt

239           0           1         0 0.00% 0.00% 0.00%   0 tHUB

240     348694   94715185         3 0.31% 0.11% 0.10%   0 MDFS MFIB Proces

241           0           1         0 0.00% 0.00% 0.00%   0 ADJ NSF process

242           0           1         0 0.00% 0.00% 0.00%   0 IPv6 ping proces

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

243           0           2         0 0.00% 0.00% 0.00%   0 KRB5 AAA

244           8           2       4000 0.00% 0.00% 0.00%   0 MKA

245         93         168       553 0.00% 0.00% 0.00%   0 SpanTree Flush

246           0           2         0 0.00% 0.00% 0.00%   0 REP Topology cha

247           0           2         0 0.00% 0.00% 0.00%   0 RIB LM VALIDATE

248     100711     9546318         10 0.00% 0.00% 0.00%   0 mDNS

249         17         32       531 0.00% 0.00% 0.00%   0 SMI Director DB

250     288336     1784723       161 0.00% 0.00% 0.00%   0 SMI CDP Update H

251           9           1       9000 0.00% 0.00% 0.00%   0 SMI Backup Proce

252           0           2         0 0.00% 0.00% 0.00%   0 SMI IBC server p

253           0           1         0 0.00% 0.00% 0.00%   0 SMI IBC client p

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

254           0           2         0 0.00% 0.00% 0.00%   0 SMI IBC Download

255   23234445   70213416       330 0.15% 0.46% 0.51%   0 Spanning Tree

256       21493     160619       133 0.00% 0.00% 0.00%   0 Spanning Tree St

257           0           2         0 0.00% 0.00% 0.00%   0 CDP Forward Proc

258       2281       16485       138 0.00% 0.00% 0.00%   0 802.1x switch

259     211222     645780       327 0.00% 0.00% 0.00%   0 DTP Protocol

260           0           3         0 0.00% 0.00% 0.00%   0 HRPC IPv6 Host r

261           0           5         0 0.00% 0.00% 0.00%   0 IPv6 Platform Ho

262           0           1         0 0.00% 0.00% 0.00%   0 axiom-module cra

263           0           1         0 0.00% 0.00% 0.00%   0 HRPC dai request

264           0           1         0 0.00% 0.00% 0.00%   0 HULC DAI Process

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

265           0           1         0 0.00% 0.00% 0.00%   0 HRPC power down

266           0           1         0 0.00% 0.00% 0.00%   0 HRPC ip device t

267           0           1         0 0.00% 0.00% 0.00%   0 HRPC ip source g

268           0           1         0 0.00% 0.00% 0.00%   0 HULC IP Source g

269           0           3         0 0.00% 0.00% 0.00%   0 HRPC rsvp reques

270       12085       3484       3468 0.00% 0.00% 0.00%   0 SpanTree Helper

271     999930     9545260       104 0.00% 0.09% 0.10%   0 PI MATM Aging Pr

272     1464790   96849497         15 0.00% 0.22% 0.33%   0 UDLD

273     438522     5103276         85 0.00% 0.00% 0.00%   0 Port-Security

274           0           2        0 0.00% 0.00% 0.00%   0 Switch Backup In

275           0           2         0 0.00% 0.00% 0.00%   0 IP Host Track Pr

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

276           0           1         0 0.00% 0.00% 0.00%   0 Link State Group

277       4832     160167         30 0.00% 0.00% 0.00%   0 MMN bkgrd proces

278       33381     960958         34 0.00% 0.00% 0.00%   0 Ethchnl

279       13928       3586       3883 0.00% 0.00% 0.00%   0 VMATM Callback

280           0           1         0 0.00% 0.00% 0.00%   0 XDR background p

281       76228     348800       218 0.00% 0.00% 0.00%   0 XDR mcast

282           0           3         0 0.00% 0.00% 0.00%   0 XDR RP Ping Back

283     446129     897096       497 0.00% 0.00% 0.00%   0 XDR receive

284         16           4       4000 0.00% 0.00% 0.00%   0 IPC LC Message H

285           0           1         0 0.00% 0.00% 0.00%   0 XDR RP Test Back

286       5371     160167         33 0.00% 0.00% 0.00%   0 FRR Background P

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

287     119371     674593       176 0.00% 0.00% 0.00%   0 CEF background p

288           0           1         0 0.00% 0.00% 0.00%   0 fib_fib_bfd_sb e

289           0           1         0 0.00% 0.00% 0.00%   0 IP IRDP

290           0           1         0 0.00% 0.00% 0.00%   0 Tunnel FIB

291           0           2         0 0.00% 0.00% 0.00%   0 Tunnel

292           0           1         0 0.00% 0.00% 0.00%   0 IPC UDP Input

293           0           1         0 0.00% 0.00% 0.00%   0 CEF RF HULC Conv

294       31709     961052         32 0.00% 0.00% 0.00%   0 IP ACL XDR LC Ba

295     180424     4790387         37 0.00% 0.00% 0.00%   0 CEF: IPv4 proces

296     2793044     9557736       292 0.00% 0.06% 0.05%   0 ADJ background

297     783353   10665500         73 0.00% 0.00% 0.00%   0 MDFS LC Process

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

298           0           2         0 0.00% 0.00% 0.00%   0 AN

299           0           1         0 0.00% 0.00% 0.00%   0 AN HELPER

300           0           1         0 0.00% 0.00% 0.00%   0 AN CONFIG DOWNLO

301           0           2         0 0.00% 0.00% 0.00%   0 Routing Topology

302       14304       5842       2448 0.00% 0.00% 0.00%   0 IP RIB Update

303     491378       8023     61246 0.00% 0.00% 0.00%   0 NIST rng proc

304     357269   18511297         19 0.00% 0.00% 0.00%   0 Timer Library

305           0           2         0 0.00% 0.00% 0.00%   0 App Route Proces

306           0           1         0 0.00% 0.00% 0.00%   0 IPv6 RIB Cleanup

307           0           1         0 0.00% 0.00% 0.00%   0 IPv6 RIB Event H

308         931       11211         83 0.00% 0.00% 0.00%   0 Auth Manager

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

309           0          1         0 0.00% 0.00% 0.00%   0 Critical Auth

310           0           1         0 0.00% 0.00% 0.00%   0 EPM MAIN PROCESS

311           0           1         0 0.00% 0.00% 0.00%   0 Timer Handler Pr

312           0           1        0 0.00% 0.00% 0.00%   0 EPM Downlad Mana

313     108885     1916098         56 0.00% 0.00% 0.00%   0 EPM ACL PLUG-IN

314     319985   19022800         16 0.00% 0.00% 0.00%   0 SVM UT Process

315           0           1         0 0.00% 0.00% 0.00%   0 CMD HANDLER

316         411       4415         93 0.00% 0.00% 0.00%   0 TCP Timer

317       6123       9293       658 0.00% 0.00% 0.00%   0 TCP Protocols

318       24378     9669446         2 0.00% 0.00% 0.00%   0 Socket Timers

319         476       32154         14 0.00% 0.00% 0.00%   0 HTTP CORE

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

320       1346     120461         11 0.00% 0.00% 0.00%   0 Cluster L2

321       38325     960959         39 0.00% 0.00% 0.00%   0 Cluster RARP

322     153135     1524247       100 0.00% 0.00% 0.00%   0 Cluster Base

323           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Supplicant

324           0          2         0 0.00% 0.00% 0.00%   0 Dot1x Supplicant

325           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Supplicant

326           0           1         0 0.00% 0.00% 0.00%   0 DSensor HTTP

327           0           2        0 0.00% 0.00% 0.00%   0 CEF RP Backgroun

328         176       8800         20 0.00% 0.00% 0.00%   0 Flow Exporter Ti

329           0           1         0 0.00% 0.00% 0.00%   0 DCT Bkgd Process

330           0           4         0 0.00% 0.00% 0.00%   0 MFIB Master back

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

331           0           2         0 0.00% 0.00% 0.00%   0 ENABLE AAA

332         888       8818       100 0.00% 0.00% 0.00%   0 MVPN Mgr Process

333           0           2         0 0.00% 0.00% 0.00%   0 Multicast Offloa

334       1975       9282       212 0.00% 0.00% 0.00%   0 RARP Input

335           9           7       1285 0.00% 0.00% 0.00%   0 static

336     1711242     618410       2767 0.00% 0.00% 0.00%   0 ADJ resolve proc

337           0           1         0 0.00% 0.00% 0.00%   0 IPv6 Static Hand

338           0           1         0 0.00% 0.00% 0.00%   0 MAB Framework

339           0          2         0 0.00% 0.00% 0.00%   0 Media Services P

340     513274     963412       532 0.00% 0.00% 0.00%   0 QoS stats proces

341           0           1         0 0.00% 0.00% 0.00%   0 LICENSE AGENT

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

342           0           2         0 0.00% 0.00% 0.00%   0 REP LSL Proc

343           0           2         0 0.00% 0.00% 0.00%   0 REP BPA/EPA Proc

344           0           5         0 0.00% 0.00% 0.00%   0 SNMP Timers

345           0           1         0 0.00% 0.00% 0.00%   0 HRPC hwccp reque

346           0           3         0 0.00% 0.00% 0.00%   0 HULC_WCCP_BACKGR

347           0           1         0 0.00% 0.00% 0.00%   0 HRPC dhcp snoopi

348           0           5         0 0.00% 0.00% 0.00%   0 HULC DHCP Snoopi

349           0           1         0 0.00% 0.00% 0.00%   0 HRPC sisf reques

350           8           3       2666 0.00% 0.00% 0.00%   0 HULC SISF Proces

351           0           1         0 0.00% 0.00% 0.00%   0 HULC SISF Source

352     4493831   19745136       227 0.15% 0.10% 0.11%   0 Inline Power

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

353     6282892   21095284       297 0.63% 0.27% 0.28%   0 Marvell wk-a Pow

354       65642     480494       136 0.00% 0.00% 0.00%   0 HULC stack power

355       20919     320321         65 0.00% 0.00% 0.00%   0 HRPC stack power

356       16431       15711      1045 0.00% 0.00% 0.00%   0 IGMPSN L2MCM

357           0           1         0 0.00% 0.00% 0.00%   0 IGMPSN MRD

358     111500     676864       164 0.00% 0.00% 0.00%   0 IGMPSN

359           0           1         0 0.00% 0.00% 0.00%   0 IGMPQR

360           0           3         0 0.00% 0.00% 0.00%   0 L2TRACE SERVER

361           0           2         0 0.00% 0.00% 0.00%   0 AUTO LAG Protoco

362       12859       15699       819 0.00% 0.00% 0.00%   0 MLDSN L2MCM

363          0           1         0 0.00% 0.00% 0.00%   0 MRD

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

364           0           1         0 0.00% 0.00% 0.00%   0 MLD_SNOOP

365           0           1         0 0.00% 0.00% 0.00%   0 TDP Discovery

366     2271936   93447201         24 0.00% 0.10% 0.11%   0 MDFS RP process

367           0           1         0 0.00% 0.00% 0.00%   0 EM Background Pr

368           0           2         0 0.00% 0.00% 0.00%  0 AAA Cached Serve

369           0           2         0 0.00% 0.00% 0.00%   0 MRIB Process

370       19938       97052       205 0.00% 0.00% 0.00%   0 HRPC hl2mcm igmp

371         25         90       277 0.00% 0.00% 0.00%   0 HRPC hl2mcm mlds

372           0           3         0 0.00% 0.00% 0.00%   0 LDAP process

373           0           2         0 0.00% 0.00% 0.00%   0 Crypto Support

374           9           1       9000 0.00% 0.00% 0.00%   0 IPSECv6 PS Proc

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

375           0           1         0 0.00% 0.00% 0.00%   0 Key chain liveke

376         61         278       219 0.00% 0.00% 0.00%   0 Collection proce

377           0          2         0 0.00% 0.00% 0.00%   0 LINE AAA

378         36         71       507 0.00% 0.00% 0.00%   0 LOCAL AAA

379     4318068   142408616         30 0.15% 0.18% 0.21%   0 MMA DB TIMER

380           8           1       8000 0.00% 0.00% 0.00%   0 MMA MSG MCOL

381           0           1         0 0.00% 0.00% 0.00%   0 Opaque Database

382         34         755         45 0.00% 0.00% 0.00%   0 TACACS+

383       3683       4438       829 0.00% 0.00% 0.00%   0 TPLUS

384     1181450     480481       2458 0.00% 0.00% 0.00%   0 HULC ISM FNF

385     120317       12370       9726 0.00% 0.00% 0.00%   0 hulc running con

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

386         427        33     12939 0.00% 0.00% 0.00%   0 crypto engine pr

387           0           1         0 0.00% 0.00% 0.00%   0 RSA background p

388         820       10714         76 0.00% 0.00% 0.00%   0 Crypto CA

389           0           1         0 0.00% 0.00% 0.00%   0 Crypto PKI-CRL

390     357905     9597571         37 0.00% 0.01% 0.00%   0 Crypto IKEv2

391           0           1         0 0.00% 0.00% 0.00%   0 IKEv2 AAA handle

392           0           1         0 0.00% 0.00% 0.00%   0 encrypt proc

393           0           2         0 0.00% 0.00% 0.00%   0 CRYPTO MAP FREE

394           0           1         0 0.00% 0.00% 0.00%   0 Crypto INT

395           0           2         0 0.00% 0.00% 0.00%   0 Crypto IKE Dispa

396           0           3         0 0.00% 0.00% 0.00%   0 Crypto IKMP

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

397           0           1         0 0.00% 0.00% 0.00%   0 IPSEC manual key

398       25451    480482         52 0.00% 0.00% 0.00%   0 IPSEC key engine

399           0           5         0 0.00% 0.00% 0.00%   0 Crypto ACL

400           0           1         0 0.00% 0.00% 0.00%   0 Crypto PAS Proc

401           0           5         0 0.00% 0.00% 0.00%   0 IPSec background

402           0           2         0 0.00% 0.00% 0.00%   0 MMON PROCESS

403     4351735   142408695         30 0.31% 0.16% 0.21%   0 MMA DP TIMER

404     4492904 299961959         14 0.00% 0.14% 0.12%   0 MMON MENG

405           0           1         0 0.00% 0.00% 0.00%   0 HRPC x_setup req

406           0           2         0 0.00% 0.00% 0.00%   0 REP Switch Helpe

407           0           1          0 0.00% 0.00% 0.00%   0 Licensing MIB pr

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

408           0         23         0 0.00% 0.00% 0.00%   0 VTP Trap Process

409         67           5     13400 0.00% 0.00% 0.00%   0 OBFL MSG obfl0

410         267       8934         29 0.00% 0.00% 0.00%   0 ASP Process Crea

411         57         391       145 0.00% 0.00% 0.00%   0 AAA SEND STOP EV

412           0           1         0 0.00% 0.00% 0.00%   0 Test AAA Client

413           8           8       1000 0.00% 0.00% 0.00%   0 DCM Core Thread

414           0           2         0 0.00% 0.00% 0.00%   0 DHCP Security He

415           0           1         0 0.00% 0.00% 0.00%   0 dcm_cli_engine

416           0           3         0 0.00% 0.00% 0.00%   0 dcm_cli_provider

417           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED Generic

418           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED Routing

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

419       20623     221652         93 0.00% 0.00% 0.00%   0 EEM ED Syslog

420           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Media Ser

421           0           1        0 0.00% 0.00% 0.00%   0 Syslog Traps

422         67           5     13400 0.00% 0.00% 0.00%   0 OBFL ENV obfl0

423       6040     160167         37 0.00% 0.00% 0.00%   0 vlmap logger

424           0           1         0 0.00% 0.00% 0.00%   0 HCD Process

425           0           1         0 0.00% 0.00% 0.00%   0 HRPC cable diagn

426           0           1         0 0.00% 0.00% 0.00%   0 HRPC archive_dnl

427           0           1         0 0.00% 0.00% 0.00%   0 HRPC rsu request

428           0           2         0 0.00% 0.00% 0.00%   0 DiagCard9/-1

429           0           1         0 0.00% 0.00% 0.00%   0 Online Diag EEM

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

430     2327145   80270462         28 0.15% 0.06% 0.07%   0 PM Callback

432       8265     770432         10 0.00% 0.00% 0.00%   0 dhcp snooping sw

434         60         106       566 0.00% 0.00% 0.00%   0 EEM Server

435         17          3       5666 0.00% 0.00% 0.00%   0 Call Home proces

436         49         131       374 0.00% 0.00% 0.00%   0 EEM Helper Threa

437       11050     160193         68 0.00% 0.00% 0.00%   0 Call Home Timer

438       81341     961300        84 0.00% 0.00% 0.00%   0 Bulkstat-Client

439           0           4         0 0.00% 0.00% 0.00%   0 CTS credentials

440           0           1         0 0.00% 0.00% 0.00%   0 HRPC eee request

441           0           6         0 0.00% 0.00% 0.00%   0 DCM snmp dp Thre

442           8           3       2666 0.00% 0.00% 0.00%   0 snmp dcm ma shim

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

443           0           2         0 0.00% 0.00% 0.00%   0 EEM Policy Direc

445           9           3       3000 0.00% 0.00% 0.00%   0 EEM ED CLI

446           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Config

447           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Counter

448           0           4         0 0.00% 0.00% 0.00%   0 EEM ED CRASH

449           0           4         0 0.00% 0.00% 0.00%   0 EEM ED DS

450           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Env

451         17           4      4250 0.00% 0.00% 0.00%   0 EM ED GOLD

452           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Interface

453           0           4         0 0.00% 0.00% 0.00%   0 EEM ED IOSWD

454           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Ipsla

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

455           0           4         0 0.00% 0.00% 0.00%   0 EEM ED None

456           0           4         0 0.00% 0.00% 0.00%   0 EEM ED OIR

457           0           2         0 0.00% 0.00% 0.00%   0 STP FAST TRANSIT

458           0           4         0 0.00% 0.00% 0.00%   0 EEM ED SNMP

459           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED SNMP Obje

460           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED SNMP Noti

461           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Test

462       9016     210491         42 0.00% 0.00% 0.00%   0 EEM ED Timer

463       58035     215404      269 0.00% 0.00% 0.00%   0 Syslog

464       27605     640628         43 0.00% 0.00% 0.00%   0 OBFL I/O Buffer

465           0           6         0 0.00% 0.00% 0.00%   0 RBM CORE

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

466           0           5         0 0.00% 0.00% 0.00%   0 CSRT RAPID TRANS

467         24           6       4000 0.00% 0.00% 0.00%   0 VLAN Manager

468         51       2680         19 0.00% 0.00% 0.00%   0 SSH Event handle

469           0           1         0 0.00% 0.00% 0.00%   0 DiagCard8/-1

470       14667     169051         86 0.00% 0.00% 0.00%   0 IP Background

471       1714       14150       121 0.00% 0.00% 0.00%   0 IP Connected Rou

472     1080191   10019513       107 0.00% 0.02% 0.00%   0 DHCPD Receive

473     2511654   96618573         25 0.00% 0.05% 0.12%   0 IGMP Input

474     5425656   206263986         26 0.15% 0.23% 0.27%   0 PIM Process

475     1869316   96220812         19 0.15% 0.12% 0.10%   0 Mwheel Process

476           0           1         0 0.00% 0.00% 0.00%   0 mcast_rw_process

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

477     305592     9717654         31 0.00% 0.01% 0.00%   0 NTP

479       44049     109179       403 0.00% 0.00% 0.00%   0 IP SNMP

480       41311       74936       551 0.00% 0.00% 0.00%   0 PDU DISPATCHER

481     135036       75081       1798 0.00% 0.00% 0.00%   0 SNMP ENGINE

482           0          2         0 0.00% 0.00% 0.00%   0 IP SNMPV6

483           0           1         0 0.00% 0.00% 0.00%   0 SNMP ConfCopyPro

484         26           7       3714 0.00% 0.00% 0.00%   0 SNMP Traps

485       1476       82982         17 0.00% 0.00% 0.00%   0 DHCPD Timer

486       5216     160166         32 0.00% 0.00% 0.00%   0 DHCPD Database

488     690433     4182565       165 0.00% 0.00% 0.00%   0 EIGRP-IPv4

489     1066113     8307387       128 0.15% 0.02% 0.00%   0 EIGRP-IPv4 Hello

 

 

 

 

AFTER when more stable:

3750STACK#sh processes cpu

CPU utilization for five seconds: 36%/0%; one minute: 40%; five minutes: 40%

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

   1       11102     132585         83 0.00% 0.00% 0.00%   0 Chunk Manager

   2       73261     1927939         37 0.00% 0.00% 0.00%   0 Load Meter

   3       2642         471       5609 0.00% 0.00% 0.00%   1 SSH Process

   4   240755181   10265024     23453 0.00% 2.03% 2.25%   0 Check heaps

   5       3240       46607         69 0.00% 0.00% 0.00%   0 Pool Manager

   6           0           1         0 0.00% 0.00% 0.00%   0 DiscardQ Backgro

   7           0           2         0 0.00% 0.00% 0.00%   0 Timers

   8         664       31374         21 0.00% 0.00% 0.00%   0 WATCH_AFS

   9     599331   38557806         15 0.00% 0.00% 0.00%   0 HUSB Console

10           8           5       1600 0.00% 0.00% 0.00%   0 License Client N

11   26820064     160201     167416 0.00% 0.28% 0.23%   0 Licensing Auto U

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

12         227         12     18916 0.00% 0.00% 0.00%   0 Image License br

13   16877935   27065967       623 0.00% 0.14% 0.12%   0 ARP Input

14     598883     9955061         60 0.00% 0.00% 0.00%   0 ARP Background

15           0           1         0 0.00% 0.00% 0.00%   0 CEF MIB API

16           0           2         0 0.00% 0.00% 0.00%   0 AAA_SERVER_DEADT

17           0           1         0 0.00% 0.00% 0.00%   0 Policy Manager

18         75         19       3947 0.00% 0.00% 0.00%   0 Entity MIB API

19           0           1         0 0.00% 0.00% 0.00%   0 IFS Agent Manage

20       68794     1916243         35 0.00% 0.00% 0.00%   0 IPC Event Notifi

21     106735     9332471         11 0.00% 0.00% 0.00%   0 IPC Mcast Pendin

22       5999     160209         37 0.00% 0.00% 0.00%   0 IPC Dynamic Cach

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

23           0           1         0 0.00% 0.00% 0.00%   0 IPC Session Serv

24           0          1         0 0.00% 0.00% 0.00%   0 IPC Zone Manager

25     112380     9332500         12 0.00% 0.00% 0.00%   0 IPC Periodic Tim

26     108540     9332467         11 0.00% 0.00% 0.00%   0 IPC Deferred Por

27           0           1          0 0.00% 0.00% 0.00%   0 IPC Process leve

28     993511     4948165       200 0.00% 0.00% 0.00%   0 IPC Seat Manager

29       5110     550518         9 0.00% 0.00% 0.00%   0 IPC Check Queue

30           8         25       320 0.00% 0.00% 0.00%   0 IPC Seat RX Cont

31           0           1         0 0.00% 0.00% 0.00%   0 IPC Seat TX Cont

32       9944     963993         10 0.00% 0.00% 0.00%   0 IPC Keep Alive M

33       53393     1922426         27 0.00% 0.00% 0.00%   0 IPC Loadometer

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

34           0           1         0 0.00% 0.00% 0.00%   0 IPC Session Deta

35         133           6     22166 0.00% 0.00% 0.00%   0 PrstVbl

36           0           1         0 0.00% 0.00% 0.00%   0 Crash writer

37           0           1         0 0.00% 0.00% 0.00%   0 Exception contro

38           0           3         0 0.00% 0.00% 0.00%   0 License IPC stat

39          0           1         0 0.00% 0.00% 0.00%   0 License IPC serv

40     170126     9639694         17 0.00% 0.00% 0.00%   0 GraphIt

41       3055         454       6729 0.00% 0.00% 0.00%   0 crypto sw pk pro

42           0           2          0 0.00% 0.00% 0.00%   0 SMART

43           0           2         0 0.00% 0.00% 0.00%   0 XML Proxy Client

44           0           1         0 0.00% 0.00% 0.00%   0 ARP Snoop

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

45       82571     9547807         8 0.00% 0.00% 0.00%   0 Dynamic ARP Insp

46           0           1         0 0.00% 0.00% 0.00%   0 Critical Bkgnd

47     8753885   11440407       765 0.00% 0.09% 0.06%   0 Net Background

48           0           6         0 0.00% 0.00% 0.00%   0 IDB Work

49       23245     427602         54 0.00% 0.00% 0.00%   0 Logger

50       91769     9548053         9 0.00% 0.00% 0.00%   0 TTY Background

51           0           4        0 0.00% 0.00% 0.00%   0 CTS CORE

52         41         66       621 0.00% 0.00% 0.00%   0 SXP CORE

53         16         21       761 0.00% 0.00% 0.00%   0 IF-MGR control p

54         236       10401         22 0.00% 0.00% 0.00%   0 IF-MGR event pro

55           0           4         0 0.00% 0.00% 0.00%   0 IP Admission HA

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

56     2141667   18587944       115 0.00% 0.00% 0.00%   0 Net Input

57     2872297     1958085       1466 0.00% 0.00% 0.00%   0 Compute load avg

58     6839507     171806     39809 0.00% 0.07% 0.05%   0 Per-minute Jobs

59     6494536     9725346       667 0.00% 0.02% 0.00%   0 Per-Second Jobs

60          0           1         0 0.00% 0.00% 0.00%   0 Inode Table Dest

61           0           1         0 0.00% 0.00% 0.00%   0 AggMgr Process

62         241       11502         20 0.00% 0.00% 0.00%   0 Transport Port A

63     1304070     2732518       477 0.00% 0.00% 0.00%   0 HC Counter Timer

64           0           1         0 0.00% 0.00% 0.00%   0 SFF8472

65           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED Identity

66           0           4         0 0.00% 0.00% 0.00%   0 EEM ED MAT

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

67     610445     2039005       299 0.00% 0.00% 0.00%   0 EEM ED ND

68         42         79       531 0.00% 0.00% 0.00%   0 USB Startup

69     509622   183394994         2 0.00% 0.00% 0.00%   0 DownWhenLooped

70           0           1         0 0.00% 0.00% 0.00%   0 Hulc Warm Upgrad

71           0           1         0 0.00% 0.00% 0.00%   0 HRPC hdwl reques

72          0           1         0 0.00% 0.00% 0.00%   0 Hulc EEM Process

73           0           1         0 0.00% 0.00% 0.00%   0 HRPC lpip reques

74           0           2         0 0.00% 0.00% 0.00%   0 HLPIP Sync Proce

75           0           1         0 0.00% 0.00% 0.00%   0 HRPC Multi-FS Sy

76           8         58       137 0.00% 0.00% 0.00%   0 HULC multifs pro

77           0           1         0 0.00% 0.00% 0.00%   0 HRPC hnetwpol re

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

78           0           1         0 0.00% 0.00% 0.00%   0 HRPC EnergyWise

79           0           1         0 0.00% 0.00% 0.00%   0 HRPC actual powe

80           0           1        0 0.00% 0.00% 0.00%   0 HRPC ipadm reque

81       17000     320479         53 0.00% 0.00% 0.00%   0 PSP Timer

82       3122         12     260166 0.00% 0.00% 0.00%   0 X2 HAMM detect p

83           0           1         0 0.00% 0.00% 0.00%   0 Madmax Process

84           0           1         0 0.00% 0.00% 0.00%   0 HRPC asic-stats

85   271029893   193525373       1400 2.24% 2.25% 2.33%   0 RedEarth I2C dri

86   622994511   119979004       5192 6.88% 6.81% 6.70%   0 RedEarth Tx Mana

87   16920487   90190970       187 0.15% 0.15% 0.15%   0 RedEarth Rx Mana

88       26712     1927943         13 0.00% 0.00% 0.00%   0 HULC Thermal Pro

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

89           0           1         0 0.00% 0.00% 0.00%   0 HRPC hsm request

90         126         135       933 0.00% 0.00% 0.00%   0 Stack Mgr

91         797         157       5076 0.00% 0.00% 0.00%   0 Stack Mgr Notifi

92     4110623     1098195       3743 0.00% 0.02% 0.00%   0 Adjust Regions

93     493024     4800159       102 0.00% 0.00% 0.00%   0 hrpc -> response

94     431672   12972009         33 0.00% 0.00% 0.00%   0 hrpc -> request

95     4414522   12505748       352 0.15% 0.02% 0.00%   0 hrpc <- response

96           0           3         0 0.00% 0.00% 0.00%   0 HRPC hcomp reque

97     2231341   48196105         46 0.00% 0.01% 0.00%   0 yeti2_emac_proce

98           0         22         0 0.00% 0.00% 0.00%   0 HULC Device Mana

99     116818     380967       306 0.00% 0.00% 0.00%   0 HRPC hdm non blo

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

100           0           4         0 0.00% 0.00% 0.00%   0 HRPC hdm blockin

101       99971     1916211         52 0.00% 0.00% 0.00%   0 HIPC bkgrd proce

102           0           1         0 0.00% 0.00% 0.00%   0 RTTYS Process

103           0           1         0 0.00% 0.00% 0.00%   0 HRPC hautosecure

104     1044661     3739080       279 0.00% 0.01% 0.00%   0 Hulc Port-Securi

105     649303     3087785       210 0.00% 0.00% 0.00%   0 HRPC hpsecure re

106          0           3         0 0.00% 0.00% 0.00%   0 HRPC hfbm reques

107         241         83       2903 0.00% 0.00% 0.00%   0 hulc cfg mgr mas

108           0           1         0 0.00% 0.00% 0.00%   0 HRPC hrcmd reque

109           0        30         0 0.00% 0.00% 0.00%   0 HRPC emac reques

110         16         10       1600 0.00% 0.00% 0.00%   0 HULC EMAC Proces

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

111           9           7       1285 0.00% 0.00% 0.00%   0 HRPC hulc misc r

112       61689     3203984         19 0.00% 0.00% 0.00%   0 HVLAN main bkgrd

113           0           2         0 0.00% 0.00% 0.00%   0 HVLAN Mapped Vla

114           0           2         0 0.00% 0.00% 0.00%   0 Vlan shutdown Pr

115           8         70       114 0.00% 0.00% 0.00%   0 HRPC vlan reques

116           0           1         0 0.00% 0.00% 0.00%   0 HULC VLAN REF Ba

117       6147     160207         38 0.00% 0.00% 0.00%   0 hulc_xps_process

118     207974     1590209       130 0.00% 0.00% 0.00%   0 HRPC ilp request

119     145439     9547678         15 0.00% 0.00% 0.00%   0 Hulc ILP Alchemy

120     791570     640798       1235 0.00% 0.00% 0.00%   0 Strider Tcam Mem

121       44939     232709       193 0.00% 0.00% 0.00%   0 HRPC hlfm reques

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

122     9191385   238955261         38 0.00% 0.06% 0.06%   0 HLFM address lea

123    131823     9547634         13 0.00% 0.00% 0.00%   0 HLFM aging proce

124       48723     2443579         19 0.00% 0.00% 0.00%   0 HLFM address ret

125       84463       84335       1001 0.00% 0.00% 0.00%   0 HULC PM Vector P

126           0           1         0 0.00% 0.00% 0.00%   0 HPM Msg Retry Pr

127           0           1         0 0.00% 0.00% 0.00%   0 Connection Mgr

128   19254027   188843923       101 0.15% 0.26% 0.20%   0 hpm main process

129       3553       12879       275 0.00% 0.00% 0.00%   0 HPM Stack Sync P

130       9981       30214       330 0.00% 0.01% 0.00%   0 HRPC pm request

131           0           5         0 0.00% 0.00% 0.00%   0 HPM if_num mappi

132   145111882     9639700     15053 1.59% 1.55% 1.48%   0 hpm counter proc

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

133   10257248   11966123       857 0.00% 0.06% 0.06%   0 HRPC pm-counters

134           0           4        0 0.00% 0.00% 0.00%   0 hpm vp events ca

135           0           1         0 0.00% 0.00% 0.00%   0 HRPC hcmp reques

136       20128       4719       4265 0.00% 0.00% 0.00%   0 HCEF ADJ Refresh

137           0           1         0 0.00% 0.00% 0.00%   0 hl3md_rpfq_thrl_

138       48427     4800164         10 0.00% 0.00% 0.00%   0 hl3mm_rp

139           0           1         0 0.00% 0.00% 0.00%   0 HACL Queue Proce

140           0           1         0 0.00% 0.00% 0.00%   0 HRPC acl request

141           9         81       111 0.00% 0.00% 0.00%   0 HACL Acl Manager

142           0           1         0 0.00% 0.00% 0.00%   0 HRPC aim request

143           0           5         0 0.00% 0.00% 0.00%   0 HRPC axiom-mgmt

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

144       53058     4800153         11 0.00% 0.00% 0.00%   0 Hulc Axiom Mgmt

145           0           1         0 0.00% 0.00% 0.00%   0 HRPC backup inte

146        1885       32181         58 0.00% 0.00% 0.00%   0 OBFL TEMP obfl0

147           0           1         0 0.00% 0.00% 0.00%   0 HRPC cdp request

148           0           2         0 0.00% 0.00% 0.00%   0 HULC CISP Proces

149     114687    9555124         12 0.00% 0.00% 0.00%   0 Hulc CTS Process

150           0           1         0 0.00% 0.00% 0.00%   0 HRPC cts request

151           0           1         0 0.00% 0.00% 0.00%   0 HULC Msg Retry P

152       2898       18755       154 0.00% 0.00% 0.00%   0 HRPC dot1x reque

153           0           1         0 0.00% 0.00% 0.00%   0 Dot1X Msg Retry

154       8803       29568       297 0.00% 0.00% 0.00%   0 HULC DOT1X Proce

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

155           0           1         0 0.00% 0.00% 0.00%   0 HRPC epm vlan pl

156           0           3         0 0.00% 0.00% 0.00%   0 HRPC fmd request

157           0          2         0 0.00% 0.00% 0.00%   0 HULC FMD Config

158           0           1         0 0.00% 0.00% 0.00%   0 HRPC foxhound re

159           0           2         0 0.00% 0.00% 0.00%   0 Hulc Foxhound Pr

160           0           1        0 0.00% 0.00% 0.00%   0 HRPC lldp reques

161           0           2         0 0.00% 0.00% 0.00%   0 MACsec Process

162           0           1         0 0.00% 0.00% 0.00%   0 HRPC macsec requ

163           0           3         0 0.00% 0.00% 0.00%   0 HRPC msp request

164           0           1         0 0.00% 0.00% 0.00%   0 HULC MSP Snoop P

165           0           3         0 0.00% 0.00% 0.00%   0 HRPC system mtu

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

166           0           5         0 0.00% 0.00% 0.00%   0 HRPC rep request

167           9           4       2250 0.00% 0.00% 0.00%   0 REP Helper Proc

168           0           1         0 0.00% 0.00% 0.00%   0 HRPC sdm request

169           0           1         0 0.00% 0.00% 0.00%   0 SMI MSG Retry Pr

170           8           1       8000 0.00% 0.00% 0.00%   0 HRPC Smart Insta

171     2023547   47418511         42 0.00% 0.00% 0.00%   0 Hulc Storm Contr

172           0           2         0 0.00% 0.00% 0.00%   0 HSTP Sync Proces

173           0           1         0 0.00% 0.00% 0.00%   0 HRPC stp_cli req

174           0         29         0 0.00% 0.00% 0.00%   0 HRPC stp_state_s

175          0           2         0 0.00% 0.00% 0.00%   0 S/W Bridge Proce

176           0         17         0 0.00% 0.00% 0.00%   0 HRPC hudld reque

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

177           0           1        0 0.00% 0.00% 0.00%   0 HRPC vqpc reques

178           0           1         0 0.00% 0.00% 0.00%   0 HRPC iec_load_ba

179       68169     9547662         7 0.00% 0.00% 0.00%   0 IEC Load balance

180           0           1         0 0.00% 0.00% 0.00%   0 HRPC l2pt qnq rp

181           0         18         0 0.00% 0.00% 0.00%   0 HRPC hled reques

 

182 2040013082   240021082       8499 19.84% 19.85% 19.93%   0 Hulc LED Process

 

183           0           1         0 0.00% 0.00% 0.00%   0 HRPC hl3mm reque

184     139348     6642525         20 0.00% 0.00% 0.00%   0 hl3mm

185     818164     7039309       116 0.00% 0.00% 0.00%   0 HL3U bkgrd proce

186     368965     3855757         95 0.00% 0.00% 0.00%   0 HRPC hl3u reques

187     195471     2199676         88 0.00% 0.00% 0.00%   0 HIPV6 bkgrd proc

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

188     376413     3855753         97 0.00% 0.00% 0.00%   0 HRPC IPv6 Unicas

189       39619     2199602         18 0.00% 0.00% 0.00%   0 HL3U PBR bkgrd p

190       1651       80323         20 0.00% 0.00% 0.00%   0 HL3U PBR n-h res

191           0           1         0 0.00% 0.00% 0.00%   0 HRPC obfl reques

192           0           3         0 0.00% 0.00% 0.00%   0 OBFL Cfg Dispatc

193           0           1         0 0.00% 0.00% 0.00%   0 HRPC dtp request

194           0           1         0 0.00% 0.00% 0.00%   0 HRPC hmedianet p

195           0           1         0 0.00% 0.00% 0.00%   0 HRPC show_forwar

196           0           1         0 0.00% 0.00% 0.00%   0 HRPC snmp reques

197           0           1         0 0.00% 0.00% 0.00%   0 HULC SNMP Proces

198   48188407     1916021     25150 0.47% 0.45% 0.47%   0 HQM Stack Proces

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

199   26982024     7653774       3525 0.31% 0.28% 0.27%   0 HRPC qos request

200          0           1         0 0.00% 0.00% 0.00%   0 HRPC span reques

201           0           8         0 0.00% 0.00% 0.00%   0 HRPC system post

202           8           9       888 0.00% 0.00% 0.00%   0 Hulc Reload Mana

203         186        443       419 0.00% 0.00% 0.00%   0 HRPC hrcli-event

204       4308       53555         80 0.00% 0.00% 0.00%   0 HCMP sync proces

205           0           2         0 0.00% 0.00% 0.00%   0 image mgr

206           0           1        0 0.00% 0.00% 0.00%   0 HRFS OIR Proc

207         77         48       1604 0.00% 0.00% 0.00%   0 HL2MCM

208         66         24       2750 0.00% 0.00% 0.00%   0 HL2MCM

209           0           6         0 0.00% 0.00% 0.00%   0 Hulc SM10G Infra

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

210           0           2         0 0.00% 0.00% 0.00%   0 SM RBCP Monitor

211         60        395       151 0.00% 0.00% 0.00%   0 AAA Server

212         60         470       127 0.00% 0.00% 0.00%   0 AAA ACCT Proc

213           0           1         0 0.00% 0.00% 0.00%   0 ACCT Periodic Pr

214           0           1         0 0.00% 0.00% 0.00%   0 AAA System Acct

215           0           1         0 0.00% 0.00% 0.00%   0 Webauth client

216           0           1         0 0.00% 0.00% 0.00%   0 Auth-proxy HTTP

217           0           1         0 0.00% 0.00% 0.00%   0 IP Admin SM Proc

218   19704230   20599284       956 0.00% 0.20% 0.17%   0 CDP Protocol

219           0           1         0 0.00% 0.00% 0.00%   0 IPv6 Access Cont

220     331711     590088       562 0.00% 0.00% 0.00%   0 mdns Timer Proce

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

221           0           2         0 0.00% 0.00% 0.00%   0 AAA Dictionary R

222       4078       80332         50 0.00% 0.00% 0.00%   0 DHCP Snooping

223          0           1         0 0.00% 0.00% 0.00%   0 DHCP Snooping db

224           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Mgr Proces

225           0           1         0 0.00% 0.00% 0.00%   0 EAP Framework

226           0          1         0 0.00% 0.00% 0.00%   0 EAP Test

227     212418     9547649         22 0.00% 0.00% 0.00%   0 FHRP Main thread

228           0           1         0 0.00% 0.00% 0.00%   0 TRACK Main threa

229           0           1         0 0.00% 0.00% 0.00%   0 TRACK Client thr

230           0           1         0 0.00% 0.00% 0.00%   0 VRRP Main thread

231     4456326   142443725         31 0.00% 0.01% 0.00%   0 VRRS Main thread

PID Runtime(ms)     Invoked     uSecs   5Sec 1Min   5Min TTY Process

232           0           2         0 0.00% 0.00% 0.00%   0 CEF switching ba

233       80352       81755       982 0.00% 0.00% 0.00%   0 IP ARP Adjacency

234     5964889   238953044         24 0.00% 0.03% 0.00%   0 IP ARP Retry Age

235     8084706   19467783       415 0.15% 0.31% 0.20%   0 IP Input

236           0           1         0 0.00% 0.00% 0.00%   0 ICMP event handl

237     345983  18515872         18 0.00% 0.00% 0.00%   0 IP ARP Track

238       17353     9570146         1 0.00% 0.00% 0.00%   0 fnvRootHubPortSt

239           0           1         0 0.00% 0.00% 0.00%   0 tHUB

240     349372   94738466         3 0.00% 0.00% 0.00%   0 MDFS MFIB Proces

241           0           1         0 0.00% 0.00% 0.00%   0 ADJ NSF process

242           0           1         0 0.00% 0.00% 0.00%   0 IPv6 ping proces

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min 5Min TTY Process

243           0           2         0 0.00% 0.00% 0.00%   0 KRB5 AAA

244           8           2       4000 0.00% 0.00% 0.00%   0 MKA

245         93         168       553 0.00% 0.00% 0.00%   0 SpanTree Flush

246           0           2         0 0.00% 0.00% 0.00%   0 REP Topology cha

247           0           2         0 0.00% 0.00% 0.00%   0 RIB LM VALIDATE

248     100745     9548687         10 0.00% 0.00% 0.00%   0 mDNS

249         17         32       531 0.00% 0.00% 0.00%   0 SMI Director DB

250     288447     1785196       161 0.00% 0.00% 0.00%   0 SMI CDP Update H

251           9           1       9000 0.00% 0.00% 0.00%   0 SMI Backup Proce

252           0           2         0 0.00% 0.00% 0.00%   0 SMI IBC server p

253           0           1         0 0.00% 0.00% 0.00%   0 SMI IBC client p

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

254           0           2         0 0.00% 0.00% 0.00%   0 SMI IBC Download

255   23241579   70230066       330 0.31% 0.24% 0.20%   0 Spanning Tree

256       21501     160659       133 0.00% 0.00% 0.00%   0 Spanning Tree St

257           0           2         0 0.00% 0.00% 0.00%   0 CDP Forward Proc

258       2290       16543       138 0.00% 0.00% 0.00%   0 802.1x switch

259     211281     645950       327 0.00% 0.00% 0.00%   0 DTP Protocol

260           0           3         0 0.00% 0.00% 0.00%   0 HRPC IPv6 Host r

261           0           5         0 0.00% 0.00% 0.00%   0 IPv6 Platform Ho

262           0           1         0 0.00% 0.00% 0.00%   0 axiom-module cra

263           0           1         0 0.00% 0.00% 0.00%   0 HRPC dai request

264          0           1         0 0.00% 0.00% 0.00%   0 HULC DAI Process

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

265           0           1         0 0.00% 0.00% 0.00%   0 HRPC power down

266           0           1        0 0.00% 0.00% 0.00%   0 HRPC ip device t

267           0           1         0 0.00% 0.00% 0.00%   0 HRPC ip source g

268           0           1         0 0.00% 0.00% 0.00%   0 HULC IP Source g

269           0           3         0 0.00% 0.00% 0.00%   0 HRPC rsvp reques

270       12187       3507       3475 0.00% 0.00% 0.00%   0 SpanTree Helper

271     1000565     9547622       104 0.00% 0.00% 0.00%   0 PI MATM Aging Pr

272     1467264   96873488         15 0.00% 0.00% 0.00%   0 UDLD

273     438631     5104137         85 0.00% 0.00% 0.00%   0 Port-Security

274           0           2         0 0.00% 0.00% 0.00%   0 Switch Backup In

275           0           2         0 0.00% 0.00% 0.00%   0 IP Host Track Pr

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

276           0           1         0 0.00% 0.00% 0.00%   0 Link State Group

277       4832     160206         30 0.00% 0.00% 0.00%   0 MMN bkgrd proces

278       33389     961196         34 0.00% 0.00% 0.00%   0 Ethchnl

279       13980       3601       3882 0.00% 0.00% 0.00%   0 VMATM Callback

280           0           1         0 0.00% 0.00% 0.00%   0 XDR background p

281       76263     348972      218 0.00% 0.00% 0.00%   0 XDR mcast

282           0           3         0 0.00% 0.00% 0.00%   0 XDR RP Ping Back

283     446295     897328       497 0.00% 0.00% 0.00%   0 XDR receive

284         16           4       4000 0.00% 0.00% 0.00%   0 IPC LC Message H

285           0           1         0 0.00% 0.00% 0.00%   0 XDR RP Test Back

286       5371     160206         33 0.00% 0.00% 0.00%   0 FRR Background P

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

287     119413     674765       176 0.00% 0.00% 0.00%   0 CEF background p

288           0           1         0 0.00% 0.00% 0.00%   0 fib_fib_bfd_sb e

289           0           1         0 0.00% 0.00% 0.00%   0 IP IRDP

290           0           1         0 0.00% 0.00% 0.00%   0 Tunnel FIB

291           0           2         0 0.00% 0.00% 0.00%   0 Tunnel

292           0           1         0 0.00% 0.00% 0.00%   0 IPC UDP Input

293           0           1          0 0.00% 0.00% 0.00%   0 CEF RF HULC Conv

294       31709     961291         32 0.00% 0.00% 0.00%   0 IP ACL XDR LC Ba

295     180515     4791579         37 0.00% 0.00% 0.00%   0 CEF: IPv4 proces

296     2793895     9560343       292 0.00% 0.01% 0.00%   0 ADJ background

297     783633   10668133         73 0.00% 0.01% 0.00%   0 MDFS LC Process

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

298           0           2         0 0.00% 0.00% 0.00%  0 AN

299           0           1         0 0.00% 0.00% 0.00%   0 AN HELPER

300           0           1         0 0.00% 0.00% 0.00%   0 AN CONFIG DOWNLO

301           0           2         0 0.00% 0.00% 0.00%   0 Routing Topology

302     14304       5842       2448 0.00% 0.00% 0.00%   0 IP RIB Update

303     491629       8026     61254 0.00% 0.00% 0.00%   0 NIST rng proc

304     357351   18515877         19 0.00% 0.00% 0.00%   0 Timer Library

305           0           2          0 0.00% 0.00% 0.00%   0 App Route Proces

306           0           1         0 0.00% 0.00% 0.00%   0 IPv6 RIB Cleanup

307           0           1         0 0.00% 0.00% 0.00%   0 IPv6 RIB Event H

308         947       11250         84 0.00% 0.00% 0.00%   0 Auth Manager

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

309           0           1         0 0.00% 0.00% 0.00%   0 Critical Auth

310           0           1         0 0.00% 0.00% 0.00%   0 EPM MAIN PROCESS

311           0           1         0 0.00% 0.00% 0.00%   0 Timer Handler Pr

312           0           1         0 0.00% 0.00% 0.00%   0 EPM Downlad Mana

313     108910     1916572         56 0.00% 0.00% 0.00%   0 EPM ACL PLUG-IN

314     320168   19027461         16 0.00% 0.00% 0.00%   0 SVM UT Process

315           0           1         0 0.00% 0.00% 0.00%   0 CMD HANDLER

316         411       4452         92 0.00% 0.00% 0.00%   0 TCP Timer

317       6123       9293       658 0.00% 0.00% 0.00%   0 TCP Protocols

318       24402     9671873         2 0.00% 0.00% 0.00%   0 Socket Timers

319         476       32162         14 0.00% 0.00% 0.00%   0 HTTP CORE

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

320       1346     120491         11 0.00% 0.00% 0.00%   0 Cluster L2

321       38342     961197         39 0.00% 0.00% 0.00%   0 Cluster RARP

322     153160     1524628       100 0.00% 0.00% 0.00%   0 Cluster Base

323           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Supplicant

324           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Supplicant

325           0           2         0 0.00% 0.00% 0.00%   0 Dot1x Supplicant

326           0           1         0 0.00% 0.00% 0.00%   0 DSensor HTTP

327           0           2         0 0.00% 0.00% 0.00%   0 CEF RP Backgroun

328         176       8836         19 0.00% 0.00% 0.00%   0 Flow Exporter Ti

329           0           1         0 0.00% 0.00% 0.00%   0 DCT Bkgd Process

330           0           4         0 0.00% 0.00% 0.00%   0 MFIB Master back

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

331           0          2         0 0.00% 0.00% 0.00%   0 ENABLE AAA

332         888       8854       100 0.00% 0.00% 0.00%   0 MVPN Mgr Process

333           0           2         0 0.00% 0.00% 0.00%   0 Multicast Offloa

334       1975       9282        212 0.00% 0.00% 0.00%   0 RARP Input

335           9           7       1285 0.00% 0.00% 0.00%   0 static

336     1711242     618410       2767 0.00% 0.00% 0.00%   0 ADJ resolve proc

337           0           1         0 0.00% 0.00% 0.00%   0 IPv6 Static Hand

338           0           1         0 0.00% 0.00% 0.00%   0 MAB Framework

339           0           2         0 0.00% 0.00% 0.00%   0 Media Services P

340     513425     963680       532 0.00% 0.00% 0.00%   0 QoS stats proces

341           0           1         0 0.00% 0.00% 0.00%   0 LICENSE AGENT

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

342           0           2         0 0.00% 0.00% 0.00%   0 REP LSL Proc

343           0           2         0 0.00% 0.00% 0.00%   0 REP BPA/EPA Proc

344           0           5         0 0.00% 0.00% 0.00%   0 SNMP Timers

345           0           1         0 0.00% 0.00% 0.00%   0 HRPC hwccp reque

346           0           3         0 0.00% 0.00% 0.00%   0 HULC_WCCP_BACKGR

347           0           1         0 0.00% 0.00% 0.00%   0 HRPC dhcp snoopi

348           0           5         0 0.00% 0.00% 0.00%   0 HULC DHCP Snoopi

349           0          1         0 0.00% 0.00% 0.00%   0 HRPC sisf reques

350           8           3       2666 0.00% 0.00% 0.00%   0 HULC SISF Proces

351           0           1         0 0.00% 0.00% 0.00%   0 HULC SISF Source

352     4495246   19749972       227 0.00% 0.04% 0.01%   0 Inline Power

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

353     6285501   21100491       297 0.00% 0.05% 0.05%   0 Marvell wk-a Pow

354       65659     480612       136 0.00% 0.00% 0.00%   0 HULC stack power

355       20927     320399         65 0.00% 0.00% 0.00%   0 HRPC stack power

356       16481       15785       1044 0.00% 0.00% 0.00%   0 IGMPSN L2MCM

357           0           1         0 0.00% 0.00% 0.00%   0 IGMPSN MRD

358     111728     677987       164 0.00% 0.00% 0.00%   0 IGMPSN

359           0           1         0 0.00% 0.00% 0.00%   0 IGMPQR

360           0           3         0 0.00% 0.00% 0.00%   0 L2TRACE SERVER

361           0           2         0 0.00% 0.00% 0.00%   0 AUTO LAG Protoco

362       12961       15772       821 0.00% 0.00% 0.00%   0 MLDSN L2MCM

363           0           1         0 0.00% 0.00% 0.00%   0 MRD

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

364           0           1         0 0.00% 0.00% 0.00%   0 MLD_SNOOP

365           0           1         0 0.00% 0.00% 0.00%   0 TDP Discovery

366     2272962   93469952         24 0.00% 0.00% 0.00%   0 MDFS RP process

367           0           1         0 0.00% 0.00% 0.00%   0 EM Background Pr

368           0           2         0 0.00% 0.00% 0.00%   0 AAA Cached Serve

369           0           2         0 0.00% 0.00% 0.00%   0 MRIB Process

370       19938       97122       205 0.00% 0.00% 0.00%   0 HRPC hl2mcm igmp

371         25         90       277 0.00% 0.00% 0.00%   0 HRPC hl2mcm mlds

372           0           3         0 0.00% 0.00% 0.00%   0 LDAP process

373           0           2         0 0.00% 0.00% 0.00%   0 Crypto Support

374           9           1       9000 0.00% 0.00% 0.00%   0 IPSECv6 PS Proc

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

375           0           1         0 0.00% 0.00% 0.00%   0 Key chain liveke

376         61         278       219 0.00% 0.00% 0.00%   0 Collection proce

377           0           2         0 0.00% 0.00% 0.00%   0 LINE AAA

378         36         71       507 0.00% 0.00% 0.00%   0 LOCAL AAA

379     4320086   142443637         30 0.00% 0.02% 0.00%   0 MMA DB TIMER

380           8           1       8000 0.00% 0.00% 0.00%   0 MMA MSG MCOL

381           0           1         0 0.00% 0.00% 0.00%   0 Opaque Database

382        34         759         44 0.00% 0.00% 0.00%   0 TACACS+

383       3700       4448       831 0.00% 0.00% 0.00%   0 TPLUS

384     1181848     480600       2459 0.00% 0.00% 0.00%   0 HULC ISM FNF

385     120317       12370       9726 0.00% 0.00% 0.00%   0 hulc running con

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

386         427         33     12939 0.00% 0.00% 0.00%   0 crypto engine pr

387           0           1         0 0.00% 0.00% 0.00%   0 RSA background p

388         820       10716         76 0.00% 0.00% 0.00%   0 Crypto CA

389           0           1         0 0.00% 0.00% 0.00%   0 Crypto PKI-CRL

390     358058     9599949         37 0.00% 0.00% 0.00%   0 Crypto IKEv2

391          0           1         0 0.00% 0.00% 0.00%   0 IKEv2 AAA handle

392           0           1         0 0.00% 0.00% 0.00%   0 encrypt proc

393           0           2         0 0.00% 0.00% 0.00%   0 CRYPTO MAP FREE

394           0           1         0 0.00% 0.00% 0.00%   0 Crypto INT

395           0           2         0 0.00% 0.00% 0.00%   0 Crypto IKE Dispa

396           0           3         0 0.00% 0.00% 0.00%   0 Crypto IKMP

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

397           0           1         0 0.00% 0.00% 0.00%   0 IPSEC manual key

398       25459     480601         52 0.00% 0.00% 0.00%   0 IPSEC key engine

399           0           5         0 0.00% 0.00% 0.00%   0 Crypto ACL

400           0           1         0 0.00% 0.00% 0.00%   0 Crypto PAS Proc

401           0           5         0 0.00% 0.00% 0.00%   0 IPSec background

402           0           2         0 0.00% 0.00% 0.00%   0 MMON PROCESS

403     4353853   142443719         30 0.15% 0.03% 0.00%   0 MMA DP TIMER

404     4494486   300035654         14 0.15% 0.02% 0.00%   0 MMON MENG

405           0           1         0 0.00% 0.00% 0.00%   0 HRPC x_setup req

406           0           2         0 0.00% 0.00% 0.00%   0 REP Switch Helpe

407           0           1         0 0.00% 0.00% 0.00%   0 Licensing MIB pr

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

408           0          23         0 0.00% 0.00% 0.00%   0 VTP Trap Process

409         67           5     13400 0.00% 0.00% 0.00%   0 OBFL MSG obfl0

410         267       8971         29 0.00% 0.00% 0.00%   0 ASP Process Crea

411         57         391        145 0.00% 0.00% 0.00%   0 AAA SEND STOP EV

412           0           1         0 0.00% 0.00% 0.00%   0 Test AAA Client

413           8           8       1000 0.00% 0.00% 0.00%   0 DCM Core Thread

414           0           2         0 0.00% 0.00% 0.00%   0 DHCP Security He

415           0           1         0 0.00% 0.00% 0.00%   0 dcm_cli_engine

416           0           3         0 0.00% 0.00% 0.00%   0 dcm_cli_provider

417           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED Generic

418           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED Routing

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

419       20623     221726         93 0.00% 0.00% 0.00%   0 EEM ED Syslog

420           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Media Ser

421           0           1         0 0.00% 0.00% 0.00%   0 Syslog Traps

422         67           5     13400 0.00% 0.00% 0.00%   0 OBFL ENV obfl0

423       6040     160206         37 0.00% 0.00% 0.00%   0 vlmap logger

424           0           1         0 0.00% 0.00% 0.00%   0 HCD Process

425           0           1         0 0.00% 0.00% 0.00%   0 HRPC cable diagn

426           0          1         0 0.00% 0.00% 0.00%   0 HRPC archive_dnl

427           0           1         0 0.00% 0.00% 0.00%   0 HRPC rsu request

428           0           2         0 0.00% 0.00% 0.00%   0 DiagCard9/-1

429           0           1          0 0.00% 0.00% 0.00%   0 Online Diag EEM

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

430     2328009   80286112         28 0.00% 0.00% 0.00%   0 PM Callback

432       8273     770624         10 0.00% 0.00% 0.00%   0 dhcp snooping sw

434         60         106       566 0.00% 0.00% 0.00%   0 EEM Server

435         17           3       5666 0.00% 0.00% 0.00%   0 Call Home proces

436         49         131       374 0.00% 0.00% 0.00%   0 EEM Helper Threa

437       11050     160232         68 0.00% 0.00% 0.00%   0 Call Home Timer

438       81383     961542         84 0.00% 0.00% 0.00%   0 Bulkstat-Client

439           0           4         0 0.00% 0.00% 0.00%   0 CTS credentials

440          0           1         0 0.00% 0.00% 0.00%   0 HRPC eee request

441           0           6         0 0.00% 0.00% 0.00%   0 DCM snmp dp Thre

442           8           3       2666 0.00% 0.00% 0.00%   0 snmp dcm ma shim

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

443           0           2         0 0.00% 0.00% 0.00%   0 EEM Policy Direc

445           9           3       3000 0.00% 0.00% 0.00%   0 EEM ED CLI

446           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Config

447           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Counter

448           0           4         0 0.00% 0.00% 0.00%   0 EEM ED CRASH

449           0           4         0 0.00% 0.00% 0.00%   0 EEM ED DS

450           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Env

451         17           4       4250 0.00% 0.00% 0.00%   0 EM ED GOLD

452           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Interface

453           0           4         0 0.00% 0.00% 0.00%   0 EEM ED IOSWD

454           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Ipsla

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

455           0           4         0 0.00% 0.00% 0.00%   0 EEM ED None

456           0           4         0 0.00% 0.00% 0.00%   0 EEM ED OIR

457           0           2         0 0.00% 0.00% 0.00%   0 STP FAST TRANSIT

458           0           4         0 0.00% 0.00% 0.00%   0 EEM ED SNMP

459           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED SNMP Obje

460           8           4       2000 0.00% 0.00% 0.00%   0 EEM ED SNMP Noti

461           0           4         0 0.00% 0.00% 0.00%   0 EEM ED Test

462       9040     210543         42 0.00% 0.00% 0.00%   0 EEM ED Timer

463       58092     215452       269 0.00% 0.01% 0.00%   0 Syslog

464       27614     640786         43 0.00% 0.00% 0.00%   0 OBFL I/O Buffer

465          0           6         0 0.00% 0.00% 0.00%   0 RBM CORE

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

466           0           5         0 0.00% 0.00% 0.00%   0 CSRT RAPID TRANS

467         24           6       4000 0.00% 0.00% 0.00%   0 VLAN Manager

468         51       2680         19 0.00% 0.00% 0.00%   0 SSH Event handle

469           0           1         0 0.00% 0.00% 0.00%   0 DiagCard8/-1

470       14691     169128         86 0.00% 0.00% 0.00%   0 IP Background

471       1763       14208       124 0.00% 0.00% 0.00%   0 IP Connected Rou

472     1080771   10022079       107 0.00% 0.00% 0.00%   0 DHCPD Receive

473     2513137   96642695         26 0.00% 0.01% 0.00%   0 IGMP Input

474     5427933   206314831         26 0.15% 0.03% 0.00%   0 PIM Process

475     1870274   96244779         19 0.00% 0.00% 0.00%   0 Mwheel Process

476           0           1         0 0.00% 0.00% 0.00%   0 mcast_rw_process

PID Runtime(ms)     Invoked     uSecs   5Sec   1Min   5Min TTY Process

477     305774     9720087         31 0.00% 0.00% 0.00%   0 NTP

479       44058     109195       403 0.00% 0.00% 0.00%   0 IP SNMP

480       41311       74944       551 0.00% 0.00% 0.00%   0 PDU DISPATCHER

481     135053       75091       1798 0.00% 0.00% 0.00%   0 SNMP ENGINE

482           0           2         0 0.00% 0.00% 0.00%   0 IP SNMPV6

483           0           1         0 0.00% 0.00% 0.00%   0 SNMP ConfCopyPro

484         26           7       3714 0.00% 0.00% 0.00%   0 SNMP Traps

485       1476       83002         17 0.00% 0.00% 0.00%   0 DHCPD Timer

486       5216     160205         32 0.00% 0.00% 0.00%   0 DHCPD Database

488    690602     4183604       165 0.00% 0.00% 0.00%   0 EIGRP-IPv4

489     1066579     8309503       128 0.00% 0.00% 0.00%   0 EIGRP-IPv4 Hello

Any advice is really appreciated.  Also, PoE consumption is very low on this stack

HI

 HULC is fictitious bug and has no real effect on traffic/cpu its just bug that's Unfortunatly  in lot of versions of IOS , saying that you have a high interrupt 36% which means the switch could not process and had to process in hardware , this can happen if ip cef is not enabled or if the switch has to much traffic coming through it while its trying to process it

Total CPU Utilization: 94% Process Utilization: 58% Interrupt Utilization: 36% 58% then minus 21% for HULC = 37% actual traffic being correctly processed and 36% being process switched by hardware

You need to run a full EEM script as its happening see what's collected , also check exactly on the network what's going on during these times , any type of backup , antivirus push outs etc that could spike the traffic, the EEM should capture the ints etc while the issues occurring

This is an EEM I use that may help you

event manager applet High_CPU

    event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.4.1 get-type exact entry-op ge entry-val "70" exit-time 10 poll-interval 5

    action 0.1 syslog msg "CPU Utilization is high"

    action 0.2 cli command "enable"

    action 0.4 cli command "show log | append flash:CPU_Profile.txt"

    action 0.5 cli command "show process cpu sorted | append flash:CPU_Profile.txt"

    action 0.6 cli command "show interfaces | append flash:CPU_Profile.txt"

    action 0.7 cli command " show ip cef switching stat | append flash:CPU_Profile.txt"

    action 0.8 cli command " show ip traffic | append flash:CPU_Profile.txt"

    action 0.9 cli command " show int switching | append flash:CPU_Profile.txt"

    action 1.0 cli command "end"

Haihua Rong
Level 1
Level 1

Hello Vijay,

Spikes are commonly seen around IOS platforms based on the inputs received by CPU and are not necessarily a problem.

It is slightly difficult to track the process which is causing spike, as it may last only for a short interval. The common cause could be, regular SNMP polling or any other kind of Process Switched traffic, regular CLI commands executed, like wr mem, sh runn or sh tech which is handled in CPU causing short spikes.

Every command executed or every external management/monitoring action to the switch will register as a short cpu spike. However the CPU is never monopolised by a single process.


Please refer to this document:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3750/software/troubleshooting/cpu_util.html#wp998829

You can configure an EEM script if you want to know the processes that are reaching 100% of cpu utilization and to confirm that are expected.

EEM is first supported on:
6500 - 12.2(18)SXF4 ?7600 - 12.2(33)SRA?3750/3560/ME-3750/ME-3400 - 12.2(40)SE?4500 - 12.2(40)SG

config t

event manager applet cpu_stats

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.3.1 get-type exact entry-op gt entry-val 100 exit-op lt exit-val 40 poll-interval 5               

action 1.1 syslog msg "------HIGH CPU DETECTED----, CPU: $_snmp_oid_val %"
action 1.2 cli command "enable"
action 1.3 cli command "show proc cpu sort | append nvram:cpu_stats"
action 1.4 cli command "sho spann detail | append nvram:cpu_stats"
action 1.5 cli command "sho log | append nvram:cpu_stats"

b) to remove the script, once it is not needed anymore
conf t
no event manager applet cpu_stats
end


Explanation:

The script will monitor the CPU every 5 seconds100% in the event line is the utilization that will trigger the script.
Once is triggered the CPU has to come down to 40% and it will be triggered again.

Haihua

Hi Leo ,

We are running  version of

c3750e-universalk9-mz.122-55.SE5.bin .

Hi Haihu ,

We have identified the process spiking the CPU is " HULC' process . The output is as follws

processes cpu | exclude 0.00

CPU utilization for five seconds: 99%/5%; one minute: 32%; five minutes: 29%

PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process

   4   258638827  11402314      22683  0.45%  0.71%  1.09%   0 Check heaps     

  10    13921000  17727828        785  0.15%  0.38%  0.26%   0 ARP Input       

  38    21012754  20354975       1032  0.15%  0.03%  0.03%   0 Per-Second Jobs 

  74    97782083 149788981        652  0.76%  0.45%  0.47%   0 RedEarth I2C dri

  75   605768871  99084729       6113  1.99%  2.40%  2.42%   0 RedEarth Tx Mana

  76     1965792  74894378         26  0.15%  0.03%  0.02%   0 RedEarth Rx Mana

  81   715470342   7252398      98655  3.83%  3.12%  3.22%   0 Adjust Regions  

  84    11864720  42473212        279  0.15%  0.06%  0.05%   0 hrpc <- response

118   376140309  40324036       9327  1.53%  1.53%  1.57%   0 hpm counter proc

119    13391666  42223280        317  0.15%  0.04%  0.04%   0 HRPC pm-counters

158   603839277 465763712       1296  0.45%  0.62%  0.76%   0 Hulc LED Process

159     6742985  20431422        330  0.61%  0.16%  0.11%   0 HL3U bkgrd proce

169    78496155   4137435      18972  0.45%  0.35%  0.36%   0 HQM Stack Proces

170    31111496  17635930       1764  0.15%  0.21%  0.21%   0 HRPC qos request

200    92929040 125560792        740  2.75%  2.23%  2.02%   0 IP Input        

219    85511651 248146851        344  0.76%  0.70%  0.72%   0 Spanning Tree   

261    13554971  46977580        288  0.61%  0.24%  0.21%   0 DHCPD Receive   

287     1036221      5476     189229 73.45%  5.88%  1.25%   0 hulc running con

323       62212      9439       6590  1.27%  1.10%  2.38%   0 Exec            

342    28032934  23540805       1190  0.47%  0.11%  0.06%   0 SNMP ENGINE   

Are we hitting the bug ,

CSCtn42790 .  Wther upgrading the image will solve the issue ? Kindly confirm.

BR//

Vijay

Vijay,

Try 12.2(55)SE8.

ehshum
Level 1
Level 1

Hi All,

 

I am having the same problem on two 3750 stacks. IOS versions used:

 

Stack1: 4x 3750X with IOS Version 12.2(55)SE8

Stack2: 4x 3750G with IOS Version Version 12.2(55)SE5

 

Symptom:

 

High CPU Spikes happening daily due to the hulc running process. This matches exactly with

 

http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/68461-high-cpu-utilization-cat3750.html#topic4

High CPU During a Configuration Change

 

There are no cpu spike when I run command like "show run", "show start", "write mem" but I was able to reproduce the issue by issuing the below commands:

 

conf t

exit

 

Specifically, the CPU will spike when the "exit" command is issued. The CPU will then spike up momentarily as shown in the history graph below:

 

sh proc cpu history
                 11111                                       
    1113333399999000003333111111111111111111111111111111111111
    2222222299999000002222444445555555555666644444555553333355
100         **********                                       
90         **********                                       
80         **********                                       
70         **********                                       
60         **********                                       
50         **********                                       
40         **********                                       
30    *******************                                   
20    *******************     **************     *****     **
10 **********************************************************
0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per second (last 60 seconds)
    1 1                                                      
    0302322222222222222222222222222222224222222222222212222222
    0005046879364633702523353689946454658433741458156594795536
100 * *                                                      
90 * *                                                      
80 * *                                                      
70 * *                                                      
60 * *                                                      
50 * *                                 *                    
40 * *                                 *                    
30 #*#** **** * *  *  *   * **** * * ***   *   ** ***  **** *
20 #########*#*##*####*###########*##**##*###*#*###**####*#*#
10 ##########################################################
   0....5....1....1....2....2....3....3....4....4....5....5....
             0    5    0    5    0    5    0    5    0    5   
               CPU% per minute (last 60 minutes)
              * = maximum CPU%   # = average CPU%

 

Notes:

1) I have checked the bug toolkit and there seems to be a few bugs for 3750 platform due to hulc but none is matching my environment

2) There are no spanning tree issues on the switch

3) The 72hours baseline is at 30%-40% with the high CPU spikes occuring daily.

 

Do the stack run a config sync periodically which might cause this? I am quite sure that during the spikes there are no users logged in the switch and doing any configuration change.

 

Please advise. Thanks in advance.

Review Cisco Networking for a $25 gift card