12.2(58)SE2 on a 2960 cause memory leaks - Page 4

pasztor.richard · ‎09-07-2011

Hi all,

I had a very disappointed experience today during the planned upgrade on my 2960 to IOS version 12.2(58)SE2 after business hours. We have a small office with 3x 2960 and a couple of 2950. There is a single 2960 called as the "central" as its the STP root, and most other switches are directly connected to the central. The remaining switches are conected to one of the switches conneting to the central.

I tested the aforementioned IOS on 2x 2960 las t week, but did not upgrade the central one. The 2x switches ran fine for a week, so I decided to upgrade the central one: basically it was a disaster. After the restart, the central became immediately unavailable via SSH (though admin IP pingable). I connected to the console, and realized that even console does not work, as I received the error message: "Low memory", and the console session could not be started. I restarted the switch once again (had to cut the power) and this time I managed to connect via the serial: the free memory was approx 2 MB, though its was over 5 MB if running 12.2(55)SE1. After 1-2 mins, free memory decreased down to 1.8 megs. SSH was still not functioning. At this time, fearing of a sudden memory leak during the night, I had to rollback to the previous version.

My config is quite simple: RSTP root, DHCP snooping, 1 LACP etherchannel, only 5x VLANs with VTP server, MOTD and BANNER (yeah I know its a risky feature since the previous IOS) I wonder what can be the problem here?

I have attached the running config of the central 2960 (removed some sensitive entries)

f1sh3ggst3w · ‎09-17-2013

I see Cisco has updated the 15.0(2) software to SE4. Has anyone tried it and if so, is it working better?

I had been running the 15.0(2)SE2 since April with no apparant problems until about three weeks ago. Then I started experiencing the problem listed in this thread with a few of the 2960's we have. We also have seen where connections randomly drop out, meaning you can be surfing the Internet, then it just stops and IE (or whatever browser) says it can't connect to the host. You usually can wait a few seconds, hit refresh and the page loads. The connections are individual computers, not a whole switch at a time. One computer can be surfing just fine but the one next to it times out. I've been pulling my hair out trying to find the answer to this one.

Leo Laohoo · ‎09-17-2013

I see Cisco has updated the 15.0(2) software to SE4.  Has anyone tried it and if so, is it working better?

All my 2960/G/S are running 15.0(2)SE4.

All my 3750/G/E/X and 3560/G/E/X are running 12.2(55)SE8.

I am currently testing 15.2(1)E on a 3560CG (compact switch). Will be testing this version on the 2960S.

15.2(1)E is only valid for newer switches. This version will no longer be supported on legacy 3750/G, 3560/G and 2960/G.

So far, I am pleased at the performance of 15.0(2)SE4 on the 2960/G/S. I will not be using this version on Cat 3K as I've been burnt before because our Cat6K are all distribution while our 2960/G/S are the access layer switches.

Michael Broucek · ‎09-25-2013

Today we ran into the same problem on 2960S-48FPS-L in Stack configuration. IOS: 15.2(1)E

ssh/telnet not available via console "%% Low on memory; try again later"

another 2960s stack is running on 15.0(1)SE2 uptime: 1 year, 25 weeks, so far no problems

Leo, how long you are running on 15.0(2)SE4 with 2960s? Looking for a release which is more stable than both installed

Leo Laohoo · ‎09-26-2013

Leo, how long you are running on 15.0(2)SE4 with 2960s? Looking for a release which is more stable than both installed

Tested 15.0(2)SE4 since the day it was released and all of my 2960/G/S and 3560CG are running this version. Do not see any issues with performance or CPU.

Jonathan Hall · ‎09-26-2013

Aubrey Jones wrote:

I see Cisco has updated the 15.0(2) software to SE4. Has anyone tried it and if so, is it working better?

I've just tried upgrading one of my WS-C2960-24TT-L to 15.0(2)SE4 and process memory used was at 84% straight away. So I'm rolling back to 12.2(55)SE8 with its 69% memory used.

Leo Laohoo · ‎09-27-2013

I've just tried upgrading one of my WS-C2960-24TT-L to 15.0(2)SE4 and process memory used was at 84% straight away. So I'm rolling back to 12.2(55)SE8 with its 69% memory used.

Jonathan,

I'm keen to learn/know what you've got in your 2960 (non-S or non-X, I presume?). All of my switches are currently running PoE and 802.1x. Either one can sometimes cause CPU to spike.

I'm not saying 12.2(55)SE8 is bad, on the contrary I've always been recommending people load the 12.2(55)SE5 and later IOS because this is one of the most well-behaved IOS versions I've ever come across.

Back to your issue, do you have SNMP traps enabled? If you do, do you have ALL SNMP traps enabled?

Jonathan Hall · ‎09-30-2013

Hi Leo,

Yes a WS-C2960-24TT-L, not an S or X. There's no POE, 802.1x or SNMP traps configured. What I'm seeing with 15.0(2)SE4, and to a lesser extent 12.2(58)SE2, is not a spike in CPU use. Rather it's an increase in the amount of processor pool memory.

Here's the proccess memory from a 2960 running 12.2(55)SE8:-

Processor Pool Total: 22324560 Used: 15462024 Free: 6862536

I/O Pool Total: 4194304 Used: 2493704 Free: 1700600

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

And here from a 2960 running 12.2(58)SE2:-

Processor Pool Total: 21586420 Used: 17291128 Free: 4295292

I/O Pool Total: 4194304 Used: 2375332 Free: 1818972

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

12.2(58)SE2 uses over 80% of the available processor memory and seems to have a memory leak which causes this to grow further over time. 12.(55)SE8 uses a more reasonable 69%.

Jonathan

Leo Laohoo · ‎09-30-2013

Hi Jonathan,

If you have a spare switch running the 15.0(2)SE4 IOS, can you post the outcome of the command "sh proc cpu | ex 0.00" and "sh proc mem | ex 0.00"?

I would like to see the functions causing this. Again, this is odd.

f1sh3ggst3w · ‎09-30-2013

Leo, here is one of mine for an extra reference. This is from a WS-C2960-48TT-L running 15.0(2)SE4.

.

sh proc cpu | ex 0.00

CPU utilization for five seconds: 19%/0%; one minute: 22%; five minutes: 22%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 12817472 669631 19141 0.31% 1.11% 0.83% 0 Check heaps

72 6408920 77899014 82 0.47% 0.48% 0.35% 0 Fifo Error Detec

85 1734802 43343846 40 0.15% 0.17% 0.14% 0 HLFM address lea

87 1057856 43573689 24 0.31% 0.10% 0.06% 0 HLFM address ret

130 135747765 38124082 3560 13.11% 13.89% 14.26% 0 Hulc LED Process

139 793281 304962 2601 0.15% 0.08% 0.06% 0 HQM Stack Proces

sh proc mem | ex 0.00

Processor Pool Total: 21505748 Used: 18086080 Free: 3419668

I/O Pool Total: 4194304 Used: 2384460 Free: 1809844

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 23584456 6303228 14152864 0 0 *Init*

0 0 12180 282956 12180 0 0 *Sched*

0 0 16994536 16145860 813224 3795303 1341026 *Dead*

0 0 0 0 394440 0 0 *MallocLite*

1 0 254912 242652 19420 0 0 Chunk Manager

2 0 232 232 4160 0 0 Load Meter

3 0 0 0 10032 0 0 Connection Mgr

4 0 4688 232 11668 0 0 Check heaps

5 0 91420708 94799096 42192 60305908 62229899 Pool Manager

6 0 0 0 7160 0 0 DiscardQ Backgro

7 0 232 232 7160 0 0 Timers

8 0 0 0 4160 0 0 WATCH_AFS

9 0 196211500 196254948 7620 0 0 crypto sw pk pro

10 0 0 0 25160 0 0 Crash writer

11 0 340 0 7500 0 0 Exception contro

12 0 174206580 174160148 53592 0 0 ARP Input

13 0 4108 4108 7160 0 0 ARP Background

14 0 0 0 7160 0 0 AAA_SERVER_DEADT

15 0 0 0 13160 0 0 Policy Manager

16 0 47476 59500 44776 0 8840 Entity MIB API

17 0 0 0 7160 0 0 IFS Agent Manage

18 0 0 0 7160 0 0 IPC Event Notifi

19 0 0 0 7160 0 0 IPC Mcast Pendin

20 0 0 0 7220 0 0 IPC Dynamic Cach

21 0 0 0 7160 0 0 IPC Session Serv

22 0 0 0 7160 0 0 IPC Zone Manager

23 0 0 0 7160 0 0 IPC Periodic Tim

24 0 0 0 7160 0 0 IPC Deferred Por

25 0 0 0 7160 0 0 IPC Process leve

26 0 1688 0 8848 0 0 IPC Seat Manager

27 0 0 0 7160 0 0 IPC Check Queue

28 0 0 0 7160 0 0 IPC Seat RX Cont

29 0 0 0 7160 0 0 IPC Seat TX Cont

30 0 0 0 7160 0 0 IPC Keep Alive M

31 0 0 0 7160 0 0 IPC Loadometer

32 0 72540 72540 7160 0 0 PrstVbl

33 0 232 232 13160 0 0 XML Proxy Client

34 0 0 0 7160 0 0 Dynamic ARP Insp

35 0 0 0 7160 0 0 ARP Snoop

36 0 232 232 7160 0 0 SMART

37 0 232 232 7160 0 0 GraphIt

38 0 0 0 7224 0 0 Critical Bkgnd

39 0 38352 288 19304 0 0 Net Background

40 0 0 0 13160 0 0 IDB Work

41 0 34936 232 13160 20304 0 Logger

42 0 232 232 7160 0 0 TTY Background

43 0 96644 84496 59352 0 0 IF-MGR control p

44 0 0 0 7160 0 0 IF-MGR event pro

45 0 0 0 7160 0 0 Net Input

46 0 232 232 7160 0 0 Compute load avg

47 0 2892 0 7160 1692 0 Per-minute Jobs

48 0 43304 26400 27064 0 0 Per-Second Jobs

49 0 0 0 7160 0 0 AggMgr Process

50 0 0 0 7160 0 0 Transport Port A

51 0 0 0 7160 0 0 HC Counter Timer

52 0 0 0 7160 0 0 SFF8472

53 0 232 232 7160 0 0 DownWhenLooped

54 0 0 0 7160 0 0 HULC ACL Tcam Me

55 0 0 0 7160 0 0 HRPC lpip reques

56 0 232 232 7160 0 0 HLPIP Sync Proce

57 0 0 0 7160 0 0 HRPC EnergyWise

58 0 0 0 7160 0 0 HRPC actual powe

59 0 0 0 7160 0 0 PSP Timer

60 0 0 0 7160 0 0 HULC QM Tcam Mem

61 0 34858284 34858284 7160 0 0 Authenticate Pro

62 0 0 0 7160 0 0 HULC PBR Tcam Me

63 0 368232 15779268 377792 0 0 EEM ED ND

64 0 4556 0 14716 0 0 EEM ED Identity

65 0 4556 0 14716 0 0 EEM ED MAT

66 0 7812 0 7812 0 0 HL2MCM

67 0 7820 0 7820 0 0 HL2MCM

68 0 0 0 7160 0 0 HRPC asic-stats

69 0 0 0 4160 0 0 HRPC hsm request

70 0 4088 0 17312 0 0 Stack Mgr

71 0 889176 142984 499632 0 0 Stack Mgr Notifi

72 0 9952 232 16880 0 0 Fifo Error Detec

73 0 1224 1224 4160 0 0 Adjust Regions

74 0 0 0 7160 0 0 hrpc -> response

75 0 0 0 7160 0 0 hrpc -> request

76 0 65848 532 7160 0 0 hrpc <- response

77 0 0 0 7160 0 0 HRPC hcomp reque

78 0 656 232 7160 0 0 HULC Device Mana

79 0 108 108 7160 0 0 HRPC hdm non blo

80 0 176 176 7160 0 0 HRPC hdm blockin

81 0 0 0 13160 0 0 HIPC bkgrd proce

82 0 232 232 7160 0 0 Hulc Port-Securi

83 0 0 0 7160 0 0 HRPC hpsecure re

84 0 0 0 7160 0 0 HRPC hlfm reques

85 0 52168948 232 65300 0 0 HLFM address lea

86 0 232 232 7160 0 0 HLFM aging proce

87 0 232 232 7160 0 0 HLFM address ret

88 0 0 0 7160 0 0 HRPC hulc misc r

89 0 0 0 7160 0 0 HULC Tcam Memory

90 0 0 0 13160 0 0 HVLAN main bkgrd

91 0 232 232 7160 0 0 HVLAN Mapped Vla

92 0 232 232 7160 0 0 Vlan shutdown Pr

93 0 65740 0 7160 0 0 HRPC vlan reques

94 0 0 0 7160 0 0 HULC VLAN REF Ba

95 0 0 0 7160 0 0 HRPC ilp request

96 0 0 0 7160 0 0 HULC PM Vector P

97 0 0 0 7160 0 0 HPM Msg Retry Pr

98 0 3018596 3477228 7540 0 0 SpanTree Helper

99 0 11338840 9542944 21036 0 0 hpm main process

100 0 232 232 7160 0 0 HPM Stack Sync P

101 0 0 0 25160 0 0 HRPC pm request

102 0 232 232 7160 0 0 hpm counter proc

103 0 0 0 7160 0 0 HRPC pm-counters

104 0 0 0 7160 0 0 hpm vp events ca

105 0 0 0 7160 0 0 HRPC hcmp reques

106 0 0 0 7160 0 0 HACL Queue Proce

107 0 0 0 7160 0 0 HRPC acl request

108 0 232 1460 61160 0 0 HACL Acl Manager

109 0 0 0 7160 0 0 HRPC backup inte

111 0 0 0 7160 0 0 HRPC cdp request

112 0 0 0 7160 0 0 HULC CISP Proces

113 0 0 0 7160 0 0 HRPC lldp reques

114 0 0 0 7160 0 0 HRPC dot1x reque

115 0 10044 0 17204 0 0 HULC DOT1X Proce

116 0 0 0 4160 0 0 HRPC system mtu

117 0 0 0 7160 0 0 SMI MSG Retry Pr

118 0 0 0 10160 0 0 HRPC Smart Insta

119 0 0 0 7160 0 0 HRPC sdm request

120 0 232 232 7160 0 0 Hulc Storm Contr

121 0 232 232 7160 0 0 HSTP Sync Proces

122 0 0 0 7160 0 0 HRPC stp_cli req

123 0 0 0 7160 0 0 HRPC stp_state_s

124 0 232 232 7160 0 0 S/W Bridge Proce

125 0 0 0 7160 0 0 HRPC hudld reque

126 0 0 0 7160 0 0 HRPC vqpc reques

127 0 0 0 7160 0 0 HRPC aim request

128 0 0 0 13160 0 0 HCEF ADJ Refresh

129 0 0 0 7160 0 0 HRPC hled reques

130 0 232 232 10160 0 0 Hulc LED Process

131 0 0 0 13160 0 0 HL3U bkgrd proce

132 0 0 0 7160 0 0 HRPC hl3u reques

133 0 0 0 61160 0 0 HL3U PBR bkgrd p

134 0 0 0 7160 0 0 HL3U PBR n-h res

135 0 0 0 7160 0 0 HRPC dtp request

136 0 0 0 7160 0 0 HRPC show_forwar

137 0 0 0 7160 0 0 HRPC snmp reques

138 0 0 0 7160 0 0 HULC SNMP Proces

139 0 3754693148 3753092060 8996 933984 0 HQM Stack Proces

140 0 2435742844 2435725492 61160 10152 0 HRPC qos request

141 0 0 0 7160 0 0 HRPC span reques

142 0 0 0 7160 0 0 HRPC system post

143 0 0 0 7160 0 0 Hulc Reload Mana

144 0 0 0 7160 0 0 HRPC hrcli-event

145 0 232 232 10160 0 0 image mgr

146 0 232 50063844 7160 0 0 PI MATM Aging Pr

147 0 71328 0 78488 0 0 Switch Backup In

148 0 0 0 7160 0 0 MMN bkgrd proces

149 0 4506624 5480564 189996 0 0 Auth Manager

150 0 0 0 7160 0 0 AUTH POLICY Fram

151 0 843236 841652 8744 0 0 Dot1x Mgr Proces

152 0 12025296 9410104 35208 0 0 802.1x switch

153 0 0 0 7160 0 0 802.1x Webauth F

154 0 880 232 7808 0 0 DTP Protocol

155 0 0 0 7160 0 0 EAP Framework

156 0 0 0 7160 0 0 EAP Test

157 0 0 0 7160 0 0 HRPC dai request

158 0 0 0 4160 0 0 HULC DAI Process

159 0 0 0 7160 0 0 HRPC ip device t

160 0 0 0 7160 0 0 HRPC ip source g

161 0 0 0 7160 0 0 HULC IP Source g

162 0 524 232 7452 0 0 UDLD

163 0 216 0 7376 0 0 Port-Security

164 0 232 232 10160 0 0 IP Host Track Pr

165 0 0 0 7160 0 0 Link State Group

166 0 0 171716 7160 0 0 VMATM Callback

167 0 232 232 7160 0 0 CMD HANDLER

168 0 216 0 13376 0 0 CEF switching ba

169 0 216 0 4376 0 0 ADJ NSF process

170 0 0 0 7160 0 0 HRPC sisf reques

171 0 0 0 7160 0 0 HULC SISF Proces

172 0 0 0 7160 0 0 HULC SISF Source

173 0 232 232 7160 0 0 AAA Server

174 0 0 0 7160 0 0 AAA ACCT Proc

175 0 49352 0 56512 0 0 ACCT Periodic Pr

176 0 0 0 7172 0 0 AAA System Acct

177 0 0 0 7160 0 0 Auth-proxy AAA B

178 0 0 0 7160 0 0 IP Admin SM Proc

179 0 32553284 2072 562512 263952 0 CDP Protocol

180 1 41492 35112 31412 0 0 Virtual Exec

181 0 0 0 7176 0 0 Critical Auth

182 0 232 232 7160 0 0 AAA Dictionary R

183 0 0 0 7160 0 0 DHCP Snooping

184 0 0 0 7160 0 0 DHCP Snooping db

185 0 7944 2672 18432 0 0 IP ARP Adjacency

186 0 33012 0 40172 0 0 IP ARP Retry Age

187 0 65371536 3204 33596 0 0 IP Input

188 0 0 0 7160 0 0 ICMP event handl

189 0 232 232 10160 0 0 IP ARP Track

190 0 0 0 7160 0 0 IPv6 ping proces

192 0 13204 864 38404 0 0 SMI Director DB

193 0 0 15773108 7160 0 0 SMI CDP Update H

194 0 680 0 25840 0 0 SMI Backup Proce

195 0 3892 232 10820 0 0 SMI IBC server p

196 0 372 0 13532 0 0 SMI IBC client p

197 0 4884 232 17812 0 0 SMI IBC Download

198 0 3416728 232 7376 0 0 Spanning Tree

199 0 5044 0 12204 0 0 Spanning Tree St

200 0 232 232 7160 0 0 Ethchnl

201 0 0 65347896 13160 0 0 TCP Timer

202 0 74820 0 13160 0 0 TCP Protocols

203 0 0 0 7160 0 0 Socket Timers

204 0 292486968 292306240 60612 76140 0 HTTP CORE

205 0 232 232 7160 0 0 Cluster L2

206 0 0 0 7160 0 0 Cluster RARP

207 0 13131112 13130288 7984 0 0 Cluster Base

208 0 6188 68 19280 0 0 CEF background p

209 0 216 0 13420 0 0 fib_fib_bfd_sb e

210 0 0 0 10160 0 0 IP IRDP

211 0 0 0 7160 0 0 RARP Input

212 0 448 232 7376 0 0 Dot1x Supplicant

213 0 448 232 7376 0 0 Dot1x Supplicant

214 0 448 232 7376 0 0 Dot1x Supplicant

215 0 0 0 7160 0 0 MAB Framework

216 0 0 0 7160 0 0 HRPC dhcp snoopi

217 0 0 0 7160 0 0 HULC DHCP Snoopi

218 0 1292352 3358960 8108 0 0 IGMPSN L2MCM

219 0 0 0 7160 0 0 IGMPSN MRD

220 0 0 0 7160 0 0 IGMPSN

221 0 0 0 7160 0 0 IGMPQR

222 0 14852 232 11748 0 0 L2TRACE SERVER

223 0 1292724 3357740 8384 0 0 MLDSN L2MCM

224 0 0 0 7160 0 0 MRD

225 0 0 0 7160 0 0 MLD_SNOOP

226 0 0 0 13160 0 0 Routing Topology

227 0 26384 7208 19244 0 0 CEF: IPv4 proces

228 0 216 0 7376 0 0 ADJ background

229 0 216 0 13384 0 0 ADJ resolve proc

230 0 1120 0 14280 0 0 IP RIB Update

231 0 216 0 13376 0 0 Collection proce

232 0 11805460 11798216 8536 2268 0 NIST rng proc

233 0 232 232 10160 0 0 static

234 0 0 0 10160 0 0 IPv6 RIB Event H

235 0 13733220 13733220 7160 0 0 QoS stats proces

236 0 0 0 7160 0 0 SNMP Timers

237 0 1936 232 8864 0 0 TPLUS

238 0 0 0 7032 0 0 HCMP sync proces

239 0 0 0 7160 0 0 EPM MAIN PROCESS

240 0 232 232 7160 0 0 AAA Cached Serve

241 0 232 232 7160 0 0 LOCAL AAA

242 0 232 232 7160 0 0 ENABLE AAA

243 0 232 232 7160 0 0 LINE AAA

244 0 1936 232 8864 0 0 LDAP process

246 0 0 0 7160 0 0 HRPC hl2mcm igmp

247 0 0 0 7160 0 0 HRPC hl2mcm mlds

248 0 0 0 7160 0 0 HRPC x_setup req

249 0 0 0 7160 0 0 crypto engine pr

250 0 0 0 13160 0 0 encrypt proc

251 0 219248 184236 48968 0 0 Crypto CA

252 0 0 0 13160 0 0 Crypto PKI-CRL

253 0 4884 232 11812 0 0 VTP Trap Process

254 0 232 232 25160 0 0 ASP Process Crea

255 0 232 232 7160 0 0 DHCP Security He

256 0 0 0 7160 0 0 HCD Process

257 0 0 0 7160 0 0 HRPC cable diagn

259 0 3140160 6387500 7160 0 0 PM Callback

260 0 232 232 7160 0 0 dhcp snooping sw

261 0 0 220 7160 0 0 DiagCard1/-1

262 0 232 728580 7160 0 0 AAA SEND STOP EV

263 0 0 0 13160 0 0 Test AAA Client

264 0 0 0 7160 0 0 Syslog Traps

266 0 216 0 7376 0 0 Online Diag EEM

267 0 85072 0 92292 0 0 EEM Server

268 0 11476 1720 16916 0 0 EEM Policy Direc

270 0 4556 0 14716 0 0 EEM ED OIR

271 0 4556 0 14716 0 0 EEM ED Timer

272 0 693476 693476 13160 0 0 Syslog

273 0 232 232 4160 0 0 STP FAST TRANSIT

274 0 232 232 4160 0 0 CSRT RAPID TRANS

275 0 66196 16772 61988 0 0 Call Home proces

276 0 4556 0 14716 0 0 EEM ED Test

277 0 4556 0 14716 0 0 EEM ED Config

278 0 4556 0 14716 0 0 EEM ED Env

279 0 4556 0 14716 0 0 EM ED GOLD

280 0 31116 6428 17244 2340 0 VLAN Manager

281 0 0 0 7160 0 0 Call Home Timer

282 0 47548 0 15632 0 0 SSH Event handle

284 0 1506733408 1491135436 28592 507600 0 IP SNMP

285 0 52460132 0 13180 0 0 PDU DISPATCHER

286 0 449321228 516509596 13176 0 0 SNMP ENGINE

287 0 11200 232 24172 0 0 IP SNMPV6

288 0 0 0 13184 0 0 SNMP ConfCopyPro

289 0 0 0 13220 0 0 SNMP Traps

290 0 63932 17724 52000 0 0 NTP

20462460 Total

Leo Laohoo · ‎10-01-2013

Thanks for the output, Aubrey.

I can't find anything wrong with your CPU utilization. CPU that low gives me the impression your switch doesn't have any config in it. That's a good sign, by the way!

f1sh3ggst3w · ‎10-02-2013

Leo, you are correct. There are no vlan's, etc set up on this switch. Here is an output for sh proc cpu with a little more utilization:

sh proc cpu | ex 0.00

CPU utilization for five seconds: 32%/0%; one minute: 29%; five minutes: 30%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

4 14118619 740918 19055 0.45% 0.82% 0.76% 0 Check heaps

12 2325948 4950179 469 0.15% 0.18% 0.15% 0 ARP Input

61 2527294 22262 113525 4.13% 0.81% 0.44% 0 Authenticate Pro

72 28039253 82132498 341 1.37% 0.99% 0.94% 0 Fifo Error Detec

85 7740513 46535001 166 0.30% 0.25% 0.27% 0 HLFM address lea

87 5553862 46759142 118 0.30% 0.25% 0.22% 0 HLFM address ret

130 250039996 42432539 5892 19.32% 19.75% 20.38% 0 Hulc LED Process

140 1843723 678975 2715 0.15% 0.08% 0.08% 0 HRPC qos request

180 67 120 558 0.91% 0.09% 0.02% 1 Virtual Exec

186 4994872 46534740 107 0.15% 0.16% 0.16% 0 IP ARP Retry Age

187 4632559 14793098 313 0.15% 0.25% 0.26% 0 IP Input

259 1284353 9533753 134 0.15% 0.04% 0.04% 0 PM Callback

In my monitoring, I haven't seen it above 37%. Here is that one:

CPU utilization for five seconds: 37%/0%; one minute: 35%; five minutes: 32%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

39 740083 494930 1495 0.15% 0.07% 0.03% 0 Net Background

72 28040571 82139912 341 0.47% 0.66% 0.85% 0 Fifo Error Detec

85 7740962 46539171 166 0.15% 0.22% 0.25% 0 HLFM address lea

87 5554156 46763363 118 0.31% 0.15% 0.18% 0 HLFM address ret

130 250079837 42436520 5893 30.71% 25.06% 21.84% 0 Hulc LED Process

187 4633144 14795705 313 0.31% 0.32% 0.28% 0 IP Input

Jonathan Hall · ‎10-02-2013

Hi Leo,

Sorry for the delay. Had trouble tracking down a spare switch. The output you requested is below. This is for a

WS-C2960-24TT-L with default config and just one interface active (so I could update the firmware).

Switch#sh proc cpu | ex 0.00

CPU utilization for five seconds: 4%/0%; one minute: 4%; five minutes: 1%

PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process

Switch#sh proc mem | ex 0.00

Processor Pool Total: 21505748 Used: 16838900 Free: 4666848

I/O Pool Total: 4194304 Used: 2419708 Free: 1774596

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

PID TTY Allocated Freed Holding Getbufs Retbufs Process

0 0 22370528 5917764 13784140 0 0 *Init*

0 0 12180 172432 12180 0 0 *Sched*

0 0 15665136 15088332 581768 3602595 1182914 *Dead*

0 0 0 0 394440 0 0 *MallocLite*

1 0 6588 1544 13748 0 0 Chunk Manager

2 0 232 232 4160 0 0 Load Meter

3 0 0 0 10032 0 0 Connection Mgr

4 0 4716 232 11696 0 0 Check heaps

5 0 3396 3396 10556 2196 2196 Pool Manager

6 0 0 0 7160 0 0 DiscardQ Backgro

7 0 232 232 7160 0 0 Timers

8 0 0 0 4160 0 0 WATCH_AFS

9 0 503532 506956 7692 0 0 crypto sw pk pro

10 0 0 0 25160 0 0 Crash writer

11 0 340 0 7500 0 0 Exception contro

12 0 0 0 7160 0 0 ARP Input

13 0 3712 3712 7160 0 0 ARP Background

14 0 0 0 7160 0 0 AAA_SERVER_DEADT

15 0 0 0 13160 0 0 Policy Manager

16 0 21668 5472 23356 0 0 Entity MIB API

17 0 0 0 7160 0 0 IFS Agent Manage

18 0 0 0 7160 0 0 IPC Event Notifi

19 0 0 0 7160 0 0 IPC Mcast Pendin

20 0 0 0 7220 0 0 IPC Dynamic Cach

21 0 0 0 7160 0 0 IPC Session Serv

22 0 0 0 7160 0 0 IPC Zone Manager

23 0 0 0 7160 0 0 IPC Periodic Tim

24 0 0 0 7160 0 0 IPC Deferred Por

25 0 0 0 7160 0 0 IPC Process leve

26 0 1688 0 8848 0 0 IPC Seat Manager

27 0 0 0 7160 0 0 IPC Check Queue

28 0 0 0 7160 0 0 IPC Seat RX Cont

29 0 0 0 7160 0 0 IPC Seat TX Cont

30 0 0 0 7160 0 0 IPC Keep Alive M

31 0 0 0 7160 0 0 IPC Loadometer

32 0 232 232 7160 0 0 PrstVbl

33 0 232 232 13160 0 0 XML Proxy Client

34 0 0 0 7160 0 0 Dynamic ARP Insp

35 0 0 0 7160 0 0 ARP Snoop

36 0 232 232 7160 0 0 SMART

37 0 232 232 7160 0 0 GraphIt

38 0 0 0 7224 0 0 Critical Bkgnd

39 0 36156 288 17156 0 0 Net Background

40 0 0 0 13160 0 0 IDB Work

41 0 26260 232 39188 15228 0 Logger

42 0 25664 232 7496 0 0 TTY Background

43 0 94916 84496 57624 0 0 IF-MGR control p

44 0 0 0 7160 0 0 IF-MGR event pro

45 0 0 0 7160 0 0 Net Input

46 0 232 232 7160 0 0 Compute load avg

47 0 0 0 7160 0 0 Per-minute Jobs

48 0 35912 27456 18616 0 0 Per-Second Jobs

49 0 0 0 7160 0 0 AggMgr Process

50 0 0 0 7160 0 0 Transport Port A

51 0 0 0 7160 0 0 HC Counter Timer

52 0 0 0 7160 0 0 SFF8472

53 0 232 232 7160 0 0 DownWhenLooped

54 0 0 0 7160 0 0 HULC ACL Tcam Me

55 0 0 0 7160 0 0 HRPC lpip reques

56 0 232 232 7160 0 0 HLPIP Sync Proce

57 0 0 0 7160 0 0 HRPC EnergyWise

58 0 0 0 7160 0 0 HRPC actual powe

59 0 0 0 7160 0 0 PSP Timer

60 0 0 0 7160 0 0 HULC QM Tcam Mem

61 0 3736 3736 7160 0 0 Authenticate Pro

62 0 0 0 7160 0 0 HULC PBR Tcam Me

63 0 368232 1588 377792 0 0 EEM ED ND

64 0 4556 0 14716 0 0 EEM ED Identity

65 0 4556 0 14716 0 0 EEM ED MAT

66 0 7784 0 7784 0 0 HL2MCM

67 0 7784 0 7784 0 0 HL2MCM

68 0 0 0 7160 0 0 HRPC asic-stats

69 0 0 0 4160 0 0 HRPC hsm request

70 0 4088 0 17248 0 0 Stack Mgr

71 0 643660 139340 281928 0 0 Stack Mgr Notifi

72 0 5344 232 12272 0 0 Fifo Error Detec

73 0 1336 1336 4160 0 0 Adjust Regions

74 0 0 0 7160 0 0 hrpc -> response

75 0 0 0 7160 0 0 hrpc -> request

76 0 65848 436 7160 0 0 hrpc <- response

77 0 0 0 7160 0 0 HRPC hcomp reque

78 0 560 232 7160 0 0 HULC Device Mana

79 0 108 108 7160 0 0 HRPC hdm non blo

80 0 176 176 7160 0 0 HRPC hdm blockin

81 0 0 0 13160 0 0 HIPC bkgrd proce

82 0 232 232 7160 0 0 Hulc Port-Securi

83 0 0 0 7160 0 0 HRPC hpsecure re

84 0 0 0 7160 0 0 HRPC hlfm reques

85 0 232 232 7160 0 0 HLFM address lea

86 0 232 232 7160 0 0 HLFM aging proce

87 0 232 232 7160 0 0 HLFM address ret

88 0 0 0 7160 0 0 HRPC hulc misc r

89 0 0 0 7160 0 0 HULC Tcam Memory

90 0 0 0 13160 0 0 HVLAN main bkgrd

91 0 232 232 7160 0 0 HVLAN Mapped Vla

92 0 232 232 7160 0 0 Vlan shutdown Pr

93 0 65740 0 7160 0 0 HRPC vlan reques

94 0 0 0 7160 0 0 HULC VLAN REF Ba

95 0 0 0 7160 0 0 HRPC ilp request

96 0 0 0 7160 0 0 HULC PM Vector P

97 0 0 0 7160 0 0 HPM Msg Retry Pr

98 0 2840 2508 7540 0 0 SpanTree Helper

99 0 2412 232 12340 0 0 hpm main process

100 0 232 232 7160 0 0 HPM Stack Sync P

101 0 0 0 25160 0 0 HRPC pm request

102 0 232 232 7160 0 0 hpm counter proc

103 0 0 0 7160 0 0 HRPC pm-counters

104 0 0 0 7172 0 0 hpm vp events ca

105 0 0 0 7160 0 0 HRPC hcmp reques

106 0 0 0 7160 0 0 HACL Queue Proce

107 0 0 0 7160 0 0 HRPC acl request

108 0 232 1412 61160 0 0 HACL Acl Manager

109 0 0 0 7160 0 0 HRPC backup inte

111 0 0 0 7160 0 0 HRPC cdp request

112 0 0 0 7160 0 0 HULC CISP Proces

113 0 0 0 7160 0 0 HRPC lldp reques

114 0 0 0 7160 0 0 HRPC dot1x reque

115 0 10044 0 17204 0 0 HULC DOT1X Proce

116 0 0 0 4160 0 0 HRPC system mtu

117 0 0 0 7160 0 0 SMI MSG Retry Pr

118 0 0 0 10160 0 0 HRPC Smart Insta

119 0 0 0 7160 0 0 HRPC sdm request

120 0 232 232 7160 0 0 Hulc Storm Contr

121 0 232 232 7160 0 0 HSTP Sync Proces

122 0 0 0 7160 0 0 HRPC stp_cli req

123 0 0 0 7160 0 0 HRPC stp_state_s

124 0 232 232 7160 0 0 S/W Bridge Proce

125 0 0 0 7160 0 0 HRPC hudld reque

126 0 0 0 7160 0 0 HRPC vqpc reques

127 0 0 0 7160 0 0 HRPC aim request

128 0 0 0 13160 0 0 HCEF ADJ Refresh

129 0 0 0 7160 0 0 HRPC hled reques

130 0 232 232 10160 0 0 Hulc LED Process

131 0 0 0 13160 0 0 HL3U bkgrd proce

132 0 0 0 7160 0 0 HRPC hl3u reques

133 0 0 0 61160 0 0 HL3U PBR bkgrd p

134 0 0 0 7160 0 0 HL3U PBR n-h res

135 0 0 0 7160 0 0 HRPC dtp request

136 0 0 0 7160 0 0 HRPC show_forwar

137 0 0 0 7160 0 0 HRPC snmp reques

138 0 0 0 7160 0 0 HULC SNMP Proces

139 0 546652 544816 8996 0 0 HQM Stack Proces

140 0 264360 264360 61160 0 0 HRPC qos request

141 0 0 0 7160 0 0 HRPC span reques

142 0 0 0 7160 0 0 HRPC system post

143 0 0 0 7160 0 0 Hulc Reload Mana

144 0 0 0 7160 0 0 HRPC hrcli-event

145 0 232 232 10160 0 0 image mgr

146 0 232 232 7160 0 0 PI MATM Aging Pr

147 0 71328 0 78488 0 0 Switch Backup In

148 0 0 0 7160 0 0 MMN bkgrd proces

149 0 135632 10492 138300 0 0 Auth Manager

150 0 0 0 7160 0 0 AUTH POLICY Fram

151 0 232 232 7160 0 0 Dot1x Mgr Proces

152 0 34912 4856 27804 0 0 802.1x switch

153 0 0 0 7160 0 0 802.1x Webauth F

154 0 880 232 7808 0 0 DTP Protocol

155 0 0 0 7160 0 0 EAP Framework

156 0 0 0 7160 0 0 EAP Test

157 0 0 0 7160 0 0 HRPC dai request

158 0 0 0 4160 0 0 HULC DAI Process

159 0 0 0 7160 0 0 HRPC ip device t

160 0 0 0 7160 0 0 HRPC ip source g

161 0 0 0 7160 0 0 HULC IP Source g

162 0 524 232 7452 0 0 UDLD

163 0 216 0 7376 0 0 Port-Security

164 0 232 232 10160 0 0 IP Host Track Pr

165 0 0 0 7160 0 0 Link State Group

166 0 0 0 7160 0 0 VMATM Callback

167 0 232 232 7160 0 0 CMD HANDLER

168 0 216 0 13376 0 0 CEF switching ba

169 0 216 0 4376 0 0 ADJ NSF process

170 0 0 0 7160 0 0 HRPC sisf reques

171 0 0 0 7160 0 0 HULC SISF Proces

172 0 0 0 7160 0 0 HULC SISF Source

173 0 232 232 7160 0 0 AAA Server

174 0 0 0 7160 0 0 AAA ACCT Proc

175 0 49352 0 56512 0 0 ACCT Periodic Pr

176 0 0 0 7172 0 0 AAA System Acct

177 0 0 0 7172 0 0 Auth-proxy AAA B

178 0 0 0 7160 0 0 IP Admin SM Proc

179 0 537656 300 545616 0 0 CDP Protocol

180 0 28508 22156 31448 0 0 Exec

181 0 0 0 7176 0 0 Critical Auth

182 0 232 232 7160 0 0 AAA Dictionary R

183 0 0 0 7160 0 0 DHCP Snooping

184 0 0 0 7160 0 0 DHCP Snooping db

185 0 1104 0 14264 0 0 IP ARP Adjacency

186 0 33012 0 40172 0 0 IP ARP Retry Age

187 0 0 0 13160 0 0 IP Input

188 0 0 0 7160 0 0 ICMP event handl

189 0 232 232 10160 0 0 IP ARP Track

190 0 0 0 7160 0 0 IPv6 ping proces

192 0 13204 236 38404 0 0 SMI Director DB

193 0 0 752 7160 0 0 SMI CDP Update H

194 0 680 0 25840 0 0 SMI Backup Proce

195 0 3892 232 10820 0 0 SMI IBC server p

196 0 372 0 13532 0 0 SMI IBC client p

197 0 4884 232 17812 0 0 SMI IBC Download

198 0 1736 232 7376 0 0 Spanning Tree

199 0 5044 0 12204 0 0 Spanning Tree St

200 0 232 232 7160 0 0 Ethchnl

201 0 0 0 13160 0 0 TCP Timer

202 0 0 0 13160 0 0 TCP Protocols

203 0 0 0 7160 0 0 Socket Timers

204 0 11860 0 22020 0 0 HTTP CORE

205 0 232 232 7160 0 0 Cluster L2

206 0 0 0 7160 0 0 Cluster RARP

207 0 1504 968 7696 0 0 Cluster Base

208 0 6264 76 19348 0 0 CEF background p

209 0 216 0 13420 0 0 fib_fib_bfd_sb e

210 0 0 0 10160 0 0 IP IRDP

211 0 0 0 7160 0 0 RARP Input

212 0 448 232 7376 0 0 Dot1x Supplicant

213 0 448 232 7376 0 0 Dot1x Supplicant

214 0 448 232 7376 0 0 Dot1x Supplicant

215 0 0 0 7160 0 0 MAB Framework

216 0 0 0 7160 0 0 HRPC dhcp snoopi

217 0 0 0 7160 0 0 HULC DHCP Snoopi

218 0 1856 3648 8016 0 0 IGMPSN L2MCM

219 0 1264 1264 7264 0 0 IGMPSN MRD

220 0 0 0 7160 0 0 IGMPSN

221 0 0 0 7160 0 0 IGMPQR

222 0 14848 232 11744 0 0 L2TRACE SERVER

223 0 1820 3180 8404 0 0 MLDSN L2MCM

224 0 0 0 7160 0 0 MRD

225 0 0 0 7160 0 0 MLD_SNOOP

226 0 0 0 13160 0 0 Routing Topology

227 0 26384 7208 19252 0 0 CEF: IPv4 proces

228 0 216 0 7376 0 0 ADJ background

229 0 216 0 13376 0 0 ADJ resolve proc

230 0 1120 0 14280 0 0 IP RIB Update

231 0 0 0 7160 0 0 HRPC hl2mcm igmp

232 0 0 0 7160 0 0 HRPC hl2mcm mlds

233 0 232 232 10160 0 0 static

234 0 0 0 10160 0 0 IPv6 RIB Event H

235 0 736 736 7160 0 0 QoS stats proces

236 0 0 0 7160 0 0 SNMP Timers

237 0 1936 232 8864 0 0 TPLUS

238 0 0 0 7032 0 0 HCMP sync proces

239 0 0 0 7160 0 0 EPM MAIN PROCESS

240 0 232 232 7160 0 0 AAA Cached Serve

241 0 232 232 7160 0 0 LOCAL AAA

242 0 232 232 7160 0 0 ENABLE AAA

243 0 232 232 7160 0 0 LINE AAA

244 0 1936 232 8864 0 0 LDAP process

246 0 216 0 13376 0 0 Collection proce

247 0 31780 30404 8536 0 0 NIST rng proc

248 0 0 0 7160 0 0 HRPC x_setup req

249 0 0 0 7160 0 0 crypto engine pr

250 0 0 0 13160 0 0 encrypt proc

251 0 221240 186220 48908 0 0 Crypto CA

252 0 0 0 13160 0 0 Crypto PKI-CRL

253 0 4884 232 11812 0 0 VTP Trap Process

254 0 232 232 25160 0 0 ASP Process Crea

255 0 232 232 7160 0 0 DHCP Security He

256 0 0 0 7160 0 0 HCD Process

257 0 0 0 7160 0 0 HRPC cable diagn

259 0 12400 45256 7160 0 0 PM Callback

260 0 232 232 7160 0 0 dhcp snooping sw

261 0 0 220 7160 0 0 DiagCard1/-1

262 0 232 232 7160 0 0 AAA SEND STOP EV

263 0 0 0 13160 0 0 Test AAA Client

264 0 0 0 7160 0 0 Syslog Traps

266 0 216 0 7376 0 0 Online Diag EEM

267 0 85072 0 92232 0 0 EEM Server

268 0 11476 1720 16916 0 0 EEM Policy Direc

269 0 0 0 4160 0 0 Rom Random Updat

270 0 4556 0 14716 0 0 EEM ED OIR

271 0 4556 0 14716 0 0 EEM ED Timer

272 0 3816 3816 13160 0 0 Syslog

273 0 232 232 4160 0 0 STP FAST TRANSIT

274 0 232 232 4160 0 0 CSRT RAPID TRANS

275 0 66248 16772 62040 0 0 Call Home proces

276 0 4556 0 14716 0 0 EEM ED Test

277 0 4556 0 14716 0 0 EEM ED Config

278 0 4556 0 14716 0 0 EEM ED Env

279 0 4556 0 14716 0 0 EM ED GOLD

280 0 29072 6428 28340 2340 0 VLAN Manager

281 0 0 0 7160 0 0 Call Home Timer

282 0 8484 0 15640 0 0 SSH Event handle

19198748 Total

Leo Laohoo · ‎10-02-2013

Jonathan,

I don't see anything suspicious either. CPU utilization is very low. You sure you have a config on this switch?

Jonathan Hall · ‎10-02-2013

Hi Leo,

My issue isn't CPU utilisation, it's memory utilisation.

Here's the proccess memory running 12.2(55)SE8:-

Processor Pool Total: 22324560 Used: 14177896 Free: 8146664

I/O Pool Total: 4194304 Used: 2389856 Free: 1804448

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

And running 15.0(2)SE4:-

Processor Pool Total: 21505748 Used: 16838900 Free: 4666848

I/O Pool Total: 4194304 Used: 2419708 Free: 1774596

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

Thanks,

Jonathan

Leo Laohoo · ‎10-02-2013

Processor Pool Total: 66575784 Used: 46250672 Free: 20325112

I/O Pool Total: 14680064 Used: 12279420 Free: 2400644

Driver te Pool Total: 1048576 Used: 40 Free: 1048536

Above is mine from a stack of 4 2960S running 15.0(2)SE4 with PoE and 802.1x.