01-08-2014 10:52 AM - edited 03-03-2019 07:15 AM
I have in my network has a WS-C2960S-48FPD-L that is using more than 90% of memory. The IOS version installed is 150-2.SE2 (c2960s-universalk9-mz.150-2.SE2.bin), could not find out if this is a BUG. The processing is around 23%. Below are the "show processes cpu sorted 5min" and "show processes memory sorted allocated". Thanks for help.
SW_ACC_02_TR#show processes cpu sorted 5min
CPU utilization for five seconds: 24%/0%; one minute: 26%; five minutes: 25%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
155 4195888138 639952845 6556 15.59% 16.41% 16.21% 0 Hulc LED Process
4 248001934 13719097 18077 0.00% 1.14% 0.94% 0 Check heaps
124 169412938 25660800 6602 0.49% 0.53% 0.56% 0 hpm counter proc
212 118535249 574852125 206 0.59% 0.51% 0.48% 0 Spanning Tree
120 41576380 332054932 125 0.29% 0.31% 0.27% 0 hpm main process
166 59299274 5119156 11583 0.19% 0.22% 0.20% 0 HQM Stack Proces
85 31122391 1161331958 26 0.39% 0.17% 0.18% 0 RedEarth Rx Mana
10 46775005 426883 109574 0.00% 0.13% 0.15% 0 Licensing Auto U
284 19497943 56623180 344 0.19% 0.14% 0.15% 0 Marvell wk-a Pow
103 20399008 703273565 29 0.00% 0.13% 0.12% 0 HLFM address lea
78 15964116 925562055 17 0.00% 0.10% 0.10% 0 Draught link sta
84 20573751 272991015 75 0.09% 0.09% 0.09% 0 RedEarth Tx Mana
358 8190900 2585781 3167 0.00% 0.12% 0.08% 0 SNMP ENGINE
56 35021439 25699423 1362 0.00% 0.08% 0.07% 0 Per-Second Jobs
198 12025895 703271618 17 0.09% 0.04% 0.06% 0 IP ARP Retry Age
105 12471548 706772431 17 0.00% 0.05% 0.05% 0 HLFM address ret
186 19272066 15440290 1248 0.00% 0.06% 0.05% 0 CDP Protocol
44 14077734 31430059 447 0.00% 0.04% 0.05% 0 Net Background
12 11847905 35899234 330 0.00% 0.05% 0.04% 0 ARP Input
199 7505111 33071459 226 0.00% 0.02% 0.03% 0 IP Input
228 9706327 25585689 379 0.00% 0.02% 0.03% 0 PI MATM Aging Pr
356 3004646 4774701 629 0.00% 0.04% 0.01% 0 IP SNMP
229 6628564 256439705 25 0.00% 0.02% 0.01% 0 UDLD
357 1236671 2523911 489 0.00% 0.03% 0.00% 0 PDU DISPATCHER
161 230 2446 94 0.39% 0.04% 0.00% 1 Virtual Exec
147 4448561 125135068 35 0.00% 0.04% 0.00% 0 Hulc Storm Contr
55 9622045 444117 21665 0.39% 0.04% 0.00% 0 Per-minute Jobs
53 1861943 34905451 53 0.09% 0.00% 0.00% 0 Net Input
167 5308849 10236218 518 0.09% 0.01% 0.00% 0 HRPC qos request
65 3560170 479944329 7 0.09% 0.01% 0.00% 0 DownWhenLooped
_________________________________________________________________________________________________
SW_ACC_02_TR#show processes memory sorted allocated
Processor Pool Total: 66607236 Used: 57945988 Free: 8661248
I/O Pool Total: 14680064 Used: 12340676 Free: 2339388
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
PID TTY Allocated Freed Holding Getbufs Retbufs Process
38 0 3637735668 3638354640 7692 0 0 crypto sw pk pro
167 0 3416215012 3415330192 90424 492372 0 HRPC qos request
166 0 2108866696 968241432 170224 3395844 0 HQM Stack Proces
356 0 2088379404 2074798724 25744 7004880 0 IP SNMP
358 0 2069043972 2520358128 16416 258876 0 SNMP ENGINE
10 0 1800214384 1800214384 7160 0 0 Licensing Auto U
186 0 874988176 125196 865416 24938388 0 CDP Protocol
357 0 418086688 0 33248 0 0 PDU DISPATCHER
183 0 249277624 211662804 18074212 0 0 Auth Manager
257 0 230142288 230141476 7972 0 0 Cluster Base
271 0 225815492 225815492 7160 0 0 QoS stats proces
290 0 199977112 199958180 8732 10152 0 NIST rng proc
12 0 121517796 121481096 43860 0 0 ARP Input
120 0 97428096 92337752 58976 0 0 hpm main process
92 0 93195956 1196348440 7160 0 99828 hrpc <- response
0 0 76977196 73891632 1827512 12832067 928178 *Dead*
0 0 73660260 27113924 41827516 0 0 *Init*
5 0 68336032 132544800 32920 61954432 99416795 Pool Manager
214 0 64277948 60183624 68312 0 0 802.1x switch
107 0 46027340 45332908 7160 406080 0 HRPC hulc misc r
82 0 37271008 37271008 10160 0 0 Adjust Regions
363 0 28303364 28300096 31596 0 0 hulc running con
212 0 18442748 876 7376 0 0 Spanning Tree
199 0 17137088 22760 31028 0 0 IP Input
103 0 13616600 232 157084 0 0 HLFM address lea
187 0 8341472 6196184 8300 0 0 SpanTree Helper
332 0 6298952 12438752 11568 0 0 PM Callback
285 0 4720612 14241044 33776 0 0 MLDSN L2MCM
278 0 4699604 14251332 30388 0 0 IGMPSN L2MCM
258 0 3213116 856496 56308 0 0 DHCPD Receive
303 0 2247272 600 129208 1248696 0 Crypto IKEv2
346 0 2008824 2011716 13160 0 1692 Syslog
44 0 1973632 1837448 95588 0 0 Net Background
252 0 1811708 17030976 13160 0 0 TCP Protocols
80 0 1754692 1744796 15896 0 0 OBFL TEMP obfl0
89 0 1213028 150984 725664 0 0 Stack Mgr Notifi
333 0 1022008 1017008 18032 0 0 OBFL VOLT obfl0
1 0 642016 470972 195248 0 0 Chunk Manager
354 0 614096 0 15632 0 0 SSH Event handle
216 0 589100 227808 34860 0 0 DTP Protocol
PID TTY Allocated Freed Holding Getbufs Retbufs Process
32 0 549552 549552 7160 0 0 PrstVbl
302 0 422492 421180 8472 0 0 crypto engine pr
61 0 368228 600 377788 0 0 EEM ED Identity
176 0 294744 0 294744 0 0 HL2MCM
177 0 294744 0 294744 0 0 HL2MCM
325 0 265100 0 275260 100548 0 EEM ED Syslog
315 0 236408 201388 48884 0 0 Crypto CA
350 0 113924 90852 15020 3120 0 VLAN Manager
336 0 107464 0 114624 0 0 EEM Server
49 0 101540 84896 63848 0 0 IF-MGR control p
161 1 93412 75972 42524 0 0 Virtual Exec
108 0 92752 0 99912 53760 0 HRPC emac reques
16 0 82600 79336 73296 0 11180 Entity MIB API
55 0 78164 412 7160 45684 0 Per-minute Jobs
204 0 76192 74416 14808 0 0 OBFL MSG obfl0
311 0 71960 5324 13484 0 0 IPSEC key engine
231 0 71328 0 78488 0 0 Switch Backup In
337 0 68292 18904 48984 0 0 Call Home proces
86 0 65972 232 7160 0 0 HULC Thermal Pro
143 0 65812 232 72740 0 0 REP Helper Proc
64 0 58096 232 39864 0 0 USB Startup
237 0 56128 380 69044 0 0 CEF background p
188 0 54936 9784 39956 0 0 CEF: IPv4 proces
254 0 52588 40728 22020 0 0 HTTP CORE
13 0 51560 51560 7176 0 0 ARP Background
291 0 49584 232 56512 0 0 Timer Library
180 0 49368 0 56528 0 0 ACCT Periodic Pr
46 0 43612 232 13160 25380 0 Logger
198 0 33020 0 40180 0 0 IP ARP Retry Age
56 0 22152 0 27020 0 0 Per-Second Jobs
84 0 21584 21836 7160 0 0 RedEarth Tx Mana
197 0 20388 3196 30352 0 0 IP ARP Adjacency
282 0 14912 232 11752 0 0 L2TRACE SERVER
361 0 14764 10932 14836 0 0 SNMP Traps
338 0 13020 1720 18460 0 0 EEM Policy Direc
85 0 12844 22640 7504 0 0 RedEarth Rx Mana
301 0 12408 0 37440 0 0 LICENSE AGENT
0 0 12316 3117752 12316 0 0 *Sched*
309 0 12268 232 25196 0 0 Crypto IKMP
347 0 12164 464 24860 0 0 RBM CORE
208 0 10724 0 35884 0 0 SMI Backup Proce
329 0 10652 392 20292 0 0 OBFL ENV obfl0
274 0 10276 232 17204 0 0 REP BPA/EPA Proc
206 0 8204 3408 33364 0 0 SMI Director DB
112 0 5044 0 12204 0 0 HRPC vlan reques
PID TTY Allocated Freed Holding Getbufs Retbufs Process
320 0 4884 232 11812 0 0 VTP Trap Process
211 0 4884 232 17812 0 0 SMI IBC Download
294 0 4884 232 11848 0 0 Crypto Support
4 0 4772 500 11752 0 0 Check heaps
340 0 4556 0 14716 0 0 EEM ED Config
63 0 4556 547248440 14716 0 0 EEM ED ND
62 0 4556 0 14716 0 0 EEM ED MAT
341 0 4556 0 14716 0 0 EEM ED Env
343 0 4556 0 14716 0 0 EEM ED OIR
342 0 4556 0 14716 0 0 EM ED GOLD
345 0 4556 0 14716 0 0 EEM ED Timer
344 0 4556 0 14716 0 0 EEM ED Test
52 0 4140 4140 10160 0 0 IP Admission HA
209 0 3892 232 10820 0 0 SMI IBC server p
115 0 2244 0 7160 0 0 Hulc ILP Alchemy
300 0 1936 232 8864 0 0 TPLUS
48 0 1936 7620 14864 0 0 SXP CORE
293 0 1936 232 8864 0 0 LDAP process
26 0 1688 0 8848 0 0 IPC Seat Manager
359 0 1156 232 14092 0 0 IP SNMPV6
264 0 1120 0 14280 0 0 IP RIB Update
283 0 924 0 8084 0 0 Inline Power
193 0 868 708 7320 0 0 Dot1x Mgr Proces
95 0 640 232 7160 0 0 HULC Device Mana
Solved! Go to Solution.
01-08-2014 03:02 PM
The IOS version installed is 150-2.SE2 (c2960s-universalk9-mz.150-2.SE2.bin), could not find out if this is a BUG
Avoid using 15.0(2)SE2 at all cost. There are so many CPU and memory bugs it's well known joke. Hey, you should try 15.0(2)SE3, if you can still find this IOS, and then you'll be laughing all the way to the loonie-bin.
Use either 12.2(55)SE8 or 15.0(2)SE4 instead.
01-09-2014 10:23 AM
Hi,
To troubleshoot memory please follow the below steps:
First, please check if the memory usage is genuine or not, this can be checked by checking the memory status of any other similar switch in the network having similar devices attached. This check is important because even a 95% of memory usage could be normal depending upon the requirements of your network.
If the memory usage is really abnormal and the switch is crashing then note the time between the two crashes.
If the two crashes are in a GAP of 7 days then take the outpust of "show process memory sorted" 5 times at a gap of 24-36 hours.
It will give you an idea about the process which is constantly holding more and more memory without releasing it.
Once you know the process, you can track the relevant bug.
Moreover, on the 2960 switch the 15.X train has a lot of memory bugs, I would recommend you to downgrade it to 12.2(55) train as it is one of the most stable IOS images on this platform.
Thanks.
01-08-2014 03:02 PM
The IOS version installed is 150-2.SE2 (c2960s-universalk9-mz.150-2.SE2.bin), could not find out if this is a BUG
Avoid using 15.0(2)SE2 at all cost. There are so many CPU and memory bugs it's well known joke. Hey, you should try 15.0(2)SE3, if you can still find this IOS, and then you'll be laughing all the way to the loonie-bin.
Use either 12.2(55)SE8 or 15.0(2)SE4 instead.
01-09-2014 10:23 AM
Hi,
To troubleshoot memory please follow the below steps:
First, please check if the memory usage is genuine or not, this can be checked by checking the memory status of any other similar switch in the network having similar devices attached. This check is important because even a 95% of memory usage could be normal depending upon the requirements of your network.
If the memory usage is really abnormal and the switch is crashing then note the time between the two crashes.
If the two crashes are in a GAP of 7 days then take the outpust of "show process memory sorted" 5 times at a gap of 24-36 hours.
It will give you an idea about the process which is constantly holding more and more memory without releasing it.
Once you know the process, you can track the relevant bug.
Moreover, on the 2960 switch the 15.X train has a lot of memory bugs, I would recommend you to downgrade it to 12.2(55) train as it is one of the most stable IOS images on this platform.
Thanks.
11-08-2019 01:34 AM
In my case, i have already know that HL2MCM process is holding the highest memory without releasing it.
What can i do next to resolve this memory leaking issue.
ios 15.2(2) E7
01-10-2014 06:21 AM
Hello guys I did the replacement of 150-2.SE2 by the 12.2 (55) SE8, the problem was solved. I am very happy to have the help of you. Thank you.
Thank you Leo Laohoo!
Thank you Bhishma Khanna!
01-10-2014 06:24 AM
No Problem,
Thanks for rating the helpful posts.
01-10-2014 05:09 PM
Hello guys I did the replacement of 150-2.SE2 by the 12.2 (55) SE8, the problem was solved. I am very happy to have the help of you. Thank you.
Thank you for taking the time to give us the feedback(s) of your network improvement.
And thank you for taking the time to rate our posts.
FYI: 12.2(55)SE, starting from SE6, is considered to be one of the most stable IOS versions that ever came out of Cisco.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide