08-18-2021 01:45 AM
Hi All,
I have a WS-C2960-48PST-L (15.0(2)SE4) which is working fine on the surface but SolarWinds is reporting that the memory usage has been at 80% for a long period of time.
Using all of the memory commands I can see that there are a lot of dead processes.
ms-h000717#sh processes mem sorted
Processor Pool Total: 21751188 Used: 19248020 Free: 2503168
I/O Pool Total: 4194304 Used: 2386196 Free: 1808108
Driver te Pool Total: 1048576 Used: 40 Free: 1048536
PID TTY Allocated Freed Holding Getbufs Retbufs Process
0 0 24235572 6342940 14722908 0 0 *Init*
0 0 4129224 3158900 969332 3799839 1341026 *Dead*
152 0 12608160 11503620 556472 0 0 Auth Manager
73 0 772480 143256 511840 0 0 Stack Mgr Notifi
0 0 0 0 394440 0 0 *MallocLite*
63 0 368252 1299944 377812 0 0 EEM ED ND
182 0 3655008 1402072 372708 40608 0 CDP Protocol
113 1 3479604 3379248 109508 0 0 SSH Process
1 0 147532 52840 101852 0 0 Chunk Manager
272 0 85072 0 92232 0 0 EEM Server
150 0 71328 0 78488 0 0 Switch Backup In
111 0 5276 1884 66204 0 0 HACL Acl Manager
280 0 66144 16772 61936 0 0 Call Home proces
136 0 0 0 61160 0 0 HL3U PBR bkgrd p
143 0 609511120 609502444 61160 5076 0 HRPC qos request
43 0 97160 84504 59860 0 0 IF-MGR control p
178 0 49352 0 56512 0 0 ACCT Periodic Pr
288 0 69464 1176 52740 0 0 DHCPD Receive
295 0 64440 18336 51992 0 0 NTP
264 0 3243304 4203828 51240 0 0 PM Callback
256 0 221196 185512 48912 0 0 Crypto CA
16 0 48448 64152 45576 0 9620 Entity MIB API
12 0 38104 1404 43860 0 0 ARP Input
5 0 4854668 5504100 42192 3199868 3528879 Pool Manager
189 0 33012 0 40172 0 0 IP ARP Retry Age
87 0 80092 232 38904 0 0 HLFM address lea
195 0 13204 576 38404 0 0 SMI Director DB
142 0 656425908 656189296 35528 131976 0 HQM Stack Proces
157 0 3424764 2692064 33948 0 0 DTP Protocol
48 0 55748 27456 26964 0 0 Per-Second Jobs
197 0 680 0 25840 0 0 SMI Backup Proce
259 0 232 312 25160 0 0 ASP Process Crea
10 0 0 0 25160 0 0 Crash writer
104 0 0 0 25160 0 0 HRPC pm request
102 0 4124848 3717296 24464 0 0 hpm main process
289 0 10227624 9222680 24176 5076 0 IP SNMP
176 0 16816 232 23744 0 0 AAA Server
291 0 59813768 81125488 23204 0 0 SNMP ENGINE
39 0 41204 716 19608 0 0 Net Background
211 0 6424 76 19508 0 0 CEF background p
232 0 29912 10736 19244 0 0 CEF: IPv4 proces
292 0 5808 232 18744 0 0 IP SNMPV6
199 0 5416 0 18576 0 0 SMI IBC client p
200 0 4884 232 17812 0 0 SMI IBC Download
74 0 10336 232 17264 0 0 Fifo Error Detec
281 0 6868 0 17028 0 0 EEM ED Test
273 0 11476 1720 16916 0 0 EEM Policy Direc
188 0 2588 0 15748 0 0 IP ARP Adjacency
287 0 10308 0 15592 0 0 SSH Event handle
277 0 210008 207772 15396 0 0 Syslog
285 0 381240 89620 15372 3120 0 VLAN Manager
205 0 44940 0 15068 0 0 TCP Protocols
276 0 4556 0 14756 0 0 EEM ED Timer
275 0 4556 0 14716 0 0 EEM ED OIR
64 0 4556 0 14716 0 0 EEM ED Identity
282 0 4556 0 14716 0 0 EEM ED Config
284 0 4556 0 14716 0 0 EM ED GOLD
283 0 4556 0 14716 0 0 EEM ED Env
PID TTY Allocated Freed Holding Getbufs Retbufs Process
65 0 4556 0 14716 0 0 EEM ED MAT
190 0 3852 2568 14444 0 0 IP Input
235 0 1120 0 14280 0 0 IP RIB Update
155 0 1201856 1049476 13428 0 0 802.1x switch
212 0 216 0 13420 0 0 fib_fib_bfd_sb e
234 0 216 0 13384 0 0 ADJ resolve proc
236 0 216 0 13376 0 0 Collection proce
171 0 216 0 13376 0 0 CEF switching ba
268 0 0 0 13160 0 0 Test AAA Client
33 0 232 232 13160 0 0 XML Proxy Client
255 0 0 0 13160 0 0 encrypt proc
15 0 0 0 13160 0 0 Policy Manager
290 0 21319972 983508 13160 0 0 PDU DISPATCHER
41 0 60964 232 13160 35532 0 Logger
293 0 0 0 13160 0 0 SNMP ConfCopyPro
257 0 0 0 13160 0 0 Crypto PKI-CRL
294 0 0 0 13160 0 0 SNMP Traps
72 0 0 0 13160 0 0 Stack Mgr
40 0 0 0 13160 0 0 IDB Work
231 0 0 0 13160 0 0 Routing Topology
204 0 0 0 13160 0 0 TCP Timer
131 0 0 0 13160 0 0 HCEF ADJ Refresh
83 0 0 0 13160 0 0 HIPC bkgrd proce
92 0 0 0 13160 0 0 HVLAN main bkgrd
134 0 0 0 13160 0 0 HL3U bkgrd proce
0 0 12180 196488 12180 0 0 *Sched*
207 0 12224 9984 12036 0 0 HTTP CORE
258 0 4884 232 11812 0 0 VTP Trap Process
4 0 4772 232 11752 0 0 Check heaps
225 0 14800 232 11696 0 0 L2TRACE SERVER
198 0 3892 232 10820 0 0 SMI IBC server p
133 0 232 232 10160 0 0 Hulc LED Process
213 0 0 0 10160 0 0 IP IRDP
121 0 0 0 10160 0 0 HRPC Smart Insta
192 0 232 232 10160 0 0 IP ARP Track
239 0 0 0 10160 0 0 IPv6 RIB Event H
167 0 232 232 10160 0 0 IP Host Track Pr
238 0 232 232 10160 0 0 static
148 0 232 232 10160 0 0 image mgr
3 0 0 0 10032 0 0 Connection Mgr
66 0 9856 0 9856 0 0 HL2MCM
226 0 514300 1124000 9452 0 0 MLDSN L2MCM
221 0 510868 1119868 8944 0 0 IGMPSN L2MCM
249 0 1936 232 8864 0 0 LDAP process
242 0 1936 232 8864 0 0 TPLUS
26 0 1688 0 8848 0 0 IPC Seat Manager
237 0 641420 640044 8536 0 0 NIST rng proc
67 0 7820 0 7820 0 0 HL2MCM
210 0 329280 328680 7760 0 0 Cluster Base
9 0 11099952 11117100 7692 0 0 crypto sw pk pro
70 0 23176 43124 7528 0 0 RedEarth Rx Mana
11 0 340 0 7500 0 0 Exception contro
165 0 524 232 7452 0 0 UDLD
215 0 448 232 7376 0 0 Dot1x Supplicant
233 0 216 0 7376 0 0 ADJ background
216 0 448 232 7376 0 0 Dot1x Supplicant
271 0 216 0 7376 0 0 Online Diag EEM
201 0 940748 232 7376 0 0 Spanning Tree
166 0 216 0 7376 0 0 Port-Security
217 0 448 232 7376 0 0 Dot1x Supplicant
230 0 152 0 7312 0 0 Marvell wk-a Pow
296 0 216 0 7248 0 0 DHCPD Database
38 0 0 0 7224 0 0 Critical Bkgnd
PID TTY Allocated Freed Holding Getbufs Retbufs Process
80 0 560 232 7224 0 0 HULC Device Mana
179 0 0 0 7172 0 0 AAA System Acct
117 0 0 0 7160 0 0 HRPC dot1x reque
118 0 0 0 7160 0 0 HULC DOT1X Proce
120 0 0 0 7160 0 0 SMI MSG Retry Pr
116 0 0 0 7160 0 0 HRPC lldp reques
124 0 232 232 7160 0 0 HSTP Sync Proces
125 0 0 0 7160 0 0 HRPC stp_cli req
122 0 0 0 7160 0 0 HRPC sdm request
127 0 232 232 7160 0 0 S/W Bridge Proce
123 0 232 232 7160 0 0 Hulc Storm Contr
129 0 0 0 7160 0 0 HRPC vqpc reques
126 0 0 0 7160 0 0 HRPC stp_state_s
115 0 0 0 7160 0 0 HULC CISP Proces
132 0 0 0 7160 0 0 HRPC hled reques
114 0 0 0 7160 0 0 HRPC cdp request
112 0 0 0 7160 0 0 HRPC backup inte
135 0 0 0 7160 0 0 HRPC hl3u reques
128 0 0 0 7160 0 0 HRPC hudld reque
137 0 0 0 7160 0 0 HL3U PBR n-h res
138 0 0 0 7160 0 0 HRPC dtp request
130 0 0 0 7160 0 0 HRPC aim request
140 0 0 0 7160 0 0 HRPC snmp reques
141 0 0 0 7160 0 0 HULC SNMP Proces
109 0 0 0 7160 0 0 HACL Queue Proce
108 0 0 0 7160 0 0 HRPC hcmp reques
110 0 0 0 7160 0 0 HRPC acl request
145 0 0 0 7160 0 0 HRPC system post
146 0 0 0 7160 0 0 Hulc Reload Mana
147 0 0 0 7160 0 0 HRPC hrcli-event
106 0 0 0 7160 0 0 HRPC pm-counters
149 0 232 48348 7160 0 0 PI MATM Aging Pr
105 0 232 232 7160 0 0 hpm counter proc
151 0 0 0 7160 0 0 MMN bkgrd proces
103 0 232 232 7160 0 0 HPM Stack Sync P
153 0 0 0 7160 0 0 AUTH POLICY Fram
154 0 232 232 7160 0 0 Dot1x Mgr Proces
139 0 0 0 7160 0 0 HRPC show_forwar
156 0 0 0 7160 0 0 802.1x Webauth F
100 0 0 0 7160 0 0 HPM Msg Retry Pr
158 0 0 0 7160 0 0 EAP Framework
159 0 0 0 7160 0 0 EAP Test
160 0 0 0 7160 0 0 HRPC dai request
107 0 0 0 7160 0 0 hpm vp events ca
162 0 0 0 7160 0 0 HRPC ip device t
163 0 0 0 7160 0 0 HRPC ip source g
164 0 0 0 7160 0 0 HULC IP Source g
99 0 0 0 7160 0 0 HULC PM Vector P
98 0 5672 0 7160 0 0 Hulc ILP Alchemy
97 0 0 0 7160 0 0 HRPC ilp request
168 0 0 0 7160 0 0 Link State Group
169 0 0 0 7160 0 0 VMATM Callback
170 0 232 232 7160 0 0 CMD HANDLER
96 0 0 0 7160 0 0 HULC VLAN REF Ba
206 0 0 0 7160 0 0 Socket Timers
173 0 0 0 7160 0 0 HRPC sisf reques
174 0 0 0 7160 0 0 HULC SISF Proces
175 0 0 0 7160 0 0 HULC SISF Source
95 0 0 0 7160 0 0 HRPC vlan reques
177 0 0 0 7160 0 0 AAA ACCT Proc
94 0 232 232 7160 0 0 Vlan shutdown Pr
53 0 232 232 7160 0 0 DownWhenLooped
180 0 0 0 7160 0 0 Auth-proxy AAA B
PID TTY Allocated Freed Holding Getbufs Retbufs Process
181 0 0 0 7160 0 0 IP Admin SM Proc
93 0 232 232 7160 0 0 HVLAN Mapped Vla
184 0 0 0 7160 0 0 Critical Auth
185 0 232 232 7160 0 0 AAA Dictionary R
186 0 0 0 7160 0 0 DHCP Snooping
187 0 0 0 7160 0 0 DHCP Snooping db
91 0 0 0 7160 0 0 HULC Tcam Memory
90 0 0 0 7160 0 0 HRPC hulc misc r
89 0 232 232 7160 0 0 HLFM address ret
191 0 0 0 7160 0 0 ICMP event handl
88 0 232 232 7160 0 0 HLFM aging proce
193 0 0 0 7160 0 0 IPv6 ping proces
86 0 0 0 7160 0 0 HRPC hlfm reques
196 0 0 405372 7160 0 0 SMI CDP Update H
85 0 0 0 7160 0 0 HRPC hpsecure re
84 0 232 232 7160 0 0 Hulc Port-Securi
82 0 176 176 7160 0 0 HRPC hdm blockin
81 0 68 68 7160 0 0 HRPC hdm non blo
49 0 0 0 7160 0 0 AggMgr Process
202 0 0 0 7160 0 0 Spanning Tree St
203 0 232 232 7160 0 0 Ethchnl
79 0 0 0 7160 0 0 HRPC hcomp reque
78 0 68 396 7160 0 0 hrpc <- response
248 0 232 232 7160 0 0 LINE AAA
77 0 0 0 7160 0 0 hrpc -> request
208 0 232 232 7160 0 0 Cluster L2
209 0 0 0 7160 0 0 Cluster RARP
76 0 0 0 7160 0 0 hrpc -> response
251 0 0 0 7160 0 0 HRPC hl2mcm igmp
254 0 0 0 7160 0 0 crypto engine pr
69 0 41540 41272 7160 0 0 RedEarth Tx Mana
214 0 0 0 7160 0 0 RARP Input
68 0 0 0 7160 0 0 HRPC asic-stats
62 0 0 0 7160 0 0 HULC PBR Tcam Me
61 0 1814956 1814956 7160 0 0 Authenticate Pro
218 0 0 0 7160 0 0 MAB Framework
219 0 0 0 7160 0 0 HRPC dhcp snoopi
220 0 0 0 7160 0 0 HULC DHCP Snoopi
60 0 0 0 7160 0 0 HULC QM Tcam Mem
222 0 0 0 7160 0 0 IGMPSN MRD
223 0 0 0 7160 0 0 IGMPSN
224 0 0 0 7160 0 0 IGMPQR
59 0 0 0 7160 0 0 PSP Timer
58 0 0 0 7160 0 0 HRPC actual powe
227 0 0 0 7160 0 0 MRD
228 0 0 0 7160 0 0 MLD_SNOOP
229 0 0 0 7160 0 0 Inline Power
57 0 0 0 7160 0 0 HRPC EnergyWise
56 0 232 232 7160 0 0 HLPIP Sync Proce
55 0 0 0 7160 0 0 HRPC lpip reques
54 0 0 0 7160 0 0 HULC ACL Tcam Me
44 0 0 0 7160 0 0 IF-MGR event pro
52 0 0 0 7160 0 0 SFF8472
51 0 0 0 7160 0 0 HC Counter Timer
50 0 0 364 7160 0 0 Transport Port A
266 0 0 220 7160 0 0 DiagCard1/-1
47 0 0 424 7160 0 0 Per-minute Jobs
240 0 702252 702252 7160 0 0 QoS stats proces
241 0 0 0 7160 0 0 SNMP Timers
46 0 232 232 7160 0 0 Compute load avg
34 0 0 0 7160 0 0 Dynamic ARP Insp
244 0 0 0 7160 0 0 EPM MAIN PROCESS
245 0 232 232 7160 0 0 AAA Cached Serve
PID TTY Allocated Freed Holding Getbufs Retbufs Process
246 0 1000 1000 7160 0 0 LOCAL AAA
247 0 232 232 7160 0 0 ENABLE AAA
32 0 77340 77340 7160 0 0 PrstVbl
45 0 0 0 7160 0 0 Net Input
30 0 0 0 7160 0 0 IPC Keep Alive M
252 0 0 0 7160 0 0 HRPC hl2mcm mlds
253 0 0 0 7160 0 0 HRPC x_setup req
29 0 0 0 7160 0 0 IPC Seat TX Cont
27 0 0 0 7160 0 0 IPC Check Queue
42 0 232 232 7160 0 0 TTY Background
28 0 0 0 7160 0 0 IPC Seat RX Cont
37 0 232 232 7160 0 0 GraphIt
36 0 232 232 7160 0 0 SMART
260 0 232 232 7160 0 0 DHCP Security He
261 0 0 0 7160 0 0 HCD Process
262 0 0 0 7160 0 0 HRPC cable diagn
35 0 0 0 7160 0 0 ARP Snoop
265 0 232 232 7160 0 0 dhcp snooping sw
6 0 0 0 7160 0 0 DiscardQ Backgro
267 0 173388 902180 7160 0 0 AAA SEND STOP EV
24 0 0 0 7160 0 0 IPC Deferred Por
269 0 0 0 7160 0 0 Syslog Traps
23 0 0 0 7160 0 0 IPC Periodic Tim
22 0 0 0 7160 0 0 IPC Zone Manager
31 0 0 0 7160 0 0 IPC Loadometer
21 0 0 0 7160 0 0 IPC Session Serv
7 0 232 232 7160 0 0 Timers
18 0 0 0 7160 0 0 IPC Event Notifi
13 0 5416 5416 7160 0 0 ARP Background
144 0 0 0 7160 0 0 HRPC span reques
14 0 0 0 7160 0 0 AAA_SERVER_DEADT
25 0 0 0 7160 0 0 IPC Process leve
19 0 0 0 7160 0 0 IPC Mcast Pendin
17 0 0 0 7160 0 0 IFS Agent Manage
20 0 0 0 7160 0 0 IPC Dynamic Cach
286 0 0 0 7160 0 0 Call Home Timer
243 0 0 0 7032 0 0 HCMP sync proces
101 0 1377288 1368068 7032 0 0 SpanTree Helper
270 0 0 0 7032 0 0 DHCPD Timer
172 0 216 0 4376 0 0 ADJ NSF process
71 0 0 0 4208 0 0 HRPC hsm request
279 0 232 232 4160 0 0 CSRT RAPID TRANS
161 0 0 0 4160 0 0 HULC DAI Process
8 0 0 0 4160 0 0 WATCH_AFS
75 0 1336 1336 4160 0 0 Adjust Regions
119 0 0 0 4160 0 0 HRPC system mtu
2 0 232 232 4160 0 0 Load Meter
278 0 232 232 4160 0 0 STP FAST TRANSIT
21629056 TotalCan anyway assist me in finding out what is causing this high usage, preferably down to what port is causing it?
Many thanks.
08-18-2021 01:58 AM
Hello,
this is often related to buggy software. Try and upgrade to 15.0.2-SE11...
https://software.cisco.com/download/home/282202842/type/280805680/release/15.0.2-SE11
08-18-2021 02:06 AM
As mentioned other poster, 15.0(X) some version have bugs, do you have any SNMP polling frequenty, worth checking disable and see if that improves.
Not sure may be bug :
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut06609/?rfs=iqvred
worth checking this thread :
=====Preenayamo Vasudevam=====
***** Rate All Helpful Responses *****
08-18-2021 02:17 AM
Hi @Georg Pauwen thanks for the quick suggestion.
I have the updated software to try but last time I did this I encountered an issue with not enough free storage space.
Last time I unticked the option to backup existing firmware image but the whole switch crashed and I ended up swapping it out with another device (this one) both of them have the same firmware but also the same 80% usage issue.
08-18-2021 02:35 AM
Hello,
that doesn't sound good, you don't want to have to replace the entire switch...
Check if your switch supports the global command below:
no macro auto monitor
If so, configure that and monitor if that changes the memory usage...
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide