08-18-2021 01:45 AM
Hi All,
I have a WS-C2960-48PST-L (15.0(2)SE4) which is working fine on the surface but SolarWinds is reporting that the memory usage has been at 80% for a long period of time.
Using all of the memory commands I can see that there are a lot of dead processes.
ms-h000717#sh processes mem sorted Processor Pool Total: 21751188 Used: 19248020 Free: 2503168 I/O Pool Total: 4194304 Used: 2386196 Free: 1808108 Driver te Pool Total: 1048576 Used: 40 Free: 1048536 PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 24235572 6342940 14722908 0 0 *Init* 0 0 4129224 3158900 969332 3799839 1341026 *Dead* 152 0 12608160 11503620 556472 0 0 Auth Manager 73 0 772480 143256 511840 0 0 Stack Mgr Notifi 0 0 0 0 394440 0 0 *MallocLite* 63 0 368252 1299944 377812 0 0 EEM ED ND 182 0 3655008 1402072 372708 40608 0 CDP Protocol 113 1 3479604 3379248 109508 0 0 SSH Process 1 0 147532 52840 101852 0 0 Chunk Manager 272 0 85072 0 92232 0 0 EEM Server 150 0 71328 0 78488 0 0 Switch Backup In 111 0 5276 1884 66204 0 0 HACL Acl Manager 280 0 66144 16772 61936 0 0 Call Home proces 136 0 0 0 61160 0 0 HL3U PBR bkgrd p 143 0 609511120 609502444 61160 5076 0 HRPC qos request 43 0 97160 84504 59860 0 0 IF-MGR control p 178 0 49352 0 56512 0 0 ACCT Periodic Pr 288 0 69464 1176 52740 0 0 DHCPD Receive 295 0 64440 18336 51992 0 0 NTP 264 0 3243304 4203828 51240 0 0 PM Callback 256 0 221196 185512 48912 0 0 Crypto CA 16 0 48448 64152 45576 0 9620 Entity MIB API 12 0 38104 1404 43860 0 0 ARP Input 5 0 4854668 5504100 42192 3199868 3528879 Pool Manager 189 0 33012 0 40172 0 0 IP ARP Retry Age 87 0 80092 232 38904 0 0 HLFM address lea 195 0 13204 576 38404 0 0 SMI Director DB 142 0 656425908 656189296 35528 131976 0 HQM Stack Proces 157 0 3424764 2692064 33948 0 0 DTP Protocol 48 0 55748 27456 26964 0 0 Per-Second Jobs 197 0 680 0 25840 0 0 SMI Backup Proce 259 0 232 312 25160 0 0 ASP Process Crea 10 0 0 0 25160 0 0 Crash writer 104 0 0 0 25160 0 0 HRPC pm request 102 0 4124848 3717296 24464 0 0 hpm main process 289 0 10227624 9222680 24176 5076 0 IP SNMP 176 0 16816 232 23744 0 0 AAA Server 291 0 59813768 81125488 23204 0 0 SNMP ENGINE 39 0 41204 716 19608 0 0 Net Background 211 0 6424 76 19508 0 0 CEF background p 232 0 29912 10736 19244 0 0 CEF: IPv4 proces 292 0 5808 232 18744 0 0 IP SNMPV6 199 0 5416 0 18576 0 0 SMI IBC client p 200 0 4884 232 17812 0 0 SMI IBC Download 74 0 10336 232 17264 0 0 Fifo Error Detec 281 0 6868 0 17028 0 0 EEM ED Test 273 0 11476 1720 16916 0 0 EEM Policy Direc 188 0 2588 0 15748 0 0 IP ARP Adjacency 287 0 10308 0 15592 0 0 SSH Event handle 277 0 210008 207772 15396 0 0 Syslog 285 0 381240 89620 15372 3120 0 VLAN Manager 205 0 44940 0 15068 0 0 TCP Protocols 276 0 4556 0 14756 0 0 EEM ED Timer 275 0 4556 0 14716 0 0 EEM ED OIR 64 0 4556 0 14716 0 0 EEM ED Identity 282 0 4556 0 14716 0 0 EEM ED Config 284 0 4556 0 14716 0 0 EM ED GOLD 283 0 4556 0 14716 0 0 EEM ED Env PID TTY Allocated Freed Holding Getbufs Retbufs Process 65 0 4556 0 14716 0 0 EEM ED MAT 190 0 3852 2568 14444 0 0 IP Input 235 0 1120 0 14280 0 0 IP RIB Update 155 0 1201856 1049476 13428 0 0 802.1x switch 212 0 216 0 13420 0 0 fib_fib_bfd_sb e 234 0 216 0 13384 0 0 ADJ resolve proc 236 0 216 0 13376 0 0 Collection proce 171 0 216 0 13376 0 0 CEF switching ba 268 0 0 0 13160 0 0 Test AAA Client 33 0 232 232 13160 0 0 XML Proxy Client 255 0 0 0 13160 0 0 encrypt proc 15 0 0 0 13160 0 0 Policy Manager 290 0 21319972 983508 13160 0 0 PDU DISPATCHER 41 0 60964 232 13160 35532 0 Logger 293 0 0 0 13160 0 0 SNMP ConfCopyPro 257 0 0 0 13160 0 0 Crypto PKI-CRL 294 0 0 0 13160 0 0 SNMP Traps 72 0 0 0 13160 0 0 Stack Mgr 40 0 0 0 13160 0 0 IDB Work 231 0 0 0 13160 0 0 Routing Topology 204 0 0 0 13160 0 0 TCP Timer 131 0 0 0 13160 0 0 HCEF ADJ Refresh 83 0 0 0 13160 0 0 HIPC bkgrd proce 92 0 0 0 13160 0 0 HVLAN main bkgrd 134 0 0 0 13160 0 0 HL3U bkgrd proce 0 0 12180 196488 12180 0 0 *Sched* 207 0 12224 9984 12036 0 0 HTTP CORE 258 0 4884 232 11812 0 0 VTP Trap Process 4 0 4772 232 11752 0 0 Check heaps 225 0 14800 232 11696 0 0 L2TRACE SERVER 198 0 3892 232 10820 0 0 SMI IBC server p 133 0 232 232 10160 0 0 Hulc LED Process 213 0 0 0 10160 0 0 IP IRDP 121 0 0 0 10160 0 0 HRPC Smart Insta 192 0 232 232 10160 0 0 IP ARP Track 239 0 0 0 10160 0 0 IPv6 RIB Event H 167 0 232 232 10160 0 0 IP Host Track Pr 238 0 232 232 10160 0 0 static 148 0 232 232 10160 0 0 image mgr 3 0 0 0 10032 0 0 Connection Mgr 66 0 9856 0 9856 0 0 HL2MCM 226 0 514300 1124000 9452 0 0 MLDSN L2MCM 221 0 510868 1119868 8944 0 0 IGMPSN L2MCM 249 0 1936 232 8864 0 0 LDAP process 242 0 1936 232 8864 0 0 TPLUS 26 0 1688 0 8848 0 0 IPC Seat Manager 237 0 641420 640044 8536 0 0 NIST rng proc 67 0 7820 0 7820 0 0 HL2MCM 210 0 329280 328680 7760 0 0 Cluster Base 9 0 11099952 11117100 7692 0 0 crypto sw pk pro 70 0 23176 43124 7528 0 0 RedEarth Rx Mana 11 0 340 0 7500 0 0 Exception contro 165 0 524 232 7452 0 0 UDLD 215 0 448 232 7376 0 0 Dot1x Supplicant 233 0 216 0 7376 0 0 ADJ background 216 0 448 232 7376 0 0 Dot1x Supplicant 271 0 216 0 7376 0 0 Online Diag EEM 201 0 940748 232 7376 0 0 Spanning Tree 166 0 216 0 7376 0 0 Port-Security 217 0 448 232 7376 0 0 Dot1x Supplicant 230 0 152 0 7312 0 0 Marvell wk-a Pow 296 0 216 0 7248 0 0 DHCPD Database 38 0 0 0 7224 0 0 Critical Bkgnd PID TTY Allocated Freed Holding Getbufs Retbufs Process 80 0 560 232 7224 0 0 HULC Device Mana 179 0 0 0 7172 0 0 AAA System Acct 117 0 0 0 7160 0 0 HRPC dot1x reque 118 0 0 0 7160 0 0 HULC DOT1X Proce 120 0 0 0 7160 0 0 SMI MSG Retry Pr 116 0 0 0 7160 0 0 HRPC lldp reques 124 0 232 232 7160 0 0 HSTP Sync Proces 125 0 0 0 7160 0 0 HRPC stp_cli req 122 0 0 0 7160 0 0 HRPC sdm request 127 0 232 232 7160 0 0 S/W Bridge Proce 123 0 232 232 7160 0 0 Hulc Storm Contr 129 0 0 0 7160 0 0 HRPC vqpc reques 126 0 0 0 7160 0 0 HRPC stp_state_s 115 0 0 0 7160 0 0 HULC CISP Proces 132 0 0 0 7160 0 0 HRPC hled reques 114 0 0 0 7160 0 0 HRPC cdp request 112 0 0 0 7160 0 0 HRPC backup inte 135 0 0 0 7160 0 0 HRPC hl3u reques 128 0 0 0 7160 0 0 HRPC hudld reque 137 0 0 0 7160 0 0 HL3U PBR n-h res 138 0 0 0 7160 0 0 HRPC dtp request 130 0 0 0 7160 0 0 HRPC aim request 140 0 0 0 7160 0 0 HRPC snmp reques 141 0 0 0 7160 0 0 HULC SNMP Proces 109 0 0 0 7160 0 0 HACL Queue Proce 108 0 0 0 7160 0 0 HRPC hcmp reques 110 0 0 0 7160 0 0 HRPC acl request 145 0 0 0 7160 0 0 HRPC system post 146 0 0 0 7160 0 0 Hulc Reload Mana 147 0 0 0 7160 0 0 HRPC hrcli-event 106 0 0 0 7160 0 0 HRPC pm-counters 149 0 232 48348 7160 0 0 PI MATM Aging Pr 105 0 232 232 7160 0 0 hpm counter proc 151 0 0 0 7160 0 0 MMN bkgrd proces 103 0 232 232 7160 0 0 HPM Stack Sync P 153 0 0 0 7160 0 0 AUTH POLICY Fram 154 0 232 232 7160 0 0 Dot1x Mgr Proces 139 0 0 0 7160 0 0 HRPC show_forwar 156 0 0 0 7160 0 0 802.1x Webauth F 100 0 0 0 7160 0 0 HPM Msg Retry Pr 158 0 0 0 7160 0 0 EAP Framework 159 0 0 0 7160 0 0 EAP Test 160 0 0 0 7160 0 0 HRPC dai request 107 0 0 0 7160 0 0 hpm vp events ca 162 0 0 0 7160 0 0 HRPC ip device t 163 0 0 0 7160 0 0 HRPC ip source g 164 0 0 0 7160 0 0 HULC IP Source g 99 0 0 0 7160 0 0 HULC PM Vector P 98 0 5672 0 7160 0 0 Hulc ILP Alchemy 97 0 0 0 7160 0 0 HRPC ilp request 168 0 0 0 7160 0 0 Link State Group 169 0 0 0 7160 0 0 VMATM Callback 170 0 232 232 7160 0 0 CMD HANDLER 96 0 0 0 7160 0 0 HULC VLAN REF Ba 206 0 0 0 7160 0 0 Socket Timers 173 0 0 0 7160 0 0 HRPC sisf reques 174 0 0 0 7160 0 0 HULC SISF Proces 175 0 0 0 7160 0 0 HULC SISF Source 95 0 0 0 7160 0 0 HRPC vlan reques 177 0 0 0 7160 0 0 AAA ACCT Proc 94 0 232 232 7160 0 0 Vlan shutdown Pr 53 0 232 232 7160 0 0 DownWhenLooped 180 0 0 0 7160 0 0 Auth-proxy AAA B PID TTY Allocated Freed Holding Getbufs Retbufs Process 181 0 0 0 7160 0 0 IP Admin SM Proc 93 0 232 232 7160 0 0 HVLAN Mapped Vla 184 0 0 0 7160 0 0 Critical Auth 185 0 232 232 7160 0 0 AAA Dictionary R 186 0 0 0 7160 0 0 DHCP Snooping 187 0 0 0 7160 0 0 DHCP Snooping db 91 0 0 0 7160 0 0 HULC Tcam Memory 90 0 0 0 7160 0 0 HRPC hulc misc r 89 0 232 232 7160 0 0 HLFM address ret 191 0 0 0 7160 0 0 ICMP event handl 88 0 232 232 7160 0 0 HLFM aging proce 193 0 0 0 7160 0 0 IPv6 ping proces 86 0 0 0 7160 0 0 HRPC hlfm reques 196 0 0 405372 7160 0 0 SMI CDP Update H 85 0 0 0 7160 0 0 HRPC hpsecure re 84 0 232 232 7160 0 0 Hulc Port-Securi 82 0 176 176 7160 0 0 HRPC hdm blockin 81 0 68 68 7160 0 0 HRPC hdm non blo 49 0 0 0 7160 0 0 AggMgr Process 202 0 0 0 7160 0 0 Spanning Tree St 203 0 232 232 7160 0 0 Ethchnl 79 0 0 0 7160 0 0 HRPC hcomp reque 78 0 68 396 7160 0 0 hrpc <- response 248 0 232 232 7160 0 0 LINE AAA 77 0 0 0 7160 0 0 hrpc -> request 208 0 232 232 7160 0 0 Cluster L2 209 0 0 0 7160 0 0 Cluster RARP 76 0 0 0 7160 0 0 hrpc -> response 251 0 0 0 7160 0 0 HRPC hl2mcm igmp 254 0 0 0 7160 0 0 crypto engine pr 69 0 41540 41272 7160 0 0 RedEarth Tx Mana 214 0 0 0 7160 0 0 RARP Input 68 0 0 0 7160 0 0 HRPC asic-stats 62 0 0 0 7160 0 0 HULC PBR Tcam Me 61 0 1814956 1814956 7160 0 0 Authenticate Pro 218 0 0 0 7160 0 0 MAB Framework 219 0 0 0 7160 0 0 HRPC dhcp snoopi 220 0 0 0 7160 0 0 HULC DHCP Snoopi 60 0 0 0 7160 0 0 HULC QM Tcam Mem 222 0 0 0 7160 0 0 IGMPSN MRD 223 0 0 0 7160 0 0 IGMPSN 224 0 0 0 7160 0 0 IGMPQR 59 0 0 0 7160 0 0 PSP Timer 58 0 0 0 7160 0 0 HRPC actual powe 227 0 0 0 7160 0 0 MRD 228 0 0 0 7160 0 0 MLD_SNOOP 229 0 0 0 7160 0 0 Inline Power 57 0 0 0 7160 0 0 HRPC EnergyWise 56 0 232 232 7160 0 0 HLPIP Sync Proce 55 0 0 0 7160 0 0 HRPC lpip reques 54 0 0 0 7160 0 0 HULC ACL Tcam Me 44 0 0 0 7160 0 0 IF-MGR event pro 52 0 0 0 7160 0 0 SFF8472 51 0 0 0 7160 0 0 HC Counter Timer 50 0 0 364 7160 0 0 Transport Port A 266 0 0 220 7160 0 0 DiagCard1/-1 47 0 0 424 7160 0 0 Per-minute Jobs 240 0 702252 702252 7160 0 0 QoS stats proces 241 0 0 0 7160 0 0 SNMP Timers 46 0 232 232 7160 0 0 Compute load avg 34 0 0 0 7160 0 0 Dynamic ARP Insp 244 0 0 0 7160 0 0 EPM MAIN PROCESS 245 0 232 232 7160 0 0 AAA Cached Serve PID TTY Allocated Freed Holding Getbufs Retbufs Process 246 0 1000 1000 7160 0 0 LOCAL AAA 247 0 232 232 7160 0 0 ENABLE AAA 32 0 77340 77340 7160 0 0 PrstVbl 45 0 0 0 7160 0 0 Net Input 30 0 0 0 7160 0 0 IPC Keep Alive M 252 0 0 0 7160 0 0 HRPC hl2mcm mlds 253 0 0 0 7160 0 0 HRPC x_setup req 29 0 0 0 7160 0 0 IPC Seat TX Cont 27 0 0 0 7160 0 0 IPC Check Queue 42 0 232 232 7160 0 0 TTY Background 28 0 0 0 7160 0 0 IPC Seat RX Cont 37 0 232 232 7160 0 0 GraphIt 36 0 232 232 7160 0 0 SMART 260 0 232 232 7160 0 0 DHCP Security He 261 0 0 0 7160 0 0 HCD Process 262 0 0 0 7160 0 0 HRPC cable diagn 35 0 0 0 7160 0 0 ARP Snoop 265 0 232 232 7160 0 0 dhcp snooping sw 6 0 0 0 7160 0 0 DiscardQ Backgro 267 0 173388 902180 7160 0 0 AAA SEND STOP EV 24 0 0 0 7160 0 0 IPC Deferred Por 269 0 0 0 7160 0 0 Syslog Traps 23 0 0 0 7160 0 0 IPC Periodic Tim 22 0 0 0 7160 0 0 IPC Zone Manager 31 0 0 0 7160 0 0 IPC Loadometer 21 0 0 0 7160 0 0 IPC Session Serv 7 0 232 232 7160 0 0 Timers 18 0 0 0 7160 0 0 IPC Event Notifi 13 0 5416 5416 7160 0 0 ARP Background 144 0 0 0 7160 0 0 HRPC span reques 14 0 0 0 7160 0 0 AAA_SERVER_DEADT 25 0 0 0 7160 0 0 IPC Process leve 19 0 0 0 7160 0 0 IPC Mcast Pendin 17 0 0 0 7160 0 0 IFS Agent Manage 20 0 0 0 7160 0 0 IPC Dynamic Cach 286 0 0 0 7160 0 0 Call Home Timer 243 0 0 0 7032 0 0 HCMP sync proces 101 0 1377288 1368068 7032 0 0 SpanTree Helper 270 0 0 0 7032 0 0 DHCPD Timer 172 0 216 0 4376 0 0 ADJ NSF process 71 0 0 0 4208 0 0 HRPC hsm request 279 0 232 232 4160 0 0 CSRT RAPID TRANS 161 0 0 0 4160 0 0 HULC DAI Process 8 0 0 0 4160 0 0 WATCH_AFS 75 0 1336 1336 4160 0 0 Adjust Regions 119 0 0 0 4160 0 0 HRPC system mtu 2 0 232 232 4160 0 0 Load Meter 278 0 232 232 4160 0 0 STP FAST TRANSIT 21629056 Total
Can anyway assist me in finding out what is causing this high usage, preferably down to what port is causing it?
Many thanks.
08-18-2021 01:58 AM
Hello,
this is often related to buggy software. Try and upgrade to 15.0.2-SE11...
https://software.cisco.com/download/home/282202842/type/280805680/release/15.0.2-SE11
08-18-2021 02:06 AM
As mentioned other poster, 15.0(X) some version have bugs, do you have any SNMP polling frequenty, worth checking disable and see if that improves.
Not sure may be bug :
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut06609/?rfs=iqvred
worth checking this thread :
08-18-2021 02:17 AM
Hi @Georg Pauwen thanks for the quick suggestion.
I have the updated software to try but last time I did this I encountered an issue with not enough free storage space.
Last time I unticked the option to backup existing firmware image but the whole switch crashed and I ended up swapping it out with another device (this one) both of them have the same firmware but also the same 80% usage issue.
08-18-2021 02:35 AM
Hello,
that doesn't sound good, you don't want to have to replace the entire switch...
Check if your switch supports the global command below:
no macro auto monitor
If so, configure that and monitor if that changes the memory usage...
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide