cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1446
Views
0
Helpful
4
Replies

C2960 80% memory usage and not picking up trunked config

BeckyBoo123
Level 1
Level 1

Hi All,

 

I have a WS-C2960-48PST-L (15.0(2)SE4) which is working fine on the surface but SolarWinds is reporting that the memory usage has been at 80% for a long period of time.

Using all of the memory commands I can see that there are a lot of dead processes.

 

ms-h000717#sh processes mem sorted
Processor Pool Total:   21751188 Used:   19248020 Free:    2503168
      I/O Pool Total:    4194304 Used:    2386196 Free:    1808108
Driver te Pool Total:    1048576 Used:         40 Free:    1048536

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0   24235572    6342940   14722908          0          0 *Init*          
   0   0    4129224    3158900     969332    3799839    1341026 *Dead*          
 152   0   12608160   11503620     556472          0          0 Auth Manager    
  73   0     772480     143256     511840          0          0 Stack Mgr Notifi
   0   0          0          0     394440          0          0 *MallocLite*    
  63   0     368252    1299944     377812          0          0 EEM ED ND       
 182   0    3655008    1402072     372708      40608          0 CDP Protocol    
 113   1    3479604    3379248     109508          0          0 SSH Process     
   1   0     147532      52840     101852          0          0 Chunk Manager   
 272   0      85072          0      92232          0          0 EEM Server      
 150   0      71328          0      78488          0          0 Switch Backup In
 111   0       5276       1884      66204          0          0 HACL Acl Manager
 280   0      66144      16772      61936          0          0 Call Home proces
 136   0          0          0      61160          0          0 HL3U PBR bkgrd p
 143   0  609511120  609502444      61160       5076          0 HRPC qos request
  43   0      97160      84504      59860          0          0 IF-MGR control p
 178   0      49352          0      56512          0          0 ACCT Periodic Pr
 288   0      69464       1176      52740          0          0 DHCPD Receive   
 295   0      64440      18336      51992          0          0 NTP             
 264   0    3243304    4203828      51240          0          0 PM Callback     
 256   0     221196     185512      48912          0          0 Crypto CA       
  16   0      48448      64152      45576          0       9620 Entity MIB API  
  12   0      38104       1404      43860          0          0 ARP Input       
   5   0    4854668    5504100      42192    3199868    3528879 Pool Manager    
 189   0      33012          0      40172          0          0 IP ARP Retry Age
  87   0      80092        232      38904          0          0 HLFM address lea
 195   0      13204        576      38404          0          0 SMI Director DB 
 142   0  656425908  656189296      35528     131976          0 HQM Stack Proces
 157   0    3424764    2692064      33948          0          0 DTP Protocol    
  48   0      55748      27456      26964          0          0 Per-Second Jobs 
 197   0        680          0      25840          0          0 SMI Backup Proce
 259   0        232        312      25160          0          0 ASP Process Crea
  10   0          0          0      25160          0          0 Crash writer    
 104   0          0          0      25160          0          0 HRPC pm request 
 102   0    4124848    3717296      24464          0          0 hpm main process
 289   0   10227624    9222680      24176       5076          0 IP SNMP         
 176   0      16816        232      23744          0          0 AAA Server      
 291   0   59813768   81125488      23204          0          0 SNMP ENGINE     
  39   0      41204        716      19608          0          0 Net Background  
 211   0       6424         76      19508          0          0 CEF background p
 232   0      29912      10736      19244          0          0 CEF: IPv4 proces
 292   0       5808        232      18744          0          0 IP SNMPV6       
 199   0       5416          0      18576          0          0 SMI IBC client p
 200   0       4884        232      17812          0          0 SMI IBC Download
  74   0      10336        232      17264          0          0 Fifo Error Detec
 281   0       6868          0      17028          0          0 EEM ED Test     
 273   0      11476       1720      16916          0          0 EEM Policy Direc
 188   0       2588          0      15748          0          0 IP ARP Adjacency
 287   0      10308          0      15592          0          0 SSH Event handle
 277   0     210008     207772      15396          0          0 Syslog          
 285   0     381240      89620      15372       3120          0 VLAN Manager    
 205   0      44940          0      15068          0          0 TCP Protocols   
 276   0       4556          0      14756          0          0 EEM ED Timer    
 275   0       4556          0      14716          0          0 EEM ED OIR      
  64   0       4556          0      14716          0          0 EEM ED Identity 
 282   0       4556          0      14716          0          0 EEM ED Config   
 284   0       4556          0      14716          0          0 EM ED GOLD      
 283   0       4556          0      14716          0          0 EEM ED Env      
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
  65   0       4556          0      14716          0          0 EEM ED MAT      
 190   0       3852       2568      14444          0          0 IP Input        
 235   0       1120          0      14280          0          0 IP RIB Update   
 155   0    1201856    1049476      13428          0          0 802.1x switch   
 212   0        216          0      13420          0          0 fib_fib_bfd_sb e
 234   0        216          0      13384          0          0 ADJ resolve proc
 236   0        216          0      13376          0          0 Collection proce
 171   0        216          0      13376          0          0 CEF switching ba
 268   0          0          0      13160          0          0 Test AAA Client 
  33   0        232        232      13160          0          0 XML Proxy Client
 255   0          0          0      13160          0          0 encrypt proc    
  15   0          0          0      13160          0          0 Policy Manager  
 290   0   21319972     983508      13160          0          0 PDU DISPATCHER  
  41   0      60964        232      13160      35532          0 Logger          
 293   0          0          0      13160          0          0 SNMP ConfCopyPro
 257   0          0          0      13160          0          0 Crypto PKI-CRL  
 294   0          0          0      13160          0          0 SNMP Traps      
  72   0          0          0      13160          0          0 Stack Mgr       
  40   0          0          0      13160          0          0 IDB Work        
 231   0          0          0      13160          0          0 Routing Topology
 204   0          0          0      13160          0          0 TCP Timer       
 131   0          0          0      13160          0          0 HCEF ADJ Refresh
  83   0          0          0      13160          0          0 HIPC bkgrd proce
  92   0          0          0      13160          0          0 HVLAN main bkgrd
 134   0          0          0      13160          0          0 HL3U bkgrd proce
   0   0      12180     196488      12180          0          0 *Sched*         
 207   0      12224       9984      12036          0          0 HTTP CORE       
 258   0       4884        232      11812          0          0 VTP Trap Process
   4   0       4772        232      11752          0          0 Check heaps     
 225   0      14800        232      11696          0          0 L2TRACE SERVER  
 198   0       3892        232      10820          0          0 SMI IBC server p
 133   0        232        232      10160          0          0 Hulc LED Process
 213   0          0          0      10160          0          0 IP IRDP         
 121   0          0          0      10160          0          0 HRPC Smart Insta
 192   0        232        232      10160          0          0 IP ARP Track    
 239   0          0          0      10160          0          0 IPv6 RIB Event H
 167   0        232        232      10160          0          0 IP Host Track Pr
 238   0        232        232      10160          0          0 static          
 148   0        232        232      10160          0          0 image mgr       
   3   0          0          0      10032          0          0 Connection Mgr  
  66   0       9856          0       9856          0          0 HL2MCM          
 226   0     514300    1124000       9452          0          0 MLDSN L2MCM     
 221   0     510868    1119868       8944          0          0 IGMPSN L2MCM    
 249   0       1936        232       8864          0          0 LDAP process    
 242   0       1936        232       8864          0          0 TPLUS           
  26   0       1688          0       8848          0          0 IPC Seat Manager
 237   0     641420     640044       8536          0          0 NIST rng proc   
  67   0       7820          0       7820          0          0 HL2MCM          
 210   0     329280     328680       7760          0          0 Cluster Base    
   9   0   11099952   11117100       7692          0          0 crypto sw pk pro
  70   0      23176      43124       7528          0          0 RedEarth Rx Mana
  11   0        340          0       7500          0          0 Exception contro
 165   0        524        232       7452          0          0 UDLD            
 215   0        448        232       7376          0          0 Dot1x Supplicant
 233   0        216          0       7376          0          0 ADJ background  
 216   0        448        232       7376          0          0 Dot1x Supplicant
 271   0        216          0       7376          0          0 Online Diag EEM 
 201   0     940748        232       7376          0          0 Spanning Tree   
 166   0        216          0       7376          0          0 Port-Security   
 217   0        448        232       7376          0          0 Dot1x Supplicant
 230   0        152          0       7312          0          0 Marvell wk-a Pow
 296   0        216          0       7248          0          0 DHCPD Database  
  38   0          0          0       7224          0          0 Critical Bkgnd  
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
  80   0        560        232       7224          0          0 HULC Device Mana
 179   0          0          0       7172          0          0 AAA System Acct 
 117   0          0          0       7160          0          0 HRPC dot1x reque
 118   0          0          0       7160          0          0 HULC DOT1X Proce
 120   0          0          0       7160          0          0 SMI MSG Retry Pr
 116   0          0          0       7160          0          0 HRPC lldp reques
 124   0        232        232       7160          0          0 HSTP Sync Proces
 125   0          0          0       7160          0          0 HRPC stp_cli req
 122   0          0          0       7160          0          0 HRPC sdm request
 127   0        232        232       7160          0          0 S/W Bridge Proce
 123   0        232        232       7160          0          0 Hulc Storm Contr
 129   0          0          0       7160          0          0 HRPC vqpc reques
 126   0          0          0       7160          0          0 HRPC stp_state_s
 115   0          0          0       7160          0          0 HULC CISP Proces
 132   0          0          0       7160          0          0 HRPC hled reques
 114   0          0          0       7160          0          0 HRPC cdp request
 112   0          0          0       7160          0          0 HRPC backup inte
 135   0          0          0       7160          0          0 HRPC hl3u reques
 128   0          0          0       7160          0          0 HRPC hudld reque
 137   0          0          0       7160          0          0 HL3U PBR n-h res
 138   0          0          0       7160          0          0 HRPC dtp request
 130   0          0          0       7160          0          0 HRPC aim request
 140   0          0          0       7160          0          0 HRPC snmp reques
 141   0          0          0       7160          0          0 HULC SNMP Proces
 109   0          0          0       7160          0          0 HACL Queue Proce
 108   0          0          0       7160          0          0 HRPC hcmp reques
 110   0          0          0       7160          0          0 HRPC acl request
 145   0          0          0       7160          0          0 HRPC system post
 146   0          0          0       7160          0          0 Hulc Reload Mana
 147   0          0          0       7160          0          0 HRPC hrcli-event
 106   0          0          0       7160          0          0 HRPC pm-counters
 149   0        232      48348       7160          0          0 PI MATM Aging Pr
 105   0        232        232       7160          0          0 hpm counter proc
 151   0          0          0       7160          0          0 MMN bkgrd proces
 103   0        232        232       7160          0          0 HPM Stack Sync P
 153   0          0          0       7160          0          0 AUTH POLICY Fram
 154   0        232        232       7160          0          0 Dot1x Mgr Proces
 139   0          0          0       7160          0          0 HRPC show_forwar
 156   0          0          0       7160          0          0 802.1x Webauth F
 100   0          0          0       7160          0          0 HPM Msg Retry Pr
 158   0          0          0       7160          0          0 EAP Framework   
 159   0          0          0       7160          0          0 EAP Test        
 160   0          0          0       7160          0          0 HRPC dai request
 107   0          0          0       7160          0          0 hpm vp events ca
 162   0          0          0       7160          0          0 HRPC ip device t
 163   0          0          0       7160          0          0 HRPC ip source g
 164   0          0          0       7160          0          0 HULC IP Source g
  99   0          0          0       7160          0          0 HULC PM Vector P
  98   0       5672          0       7160          0          0 Hulc ILP Alchemy
  97   0          0          0       7160          0          0 HRPC ilp request
 168   0          0          0       7160          0          0 Link State Group
 169   0          0          0       7160          0          0 VMATM Callback  
 170   0        232        232       7160          0          0 CMD HANDLER     
  96   0          0          0       7160          0          0 HULC VLAN REF Ba
 206   0          0          0       7160          0          0 Socket Timers   
 173   0          0          0       7160          0          0 HRPC sisf reques
 174   0          0          0       7160          0          0 HULC SISF Proces
 175   0          0          0       7160          0          0 HULC SISF Source
  95   0          0          0       7160          0          0 HRPC vlan reques
 177   0          0          0       7160          0          0 AAA ACCT Proc   
  94   0        232        232       7160          0          0 Vlan shutdown Pr
  53   0        232        232       7160          0          0 DownWhenLooped  
 180   0          0          0       7160          0          0 Auth-proxy AAA B
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
 181   0          0          0       7160          0          0 IP Admin SM Proc
  93   0        232        232       7160          0          0 HVLAN Mapped Vla
 184   0          0          0       7160          0          0 Critical Auth   
 185   0        232        232       7160          0          0 AAA Dictionary R
 186   0          0          0       7160          0          0 DHCP Snooping   
 187   0          0          0       7160          0          0 DHCP Snooping db
  91   0          0          0       7160          0          0 HULC Tcam Memory
  90   0          0          0       7160          0          0 HRPC hulc misc r
  89   0        232        232       7160          0          0 HLFM address ret
 191   0          0          0       7160          0          0 ICMP event handl
  88   0        232        232       7160          0          0 HLFM aging proce
 193   0          0          0       7160          0          0 IPv6 ping proces
  86   0          0          0       7160          0          0 HRPC hlfm reques
 196   0          0     405372       7160          0          0 SMI CDP Update H
  85   0          0          0       7160          0          0 HRPC hpsecure re
  84   0        232        232       7160          0          0 Hulc Port-Securi
  82   0        176        176       7160          0          0 HRPC hdm blockin
  81   0         68         68       7160          0          0 HRPC hdm non blo
  49   0          0          0       7160          0          0 AggMgr Process  
 202   0          0          0       7160          0          0 Spanning Tree St
 203   0        232        232       7160          0          0 Ethchnl         
  79   0          0          0       7160          0          0 HRPC hcomp reque
  78   0         68        396       7160          0          0 hrpc <- response
 248   0        232        232       7160          0          0 LINE AAA        
  77   0          0          0       7160          0          0 hrpc -> request 
 208   0        232        232       7160          0          0 Cluster L2      
 209   0          0          0       7160          0          0 Cluster RARP    
  76   0          0          0       7160          0          0 hrpc -> response
 251   0          0          0       7160          0          0 HRPC hl2mcm igmp
 254   0          0          0       7160          0          0 crypto engine pr
  69   0      41540      41272       7160          0          0 RedEarth Tx Mana
 214   0          0          0       7160          0          0 RARP Input      
  68   0          0          0       7160          0          0 HRPC asic-stats 
  62   0          0          0       7160          0          0 HULC PBR Tcam Me
  61   0    1814956    1814956       7160          0          0 Authenticate Pro
 218   0          0          0       7160          0          0 MAB Framework   
 219   0          0          0       7160          0          0 HRPC dhcp snoopi
 220   0          0          0       7160          0          0 HULC DHCP Snoopi
  60   0          0          0       7160          0          0 HULC QM Tcam Mem
 222   0          0          0       7160          0          0 IGMPSN MRD      
 223   0          0          0       7160          0          0 IGMPSN          
 224   0          0          0       7160          0          0 IGMPQR          
  59   0          0          0       7160          0          0 PSP Timer       
  58   0          0          0       7160          0          0 HRPC actual powe
 227   0          0          0       7160          0          0 MRD             
 228   0          0          0       7160          0          0 MLD_SNOOP       
 229   0          0          0       7160          0          0 Inline Power    
  57   0          0          0       7160          0          0 HRPC EnergyWise 
  56   0        232        232       7160          0          0 HLPIP Sync Proce
  55   0          0          0       7160          0          0 HRPC lpip reques
  54   0          0          0       7160          0          0 HULC ACL Tcam Me
  44   0          0          0       7160          0          0 IF-MGR event pro
  52   0          0          0       7160          0          0 SFF8472         
  51   0          0          0       7160          0          0 HC Counter Timer
  50   0          0        364       7160          0          0 Transport Port A
 266   0          0        220       7160          0          0 DiagCard1/-1    
  47   0          0        424       7160          0          0 Per-minute Jobs 
 240   0     702252     702252       7160          0          0 QoS stats proces
 241   0          0          0       7160          0          0 SNMP Timers     
  46   0        232        232       7160          0          0 Compute load avg
  34   0          0          0       7160          0          0 Dynamic ARP Insp
 244   0          0          0       7160          0          0 EPM MAIN PROCESS
 245   0        232        232       7160          0          0 AAA Cached Serve
 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
 246   0       1000       1000       7160          0          0 LOCAL AAA       
 247   0        232        232       7160          0          0 ENABLE AAA      
  32   0      77340      77340       7160          0          0 PrstVbl         
  45   0          0          0       7160          0          0 Net Input       
  30   0          0          0       7160          0          0 IPC Keep Alive M
 252   0          0          0       7160          0          0 HRPC hl2mcm mlds
 253   0          0          0       7160          0          0 HRPC x_setup req
  29   0          0          0       7160          0          0 IPC Seat TX Cont
  27   0          0          0       7160          0          0 IPC Check Queue 
  42   0        232        232       7160          0          0 TTY Background  
  28   0          0          0       7160          0          0 IPC Seat RX Cont
  37   0        232        232       7160          0          0 GraphIt         
  36   0        232        232       7160          0          0 SMART           
 260   0        232        232       7160          0          0 DHCP Security He
 261   0          0          0       7160          0          0 HCD Process     
 262   0          0          0       7160          0          0 HRPC cable diagn
  35   0          0          0       7160          0          0 ARP Snoop       
 265   0        232        232       7160          0          0 dhcp snooping sw
   6   0          0          0       7160          0          0 DiscardQ Backgro
 267   0     173388     902180       7160          0          0 AAA SEND STOP EV
  24   0          0          0       7160          0          0 IPC Deferred Por
 269   0          0          0       7160          0          0 Syslog Traps    
  23   0          0          0       7160          0          0 IPC Periodic Tim
  22   0          0          0       7160          0          0 IPC Zone Manager
  31   0          0          0       7160          0          0 IPC Loadometer  
  21   0          0          0       7160          0          0 IPC Session Serv
   7   0        232        232       7160          0          0 Timers          
  18   0          0          0       7160          0          0 IPC Event Notifi
  13   0       5416       5416       7160          0          0 ARP Background  
 144   0          0          0       7160          0          0 HRPC span reques
  14   0          0          0       7160          0          0 AAA_SERVER_DEADT
  25   0          0          0       7160          0          0 IPC Process leve
  19   0          0          0       7160          0          0 IPC Mcast Pendin
  17   0          0          0       7160          0          0 IFS Agent Manage
  20   0          0          0       7160          0          0 IPC Dynamic Cach
 286   0          0          0       7160          0          0 Call Home Timer 
 243   0          0          0       7032          0          0 HCMP sync proces
 101   0    1377288    1368068       7032          0          0 SpanTree Helper 
 270   0          0          0       7032          0          0 DHCPD Timer     
 172   0        216          0       4376          0          0 ADJ NSF process 
  71   0          0          0       4208          0          0 HRPC hsm request
 279   0        232        232       4160          0          0 CSRT RAPID TRANS
 161   0          0          0       4160          0          0 HULC DAI Process
   8   0          0          0       4160          0          0 WATCH_AFS       
  75   0       1336       1336       4160          0          0 Adjust Regions  
 119   0          0          0       4160          0          0 HRPC system mtu 
   2   0        232        232       4160          0          0 Load Meter      
 278   0        232        232       4160          0          0 STP FAST TRANSIT
                                 21629056 Total

Can anyway assist me in finding out what is causing this high usage, preferably down to what port is causing it?

Many thanks.

4 Replies 4

Hello,

 

this is often related to buggy software. Try and upgrade to 15.0.2-SE11...

 

https://software.cisco.com/download/home/282202842/type/280805680/release/15.0.2-SE11

balaji.bandi
Hall of Fame
Hall of Fame

As mentioned other poster, 15.0(X) some version have bugs, do you have any SNMP polling frequenty, worth checking disable and see if that improves.

 

Not sure may be bug :

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCut06609/?rfs=iqvred

 

worth checking this thread :

 

https://community.cisco.com/t5/switching/memory-leak-on-catalyst-2960-2960s-similar-to-bug-cscts52797/td-p/2245672

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

BeckyBoo123
Level 1
Level 1

Hi @Georg Pauwen thanks for the quick suggestion.

I have the updated software to try but last time I did this I encountered an issue with not enough free storage space.

 

 

Last time I unticked the option to backup existing firmware image but the whole switch crashed and I ended up swapping it out with another device (this one) both of them have the same firmware but also the same 80% usage issue.

Hello,

 

that doesn't sound good, you don't want to have to replace the entire switch...

 

Check if your switch supports the global command below:

 

no macro auto monitor

 

If so, configure that and monitor if that changes the memory usage...