cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
1658
Views
25
Helpful
10
Replies

ISR 4431 High Memory Utilization

salemmahara
Level 3
Level 3

Hello everyone,

we have a bunch of ISR 4431 running universalk9.03.16.04b.S.155-3.S4b-ext with no load at all. We have a big problem with memory utilization (As always, CPU and Memory management is one the biggest problems with Cisco softwares).

 

Here is the output of different commands. That would be great if you could give us a hand:

 

Router#show processes memory sorted 
Processor Pool Total: 1688360784 Used:  264057808 Free: 1424302976
 lsmpi_io Pool Total:    6295128 Used:    6294296 Free:        832

 PID TTY  Allocated      Freed    Holding    Getbufs    Retbufs Process
   0   0  353660936  108857696  227620776          0          0 *Init*          
  79   0  533775128     302040    7594720          0          0 IOSD ipc task   
 389   0    3889216       5160    3926048     799092          0 EEM ED Syslog   
   0   0  847546560  844385288    3117360    8177103          0 *Dead*          
 409   0    1438984      26792    1442056          0          0 EEM Server      
 155   0    3218992      91360    1053512          0          0 CWAN OIR Handler
 185   0    1036624      66816    1023520          0          0 IP ARP Adjacency
   1   0     789776          0     819640          0          0 Chunk Manager   
   6   0     522608      21808     461360          0          0 RF Slave Main Th
 390   0     376128       5160     412832      67468          0 EEM ED Generic  
   0   0          0          0     395592          0          0 *MallocLite*    
 162   0 2397163680 1496916312     381456          0          0 CDP Protocol    
 189   0     614472     339384     250592          0          0 mDNS            
 277   0       1808          0     247672          0          0 COPS            
 508   0     167720        448     221136          0          0 MRIB Process    
  10   0 19150047736 19150089016     215080 18311499785 18311529513 Pool Manager    
  15   0 20955311736 20955132000     195336          0          0 ARP Input       
 374   0    1349008    1199016     194168          0          0 Crypto CA       
 334   0     153800        448     183216          0          0 RUDPV1 Main Proc
 271   0     130168       3216     180968          0          0 IP RIB Update 

What is the problem with this devices? Holding memory? Or maybe Getbufs/ Retbufs?

 

Another command:

Router#show platform software status control-processor brief 
Load Average
 Slot  Status  1-Min  5-Min 15-Min
  RP0 Healthy   0.98   0.58   0.25

Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Healthy  3965680  3678204 (93%)   287476 ( 7%)   1739072 (44%)

CPU Utilization
 Slot  CPU   User System   Nice   Idle    IRQ   SIRQ IOwait
  RP0    0   1.70   0.80   0.00  97.49   0.00   0.00   0.00
         1   2.00   3.00   0.00  95.00   0.00   0.00   0.00
         2   0.70   9.20   0.00  90.10   0.00   0.00   0.00
         3   1.10   0.50   0.00  98.40   0.00   0.00   0.00
         4   0.10   0.50   0.00  99.40   0.00   0.00   0.00
         5   0.30   0.20   0.00  99.50   0.00   0.00   0.00
         6   1.20   0.40   0.00  98.40   0.00   0.00   0.00
         7   0.30   0.30   0.00  99.39   0.00   0.00   0.00

These routers are truley important so we are not allowes to upgrade them to any tricky/buggy version. We are also on RTU license and switching to Cisco One is another challenge. So, if you recommend an upgrade please think about the consideration.

 

***If someone from Cisco read this topic:

PLEASE PLEASE PLEASE

simplify licensing and donwloading/ upgrading your products. Out troubles are getting bigger everyday and those centralized mindsets are not compatible with complex scenarios. We purchase a product, so then we need the upgrades as simple as possible. Why you need different kinds of accounts and contracts? It's just an obstacle 

 

10 Replies 10

Leo Laohoo
Hall of Fame
Hall of Fame

@salemmahara wrote:
Memory (kB)
 Slot  Status    Total     Used (Pct)     Free (Pct) Committed (Pct)
  RP0 Healthy  3965680  3678204 (93%)   287476 ( 7%)   1739072 (44%)

F*ck that is high. 

Please post the complete output to the command "sh process memory platform sort location r0".  

NOTE:  I only want to see the "first page".  

Hello @Leo Laohoo 

Here is the output:

Router#sho processes memory platform sorted location r0
System memory: 3965680K total, 1739400K used, 2226280K free
Lowest: 286320K
   Pid    Text      Data   Stack   Dynamic       RSS     Total              Name  
--------------------------------------------------------------------------------
  3149   11416    176004     204     20716   8698952  10370144     fman_fp_image  
  2255      78    168268     132     19892   8687984   9912076        cpp_cp_svr  
  2897      62    113820      84      2272   8633536   9322480        cpp_sp_svr  
  2695     323     93876      84      3208   8613592   9294812   cpp_ha_top_leve  
  2421     144     91428     148      2520   8611148   9281584        cpp_driver  
  2000      59     23692      84       988   8543408   9202552       cpp_cdm_svr  
 26602  382300    742664     216        56    742908   2445128   linux_iosd-imag  
 25317    6416     81912     184      1688     81912   1058628           fman_rp  
  1602     101     19816      84      3276     19828    508856           cman_fp  
 23676     247     76312      84      4956     84764    371368               bsm  
 31325     749     16604      84       876     16604    179608              vman  
 29476     385    118456     100    107968    118464    127032             smand  
  7727      29      9888      84       280     75444     79808          ngiolite  
  7614     358     27676      84      3512     27688     77832              iomd  
 24598     956     32920      84      3680     32940     53272             cmand  
 26413     303     30984      84      1084     30984     41564              hman  
  5062     303     29200      84       844     29200     39224              hman  
  3311     303     29200      84       844     29200     39224              hman  
  4718     570     11312      84       948     11312     29648              cmcc  
 10526     101      1316     420       132      1316     22424         automount  
 24021     892      1056      84         0      1056     21832          libvirtd  
 27771      85       808      84       132       808     18224             pcscd  
 28529      29      6364      84       676      6364     17396               psd  
 27504     103      6152      84      1232      6160     16212              lman  
 28347      33      5596      84      1572      5604     14516             plogd  
 30884      18      4108      84      1160      4108     10152         tams_proc  
 30745      12      3960      84       676      3964      9548         tamd_proc  
 30228      11      3180      84       536      3180      9308   tam_svcs_esg_cf  
 28034     635      4712      84      3528      4712      5972       periodic.sh  
 30983       4      1656      84       132      1656      4812             rotee  
 30398       4      1656      84       132      1656      4812             rotee  
 30113       4      1656      84       132      1656      4812             rotee 


@salemmahara wrote:
   Pid    Text      Data   Stack   Dynamic       RSS     Total              Name  
--------------------------------------------------------------------------------
  3149   11416    176004     204     20716   8698952  10370144     fman_fp_image  
  2255      78    168268     132     19892   8687984   9912076        cpp_cp_svr  
  2897      62    113820      84      2272   8633536   9322480        cpp_sp_svr  
  2695     323     93876      84      3208   8613592   9294812   cpp_ha_top_leve  
  2421     144     91428     148      2520   8611148   9281584        cpp_driver  
  2000      59     23692      84       988   8543408   9202552       cpp_cdm_svr  

The items in RED are the causes of the memory leak.  You have hit multiple memory leak bugs and I cannot begin to enumerate what might cause them to be that high. 

The most common causes of these are:  

  • DMVPN
  • NBAR

If you can upgrade the firmware of the router, upgrade to the latest 3.16.X train, 3.16.10.

Hello Leo,

these routers are running HSRP on both sides plus some static routing and 20-30 lines ACL. A set of simple NATs.

Is 3.16.10s safe to upgrade while I’m using my RTU license.


@salemmahara wrote:

Is 3.16.10s safe to upgrade while I’m using my RTU license.


3.X.X does not support Cisco Smart License.  

Are this routet connect to isp via bgp?

Hello,

Not at all

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking products for a $25 gift card