High memory utilization on Cisco ISE 3.0

[rakesh nair]

Hi Team,

Can you help me to verify what process is making memory overutilization in my ISE server.

ise-mgt01/admin# show memory
total memory: 98833168 kB
free memory: 1853356 kB
cached: 30809084 kB
swap-cached: 13292 kB

output of free command:
total used free shared buff/cache available
Mem: 98833168 31477140 1852612 17325252 65503416 49447604
Swap: 8191996 74880 8117116

ise-mgt01/admin

[adamscottmaster2013]

Not possible unless you have "root" access to the ISE.

[rakesh nair]

How to go to root profile

[Rodrigo Diaz]

Hi @rakesh nair , 

To review the process that might be causing you problem in your ISE , review the command "tech top" from the command line the utilization of ISE resources among the processes that are running. 

You should see a column indicating the %MEM used by the process, you should get an idea from there. 

Let me know if that helped you. 

[rakesh nair]

Thanks Rodigro, this is the output we see

Tasks: 561 total, 1 running, 560 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.2 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 98833168 total, 1847680 free, 31471720 used, 65513768 buff/cache
KiB Swap: 8191996 total, 8116860 free, 75136 used. 49455832 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14076 iseadmi+ 20 0 26.4g 5.2g 61432 S 9.3 5.5 167179:42 jsvc
19038 iseelas+ 20 0 14.3g 12.7g 180032 S 7.9 13.5 13788:07 java
16595 iserabb+ 20 0 1944172 170280 3264 S 1.0 0.2 8849:45 beam.smp
9 root 20 0 0 0 0 S 0.3 0.0 208:17.08 rcu_sched
1949 iseprof+ 20 0 702624 553588 1388 S 0.3 0.6 1797:35 redis-serv+
6701 root 20 0 0 0 0 S 0.3 0.0 58:02.07 jbd2/sda7-8
9968 100 20 0 433108 82380 11020 S 0.3 0.1 110:57.44 nginx
15869 root 20 0 164516 2804 1632 R 0.3 0.0 0:00.46 top

[Arne Bier]

High memory utilisation is not an indication that there is a problem. Unless you're constantly swapping RAM pages to/from disk, the operating system is making good use of the RAM it has at its disposal. A good operating system will put any additional RAM to good use (e.g. caching).  ISE will allocate however much RAM it needs to run the apps. The only time we should be concerned is if the ISE applications memory requirements are steadily increasing over time ... memory leaks. Those situations could lead to memory exhaustion. But in my opinion, if ISE is sitting at 90% memory util all day long, I don't worry. 

[adamscottmaster2013]

@Arne Bier:  Agreed.  But you can get TAC to root into the box and check to make sure there are no potential memory leak.  Not that hard to do when you have "root" access :-).

The reason ISE always operates at 90% memory utilization is because the Oracle database backend.  Oracle will grab as much memory as it can, depending on the platform of 32GB, 64GB, 98GB or 128GB of RAM.  This parameter can be configured in Oracle database, if you have "root" access and also access to the Oracle backend.

[rakesh nair]

I am working with TAC to see if this is concerning or expected behaviour , also we are planning to upgrade the ISE server too

[Amar_Tufo]

Has TAC come to a useful conclusion? I'm asking because we do have the same issues after the 3.2 upgrade, high RAM usage for two nodes, yet we do not have troubles with the overal situation, except that at one moment for no explainable reason the second node went unresponsive to web gui access and cli access, which forced us to a total reinstall. 

[rakesh nair]

WE have upgraded ISE server to version 3.2 and update patch 2 , i dont see memory issue currently.

[Marcelo Morais]

Hi @rakesh nair ,

 please take a look at Field Notice: FN74005 - Identity Services Engine: Java Heap Size May Significantly Impact System Performance - Software Upgrade Recommended, an issue till ISE 3.0 Patch 6, but fixed on ISE 3.2 Patch 2.

Hope this helps !!!

[rakesh nair]

Currently im seeing this on 3.2 for tech top command, but when i do sh cpu usage command , its not showing cpu output now, but old version, it was giving output.

se-mgt01/admin#show cpu usage
ise-mgt01/admin#

[davidgfriedman]

What patch?  I have some systems on 3.2 patch 4 working properly with "tech top", "show cpu", "show cpu statistics", "show cpu usage", and "show memory". BTW, does anyone else love "tech top" then hitting 1 to show you each individual CPU's status? 

[rakesh nair]

3.2 patch 4  we are using . the show cpu usage output is not showing only on PAN and SAN nodes , but its giving output on monitoring and PSN nodes

[hslai · Accepted solution]

@rakesh nair If not already done, please continue working with Cisco TAC to track down the issue with the show cpu usage command on the two ISE admin nodes. The admin nodes usually have more CPU cores than PSN nodes do and that might have triggered this.