topic Re: High memory utilization on Cisco ISE 3.0 in Network Access Control

High memory utilization on Cisco ISE 3.0

rakesh nair — Tue, 17 Oct 2023 11:44:44 GMT

Hi Team,

Can you help me to verify what process is making memory overutilization in my ISE server.

ise-mgt01/admin# show memory
total memory: 98833168 kB
free memory: 1853356 kB
cached: 30809084 kB
swap-cached: 13292 kB

output of free command:
total used free shared buff/cache available
Mem: 98833168 31477140 1852612 17325252 65503416 49447604
Swap: 8191996 74880 8117116

ise-mgt01/admin

Re: High memory utilization on Cisco ISE 3.0

adamscottmaster2013 — Tue, 17 Oct 2023 13:09:27 GMT

Not possible unless you have "root" access to the ISE.

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Tue, 17 Oct 2023 13:21:01 GMT

How to go to root profile

Re: High memory utilization on Cisco ISE 3.0

Rodrigo Diaz — Tue, 17 Oct 2023 14:11:42 GMT

Hi @rakesh nair ,

To review the process that might be causing you problem in your ISE , review the command "tech top" from the command line the utilization of ISE resources among the processes that are running.

You should see a column indicating the %MEM used by the process, you should get an idea from there.

Let me know if that helped you.

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Tue, 17 Oct 2023 15:10:48 GMT

Thanks Rodigro, this is the output we see

Tasks: 561 total, 1 running, 560 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.7 us, 0.2 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 98833168 total, 1847680 free, 31471720 used, 65513768 buff/cache
KiB Swap: 8191996 total, 8116860 free, 75136 used. 49455832 avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14076 iseadmi+ 20 0 26.4g 5.2g 61432 S 9.3 5.5 167179:42 jsvc
19038 iseelas+ 20 0 14.3g 12.7g 180032 S 7.9 13.5 13788:07 java
16595 iserabb+ 20 0 1944172 170280 3264 S 1.0 0.2 8849:45 beam.smp
9 root 20 0 0 0 0 S 0.3 0.0 208:17.08 rcu_sched
1949 iseprof+ 20 0 702624 553588 1388 S 0.3 0.6 1797:35 redis-serv+
6701 root 20 0 0 0 0 S 0.3 0.0 58:02.07 jbd2/sda7-8
9968 100 20 0 433108 82380 11020 S 0.3 0.1 110:57.44 nginx
15869 root 20 0 164516 2804 1632 R 0.3 0.0 0:00.46 top

Re: High memory utilization on Cisco ISE 3.0

Arne Bier — Tue, 17 Oct 2023 23:11:39 GMT

High memory utilisation is not an indication that there is a problem. Unless you're constantly swapping RAM pages to/from disk, the operating system is making good use of the RAM it has at its disposal. A good operating system will put any additional RAM to good use (e.g. caching). ISE will allocate however much RAM it needs to run the apps. The only time we should be concerned is if the ISE applications memory requirements are steadily increasing over time ... memory leaks. Those situations could lead to memory exhaustion. But in my opinion, if ISE is sitting at 90% memory util all day long, I don't worry.

Re: High memory utilization on Cisco ISE 3.0

adamscottmaster2013 — Wed, 18 Oct 2023 12:03:50 GMT

@Arne Bier: Agreed. But you can get TAC to root into the box and check to make sure there are no potential memory leak. Not that hard to do when you have "root" access :-).

The reason ISE always operates at 90% memory utilization is because the Oracle database backend. Oracle will grab as much memory as it can, depending on the platform of 32GB, 64GB, 98GB or 128GB of RAM. This parameter can be configured in Oracle database, if you have "root" access and also access to the Oracle backend.

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Wed, 18 Oct 2023 13:51:11 GMT

I am working with TAC to see if this is concerning or expected behaviour , also we are planning to upgrade the ISE server too

Re: High memory utilization on Cisco ISE 3.0

Amar_Tufo — Wed, 20 Dec 2023 07:14:57 GMT

Has TAC come to a useful conclusion? I'm asking because we do have the same issues after the 3.2 upgrade, high RAM usage for two nodes, yet we do not have troubles with the overal situation, except that at one moment for no explainable reason the second node went unresponsive to web gui access and cli access, which forced us to a total reinstall.

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Wed, 20 Dec 2023 15:23:42 GMT

WE have upgraded ISE server to version 3.2 and update patch 2 , i dont see memory issue currently.

Re: High memory utilization on Cisco ISE 3.0

Marcelo Morais — Wed, 20 Dec 2023 22:36:06 GMT

Hi @rakesh nair ,

please take a look at Field Notice: FN74005 - Identity Services Engine: Java Heap Size May Significantly Impact System Performance - Software Upgrade Recommended, an issue till ISE 3.0 Patch 6, but fixed on ISE 3.2 Patch 2.

Hope this helps !!!

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Thu, 21 Dec 2023 19:21:06 GMT

Currently im seeing this on 3.2 for tech top command, but when i do sh cpu usage command , its not showing cpu output now, but old version, it was giving output.

se-mgt01/admin#show cpu usage
ise-mgt01/admin#

Re: High memory utilization on Cisco ISE 3.0

davidgfriedman — Thu, 21 Dec 2023 19:53:36 GMT

What patch? I have some systems on 3.2 patch 4 working properly with "tech top", "show cpu", "show cpu statistics", "show cpu usage", and "show memory". BTW, does anyone else love "tech top" then hitting 1 to show you each individual CPU's status?

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Thu, 21 Dec 2023 20:03:29 GMT

3.2 patch 4 we are using . the show cpu usage output is not showing only on PAN and SAN nodes , but its giving output on monitoring and PSN nodes

Re: High memory utilization on Cisco ISE 3.0

hslai — Wed, 27 Dec 2023 00:09:55 GMT

@rakesh nair If not already done, please continue working with Cisco TAC to track down the issue with the show cpu usage command on the two ISE admin nodes. The admin nodes usually have more CPU cores than PSN nodes do and that might have triggered this.

Re: High memory utilization on Cisco ISE 3.0

rakesh nair — Tue, 02 Jan 2024 15:10:50 GMT

HI,

I checked with Cisco and they said its another bug.

CSCwf61673

From Cisco ISE CLI, read-only users can not run a show CPU usage command.