topic Re: ISE 2.2 : High memory utilization issue in Network Access Control

ISE 2.2 : High memory utilization issue

siddhesh.parab@orange.com — Tue, 21 Aug 2018 13:58:29 GMT

Hello Team,

We have 2 ISE nodes in production environment. we have set 70 % of threshold for memory.

current utilization id 71 %. TAC is already opened. also suppressed radius logs.

Total RAM - 16 GB Hard disk - 1.4 TB

Any suggestions ??

Re: ISE 2.2 : High memory utilization issue

howon — Tue, 21 Aug 2018 14:04:29 GMT

High memory usage itself is not an issue and could be part of normal operation. Are you seeing any other issues aside from the memory consumption? I suggest continue working with the TAC for root cause.

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Tue, 21 Aug 2018 16:40:46 GMT

Is it increasing around 1% every day?:

Operations > Reports > Diagnostics > Health Summary > Select server, select Last 30 Days or custom.

^^^^

available log history based on setting here: Administration > System > Logging > Local Log Setting

Re: ISE 2.2 : High memory utilization issue

siddhesh.parab@orange.com — Tue, 21 Aug 2018 16:58:04 GMT

Yes it is increasing every day...

Re: ISE 2.2 : High memory utilization issue

siddhesh.parab@orange.com — Tue, 21 Aug 2018 16:57:11 GMT

Hello Anthony

Thanks for your response.

TAC suggested us to upgrade patch to 09

Is their any other resolution except patch upgrade ??

Regards

Sid

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Tue, 21 Aug 2018 17:03:50 GMT

I've brought this up to TAC several times, and they say this behavior is expected.

What I've experienced is memory climbing at 1%/day until it gets up around 80%, and there it levels out.

However... without fail, at some point, we see the memory start to climb again > 80% and the application will crash. TAC has never been able to provide a root cause. They have always told us to install the latest available patch, and see if the issue still occurs. Which it does, but by then there's a newer patch out which they, again, tell us to install.

see attached screenshots. One shows the climb to 80%, the other shows mem exceeding 80%, then crash, then drops to 20% and climbs 1% every day again.

I'm curious to know what they say regarding your case. Some of the items they will certainly check and challenge (our deployment is a vm, so these may not apply to yours):

do you have the recommended disk space allocated for this node's personas? (admin / mnt / policy have different requirements) must be thick provisioned.
do you have appropriate hardware reservations in place fore CPU / Mem (does not matter what your current utilization is, these reservations must be in place.)
do you have snapshots disabled
are you at the latest available patch

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Tue, 21 Aug 2018 17:09:35 GMT

yup, been here many times... see my prior post. I'm sure patch 9 will fix for a bit because patching requires a reboot, and it will be another 50 days minimum before your memory is up over 70% again. At which point, patch 10 will likely be out.

Re: ISE 2.2 : High memory utilization issue

hslai — Sun, 02 Sep 2018 00:44:59 GMT

If your deployment hitting CSCvj53801, we should apply Patch 9 to address it, unless you want to live with restarting ISE services yourself periodically.

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Mon, 10 Sep 2018 15:33:40 GMT

Just curious, have you installed Patch 9? and does the issue still exist after installing?

Thank you,

-Tony

Re: ISE 2.2 : High memory utilization issue

siddhesh.parab@orange.com1 — Mon, 10 Sep 2018 16:04:36 GMT

No, we haven't upgrade to patch 09 yet.. however recently in last week we did password recovery activity and during this activity we had restarted both the ISE servers.

Post restart, memory utilization dropped to 41% and as of now everything looks good.

But still i have strong feeling that in future it may again start increasing day by day.

As of now memory utilization is under control. we will keep this under observation for next 1 week..and will take action accordingly

Regards

Sid

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Mon, 10 Sep 2018 16:18:02 GMT

I see. We installed patch 9 last week, and we are still seeing memory increase each day. I've opened a tac case, and will let you know if there is a resolution.

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Wed, 26 Sep 2018 13:48:42 GMT

Providing an update on this case.

TAC had me enabled the following debugs, and collect a support bundle to upload stating: "The bug fixed by patch 9 matches your issue's description perfectly but we need to keep investigating at the logs for more information."

org-apache (appserver/catalina.out)
org-apache-cxf (appserver/catalina.out)
org-apache-digester (appserver/catalina.out)
ise-psc

After reviewing the logs, TAC requested a webex for live troubleshoot, that happened yesterday. TAC performed the following during the webex:

Install root-patch
ran ps -ef grepping for 'java' 'iseadmi+'
performed kill -3 for PID = jsvc COMMAND
exported heapdump to localdisk for upload to the case

Currently waiting for log review.

Re: ISE 2.2 : High memory utilization issue

siddhesh.parab@orange.com1 — Wed, 26 Sep 2018 14:29:31 GMT

I have received below comment from cisco TAC..

The memory utilization can be up to 100%, if the ISE doesn't report "Out of memory" alarm or there is no observable issue this is not an indication of the misbehavior. ISE is build to utilize all available memory.

Re: ISE 2.2 : High memory utilization issue

anthonylofreso — Wed, 26 Sep 2018 14:40:58 GMT

In our case, memory has never reached 100%. The ISE nodes always crash before that occurs. Also, an important distinction I continue to bring up to TAC: We do not experience High mem utilization. We experience memory utilization increasing at an average of 1%/day.

Re: ISE 2.2 : High memory utilization issue

JHILL2 — Tue, 16 Apr 2019 20:45:09 GMT

So, just to rehash, I'm working on a deployment running 2.2 patch 12 whose primary ADM/MnT memory utilization is around 33%, and the secondary ADM/MNT is consistently at 61%. One of the intermittent issues that i see is having to wait >5 minutes for the UI to become responsive before i can proceed with anything within Live Logs or Context Visibility. Is this behavior seen as a symptom and mitigated within the same patch? Assuming no since I'm on patch 12. The only other symptom that's fishy that I've been able to see from a deployment perspective is the use of thin provisioning (1.2 TB).

Re: ISE 2.2 : High memory utilization issue

Damien Miller — Tue, 16 Apr 2019 21:59:54 GMT

Thin provisoning is supported, it has the same performance as thick provisioned lazy zeroed that most use.

I have a customer deployment on 2.2 p12, 180k active endpoints, the dashboard is still reasonably snappy. I would be contacting tac to help tshoot the performance issues because it should not be that slow.

Re: ISE 2.2 : High memory utilization issue

thedaoo — Wed, 25 Dec 2019 18:05:42 GMT

We applied the latest patch 16 which (recommended) solves most of the issues like swap memory high utilization related to ISE 2.2 performance via ISE GUI .

Note : It will automatically install the patch on the ISE device , it will be rebooted as well.

Re: ISE 2.2 : High memory utilization issue

bakaholic39 — Tue, 27 Aug 2024 02:51:27 GMT

Same here with 2 physical nodes in deployment, running on version 3.1.

Memory utilization gradually climbed up to 70%, and TAC advised me to upgrade to version 3.2 to fix the bug CSCwd45843.

Last month, while running on version 3.2 patch 5, memory utilization increased by 1-2% per week. TAC then recommended installing the latest patch to fix the bug CSCwi63725.

Now, after updating to the latest patch of version 3.2 (patch 6), memory utilization has climbed up to 70% again...