ISE 2.2 : High memory utilization issue

[siddhesh.parab@orange.com]

Hello Team,

We have 2 ISE nodes in production environment. we have set 70 % of threshold for memory.

current utilization id 71 %. TAC is already opened. also suppressed radius logs.

Total RAM - 16 GB Hard disk - 1.4 TB 

Any suggestions ??

[howon · Accepted solution]

High memory usage itself is not an issue and could be part of normal operation. Are you seeing any other issues aside from the memory consumption? I suggest continue working with the TAC for root cause.

[anthonylofreso]

Is it increasing around 1% every day?:

Operations > Reports > Diagnostics > Health Summary > Select server, select Last 30 Days or custom.

^^^^

available log history based on setting here: Administration > System > Logging > Local Log Setting

[siddhesh.parab@orange.com]

Yes it is increasing every day...

[anthonylofreso · Accepted solution]

I've brought this up to TAC several times, and they say this behavior is expected.

What I've experienced is memory climbing at 1%/day until it gets up around 80%, and there it levels out.

However... without fail, at some point, we see the memory start to climb again > 80% and the application will crash. TAC has never been able to provide a root cause. They have always told us to install the latest available patch, and see if the issue still occurs. Which it does, but by then there's a newer patch out which they, again, tell us to install.

see attached screenshots. One shows the climb to 80%, the other shows mem exceeding 80%, then crash, then drops to 20% and climbs 1% every day again.

I'm curious to know what they say regarding your case. Some of the items they will certainly check and challenge (our deployment is a vm, so these may not apply to yours):

• do you have the recommended disk space allocated for this node's personas? (admin / mnt / policy have different requirements) must be thick provisioned.
• do you have appropriate hardware reservations in place fore CPU / Mem (does not matter what your current utilization is, these reservations must be in place.)
• do you have snapshots disabled
• are you at the latest available patch

[siddhesh.parab@orange.com]

Hello Anthony 

Thanks for your response.

TAC suggested us to upgrade patch to 09

Is their any other resolution except patch upgrade ??

Regards

Sid

[anthonylofreso]

yup, been here many times... see my prior post. I'm sure patch 9 will fix for a bit because patching requires a reboot, and it will be another 50 days minimum before your memory is up over 70% again. At which point, patch 10 will likely be out.

[hslai]

If your deployment hitting CSCvj53801, we should apply Patch 9 to address it, unless you want to live with restarting ISE services yourself periodically.

[thedaoo · Accepted solution]

We applied  the latest patch 16 which (recommended) solves most of the issues like swap memory  high  utilization related to ISE 2.2 performance via ISE GUI .

Note : It will automatically install the patch on the ISE device , it will be rebooted as well.

[anthonylofreso]

Just curious, have you installed Patch 9? and does the issue still exist after installing?

Thank you,

-Tony

[siddhesh.parab@orange.com1]

No, we haven't upgrade to patch 09 yet.. however recently in last week we did password recovery activity and during this activity we had restarted both the ISE servers.

Post restart,  memory utilization dropped to 41% and as of now everything looks good.

But still i have strong feeling that in future it may again start increasing day by day. 

As of now memory utilization is under control. we will keep this under observation for next 1 week..and will take action accordingly 

Regards

Sid 

[anthonylofreso]

I see. We installed patch 9 last week, and we are still seeing memory increase each day. I've opened a tac case, and will let you know if there is a resolution.

[anthonylofreso · Accepted solution]

Providing an update on this case.

TAC had me enabled the following debugs, and collect a support bundle to upload stating: "The bug fixed by patch 9 matches your issue's description perfectly but we need to keep investigating at the logs for more information."

• org-apache (appserver/catalina.out)
• org-apache-cxf (appserver/catalina.out)
• org-apache-digester (appserver/catalina.out)
• ise-psc

After reviewing the logs, TAC requested a webex for live troubleshoot, that happened yesterday. TAC performed the following during the webex:

1. Install root-patch
2. ran ps -ef grepping for 'java' 'iseadmi+'
3. performed kill -3 for PID = jsvc COMMAND
4. exported heapdump to localdisk for upload to the case

Currently waiting for log review.

[siddhesh.parab@orange.com1 · Accepted solution]

I have received below comment from cisco TAC..

The memory utilization can be up to 100%, if the ISE doesn't report "Out of memory" alarm or there is no observable issue this is not an indication of the misbehavior. ISE is build to utilize all available memory.

[anthonylofreso]

In our case, memory has never reached 100%. The ISE nodes always crash before that occurs. Also, an important distinction I continue to bring up to TAC: We do not experience High mem utilization. We experience memory utilization increasing at an average of 1%/day.