ISE 2.2 : High memory utilization issue (2)

[anthonylofreso]

Building on the post from last week:

ISE 2.2 : High memory utilization issue

"What I've experienced is memory climbing at 1%/day until it gets up around 80%, and there it levels out. However... without fail, at some point, we see the memory start to climb again > 80% and the application will crash."

See attached screenshot. I only save 60 days of logs, but you can see the climb leveling off at 80% early July, and now, this past week... "we see the memory start to climb again > 80%" This is currently happening to our primary policy node.

Still deciding if I should open a TAC case. At this point I can recite the call with the engineer. They'll tell me to install patch 9, I'll say this has been an issue since 2.2 patch zero... show me the specific bug I'm hitting that's resolved in patch 9. They won't be able to.

I've brought this up with our account team. I've brought this up in whisper suites at live. I've brought this up with the ISE care program. I'm running out of avenues to provide feedback, and am not seeing any results.

[thomas · Accepted solution]

I don't see an actual question to answer and you've not stated WHICH PATCH you are running.

So let me simply confirm that, Yes, high memory utilization was a known issue (CSCvf22318, CSCvf47316) in ISE 2.2 and it is highly recommended that you go to ISE 2.2 Patch #9 which is currently the latest patch. This is why TAC was and still is making this recommendation.

[anthonylofreso · Accepted solution]

FYI Thomas, when I posted this a couple weeks ago, we were at ISE 2.2 patch 5, which I do not believe was one of the affected releases for the two bugs you had noted.

We ended up performing an emergency change window to install Patch 9 followed by the struts hotfix. As expected, the memory utilization dropped to normal levels after reboot. Since patch 9 installation, we do still see memory utilization increasing at 1% per day. This increase appears to be linear to the mem consumption of the 'jsvc' process run by 'iseadmi+' user.

I've opened a TAC case specifically for a potential memory leak, and attached output of the tech top command illustrating the issue, along with  a support bundle that captures pre & post patch logs.

If you have any further recommendations, please let us know. I'll keep this post up to date with what TAC finds.

[anthonylofreso]

Providing an update on this case.

TAC had me enabled the following debugs, and collect a support bundle to upload stating: "The bug fixed by patch 9 matches your issue's description perfectly but we need to keep investigating at the logs for more information."

• org-apache (appserver/catalina.out)
• org-apache-cxf (appserver/catalina.out)
• org-apache-digester (appserver/catalina.out)
• ise-psc

After reviewing the logs, TAC requested a webex for live troubleshoot, that happened yesterday. TAC performed the following during the webex:

1. Install root-patch
2. ran ps -ef grepping for 'java' 'iseadmi+'
3. performed kill -3 for PID = jsvc COMMAND
4. exported heapdump to localdisk for upload to the case

Currently waiting for log review.

[xtreme]

Hi anthonylofreso,

Thanks for the detailed info and update.Any word from TAC? facing a similar issue..
Abi

[anthonylofreso]

No update yet. I was going to request an update today as it has been > 72 hours since they updated the case.

[anthonylofreso · Accepted solution]

FYI, this case has been escalated to the BU.

[xtreme]

Thanks for the update.

[xtreme]

Any Luck or word from Cisco on this issue?  Has it been resolved finally?

[anthonylofreso]

We're still working the case.

TAC has us collecting the following from three different nodes every few days:

1. Install root-patch
2. ran ps -ef grepping for 'java' 'iseadmi+'
3. performed kill -3 for PID = jsvc COMMAND
4. exported heapdump to localdisk for upload to the case

hopefully soon TAC/BU will have enough data to troubleshoot.

[xtreme]

Ok..thanks.

[anthonylofreso · Accepted solution]

FYI to all, I received the following email from TAC yesterday afternoon:

Hello Anthony,

You don’t need to continue collecting information on your ISE deployment anymore. Engineer I was collaborating with has shared with me an internal defect. I will get a summary of their findings and send you as soon as I get it.

I will update this thread with their response once I receive it.

[xtreme]

Just wondering if Cisco has released any solution for this problem. 

Abi

[Jason Kunst · Accepted solution]

Please get with the tac to debug and make sure you have same issue. Log a case and defect attachment where needed

[anthonylofreso]

Agree with this.

My case is still open unfortunately. The BU agrees that there's a problem... but they just continue to have me collecting logs and for some reason are unable to figure out the issue. I'm a bit shocked at how long this has gone. Current path is to wait for the node to fail so they can collect the crash bundle.