topic Re: ISE 2.2 : High memory utilization issue (2) in Network Access Control

ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Fri, 31 Aug 2018 13:09:49 GMT

Building on the post from last week:

"What I've experienced is memory climbing at 1%/day until it gets up around 80%, and there it levels out. However... without fail, at some point, we see the memory start to climb again > 80% and the application will crash."

See attached screenshot. I only save 60 days of logs, but you can see the climb leveling off at 80% early July, and now, this past week... "we see the memory start to climb again > 80%" This is currently happening to our primary policy node.

Still deciding if I should open a TAC case. At this point I can recite the call with the engineer. They'll tell me to install patch 9, I'll say this has been an issue since 2.2 patch zero... show me the specific bug I'm hitting that's resolved in patch 9. They won't be able to.

I've brought this up with our account team. I've brought this up in whisper suites at live. I've brought this up with the ISE care program. I'm running out of avenues to provide feedback, and am not seeing any results.

Re: ISE 2.2 : High memory utilization issue (2)

thomas — Fri, 31 Aug 2018 16:12:33 GMT

I don't see an actual question to answer and you've not stated WHICH PATCH you are running.

So let me simply confirm that, Yes, high memory utilization was a known issue (CSCvf22318, CSCvf47316) in ISE 2.2 and it is highly recommended that you go to ISE 2.2 Patch #9 which is currently the latest patch. This is why TAC was and still is making this recommendation.

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Mon, 10 Sep 2018 15:32:32 GMT

FYI Thomas, when I posted this a couple weeks ago, we were at ISE 2.2 patch 5, which I do not believe was one of the affected releases for the two bugs you had noted.

We ended up performing an emergency change window to install Patch 9 followed by the struts hotfix. As expected, the memory utilization dropped to normal levels after reboot. Since patch 9 installation, we do still see memory utilization increasing at 1% per day. This increase appears to be linear to the mem consumption of the 'jsvc' process run by 'iseadmi+' user.

I've opened a TAC case specifically for a potential memory leak, and attached output of the tech top command illustrating the issue, along with a support bundle that captures pre & post patch logs.

If you have any further recommendations, please let us know. I'll keep this post up to date with what TAC finds.

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Wed, 26 Sep 2018 13:48:39 GMT

Providing an update on this case.

TAC had me enabled the following debugs, and collect a support bundle to upload stating: "The bug fixed by patch 9 matches your issue's description perfectly but we need to keep investigating at the logs for more information."

org-apache (appserver/catalina.out)
org-apache-cxf (appserver/catalina.out)
org-apache-digester (appserver/catalina.out)
ise-psc

After reviewing the logs, TAC requested a webex for live troubleshoot, that happened yesterday. TAC performed the following during the webex:

Install root-patch
ran ps -ef grepping for 'java' 'iseadmi+'
performed kill -3 for PID = jsvc COMMAND
exported heapdump to localdisk for upload to the case

Currently waiting for log review.

Re: ISE 2.2 : High memory utilization issue (2)

xtreme — Mon, 01 Oct 2018 05:34:02 GMT

Hi anthonylofreso,

Thanks for the detailed info and update.Any word from TAC? facing a similar issue..
Abi

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Mon, 01 Oct 2018 11:41:55 GMT

No update yet. I was going to request an update today as it has been > 72 hours since they updated the case.

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Tue, 02 Oct 2018 11:48:28 GMT

FYI, this case has been escalated to the BU.

Re: ISE 2.2 : High memory utilization issue (2)

xtreme — Sun, 07 Oct 2018 07:20:50 GMT

Thanks for the update.

Re: ISE 2.2 : High memory utilization issue (2)

xtreme — Wed, 24 Oct 2018 10:21:14 GMT

Any Luck or word from Cisco on this issue? Has it been resolved finally?

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Wed, 24 Oct 2018 12:50:33 GMT

We're still working the case.

TAC has us collecting the following from three different nodes every few days:

Install root-patch
ran ps -ef grepping for 'java' 'iseadmi+'
performed kill -3 for PID = jsvc COMMAND
exported heapdump to localdisk for upload to the case

hopefully soon TAC/BU will have enough data to troubleshoot.

Re: ISE 2.2 : High memory utilization issue (2)

xtreme — Wed, 24 Oct 2018 13:31:39 GMT

Ok..thanks.

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Thu, 01 Nov 2018 11:29:33 GMT

FYI to all, I received the following email from TAC yesterday afternoon:

Hello Anthony,

You don’t need to continue collecting information on your ISE deployment anymore. Engineer I was collaborating with has shared with me an internal defect. I will get a summary of their findings and send you as soon as I get it.

I will update this thread with their response once I receive it.

Re: ISE 2.2 : High memory utilization issue (2)

xtreme — Wed, 12 Dec 2018 12:34:19 GMT

Just wondering if Cisco has released any solution for this problem.

Abi

Re: ISE 2.2 : High memory utilization issue (2)

Jason Kunst — Wed, 12 Dec 2018 15:50:52 GMT

Please get with the tac to debug and make sure you have same issue. Log a case and defect attachment where needed

Re: ISE 2.2 : High memory utilization issue (2)

anthonylofreso — Wed, 12 Dec 2018 16:06:55 GMT

Agree with this.

My case is still open unfortunately. The BU agrees that there's a problem... but they just continue to have me collecting logs and for some reason are unable to figure out the issue. I'm a bit shocked at how long this has gone. Current path is to wait for the node to fail so they can collect the crash bundle.