cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
Announcements
Choose one of the topics below to view our ISE Resources to help you on your journey with ISE

This community is for technical, feature, configuration and deployment questions.
For production deployment issues, please contact the TAC! We will not comment or assist with your TAC case in these forums.
Please see How to Ask the Community for Help for other best practices.

811
Views
0
Helpful
14
Replies
Enthusiast

ISE 2.2 : High memory utilization issue (2)

Building on the post from last week:

ISE 2.2 : High memory utilization issue

 

"What I've experienced is memory climbing at 1%/day until it gets up around 80%, and there it levels out. However... without fail, at some point, we see the memory start to climb again > 80% and the application will crash."

 

See attached screenshot. I only save 60 days of logs, but you can see the climb leveling off at 80% early July, and now, this past week... "we see the memory start to climb again > 80%" This is currently happening to our primary policy node.

 

Still deciding if I should open a TAC case. At this point I can recite the call with the engineer. They'll tell me to install patch 9, I'll say this has been an issue since 2.2 patch zero... show me the specific bug I'm hitting that's resolved in patch 9. They won't be able to.

 

I've brought this up with our account team. I've brought this up in whisper suites at live. I've brought this up with the ISE care program. I'm running out of avenues to provide feedback, and am not seeing any results.

5 ACCEPTED SOLUTIONS

Accepted Solutions
Cisco Employee

Re: ISE 2.2 : High memory utilization issue (2)

I don't see an actual question to answer and you've not stated WHICH PATCH you are running.

 

So let me simply confirm that, Yes, high memory utilization was a known issue (CSCvf22318, CSCvf47316) in ISE 2.2 and it is highly recommended that you go to ISE 2.2 Patch #9 which is currently the latest patch. This is why TAC was and still is making this recommendation.

View solution in original post

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

FYI Thomas, when I posted this a couple weeks ago, we were at ISE 2.2 patch 5, which I do not believe was one of the affected releases for the two bugs you had noted.

We ended up performing an emergency change window to install Patch 9 followed by the struts hotfix. As expected, the memory utilization dropped to normal levels after reboot. Since patch 9 installation, we do still see memory utilization increasing at 1% per day. This increase appears to be linear to the mem consumption of the 'jsvc' process run by 'iseadmi+' user.

I've opened a TAC case specifically for a potential memory leak, and attached output of the tech top command illustrating the issue, along with  a support bundle that captures pre & post patch logs.

If you have any further recommendations, please let us know. I'll keep this post up to date with what TAC finds.

View solution in original post

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

FYI, this case has been escalated to the BU.

View solution in original post

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

FYI to all, I received the following email from TAC yesterday afternoon:

Hello Anthony,

You don’t need to continue collecting information on your ISE deployment anymore. Engineer I was collaborating with has shared with me an internal defect. I will get a summary of their findings and send you as soon as I get it.

I will update this thread with their response once I receive it.

View solution in original post

Cisco Employee

Re: ISE 2.2 : High memory utilization issue (2)

Please get with the tac to debug and make sure you have same issue. Log a case and defect attachment where needed

View solution in original post

14 REPLIES 14
Cisco Employee

Re: ISE 2.2 : High memory utilization issue (2)

I don't see an actual question to answer and you've not stated WHICH PATCH you are running.

 

So let me simply confirm that, Yes, high memory utilization was a known issue (CSCvf22318, CSCvf47316) in ISE 2.2 and it is highly recommended that you go to ISE 2.2 Patch #9 which is currently the latest patch. This is why TAC was and still is making this recommendation.

View solution in original post

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

FYI Thomas, when I posted this a couple weeks ago, we were at ISE 2.2 patch 5, which I do not believe was one of the affected releases for the two bugs you had noted.

We ended up performing an emergency change window to install Patch 9 followed by the struts hotfix. As expected, the memory utilization dropped to normal levels after reboot. Since patch 9 installation, we do still see memory utilization increasing at 1% per day. This increase appears to be linear to the mem consumption of the 'jsvc' process run by 'iseadmi+' user.

I've opened a TAC case specifically for a potential memory leak, and attached output of the tech top command illustrating the issue, along with  a support bundle that captures pre & post patch logs.

If you have any further recommendations, please let us know. I'll keep this post up to date with what TAC finds.

View solution in original post

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

Providing an update on this case.

TAC had me enabled the following debugs, and collect a support bundle to upload stating: "The bug fixed by patch 9 matches your issue's description perfectly but we need to keep investigating at the logs for more information."

  • org-apache (appserver/catalina.out)
  • org-apache-cxf (appserver/catalina.out)
  • org-apache-digester (appserver/catalina.out)
  • ise-psc

After reviewing the logs, TAC requested a webex for live troubleshoot, that happened yesterday. TAC performed the following during the webex:

  1. Install root-patch
  2. ran ps -ef grepping for 'java' 'iseadmi+'
  3. performed kill -3 for PID = jsvc COMMAND
  4. exported heapdump to localdisk for upload to the case

Currently waiting for log review.

Beginner

Re: ISE 2.2 : High memory utilization issue (2)

Hi anthonylofreso,

Thanks for the detailed info and update.Any word from TAC? facing a similar issue..
Abi
Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

No update yet. I was going to request an update today as it has been > 72 hours since they updated the case.

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

FYI, this case has been escalated to the BU.

View solution in original post

Beginner

Re: ISE 2.2 : High memory utilization issue (2)

Thanks for the update.

Beginner

Re: ISE 2.2 : High memory utilization issue (2)

Any Luck or word from Cisco on this issue?  Has it been resolved finally?

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

We're still working the case.

TAC has us collecting the following from three different nodes every few days:

  1. Install root-patch
  2. ran ps -ef grepping for 'java' 'iseadmi+'
  3. performed kill -3 for PID = jsvc COMMAND
  4. exported heapdump to localdisk for upload to the case

hopefully soon TAC/BU will have enough data to troubleshoot.

Highlighted
Beginner

Re: ISE 2.2 : High memory utilization issue (2)

Ok..thanks.

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

FYI to all, I received the following email from TAC yesterday afternoon:

Hello Anthony,

You don’t need to continue collecting information on your ISE deployment anymore. Engineer I was collaborating with has shared with me an internal defect. I will get a summary of their findings and send you as soon as I get it.

I will update this thread with their response once I receive it.

View solution in original post

Beginner

Re: ISE 2.2 : High memory utilization issue (2)

Just wondering if Cisco has released any solution for this problem. 

 

Abi

Cisco Employee

Re: ISE 2.2 : High memory utilization issue (2)

Please get with the tac to debug and make sure you have same issue. Log a case and defect attachment where needed

View solution in original post

Enthusiast

Re: ISE 2.2 : High memory utilization issue (2)

Agree with this.

My case is still open unfortunately. The BU agrees that there's a problem... but they just continue to have me collecting logs and for some reason are unable to figure out the issue. I'm a bit shocked at how long this has gone. Current path is to wait for the node to fail so they can collect the crash bundle.