08-31-2018 06:09 AM
Building on the post from last week:
ISE 2.2 : High memory utilization issue
"What I've experienced is memory climbing at 1%/day until it gets up around 80%, and there it levels out. However... without fail, at some point, we see the memory start to climb again > 80% and the application will crash."
See attached screenshot. I only save 60 days of logs, but you can see the climb leveling off at 80% early July, and now, this past week... "we see the memory start to climb again > 80%" This is currently happening to our primary policy node.
Still deciding if I should open a TAC case. At this point I can recite the call with the engineer. They'll tell me to install patch 9, I'll say this has been an issue since 2.2 patch zero... show me the specific bug I'm hitting that's resolved in patch 9. They won't be able to.
I've brought this up with our account team. I've brought this up in whisper suites at live. I've brought this up with the ISE care program. I'm running out of avenues to provide feedback, and am not seeing any results.
Solved! Go to Solution.
08-31-2018 09:12 AM
I don't see an actual question to answer and you've not stated WHICH PATCH you are running.
So let me simply confirm that, Yes, high memory utilization was a known issue (CSCvf22318, CSCvf47316) in ISE 2.2 and it is highly recommended that you go to ISE 2.2 Patch #9 which is currently the latest patch. This is why TAC was and still is making this recommendation.
09-10-2018 08:32 AM
FYI Thomas, when I posted this a couple weeks ago, we were at ISE 2.2 patch 5, which I do not believe was one of the affected releases for the two bugs you had noted.
We ended up performing an emergency change window to install Patch 9 followed by the struts hotfix. As expected, the memory utilization dropped to normal levels after reboot. Since patch 9 installation, we do still see memory utilization increasing at 1% per day. This increase appears to be linear to the mem consumption of the 'jsvc' process run by 'iseadmi+' user.
I've opened a TAC case specifically for a potential memory leak, and attached output of the tech top command illustrating the issue, along with a support bundle that captures pre & post patch logs.
If you have any further recommendations, please let us know. I'll keep this post up to date with what TAC finds.
10-02-2018 04:48 AM
FYI, this case has been escalated to the BU.
11-01-2018 04:29 AM
FYI to all, I received the following email from TAC yesterday afternoon:
Hello Anthony,
You don’t need to continue collecting information on your ISE deployment anymore. Engineer I was collaborating with has shared with me an internal defect. I will get a summary of their findings and send you as soon as I get it.
I will update this thread with their response once I receive it.
12-12-2018 07:50 AM
08-31-2018 09:12 AM
I don't see an actual question to answer and you've not stated WHICH PATCH you are running.
So let me simply confirm that, Yes, high memory utilization was a known issue (CSCvf22318, CSCvf47316) in ISE 2.2 and it is highly recommended that you go to ISE 2.2 Patch #9 which is currently the latest patch. This is why TAC was and still is making this recommendation.
09-10-2018 08:32 AM
FYI Thomas, when I posted this a couple weeks ago, we were at ISE 2.2 patch 5, which I do not believe was one of the affected releases for the two bugs you had noted.
We ended up performing an emergency change window to install Patch 9 followed by the struts hotfix. As expected, the memory utilization dropped to normal levels after reboot. Since patch 9 installation, we do still see memory utilization increasing at 1% per day. This increase appears to be linear to the mem consumption of the 'jsvc' process run by 'iseadmi+' user.
I've opened a TAC case specifically for a potential memory leak, and attached output of the tech top command illustrating the issue, along with a support bundle that captures pre & post patch logs.
If you have any further recommendations, please let us know. I'll keep this post up to date with what TAC finds.
09-26-2018 06:48 AM
Providing an update on this case.
TAC had me enabled the following debugs, and collect a support bundle to upload stating: "The bug fixed by patch 9 matches your issue's description perfectly but we need to keep investigating at the logs for more information."
After reviewing the logs, TAC requested a webex for live troubleshoot, that happened yesterday. TAC performed the following during the webex:
Currently waiting for log review.
09-30-2018 10:34 PM
10-01-2018 04:41 AM
No update yet. I was going to request an update today as it has been > 72 hours since they updated the case.
10-02-2018 04:48 AM
FYI, this case has been escalated to the BU.
10-07-2018 12:20 AM
Thanks for the update.
10-24-2018 03:21 AM
Any Luck or word from Cisco on this issue? Has it been resolved finally?
10-24-2018 05:50 AM
We're still working the case.
TAC has us collecting the following from three different nodes every few days:
hopefully soon TAC/BU will have enough data to troubleshoot.
10-24-2018 06:31 AM
Ok..thanks.
11-01-2018 04:29 AM
FYI to all, I received the following email from TAC yesterday afternoon:
Hello Anthony,
You don’t need to continue collecting information on your ISE deployment anymore. Engineer I was collaborating with has shared with me an internal defect. I will get a summary of their findings and send you as soon as I get it.
I will update this thread with their response once I receive it.
12-12-2018 04:34 AM
Just wondering if Cisco has released any solution for this problem.
Abi
12-12-2018 07:50 AM
12-12-2018 08:06 AM
Agree with this.
My case is still open unfortunately. The BU agrees that there's a problem... but they just continue to have me collecting logs and for some reason are unable to figure out the issue. I'm a bit shocked at how long this has gone. Current path is to wait for the node to fail so they can collect the crash bundle.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: