topic Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9 in Network Access Control

High memory utilization on ISE 3.1 after patch upgrade - patch 9

rajesh halvi — Thu, 31 Oct 2024 04:03:35 GMT

Hi Team,

We are noticing high memory utilization on cisco ISE 3.1 nodes after patch upgrade. we upgraded from patch 3 to Patch 9. The memory goes upto 80+ percent and then at one point it immediately reduces to 40%. When the memory goes high, it is impacting authentication services. Especially the guest captive portal goes down. Did anyone have same behaviour with patch 9?

Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9

ahollifield — Thu, 31 Oct 2024 11:12:15 GMT

What size node? What size deployment? Are you within the scale limits? https://cs.co/ise-scale

Any reason why 3.1 and not 3.3 or 3.4?

Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9

rajesh halvi — Fri, 01 Nov 2024 13:07:50 GMT

It is 16CPU/32GB PSN and 24CPU/96GB PAN and MNT… it is 35 nodes deployment and also another with 18 nodes. Problem happening on both deployment after upgrade. TAC said it could be due to broken ISE messaging service certificate on few nodes. My concern is why it started all of a sudden after upgrade.

Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9

ahollifield — Fri, 01 Nov 2024 13:21:04 GMT

Very common after ISE upgrades to have to re-generate the ISE root CA and ISE messaging service certificates. I have no idea why IMS is so buggy but it is

Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9

Mark Elsen — Fri, 01 Nov 2024 14:14:54 GMT

@rajesh halvi wrote >...My concern is why it started all of a sudden after upgrade.
What can sometimes happen is that the VM's (if virtual deployments are used)
are configured on the edge w.r.t resources such as memory ; if it then starts using
somewhat more memory it may spike because of a trigger that has been set in motion and includes
other stuff 'with needs'.
I have had to increase memory in the past ; but usually it was after a full upgrade however ,

回复： High memory utilization on ISE 3.1 after patch upgrade - patch 9

lightda — Fri, 08 Nov 2024 04:25:13 GMT

Hi, I've got a same problem here, with ISE 3.1 Patch 9,
after showing tech top, we found out that there's a command called "swapoff" using high CPU with 77.8%, and the swap mem of ISE was full.

Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9

rajesh halvi — Mon, 02 Dec 2024 15:17:15 GMT

Cisco BU team is saying the memory leak is happening because of queue links broken between few ISE nodes. We have ISE nodes in different company in different VRFs. so there are intercompany routing restrictions between few nodes. so few nodes will not be able to communicate over port 8671. My question - is it really necessity to have communication between all policy nodes in a deployment? note that this deployment was working since last 3 years even earlier to that.

Re: High memory utilization on ISE 3.1 after patch upgrade - patch 9

ahollifield — Mon, 02 Dec 2024 15:25:57 GMT

Yes for ISE Messaging Service and Lightweight Data Distribution.