High memory utilization on ISE 3.1 after patch upgrade - patch 9

[rajesh halvi]

Hi Team,

   We are noticing high memory utilization on cisco ISE 3.1 nodes after patch upgrade. we upgraded from patch 3 to Patch 9. The memory goes upto 80+ percent and then at one point it immediately reduces to 40%. When the memory goes high, it is impacting authentication services. Especially the guest captive portal goes down. Did anyone have same behaviour with patch 9?

[ahollifield]

What size node?  What size deployment?  Are you within the scale limits?  https://cs.co/ise-scale

Any reason why 3.1 and not 3.3 or 3.4?

[rajesh halvi]

It is 16CPU/32GB PSN and 24CPU/96GB PAN and MNT… it is 35 nodes deployment and also another with 18 nodes. Problem happening on both deployment after upgrade. TAC said it could be due to broken ISE messaging service certificate on few nodes. My concern is why it started all of a sudden after upgrade.

[ahollifield]

Very common after ISE upgrades to have to re-generate the ISE root CA and ISE messaging service certificates. I have no idea why IMS is so buggy but it is

[marce1000]

   @rajesh halvi wrote >...My concern is why it started all of a sudden after upgrade.
                                    What can sometimes happen  is that the VM's (if virtual deployments are used) 
                                    are configured on the edge w.r.t resources such as memory ; if it then starts using
                                    somewhat more memory it may spike because of a trigger that has been set in motion and includes
                                    other stuff 'with needs'.
                                    I have had to increase memory in the past ; but usually it was after a full upgrade however ,

 M.

[lightda]

Hi, I've got a same problem here, with ISE 3.1 Patch 9,
after showing tech top, we found out that there's a command called "swapoff" using high CPU with 77.8%, and the swap mem of ISE was full.

[rajesh halvi]

Cisco BU team is saying the memory leak is happening because of queue links broken between few ISE nodes. We have ISE nodes in different company in different VRFs. so there are intercompany routing restrictions between few nodes. so few nodes will not be able to communicate over port 8671. My question - is it really necessity to have communication between all policy nodes in a deployment? note that this deployment was working since last 3 years even earlier to that. 

[ahollifield]

Yes for ISE Messaging Service and Lightweight Data Distribution.