Cisco ISE - VM high CPU load after 2 days -> reboot

[#1]

Dear all

We deployed a Cisco ISE-VM at a customer and running it as a radius-server for WLAN-Clients. Everything is working fine but after 2 days the ISE has a high cpu load and does not act as a radius-server anymore. After a reboot everything is working fine.

VM-Ware Settings

- CPU

- 2 virtual sockets

- 3 cores per virtual sockets

- 16 GB Ram

- Disk size 400 (Thin Provisioning -> Flash)

- License -> 300 concurrent endpoints

ISE was installed with the ISO-File "ise-2.2.0.470.SPA.x86_64.iso" - ISE Version 2.2.

Does anyone have a similar performance issue with ISE 2.2 (vm)?

Thanks a lot and best Regards

[#2]

I have the same issue on one of my nodes. After a day or two it locks up, you cant ssh, browse etc to it. I have to restart it in VMware.

If i make the other node the primary it doesn't have the same issue. 

Seems to only happen on one node. 

[#3]

Dear Bankstown

Thanks a lot for your feedback.

The effect is excactly the same cannot access ssh nor webgui.

In my case it is a single node installation. Are you doing automatically backups as well in your esx-store?

Thanks a lot and best regards

Oliver

[#4]

Yes we are taking snapshots using Commvault. I will disable them for tonight and see if the node is responding in the morning. Fingers crossed!

[#5]

Dear Bankstown

Thanks for your answer.

In our environment without snapshots it seems to be stable so far. Let me know if it works in your case.

Finger crossed and best regards

Oliver

[jacob.fredriksson]

Does the high CPU usage strike at the same time very day?

We had a similar problem on a Monitor node where CPU would go from 20% to 100% when ISE tried to run a cleanup script that was run on a set schedule every day. This was on a node that was also installed using the ISE 2.2 ISO-file.

The root cause was that something was stuck in the database which caused the cleanup script to execute in a bad way. We fixed it by running the command "application configure ise" in CLI and then selecting options [4].

[#7]

Dear Jacob

Thanks a lot for your feedback.

We moved the ise-vm to esx-store where no automatically backups (snapshots) are made by esx and the ise is running stable since 4 days. I keep an eye on it.

Are you doing automaically backups aswell in your esx-store?

Many thanks and best regards

Oliver

[jacob.fredriksson]

Snapshots are not supported on any ISE node. When we still had ISE 2.1 we took snapshots every night and it worked fine but in ISE 2.2 our scheduled snapshots COMPLETLY killed the nodes. No web/SSH/VMware console access. When restarted they worked fine until next nightly snapshot.

After we removed the snapshot trouble we ran into the problem I described before (high CPU from cleanup-scripts) but now everything is fine.