ISE 2.4 "No Data available" in System Summary

[dodavis]

We have been live for about 3 weeks with ISE 2.4 using RADIUS authentication on our production wireless network. On Monday we started getting "No Data available" in Home>System Summary.

Even though authentications appear to still be working, the RADIUS Live Log is very rarely showing any new information.

The TAC suggests rebooting the server due to a known bug caused by memory utilization. We are in a two node HA pair.

- Am I likely to see an outage for RADIUS authentications during a reboot?

- Is it normal to have to reboot so soon after going live with ISE?

[Damien Miller]

Are you hitting this? I've been hit by this as well and am waiting a fix.
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvo87602

Now the impact relies heavily on how consistent your radius config is and which node you are reloading. If the node you have to reload is configured as the secondary radius server everywhere, and is the secondary PAN, then the impact would be non existent.

If you have NADs configured with the node you need to reload, then you will be relying on their RADIUS failover. Any authenticated endpoint will remain unchanged, new authentications have to leverage the node still up. If the primary PAN has to be reloaded, there are some features that would be unavailable during the reload, you can read about them here.
https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/admin_guide/b_ise_admin_guide_24/b_ise_admin_guide_24_new_chapter_011.html#ID59

I recently ran in to a new unpatched issue with database memory exhaustion, TAC had to adjust the platform properties to allocate the component more memory. This created missing live sessions, and partially working live logs. Live logs appeared to be working, but endpoints auth attempts were hit or miss if they were logged.

[dodavis]

Thanks for the response! That is the exact same bug the TAC thought we are running in to.

I ended up reloading the secondary server, promoting it to primary, then reloading the other server. The appears to have fixed the System Summary and RADIUS Live logs issues....for now.

I guess I will keep and eye on memory consumption and hope a patch for the bug is released soon. In the mean time I have trimmed back some of the Profiling Configuration options for the deployment in hopes of reducing the load on system resources.

If I get more information from the TAC I will post the results in hopes that it could help someone else in a similar situation.

[Damien Miller]

I have been told that they are trying to get the fix in 2.4 patch 11. I have a 3595 node that began to hit it within 5 days again after a reload. It was showing 50% utilization when health status alarms began, so keep a very close eye on node memory daily. It's now up at 80% and consistent alarms.

[Nishanna Gunasekera]

What happens when the server runs out of memory? Does it become completely unavailable and a manual (physical) reboot is necessary? I have some nodes that aren't responding to SSH and I'm concerned I'm hitting this bug. 

[dodavis · Accepted solution]

The bug I ran into had not affected SSH yet. The bug we had first caused the Home>System Summary to stop showing the system health information. Two days later the RADIUS Live Logs stopped working.

I was able to SSH into the servers to perform a reload and that resolved the problems for now.

[dmilyutov]

Exact the same I have now under 2.6 patch 6.

[Jason Kunst]

If it’s the administration or monitoring node dedicated and you were asked to reboot that then there should be no outage.

These really should be asked of the tac as well