Cisco ISE 2.3 Patch 5 - Alarms: Health Status Unavailable

[pgiouvanellis]

Hello everyone ,

We have an ISE implementation with 2 PAN nodes that are also and PSN and MnT nodes .

The last 2 days we receive alerts for one of the nodes with message "Alarms: Health Status Unavailable" .

The issue is that we are not losing the node and all the services seems to work/running fine .

The alert comes every hour but we never lost the node .

I want to refer that the specific node is a VM not physical appliance .

Is there any way to find the reason of these alerts and fix the issue .

Thank You ,

Palaiologos .

[Surendra · Accepted solution]

Check iseLocalStore.log from Operations > Troubleshoot > Download Logs > [Choose the node for which the alarms are thrown for] > Debug Logs.

Search for logs similar to “NOTICE System-Stats: ISE Process Health” and see the time difference between these logs. They are supposed to be generated every 5 minutes. If the delay is more than say 10 minutes, your MnT node will throw the error that it did not receive the health status. There is a bug CSCvh17334 for this issue as well which is fixed in 2.4.

Another reason why this could happen is if there is high load on the MnT nodes and that the logs are not processed enough. If you see any latency in the live logs or anything then this could be it, if not then it is most likely not a problem with MnT node.

[pgiouvanellis]

Its seems the last 2 days i have the problem i did not receice 

NOTICE System-Stats: ISE Process Health messages but i get 

NOTICE System-Stats: ISE Process Health Unavailable messages .

I get them every 5 minutes .

Also i noticed 

2018-12-07 08:24:25.787 +02:00 0066061587 34140 WARN  System-Management: ISE failed secure syslog connection because of unknown certificate in syslog server certificate chain, ConfigVersionId=78, DestinationPort=6514, LoggerName=SecureSyslogCollector, 

These messages but i get the same messages when the alarm was not triggered and all works normally without alarms .

So any idea ? 

Thank You,

Palaiologos

[pgiouvanellis · Accepted solution]

Hello Everyone ,

We performed a reload on the virtual machine and the issue and alarms stopped .

Thank You,

Palaiologos

[Nadav · Accepted solution]

If the issue comes up again, keep in mind that the health status is sent from each node to the MnT nodes. If you're not receiving a health status summary from an ISE node to the active MnT, that could many a few things. 

1) It could mean that your resources are very high on either MnT or the syslog client, although that would be noticeable in other ways such as show commands and VM summary within your hypervisor.

2) Maybe you're using Secure Syslog between the two, and have CRL's enabled for their trusted certificate, that one of the nodes can't download the CRL and therefore the Secure Syslog isn't trusted after initial setup. This can be seen within the localstore.log file on both MnT and syslog client. 

3) Maybe you have a firewall between the two nodes and due to the long-lived TCP connection, the connection is being dropped ungracefully by the firewall so that the flow is dropped.