ISE and complete SYSLOG message list for clever event management

[Arne Bier]

Hello

My customer wants to monitor their ISE deployment more closely and we have recommended enabling more ISE Logging Categories, especially for issues that could cause business impact.  Recently one of their ISE appliances stopped processing Radius because of a disk full issue.  It was just one SYSLOG in a sea of millions and was not spotten in time - even if it had been spotted, the 1st line guys/gals may not understand the impact or who to inform next.  The goal is to focus on the top 45 critical ones (in my opinion) and to create some logic for their Manager of Managers. 

I was looking for the canonical list of ISE SYSLOG messages and came across an ancient Excel for ISE 2.0.  Is there anything newer than this Excel (for ISE 2.4 ?) because I have logged quite a few SYSLOG event messages that are not listed, or have no Message Code in that Excel. 

At the moment I am reverse engineering ISE to compile my own list of text strings that I can give to Operations Team for their SYSLOG application filtering.  The final goal is to group and classify these alarms and create some automated rules about escallation paths etc.  - it's very tedious though.  I would appreciate if anyone else has done a similar exercise - or if not, does anyone have access to their SYSLOG server and can perform a quick grep of any "CISE_Alarm CRITICAL" and send those over to me? You might be surprised about what's in there ;)

I am spending time torturing ISE to provoke all these error conditions - but I wish I didn't have to.

[hslai]

This should have shown in ISE admin web UI in recent ISE releases, but ISE 2.4 has CSCvi88520.

Attached is a copy of the XML file from ISE 2.4. I hope it suffices.

[Arne Bier]

Hi @hslai - thanks for this - would you be okay sharing the .XML file as an attachment on this forum?  It might help others too.

[hslai]

I edited my previous response and attached the file there.

[Arne Bier]

I can't find some of the rather important SYSLOG messages in that file.  Just some random examples

CISE_Alarm CRITICAL: High Disk I/O Utilization: Server=ise01

CISE_Alarm CRITICAL: High Disk Utilization: Server=ise01

CISE_Alarm CRITICAL: High Memory usage: Server=ise01

Where is the list that contains this stuff?

[hslai]

Currently, ISE alarms are in somewhat free format and not using message catalog.

[Arne Bier]

That means I am back to square one again.  Reverse engineering based on torture testing.

I have around 1/3 of my list complete. 

Sounds to me like there is some work to be done to get error codes assigned to all of these events and to index them into this XML file.  And then update the CCO document (been neglected since ISE 2.0)

[hslai]

As they are in free format, hard to document them.

CSCvi40720 is an existing enhancement Nidhi raised on this area.

[anthonylofreso]

Man, I was excited when I saw this discussion come through... Much less excited after reading through the thread. Sounds like a significant miss on the vendor's part.

Thank you for your efforts, and happy to contribute if I can find the time. Hopefully others on the community have done some of this leg work!

[paul]

I am just catching up on this thread and have a few questions.  I haven't spend much time on the log parsing side of ISE, but there are two types of logs that can be sent in Syslogs from what I understand:

1. Native syslog messages that are listed on the Administration->System->Logging->Message Catalog screen.  All of these logs have Categories, Classes and Codes assigned to them.
2. Alarms that can also be sent via syslog.

So are you saying that for #1 where all the work has been done to create the Categories, Classes and Message Codes that that information doesn't make it into the actual Syslog Message?  Or are you perhaps seeing the Alarms turned into Syslogs and those are just the raw alarm messages sent via Syslog?

[Arne Bier]

@paul - yip it's the Alarms that don't have numbers assigned to them - that part I don't mind so much - it would still be okay if we had a list from Cisco that contained all the possible alarm strings and which category/setting is ISE triggers it.  There are alarms that I don't think I could ever simulate (unless I had root access and knew exactly what to do).  But I have managed to capture around 1/3 of the ones I am interested in.

I'd be keen to have some more from folks out there who can grep their SYSLOG files for the "CRITICAL" alarms. :-)

[anthonylofreso]

Hmm, seems to me it would be easy for someone at Cisco *cough cough* *wink wink* to pull the actual ISE application syslog config. With root access, this shouldn't be very difficult. Just do a recursive 'find' for 'CISE_Alarm' under the directory where the app is running.

Does this make sense? Or am I just blowing smoke? Maybe it's not that simple.

[Arne Bier]

I don't see any SYSLOGs being sent from an ISE node, when it loses connection to the Primary PAN.  This would be a scenario where the PAN loses connectivity to another ISE node.  I would have expected something ... anything.  e.g. replication warning or heartbeat warning.

Maybe I did something wrong. But in my case the remaining node is still sending SYSLOGs (e.g. regular System_Statistics etc.) - but it doesn't complain about the loss of comms to the PAN.  Weird.

[paul]

Arne,



So we agree that the native Syslog messages detailed out on the logging screen have message codes and categories set right? We are only talking about alarms turned into Syslog messages. Are you seeing different alarm messages/texts then what appears on the Administration->System->Settings->Alarms screen? That should be the complete list of alarms and the text that should be sent in the Syslog messages.

[Arne Bier]

Hi  @paul

I have not looked into all of the SYSLOG messages so I won't comment on whether they are correct or not.  @hslai has uploaded the latest .XML file containing all the SYSLOG messages that are sent via the proper way.  The remaining SYSLOGS are just scattered around in the code if and when needed (and those don't have ID numbers).

I took the interesting Alarm categories from ISE and then compiled an Excel like below.  The column SYSLOG example is based on my own lab output.  I doubt I will be able to fill the entire sheet - and that is why I was hoping someone out there had done something similar.

A customer can't spend all day looking at their SYSLOG waiting for something to trigger on - we need to be able to supply them with the strings that they can intelligently match on.