cisco esa log forwarding in a cluster

[Thom89]

Hello.

I have a question regarding a cluster of cisco esa appliances and behaviour of syslog forwarding (whether it's worsk as designed or possibly a bug/issue)

we have a cluster of 2 cisco esa. on cluster level they have a syslog forwarder configured. recently we noticed that nearly all logs arrive on our syslog server from only 1 of the 2 nodes.
no logs or just 1 every 2 days on average come from the other.
when I look at the dashboard it seems the 2nd node is indeed processing some mails. (not as much as the other node, but it still amounts to about 20% of all mails)

I was wondering if a esa cluster has a mechanism to send all logs( even from the mails processed by the other node) via the one (primary/prioritized) cluster node.
or whether something is wrong here?

I looked in documentation, guides, etc. but I can't find a clear black on white answer how syslog forwarding is supposed to behave.

any help( pointing me to a correct resource/link or confirming this would be greatly appreciated.

[Thom89]

update on this issue.
I also have access to the 1 firewall in between the esa appliance and the syslog server.

at first glance it looked okay( when I clear a session from either appliance a new tcp syslog session is setup within a minute.)

looking into more detail however I noticed 1 appliance is sending syslog to the correct server (let's say ip 192.168.1.21(not the actual ip)
the other node is trying to setup syslog to a different ip( 192.168.1.11) exactly 10 lower then the actual ip.
this of course is an ip either not in use or not used for a syslog server so the connection ages out/remains in init.

I have opened a tac case as I can't find any configuration on the faulty appliance for the incorrect ip( 192.168.1.11 is used nowhere in the configuration)
so I suspect a bug. but I'm not sure.
running esa 13.5.2-036 I believe

[marc.luescherFRE · Accepted solution]

We had to learn the hard way that all logging configuration should ALWAYS be done on the machine level:

a) cluster traffic might exceed the amount of traffic syslog engine can handle in the ESA

b) we never figured out how syslog in a cluster was really supposed to work and finally switched t machine mode which is predictible

c) syslog on machine level for maillog with extension of URL enable might also need you to to enable cut off settings

d) ESA Rel 14.0.1 finally fixed some syslog limitaions, now you can use TLS, define destination ports and some other fixes.

Happy to answer any other question you might have.

-Marc

[Thom89]

Yes.
so we opened a tac support case.
the cisco engineer also mentioned it's a best practice to do the syslog on machine level. not in cluster or group.
we copied the config from cluster to machine level.
on the 1 appliance having the issue I also created a new profile( different name, same settings) next to the originally copied over one.
I then deleted the original profile. did the commit --> no success still the same issue.

after some more feedback from the cisco engineer requesting to try it once more and also provide a config file I decided to try again but be way more meticulous/step by step.

what I did exactly to fix the issue:

I first went to cluster level and fully deleted the log event there (settings on each machine were overridden, but again being a bit more thorough here)

I then went to the machine with the issue and deleted the existing log event there first.
I then committed the config (with no log event configured to forward syslog)
I then recreated the exact same log event( same name, same settings, same syslog server)
and committed a 2nd time.

after that the appliance finally started sending logs to the correct destination.
actual root cause: no idea, ghosts in the machine, bugs in the chips...
but completely erasing the config and then configuring it again exactly the same seems to work as a fix.