C9300 TCP Syslogs failing periodically.

[Jared Wine]

Wondering if anyone has seen behavior such as this. TCP syslogs from a switch are failing periodically, for exactly 5 seconds at a time. The connection is immediately re-established, so this isn't really a major issue, but it's filling up the logging buffer and spamming my syslog servers. Two different syslog servers are configured on the switch, each behind their own firewalls (virtual FTDs). Both syslog servers are failing at the same time, and one right after another, so I have a hard time believing the root cause is anything other than the switch itself. Here's what the logging buffer on the switch looks like:

#show logging | i Logging

020858: Oct 10 10:28:28.784 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.11.8.21 port 514 failed
020859: Oct 10 10:28:33.786 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.11.8.21 port 514 started - reconnection
020860: Oct 10 10:28:33.786 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.12.8.21 port 514 failed
020861: Oct 10 10:28:38.787 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.12.8.21 port 514 started - reconnection
020865: Oct 12 00:21:32.113 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.11.8.21 port 514 failed
020866: Oct 12 00:21:37.115 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.11.8.21 port 514 started - reconnection
020867: Oct 12 00:21:37.115 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.12.8.21 port 514 failed
020868: Oct 12 00:21:42.116 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.12.8.21 port 514 started - reconnection
020871: Oct 14 13:25:56.000 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.11.8.21 port 514 failed
020872: Oct 14 13:26:01.001 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.11.8.21 port 514 started - reconnection
020873: Oct 14 13:26:01.002 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.12.8.21 port 514 failed
020874: Oct 14 13:26:06.004 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.12.8.21 port 514 started - reconnection
020879: Oct 16 03:30:04.072 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.11.8.21 port 514 failed
020880: Oct 16 03:30:09.074 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.11.8.21 port 514 started - reconnection
020881: Oct 16 03:30:09.074 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.12.8.21 port 514 failed
020885: Oct 16 03:30:14.076 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.12.8.21 port 514 started - reconnection
020917: Oct 16 15:30:22.934 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.11.8.21 port 514 failed
020920: Oct 16 15:30:27.935 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.11.8.21 port 514 started - reconnection
020921: Oct 16 15:30:27.935 est: %SYS-3-LOGGINGHOST_FAIL: Logging to host 10.12.8.21 port 514 failed
020922: Oct 16 15:30:32.937 est: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.12.8.21 port 514 started - reconnection

So the TCP connection fails for server-1. 5 seconds go by and the TCP connection is re-established. At that exact moment, the TCP connection to server-2 fails. Then 5 seconds later it comes up again.

Configuration on the switch looks like this:

logging userinfo
logging buffered 65536 informational
logging persistent url flash:/logfile size 1638400 filesize 65536
logging console notifications
logging facility local2
logging host 10.11.8.21 transport tcp port 514
logging host 10.12.8.21 transport tcp port 514

---------

Seems like an issue on the switch's side, but I'm not sure what it could be. Both syslog servers are behind a router and a firewall (different router and firewall for each syslog server), but the somewhat random and synchronized timing of the failures have me confused.

[Mark Elsen]

 - @Jared Wine    Check logs on the firewalls   and or check that this traffic is allowed at all times 

          M.

[balaji.bandi]

I have observed this in IOS XE and FTD: you keep getting these logs when there is no continuous TCP log sending to the Log Server (no solution found), so I have to move back to UDP 514 for now, and it works as expected.

Also, make sure your Log server listening on TCP port 514 (514 default is used for UDP in most syslog servers)

By the way, what log server ?

[Jared Wine]

@Mark Elsen Nothing in the firewall logs indicating any blocks during this time period.

@balaji.bandi Yeah the log servers (Solarwinds SEM) accept TCP/514 no problem. Switching to UDP would solve the problem (not that there actually is a problem; syslog on tcp/514 works just fine apart from these random failures), but I'd rather use TCP for the reliability.

[balaji.bandi]

@balaji.bandi Yeah, the log servers (Solarwinds SEM) accept TCP/514, no problem. Switching to UDP would solve the problem (not that there actually is a problem; syslog on tcp/514 works just fine apart from these random failures), but I'd rather use TCP for the reliability.

until 17.9.5 i have tested, not found the solution, until we suppress the logs not to send to syslog.(that is only i am thinking)