TCP reverse traffic blocked, timeout settings not helping

[davidnelson]

I have an ongoing issue where TCP traffic is being blocked in the reverse direction and it's happening too frequently for TCP timeout settings to come into play. This is an ASA-5516 running the newest version available for it: ASA 9.16(4)18.

This affects ssh, https, and other TCP traffic from various vendors. To give an example, let's suppose my ACLs are configured like this (a very small but representative snippet):

object service splunk-indexer
 service tcp destination eq 9997

object-group service SPLUNK_DATA
 service-object object splunk-indexer

access-list RMT_Network_ACL_In extended permit object-group SPLUNK_DATA object-group ALL_SPLUNK_INDEXER object-group LOC_LINUX_SERVER
access-list RMT_Network_ACL_In extended deny ip any any log

access-list RMT_Network_ACL_Out extended permit object-group SPLUNK_DATA object-group LOC_LINUX_SERVER object-group ALL_SPLUNK_INDEXER
access-list RMT_Network_ACL_Out extended deny ip any any log

For the purposes of this, ignore that it has anything to do with Splunk; this happens over ssh, https, sql, and a variety of other protocols as well.

As expected, all traffic is allowed between devices whenever each side tries to reach the other on port 9997. But, consistently in the middle of perfectly normal operations, I'll get logs like this at random intervals:

access-list RMT_Network_ACL_In denied tcp RMT_Network/192.168.40.42(9997) -> LOC_Linux_Server/192.168.10.5(55466) hit-cnt 1 first hit [0xd4939a65, 0x00000000]

What's going on here? Presumably, this is the reverse traffic being blocked, but the ASA is supposed to be stateful, so this shouldn't be possible. And again, I get these logs constantly - hundreds of times per day - from all different devices on all sorts of protocols.

My timeout settings are all default and the results of "show conn" don't provide any insight. My old ASA-5510 had essentially the same ACLs and notably did not have this issue. I never experienced these reverse traffic blocks until using the ASA-5516. I've searched for hours trying to find someone who has this same issue, but somehow I appear to be entirely unique in this.

Any help at all is appreciated.

[M02@rt37]

Hello @davidnelson,

Regarding your concern about the ASA being stateful, it's possible that the ASA is not properly maintaining the state of the connections. This could be due to various reasons, such as an excessive number of connections, high CPU or memory utilization:

--Monitor the CPU and memory utilization on the ASA to see if it is experiencing high load or resource exhaustion.

--Check the ASA logs and error messages to see if there are any relevant messages or warnings that could indicate the cause of the issue.

[MHM Cisco World]

but the ASA is supposed to be stateful<<- that correct but the ASA need to keep Conn  detail to all traffic.
where you apply this ACL? in which interface ?