TCP Deny (No Connection) from SRC to DST flags ACK on Outside IF?

[andrewjinks]

We work with an organization coming in from an AWS EC2 instance trying to connect to an API we have onsite that's behind our Firepower FTD 2130 (7.4.2.1, managed by FMC) firewalls (there is an external/public NAT'd IP address it hits first, but that seems to translate fine since it's upstream and hits our FW with the internal destination IP).  They are complaining that they're getting timeouts and can't complete the API connection (to TCP/443).  When reviewing the logs, there are several connections from that same source to the same destination and port that are successful since we have an explicit policy permitting that traffic, but at the same time, we're seeing the following Deny message in between:

Deny TCP (no connection) from <AWS-IP/Port> to <Local-DST-IP/443> flags ACK on interface <OutsideInterfaceName>

And the log before it with the same exact time stamp:

Built inbound TCP connection 528346990 for <OutsideInterfaceName:<AWS-IP/Port> (<AWS-IP/Port>) to <InsideInterfaceName>:<Local-DST-IP>/443 (Local-DST-IP>/443)

There seems to be something going on or a unique attempt at connection between EC2 instances in AWS and this API.  Other sources can connect to the API.  Can someone explain what is happening and how this could be corrected?  I'm under the impression that it is something on their end (this has been seen before, but a while back and with different people - I'm not sure what the fix was), but even if that is the case, I'd like to  be able to recommend a fix for them now.  Thanks.

[nspasov]

I suspect asymetric traffic to be causing your issue but need more info. Can you share:

1. Diagram that shows your setup
2. Output from packet-tracer
3. Confirm that the packet-tracer output is identical when you run it several times in a row

Thank you for rating helpful posts!

[andrewjinks]

Not sure I have a packet capture system handy as most of this is virtual, but to lay it out:

AWS API request on TCP/443 comes in from outside to a firewall upstream (we do not manage this) that does NAT against the public IP on the internal RFC1918 destination address, which is a Virtual Server on an F5 BIG-IP (VE edition) load balancer.  It then hits our firewall, which has a policy permitting the source (AWS) to the destination (Virt IP on F5 on port 443), which then goes to that F5 virtual server IP, then to a backend host in the server pool.  It does the round trip back through the F5, back through our FW, back out the upstream firewall and back out to AWS EC2.  Not sure where to put a packet sniffer, but the packet tracer on the Firepower Accepts the traffic.  Any advice? Thanks.

[andrewjinks]

If it helps, here is a sequence of sanitized logs from Splunk at about the same time from three sources:

3/14/25
4:08:15.000 PM	
Mar 14 12:08:15 <F5-load-balancer-internal-interface-fqdn> [14/Mar/2025:12:08:14 -0400] - <AWS-EC2-instance-source-ip> - <F5-Virtual-server-API-IP> - <back-end-host-API-IP> - 200 OK - 384

    host = <F5-load-balancer-internal-interface-fqdn>
    source = /var/log/remote/<F5-load-balancer-internal-interface-fqdn>/all.log
    sourcetype = syslog:non_prod_f5

3/14/25
4:08:15.000 PM	
Mar 14 16:08:15 <log-server-IP> <Firepower-external-interface-IP> <166>:Mar 14 16:08:15 UTC: %FTD-session-6-106015: Deny TCP (no connection) from <AWS-EC2-instance-source-ip>/7968 to <F5-Virtual-server-API-IP>/443 flags ACK  on interface OutsideConnection-Dev

    host = <Firepower-external-interface-IP>
    source = /log/cisco-asa/<log-server-IP>-2025-03-14.log
    sourcetype = cisco:asa fw

3/14/25
4:08:14.446 PM	
Mar 14 16:08:14 <upstream-Juniper-FW-IP> <upstream-Juniper-FW2-IP> <14>1 2025-03-14T16:08:14.446Z <upstream-Juniper-FW-name> SESSION_CREATE [src-addr="<AWS-EC2-instance-source-ip>" src-port="7968" dst-addr="<API-public-IP>" dst-port="443" service="https" nat-src-addr="<AWS-EC2-instance-source-ip>" nat-src-port="7968" nat-dst-addr="<F5-Virtual-server-API-IP>" nat-dst-port="443" dst-nat-rule-type="dst rule" dst-nat-rule="API-NAT-RULE" protocol-id="6" policy="PolicyName" src-zone="OUTSIDE" dst-zone="INSIDE" session-id="188988903765" pkt-in-intf="reth1.0" dst-identity-context="N/A"]

    host = <upstream-Juniper-FW2-IP>
    source = /log/junos/<upstream-Juniper-FW-IP>-2025-03-14.log
    sourcetype = juniper:junos fw

3/14/25
4:08:14.000 PM	
Mar 14 12:08:14 <F5-load-balancer-internal-interface-fqdn> [14/Mar/2025:12:08:14 -0400] - <AWS-EC2-instance-source-ip> - <F5-Virtual-server-API-IP> - <back-end-host-API-IP> - 

    host = <F5-load-balancer-internal-interface-fqdn>
    source = /var/log/remote/<F5-load-balancer-internal-interface-fqdn>/all.log
    sourcetype = syslog:non_prod_f5

3/14/25
4:08:14.000 PM	
Mar 14 16:08:15 <log-server-IP> <Firepower-external-interface-IP> <166>:Mar 14 16:08:14 UTC: %FTD-session-6-302013: Built inbound TCP connection 665256622 for OutsideConnection-Dev:<AWS-EC2-instance-source-ip>/7968 (<AWS-EC2-instance-source-ip>/7968) to ether-to-switches-Dev:<F5-Virtual-server-API-IP>/443 (<F5-Virtual-server-API-IP>/443)

    host = <Firepower-external-interface-IP>
    source = /log/cisco-asa/<log-server-IP>-2025-03-14.log
    sourcetype = cisco:asa fw

3/14/25
4:07:59.255 PM	
Mar 14 16:07:59 <upstream-Juniper-FW-IP> <upstream-Juniper-FW2-IP> <14>1 2025-03-14T16:07:59.255Z <upstream-Juniper-FW-name> SESSION_CREATE [src-addr="<AWS-EC2-instance-source-ip>" src-port="50785" dst-addr="<API-public-IP>" dst-port="443" service="https" nat-src-addr="<AWS-EC2-instance-source-ip>" nat-src-port="50785" nat-dst-addr="<F5-Virtual-server-API-IP>" nat-dst-port="443" dst-nat-rule-type="dst rule" dst-nat-rule="API-NAT-RULE" protocol-id="6" policy="PolicyName" src-zone="OUTSIDE" dst-zone="INSIDE" session-id="188989261039" pkt-in-intf="reth1.0" dst-identity-context="N/A"]

    host = <upstream-Juniper-FW2-IP>
    source = /log/junos/<upstream-Juniper-FW-IP>-2025-03-14.log
    sourcetype = juniper:junos fw