ASA error "Deny TCP (no connection)"

mikekaleny · ‎03-24-2020

Good morning all,

First of all I hope everyone is staying healthy during these crazy times.

I've been trying to solve this issue for over 3 days but just can't seem to wrap my brain around it. I have a webserver with an internal IP address of 192.168.2.100. It's configured to use a 1 to 1 static nat however I cannot load the webpage when I browse to it's public IP. The error I see in the ASDM log is below.

Deny TCP (no connection) from X.X.X.X to X.X.X.X flags ACK on interface outside2

I'm really bad at working with ASA so ANY help on this would be greatly appreciated. My show run is below

interface Vlan1
nameif inside
security-level 100
ip address 192.168.2.1 255.255.255.0
!
interface Vlan2
nameif outside
security-level 0
ip address X.X.X.X - (not in use)
!
interface Vlan12
description Optimum
nameif outside2
security-level 0
ip address 1.1.1.1 - (IP changed, in use)
!
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
object network NET-OBJ-LOCAL-NETWORK
subnet 192.168.2.0 255.255.255.0
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network websrv_ext
host 1.1.1.2 (IP changed)
object network websrv_int
host 192.168.2.100
object-group network DM_INLINE_NETWORK_2
network-object object websrv_ext
network-object object websrv_int
object-group network DM_INLINE_NETWORK_1
network-object object websrv_ext
network-object object websrv_int
access-list 101 extended permit icmp any4 any4 echo-reply
access-list 101 extended permit tcp any object websrv_int eq https
access-list 101 extended permit tcp any object websrv_int eq www
access-list 102 extended permit ip any any
access-list 102 extended permit icmp any4 any4 echo-reply
access-list 102 extended permit tcp any object websrv_ext eq www
access-list 102 extended permit tcp any object websrv_ext eq https
access-list outside2_access_in extended permit ip any4 any4
pager lines 24
logging enable
logging asdm informational
mtu inside 1500
mtu outside 1500
mtu outside2 1500
no failover
icmp unreachable rate-limit 1 burst-size 1
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
!
object network obj_any
nat (inside,outside2) dynamic interface
object network websrv_int
nat (inside,outside2) static websrv_ext
access-group 102 out interface inside
access-group 101 in interface outside2

Thanks!!

Cristian Matei · ‎03-24-2020

Hi,

1. Are you sure that the SYN, SYN-ACK and ACK go through the ASA? This could be due to some asymmetric routing, like the ASA does not see the SYN-ACK, so it drops the ACK.

2. Can you simulate the session with "packet-tracer input outside2 tcp x.x.x.x 20000 1.1.1.2 80 detailed"? What is the result?

3. Can you perform a packet capture on the ASA for the respective traffic flow, on both the inside and outside2 interfaces? See here a guide for doing it.

Regards,

Cristian Matei.

mikekaleny · ‎03-24-2020

Hi, thank you for your response. Here is the packet-tracer result. I will have to work on getting the packet capture.

ciscoasa# packet-tracer input outside2 tcp 8.8.8.8 20000 1.1.1.2 80 detail

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
object network websrv_int
nat (inside,outside2) static websrv_ext
Additional Information:
NAT divert to egress interface inside
Untranslate 1.1.1.2/80 to 192.168.2.100/80

Phase: 2
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group 101 in interface outside2
access-list 101 extended permit tcp any object websrv_int eq www
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcbc4d4b8, priority=13, domain=permit, deny=false
hits=755, user_data=0xca0792f0, cs_id=0x0, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=192.168.2.100, mask=255.255.255.255, port=80, tag=0, dscp=0x0
input_ifc=outside2, output_ifc=any

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcb996108, priority=0, domain=nat-per-session, deny=false
hits=661252, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0xcc0d09f0, priority=0, domain=inspect-ip-options, deny=true
hits=364075, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=outside2, output_ifc=any

Phase: 5
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group 102 out interface inside
access-list 102 extended permit ip any any
Additional Information:
Forward Flow based lookup yields rule:
out id=0xcc0f0130, priority=13, domain=permit, deny=false
hits=144080, user_data=0xca079700, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=inside

Phase: 6
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
object network websrv_int
nat (inside,outside2) static websrv_ext
Additional Information:
Forward Flow based lookup yields rule:
out id=0xcc9a4da8, priority=6, domain=nat-reverse, deny=false
hits=817, user_data=0xcb8eeaa0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=192.168.2.100, mask=255.255.255.255, port=0, tag=0, dscp=0x0
input_ifc=outside2, output_ifc=inside

Phase: 7
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xcb996108, priority=0, domain=nat-per-session, deny=false
hits=661254, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 8
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0xcc07a260, priority=0, domain=inspect-ip-options, deny=true
hits=333998, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 9
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 340817, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: outside2
input-status: up
input-line-status: up
output-interface: inside
output-status: up
output-line-status: up
Action: allow

Cristian Matei · ‎03-24-2020

Hi,

From the ASA perspective, all seems to be good. You sure there is no asymmetric routing somehow? Perform the packet captures, to see if SYN, SYN-ACK, ACK reach the ASA.

Regards,

Cristian Matei.

Marvin Rhoads · ‎03-24-2020

Is your route to your host (the one from which your are browsing) out the outside interface or outside2?

If it is on outside (perhaps as the default route) then it won't work due asymmetric routing as @Cristian Matei mentioned. Return traffic must use the same interface as it originally arrived on.

mikekaleny · ‎03-24-2020

Interface is disabled. The ASA is currently using outside2. I'm not sure why there would be asymmetric routing as there is only one inside and one outside interface.

Something I noticed in the log - when I have the destination IP set as 192.168.2.100, I can see the tcp connections built and teardown.

Cristian Matei · ‎03-25-2020

Hi,

You have not described the layer 3 architecture, so maybe the response from the client, somehow does not go through the ASA. Otherwise perform the packet captures on both inside and outside for the respective flow, additionally perform a packet capture of type "asp-drop all" and match on ip traffic between those two hosts.

Regards,

Cristian Matei.

mikekaleny · ‎03-25-2020

Thank you for your response. I'm not well versed in packet captures but I did see TCP Retransmissions. Just to be certain, am I setting up the packet capture the correct way?

Point of ingress - Inside interface

Source Host - webserv (192.168.2.100)

Destination Host - Public IP of client trying to load webpage

Point of egress - outside2

Source Host - Webserv Public IP

Destination Host - Public IP of client trying to load webpage

The Layer 3 architecture is simple in my network. The ASA acts as the Gateway for the inside traffic (192.168.2.0/24 subnet). There is a default route sending all traffic to the ISP equipment connecting via outside2 interface.

Cristian Matei · ‎03-25-2020

Hi,

That is correct.

Regards,

Cristian Matei.

mikekaleny · ‎03-25-2020

On the outside interface I see the [SYN] from the web-client to the public IP of the web-server. Then on the inside interface I see that [SYN] going to the private IP of the web-server (192.168.2.100).

The private IP of the web-server then sends the [SYN, ACK] out the inside interface to the web-client. The [SYN, ACK] is visible on the outside interface with the public IP of the web-server going to the web-client. The process then repeats. There is no [ACK] anywhere.

Really confused as to what's happening. Is the SYN, ACK never making it to the web-client or is the ACK never making it to the webserver? What would you recommend the next step be in troubleshooting this?

Thank you

Cristian Matei · ‎03-26-2020

Hi,

Can you perform a packet capture via wireshark on the web-client, and post it?

Regards,

Cristian Matei.

mikekaleny · ‎03-26-2020

Hi,

Yes, here are all three captures. Hope it's not too difficult to read them.

This is the capture from;

Point of egress - outside2

The source in packet capture is webclient, destination is webserver

This is a capture from;

Point of ingress - Inside interface

The source in packet capture is webclient, destination is webserver private IP

and this is a capture from the webclient. It is over the internet and not local.

The source in packet capture is webclient private IP, destination is webserver public IP

As suspected, it is not receiving the tcp syn ack. Would you know of any reason why?

Would turning the stateful feature off help?

Cristian Matei · ‎03-27-2020

Hi,

Though based on the packet capture, the SYN-ACK leaves the ASA, i would upgrade the ASA to 9.8(4)17 or 9.12(3).9. Reload and see if it's fixed. If still not, try using "sysopt connection tcpmss 1360" for example.

Regards,

Cristian Matei.

mikekaleny · ‎03-27-2020

It is an ASA 5505 and I only see 9.1.7 available. I will try that and let you know. Thank you very much for your assistance thus far.