ASA remote VPN hairpin issue

[Chess Norris]

Hello,

I would really appreciate some help here with a VPN hairpin issue that I've been encounter. 

I am helping a customer who wants to tunnel all traffic including Internet through their local firewall.

To get this to work, I added the "same-security-traffic permit intra-interface" command and a NAT outside,outside rule.

This is something I've helped a lot of customer with before, but this time I'm stuck.

The customer is running a very old 8.4 ASA release, but at least the NAT config should be pretty similar to newer versions. 

After applying the "tunnel all" policy and trying to reach some addresses on the Internet, I am getting NAT reverse path failures

I see the following in the log

Jun 09 2021 17:18:57  %ASA-5-305013: Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src outside:10.123.127.50 dst outside:178.79.133.95 (type 8, code 0) denied due to NAT reverse path failure  

The message looks to me like the NAT rule for the VPN pool not matching or is overlaping with some other NAT rules, but I can't find anything in the configuration that points to this. 

As a test I just configured a similar tunnel all policy on my lab firewall and it just started to work immediately. The only difference is that I'm running 9.8 code. 

Here is the configuration that I think should be relevant, but let me know if you need anything else. (I am hiding the real public IP of the firewall and exchanged it with a private one)

interface GigabitEthernet0/0.700
vlan 700
nameif outside
security-level 0
ip address 10.10.10.5 255.255.255.248
delay 1
!
same-security-traffic permit intra-interface
!
access-list outside_access_in extended permit icmp any any echo-reply
access-list outside_access_in extended permit icmp any any time-exceeded
access-list outside_access_in extended permit icmp any any unreachable
access-list outside_access_in extended permit ip 10.123.127.0 255.255.255.0 any
access-list outside_access_in extended deny ip any any log
!
ip local pool VPNPool 10.123.127.1-10.123.127.63 mask 255.255.255.192
!
object network VPNnet
subnet 10.123.127.0 255.255.255.192
!
object network INSIDE_NET
subnet 10.0.0.0 255.0.0.0
!
object network DMZ_NET_RANGE
subnet 10.122.167.160 255.255.255.224
!
nat (inside,outside) source static INSIDE_NET INSIDE_NET destination static VPNnet VPNnet
nat (DMZ,outside) source static DMZ_NET_RANGE DMZ_NET_RANGE destination static VPNnet VPNnet
!
object network VPNnet
nat (outside,outside) dynamic interface
!
nat (any,outside) after-auto source dynamic any interface
!
route outside 0.0.0.0 0.0.0.0 10.10.10.1
route inside 10.0.0.0 255.0.0.0 10.122.167.139 1

Thanks

/Chess

[Rob Ingram]

@Chess Norris 

If you run packet-tracer that should indicate the NAT rules matched and provide a clue as to the issue.

If necessary, run packet-tracer from the CLI and provide the output for review.

[Chess Norris]

That is another issue I have with this firewall. For some reason the packet-tracer fails due to the deny ip any any rule at the end of the ACL outside, even though I have allowed the VPN pool network in the ACL outside before the deny rule. (see below)

access-list outside_access_in extended permit ip 10.123.127.0 255.255.255.0 any
access-list outside_access_in extended deny ip any any log

[Rob Ingram]

"sysopt connection permit-vpn" configured by default on your ASA 9.8 lab but not the ASA running 8.4?

[Chess Norris]

I will have a remote meeting with the customer tomorrow and will have the chance to login and troubleshoot  their ASA again. Do you have any suggestion for specific debugs commands or other measures I can prepare for troubleshooting? 

Thanks

/Chess

[Chess Norris]

I normally disable sysopt connection permit-vpn and use the ACL outside instead to control which VPN traffic to allow, but

in this case, the customer have the "sysopt connection permit-vpn" command configured and I also verified that there is no vpn-filter in use. 

[Chess Norris · Accepted solution]

I think I found the cause to this issue. The "nat (any,outside) after-auto source dynamic any interface"  at the end was interfered with the NAT rule for the VPN pool, even though it's an after-auto nat rule that should be evaluated last.

I think I read somewhere that Cisco don't recommend using "any" in NAT configuration. After changing the NAT rule to 

"nat (inside,outside) after-auto source dynamic any interface" everything started to work.

/Chess