VPN Hairpinning (VPN to VPN routing)

[bighiller]

Hi,

Been trying for a week to make this work, but alas I cannot and so I ask for help.

I have two separate IKEv1 tunnels setup between our hub  ASA 5509 and two different AWS VPCs in different regions. The goal is to have the two VPCs route between each other via the ASA (hub and spoke). Connectivity between the each VPC and the hub (and corporate network) is fine and dandy, but for the life of me I cannot get the two VPCs to communicate. Both VPN tunnels terminate at the ASA.

My relevant config is below (hopefully I didn't scrub it too much) and the error I am getting now is:

5

Feb 01 2017

14:17:11

305013

10.25.0.114

55273

10.28.1.132

139

Asymmetric NAT rules matched for forward and reverse flows; Connection for tcp src outside:10.25.0.114/55273 dst outside:10.28.1.132/139 denied due to NAT reverse path failure

I've tried to figure out exactly which NAT rules, VPN filters and ACLs to run, but I think the order of processing is doing me in, but who knows.

Config:
interface GigabitEthernet1/1
nameif outside
security-level 0
ip address 123.123.123.123 255.255.255.240
!
interface GigabitEthernet1/2
nameif inside
security-level 100
ip address 10.0.15.58 255.255.240.0
!
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network awzn_vpc
subnet 10.25.0.0 255.255.0.0
object network local_lan
subnet 10.0.0.0 255.255.0.0
object network AWS-VPC-CA
subnet 10.28.0.0 255.255.0.0
description Canadian VPC in AWS
object-group service rdp tcp
description Microsoft RDP
port-object eq 3389
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
object-group network Urthecast_Networks
description All urthecast network
network-object object West_Pender_Network
network-object object local_lan
object-group network DM_INLINE_NETWORK_8
network-object object West_Pender_Network
network-object object local_lan
object-group network DM_INLINE_NETWORK_9
network-object object West_Pender_Network
network-object object local_lan
object-group network AWS_VPC
description Contains all AWS VPCs
network-object object AWS-VPC-CA
network-object object awzn_vpc
object-group network DM_INLINE_NETWORK_13
network-object object AWS-VPC-CA
network-object object awzn_vpc
object-group network DM_INLINE_NETWORK_18
network-object object AWS-VPC-CA
network-object object awzn_vpc
access-list acl-amzn extended permit ip any4 10.25.0.0 255.255.0.0
access-list acl-amzn extended permit ip 10.25.0.0 255.255.0.0 any
access-list acl-amzn extended permit ip host 172.172.172.172 host 123.123.123.123
access-list amzn-filter extended permit ip object awzn_vpc object-group Urthecast_Networks
access-list amzn-filter extended permit ip object-group Urthecast_Networks object awzn_vpc
access-list amzn-filter extended permit ip object awzn_vpc object AWS-VPC-CA inactive
access-list amzn-filter extended permit ip host 123.123.123.123 object awzn_vpc inactive
access-list amzn-filter extended permit ip 10.0.0.0 255.255.0.0 10.25.0.0 255.255.0.0
access-list amzn-filter extended permit ip 10.25.0.0 255.255.0.0 10.0.0.0 255.255.0.0
access-list amzn-filter extended permit ip object-group DM_INLINE_NETWORK_8 object AWS-VPC-CA
access-list amzn-filter extended permit ip object AWS-VPC-CA object-group DM_INLINE_NETWORK_9
access-list amzn-filter extended permit ip object awzn_vpc object West_Pender_Network
access-list amzn-filter extended permit ip object West_Pender_Network object awzn_vpc
access-list amzn-filter extended permit ip 10.0.0.0 255.255.0.0 object 10.22.0.0
access-list amzn-filter extended permit ip object 10.22.0.0 10.0.0.0 255.255.0.0
access-list amzn-filter extended deny ip any any
access-list ca-vpc-filter extended permit ip object AWS-VPC-CA object-group Urthecast_Networks
access-list ca-vpc-filter extended permit ip object-group Urthecast_Networks object AWS-VPC-CA
access-list ca-vpc-filter extended permit ip object awzn_vpc object AWS-VPC-CA
access-list ca-vpc-filter extended deny ip any any
access-list inside_access_in extended permit ip object-group Urthecast_Networks object-group DM_INLINE_NETWORK_13
access-list inside_access_in extended permit ip object-group DM_INLINE_NETWORK_18 any
access-list inside_access_in extended permit tcp object West_Pender_Network object-group AWS_VPC object-group rdp
access-list inside_access_in extended deny ip any any
access-list ca-acl-amzn extended permit ip any4 10.28.1.0 255.255.255.0
access-list ca-acl-amzn extended permit ip 10.28.1.0 255.255.255.0 any
access-list ca-acl-amzn extended permit ip host 173.173.173.173 host 123.123.123.123
icmp unreachable rate-limit 1 burst-size 1
icmp permit any outside
no asdm history enable
arp timeout 14400
no arp permit-nonconnected
nat (inside,outside) source static any any destination static awzn_vpc awzn_vpc no-proxy-arp route-lookup
nat (inside,outside) source static any any destination static AWS-VPC-CA AWS-VPC-CA no-proxy-arp route-lookup
!
object network obj_any
nat (any,outside) dynamic interface
access-group inside_access_in in interface inside
route outside 0.0.0.0 0.0.0.0 123.123.123.254 1
route inside 10.0.0.0 255.255.0.0 10.0.1.1 1
sla monitor 1
type echo protocol ipIcmpEcho 10.25.0.114 interface outside
frequency 5
sla monitor schedule 1 life forever start-time now
sla monitor 2
type echo protocol ipIcmpEcho 10.28.1.132 interface outside
frequency 5
sla monitor schedule 2 life forever start-time now
service sw-reset-button
crypto ipsec ikev1 transform-set transform-amzn esp-aes esp-sha-hmac
crypto ipsec ikev2 ipsec-proposal DES
protocol esp encryption des
protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal 3DES
protocol esp encryption 3des
protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES
protocol esp encryption aes
protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES192
protocol esp encryption aes-192
protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES256
protocol esp encryption aes-256
protocol esp integrity sha-1 md5
crypto ipsec security-association replay window-size 128
crypto ipsec security-association pmtu-aging infinite
crypto ipsec df-bit clear-df outside
crypto map amzn_vpn_map 1 match address acl-amzn
crypto map amzn_vpn_map 1 set pfs
crypto map amzn_vpn_map 1 set peer 172.172.172.172
crypto map amzn_vpn_map 1 set ikev1 transform-set transform-amzn
crypto map amzn_vpn_map 1 set security-association lifetime seconds 3600
crypto map amzn_vpn_map 1 set df-bit clear-df
crypto map amzn_vpn_map 2 match address ca-acl-amzn
crypto map amzn_vpn_map 2 set pfs
crypto map amzn_vpn_map 2 set peer 173.173.173.173
crypto map amzn_vpn_map 2 set ikev1 transform-set transform-amzn
crypto map amzn_vpn_map 2 set security-association lifetime seconds 3600
crypto map amzn_vpn_map interface outside
group-policy CA-VPC internal
group-policy CA-VPC attributes
vpn-filter none
vpn-tunnel-protocol ikev1
group-policy filter internal
group-policy filter attributes
vpn-filter none
vpn-tunnel-protocol ikev1
dynamic-access-policy-record DfltAccessPolicy
tunnel-group 172.172.172.172 type ipsec-l2l
tunnel-group 172.172.172.172 general-attributes
default-group-policy filter
tunnel-group 172.172.172.172 ipsec-attributes
ikev1 pre-shared-key *****
isakmp keepalive threshold 10 retry 3
tunnel-group 173.173.173.173 type ipsec-l2l
tunnel-group 173.173.173.173 general-attributes
default-group-policy CA-VPC
tunnel-group 173.173.173.173 ipsec-attributes
ikev1 pre-shared-key *****

[Rahul Govindan · Accepted solution]

You should create a NAT exemption rule between the 2 networks only for the outside interface. something similar to this:

nat (outside,outside) 1 source static <vpcsubnet1> <vpcsubnet1> destination static <vpcsubnet2> <vpcsubnet2> no-proxy-arp route-lookup

Currently you only have rules between inside and outside interfaces.

[bighiller]

Hi Rahul,

I applied the nat rule:

nat (outside,outside) 1 source static awzn_vpc awzn_vpc destination static AWS-VPC-CA AWS-VPC-CA no-proxy-arp route-lookup

It returned a warning:

WARNING: Pool (10.25.0.0-10.25.255.255) overlap with existing pool.
WARNING: Pool (10.28.0.0-10.28.255.255) overlap with existing pool.

Then if I initiate a connection from a host in awzn_vpc (10.25.0.0/16) to a host in AWS-VPC-CA (10.28.1.0/24), the connection from the corporate network (10.0.0.0/16) to the awzn_vpc subnet (10.25.0.0/16) dies and syslog throws this error:

IPSEC: Received an ESP packet (SPI= 0xDBF79EC1, sequence number= 0x1F) from 172.172.172.172 (user= 172.172.172.172) to 123.123.123.123.  The decapsulated inner packet doesn't match the negotiated policy in the SA.  The packet specifies its destination as <HEX>, its source as <HEX>, and its protocol as 255.  The SA specifies its local proxy as 10.25.0.0/255.255.0.0/ip/0 and its remote_proxy as 0.0.0.0/0.0.0.0/ip/0.

I need to clear the SA for that connection to make it come up again.

The issue is not the same from connections initiated from the 10.28.1.0/24 subnet as when I try to open from 10.28.1.0/24 to 10.25.0.0/16 there is nothing at all. No traffic seems to flow.. at least not that I can tell.

Thoughts?

I attached a screenshot of the NAT from the ADSM.

[bighiller]

Fixed.

Not sure exactly what it was, but I made sure all the protected subnets matched in our two AWS VPCs and on the ASA. Also cleaned up ACLs and disabled NAT-T on the tunnels.

Likely it was a subnet summarizing that was causing my error above.

[Rahul Govindan]

Nice. I was about to reply back that your proxies seem to be wrong. But looks like you have it all corrected. Glad to hear its working !!