Conflict between NAT and a VPN route in ASA 9.9

[cpholt]

We just replaced an ASA5510 with a new ASA5516x. Everything worked smoothly except for traffic between our network and traffic on the customer end of a site-to-site VPN tunnel.

Old system: ASA5510 version 9.1(7)15.
New system: ASA5516x version 9.9(1).

Traffic originating from our network to the customer network through the tunnel is fine.

Traffic originating from the customer network that should flow outside of the tunnel (it's destination is a public web server in our network), does not work.

After performing packet captures on both sides of the ASA before the update and after the update reveals the difference:

Before update (working case), this is the partial output of a traced capture.
This is the first packet entering the ASA from outside, leaving on the inside, the web servers response entering the ASA and then leaving back to the outside. It heads off to our ISP's router (209.xxx.142.25)

1: 09:26:13.592605 129.xxx.235.132.53828 > 209.xxx.142.28.443: S 3332805073:3332805073(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK> 
2: 09:26:13.592986 129.xxx.235.132.53828 > 192.168.2.28.443: S 3832318623:3832318623(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK> 
..
5: 09:26:13.593429 192.168.2.28.443 > 129.xxx.235.132.53828: S 3052302764:3052302764(0) ack 3832318624 win 14600 <mss 1460,nop,nop,sackOK,nop,wscale 7>


Phase: 4
Type: ROUTE-LOOKUP
Subtype: output and adjacency
Result: ALLOW
Config:
Additional Information:
found next-hop 209.xxx.142.25 using egress ifc outside
adjacency Active
next-hop mac address 68ef.bd4e.7eff hits 1012402

6: 09:26:13.593505 209.xxx.142.28.443 > 129.xxx.235.132.53828: S 4066137499:4066137499(0) ack 3332805074 win 14600 <mss 1300,nop,nop,sackOK,nop,wscale 7>

Here you can see in the phase 4 the next hop for the exiting packet is 209.xxx.142.25 (our ISP's gateway).

Here is the same capture after the update.

1: 14:52:12.701349 129.xxx.235.132.61129 > 209.xxx.142.28.443: S 775761873:775761873(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK> 
2: 14:52:12.701639 129.xxx.235.132.61129 > 192.168.2.28.443: S 776220941:776220941(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK> 
3: 14:52:12.701791 192.168.2.28.443 > 129.xxx.235.132.61129: S 533424725:533424725(0) ack 776220942 win 14600 <mss 1460,nop,nop,sackOK,nop,wscale 7> 

Phase: 4
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 129.xxx.235.132 using egress ifc outside

In this case, it's decided that the next hop is 129.xxx.235.132 (the final destination and not our ISP's router). This is the heart of the problem.

That lead us to check the output of "show route" on the ASA for routes related to this destination:

Before (working):

Gateway of last resort is 209.xxx.142.25 to network 0.0.0.0

S 129.xxx.235.132 255.255.255.255 [1/0] via 209.xxx.142.25, outside

After (failing):

Gateway of last resort is 209.xxx.142.25 to network 0.0.0.0

V 129.xx.235.132 255.255.255.255 
connected by VPN (advertised), outside

Clearly something has changed between 9.1 and 9.9 ASA version in regards to how routes for traffic for a VPN are handled. How can I adjust my config to accommodate this?

Sanitized Config:

ASA Version 9.1(7)15 
!
terminal width 180
hostname 5516xa
domain-name our.company.com
enable password ***** encrypted
xlate per-session deny tcp any4 any4
xlate per-session deny tcp any4 any6
xlate per-session deny tcp any6 any4
xlate per-session deny tcp any6 any6
xlate per-session deny udp any4 any4 eq domain
xlate per-session deny udp any4 any6 eq domain
xlate per-session deny udp any6 any4 eq domain
xlate per-session deny udp any6 any6 eq domain
passwd ***** encrypted
names
ip local pool vpnpool1 192.168.2.51-192.168.2.90 mask 255.255.255.0
!
interface Ethernet0/0
nameif inside
security-level 100
ip address 192.168.2.98 255.255.255.0 
!
interface Ethernet0/1
no nameif
no security-level
no ip address
!
interface Ethernet0/2
nameif outside
security-level 0
ip address 209.xxx.142.26 255.255.255.248 
!
interface Ethernet0/3
description LAN/STATE Failover Interface
!
interface Management0/0
management-only
nameif management
security-level 0
ip address 192.168.6.6 255.255.255.0 standby 192.168.6.7 
!
boot system disk0:/asa917-15-k8.bin
ftp mode passive
clock timezone CST -6
clock summer-time CDT recurring
dns server-group DefaultDNS
domain-name our.company.com
same-security-traffic permit inter-interface
same-security-traffic permit intra-interface
object network ourcompany-inside-isp
subnet 192.168.2.0 255.255.255.0
object service ssh
service tcp source eq ssh 
object network webschedule-server
host 192.168.2.28
description Web server.
object network webschedule-server-outside
host 209.xxx.142.28
description The public (outside) address of the web schedule server.
object service http
service tcp source eq www 
object service https
service tcp source eq https 
object network falcon
host 192.168.2.14
description Falcon 
object network nat-ourcompany-at-customer
host 192.168.5.2
description our addresses as they appear at customer thru tunnel.
object network obj_any
subnet 0.0.0.0 0.0.0.0
object network RA-DHCP-Pool
range 192.168.2.51 192.168.2.90
description RemoteAccess DHCP Pool
object network customerprd1
host 129.xxx.235.135
description Customer Server (prd1)
object network customerprd2
host 129.xxx.235.134
description Customer Server (prd2)
object network customertst
host 129.xxx.235.132
description Customer Server (tst)
object-group service traceroute udp
description traceroute udp ports
port-object range 33434 33534
object-group service allowed_outbound_services tcp
port-object eq www
port-object eq https
port-object eq ssh
port-object eq 465
port-object eq 587
port-object eq smtp
object-group network group-remote-customer
network-object object customerprd1
network-object object customerprd2
network-object object customertst
object-group network group-inhouse-customer
network-object object nat-ourcompany-at-customer
object-group network DHCP_VPN_Users
description DHCP_VPN_Users
network-object object RA-DHCP-Pool
network-object object webschedule-server
network-object object falcon

access-list group1_splitTunnelAcl standard permit 192.168.2.0 255.255.255.0 
access-list group1_splitTunnelAcl standard permit host 129.xxx.235.132 
access-list group1_splitTunnelAcl standard permit host 129.xxx.235.134 
access-list group1_splitTunnelAcl standard permit host 129.xxx.235.135 

access-list inside_access_in extended permit udp 192.168.2.0 255.255.255.0 any4 eq domain 
access-list inside_access_in extended permit icmp any4 any4 
access-list inside_access_in extended permit udp any4 any4 object-group traceroute 
access-list inside_access_in extended permit ip 192.168.2.0 255.255.255.0 any4 

access-list outside_210_cryptomap extended permit ip object nat-ourcompany-at-customer object-group group-remote-customer 

access-list outside_access_in extended permit icmp any4 any4 
access-list outside_access_in extended permit udp any4 any4 object-group traceroute 
access-list outside_access_in extended permit tcp any4 object webschedule-server-outside eq https inactive 
access-list outside_access_in extended permit tcp any4 object webschedule-server-outside eq www inactive 
access-list outside_access_in extended permit tcp any4 object webschedule-server eq www 
access-list outside_access_in extended permit tcp any4 object webschedule-server eq https 
access-list outside_access_in extended permit udp any4 192.168.2.0 255.255.255.0 eq ntp 

access-list RA-ACL extended permit ip any4 any4 

pager lines 50
mtu inside 1500
mtu outside 1500
mtu management 1500
failover
failover lan unit primary
failover lan interface failoverlink Ethernet0/3
failover replication http
failover link failoverlink Ethernet0/3
failover interface ip failoverlink 10.1.10.1 255.255.255.0 standby 10.1.10.2
no monitor-interface management
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-762.bin
asdm history enable
arp timeout 14400
no arp permit-nonconnected
nat (inside,any) source static any any destination static ourcompany-inside-isp ourcompany-inside-isp no-proxy-arp route-lookup
nat (inside,outside) source static webschedule-server webschedule-server-outside service http http
nat (inside,outside) source static webschedule-server interface service http http
nat (inside,outside) source static webschedule-server webschedule-server-outside service https https
nat (inside,outside) source static webschedule-server interface service https https
nat (inside,inside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer
nat (inside,outside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer
nat (outside,outside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer
!
object network ourcompany-inside-isp
nat (outside,outside) dynamic interface
object network obj_any
nat (inside,outside) dynamic interface
access-group inside_access_in in interface inside
access-group outside_access_in in interface outside
route outside 0.0.0.0 0.0.0.0 209.xxx.142.25 1 
timeout xlate 3:00:00
timeout pat-xlate 0:00:30
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
ldap attribute-map ldapmap2
map-name memberOf Group-Policy
map-value memberOf CN=FullVPN,CN=Users,DC=,DC=us,DC=com group2
dynamic-access-policy-record DfltAccessPolicy
aaa-server Radius protocol radius
aaa-server Radius (inside) host 192.168.2.9
key *********************
aaa-server Radius (inside) host 10.0.1.128
key *********************
user-identity default-domain LOCAL
aaa authentication ssh console Radius LOCAL
aaa authentication http console Radius LOCAL
aaa authentication serial console Radius LOCAL
http server enable 4443
http 192.168.4.0 255.255.255.0 management
http 192.168.2.0 255.255.255.0 inside
no snmp-server location
no snmp-server contact
no snmp-server enable
sysopt connection tcpmss 1300
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set strong esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set nah esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ts-memorial esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set transform-amzn esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES256-SHA1_TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES256-SHA1_TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES128-SHA1_TRANS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES128-SHA1_TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES256-SHA1 esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport
crypto ipsec security-association pmtu-aging infinite
crypto dynamic-map outside_dyn_map 20 set ikev1 transform-set ESP-AES256-SHA1_TRANS ESP-AES128-SHA1_TRANS ESP-AES256-SHA1
crypto dynamic-map outside_dyn_map 30 set pfs 
crypto map outside_map 210 match address outside_210_cryptomap
crypto map outside_map 210 set peer 129.xxx.230.17 
crypto map outside_map 210 set ikev1 transform-set strong
crypto map outside_map 210 set reverse-route
crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map
crypto map outside_map interface outside
crypto ca trustpool policy
crypto isakmp disconnect-notify
crypto ikev1 enable outside
crypto ikev1 ipsec-over-tcp port 10000 
crypto ikev1 policy 20
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
crypto ikev1 policy 40
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 28800
crypto ikev1 policy 60
authentication pre-share
encryption aes-256
hash sha
group 5
lifetime 28800
crypto ikev1 policy 70
authentication pre-share
encryption aes-256
hash sha
group 2
lifetime 28800
crypto ikev1 policy 100
authentication rsa-sig
encryption aes-256
hash sha
group 5
lifetime 86400
telnet timeout 5
ssh scopy enable
ssh stricthostkeycheck
ssh 192.168.2.0 255.255.255.0 inside
ssh 192.168.0.0 255.255.0.0 management
ssh timeout 5
ssh version 2
ssh cipher encryption custom "aes128-ctr"
ssh cipher integrity custom "hmac-sha1"
ssh key-exchange group dh-group1-sha1
console timeout 5
management-access inside
no vpn-addr-assign dhcp
vpn-addr-assign local reuse-delay 1
dhcprelay timeout 60
threat-detection basic-threat
threat-detection statistics host
threat-detection statistics port
threat-detection statistics protocol
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ntp server 216.239.35.4 source outside prefer
tftp-server inside 192.168.2.17 5516ax
ssl trust-point _wildcard inside
ssl trust-point _wildcard outside
webvpn
port 4443
enable outside
dtls port 4443
anyconnect image disk0:/anyconnect-win-4.3.02039-k9.pkg 1
anyconnect image disk0:/anyconnect-macosx-i386-4.3.02039-k9.pkg 2
anyconnect profiles Default disk0:/default.xml
anyconnect enable
port-forward test1 4001 10.23.10.109 4001 
tunnel-group-list enable
cache
disable
group-policy DfltGrpPolicy attributes
vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless
group-policy group2 internal
group-policy group2 attributes
wins-server none
dns-server value 192.168.2.17 192.168.2.14
dhcp-network-scope 192.168.2.98
vpn-idle-timeout none
vpn-session-timeout none
vpn-filter value RA-ACL
vpn-tunnel-protocol ikev1 ikev2 l2tp-ipsec ssl-client 
password-storage disable
split-tunnel-policy tunnelspecified
split-tunnel-network-list value group1_splitTunnelAcl
default-domain value our.company.com
intercept-dhcp 255.255.255.255 enable
webvpn
html-content-filter none
anyconnect keep-installer installed
anyconnect ssl rekey time 30
anyconnect ssl rekey method ssl
anyconnect dpd-interval client none
anyconnect dpd-interval gateway none
anyconnect profiles value Default type user
anyconnect ask none default anyconnect
tunnel-group DefaultL2LGroup ipsec-attributes
ikev1 pre-shared-key **********
tunnel-group DefaultRAGroup general-attributes
address-pool (inside) vpnpool1
address-pool vpnpool1
authentication-server-group Radius
default-group-policy group2
tunnel-group DefaultRAGroup ipsec-attributes
ikev1 pre-shared-key ***********
isakmp keepalive disable
tunnel-group DefaultRAGroup ppp-attributes
authentication ms-chap-v2
tunnel-group DefaultWEBVPNGroup general-attributes
authentication-server-group Radius
default-group-policy group2
dhcp-server 192.168.2.98
tunnel-group group2 type remote-access
tunnel-group group2 general-attributes
address-pool vpnpool1
authentication-server-group Radius LOCAL
default-group-policy group2
password-management
tunnel-group group2 webvpn-attributes
group-alias Group2 disable
group-alias group2 enable
tunnel-group group2 ipsec-attributes
ikev1 pre-shared-key ******
tunnel-group 129.xxx.230.17 type ipsec-l2l
tunnel-group 129.xxx.230.17 ipsec-attributes
ikev1 pre-shared-key ***********
isakmp keepalive disable
!
class-map inspection_default
match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 4096
policy-map global_policy
class inspection_default
inspect dns preset_dns_map 
inspect ftp 
inspect h323 h225 
inspect h323 ras 
inspect rsh 
inspect rtsp 
inspect esmtp 
inspect sqlnet 
inspect skinny 
inspect sunrpc 
inspect xdmcp 
inspect sip 
inspect netbios 
inspect tftp 
inspect ip-options 
class class-default
user-statistics accounting
!
service-policy global_policy global
smtp-server 192.168.2.17
prompt hostname context 
no call-home reporting anonymous
hpm topN enable
Cryptochecksum:7d5b78b68915ad22e68c090a72abb86a
: end

 

[cpholt · Accepted solution]

I found the solution after recreating this in the lab:

no crypto map outside_map 210 set reverse-route