05-07-2018 08:57 AM - edited 03-12-2019 05:16 AM
We just replaced an ASA5510 with a new ASA5516x. Everything worked smoothly except for traffic between our network and traffic on the customer end of a site-to-site VPN tunnel.
Old system: ASA5510 version 9.1(7)15.
New system: ASA5516x version 9.9(1).
Traffic originating from our network to the customer network through the tunnel is fine.
Traffic originating from the customer network that should flow outside of the tunnel (it's destination is a public web server in our network), does not work.
After performing packet captures on both sides of the ASA before the update and after the update reveals the difference:
Before update (working case), this is the partial output of a traced capture.
This is the first packet entering the ASA from outside, leaving on the inside, the web servers response entering the ASA and then leaving back to the outside. It heads off to our ISP's router (209.xxx.142.25)
1: 09:26:13.592605 129.xxx.235.132.53828 > 209.xxx.142.28.443: S 3332805073:3332805073(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK> 2: 09:26:13.592986 129.xxx.235.132.53828 > 192.168.2.28.443: S 3832318623:3832318623(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK> .. 5: 09:26:13.593429 192.168.2.28.443 > 129.xxx.235.132.53828: S 3052302764:3052302764(0) ack 3832318624 win 14600 <mss 1460,nop,nop,sackOK,nop,wscale 7> Phase: 4 Type: ROUTE-LOOKUP Subtype: output and adjacency Result: ALLOW Config: Additional Information: found next-hop 209.xxx.142.25 using egress ifc outside adjacency Active next-hop mac address 68ef.bd4e.7eff hits 1012402 6: 09:26:13.593505 209.xxx.142.28.443 > 129.xxx.235.132.53828: S 4066137499:4066137499(0) ack 3332805074 win 14600 <mss 1300,nop,nop,sackOK,nop,wscale 7>
Here you can see in the phase 4 the next hop for the exiting packet is 209.xxx.142.25 (our ISP's gateway).
Here is the same capture after the update.
1: 14:52:12.701349 129.xxx.235.132.61129 > 209.xxx.142.28.443: S 775761873:775761873(0) win 8192 <mss 1380,nop,wscale 2,nop,nop,sackOK> 2: 14:52:12.701639 129.xxx.235.132.61129 > 192.168.2.28.443: S 776220941:776220941(0) win 8192 <mss 1300,nop,wscale 2,nop,nop,sackOK> 3: 14:52:12.701791 192.168.2.28.443 > 129.xxx.235.132.61129: S 533424725:533424725(0) ack 776220942 win 14600 <mss 1460,nop,nop,sackOK,nop,wscale 7> Phase: 4 Type: ROUTE-LOOKUP Subtype: Resolve Egress Interface Result: ALLOW Config: Additional Information: found next-hop 129.xxx.235.132 using egress ifc outside
In this case, it's decided that the next hop is 129.xxx.235.132 (the final destination and not our ISP's router). This is the heart of the problem.
That lead us to check the output of "show route" on the ASA for routes related to this destination:
Before (working):
Gateway of last resort is 209.xxx.142.25 to network 0.0.0.0 S 129.xxx.235.132 255.255.255.255 [1/0] via 209.xxx.142.25, outside
After (failing):
Gateway of last resort is 209.xxx.142.25 to network 0.0.0.0 V 129.xx.235.132 255.255.255.255 connected by VPN (advertised), outside
Clearly something has changed between 9.1 and 9.9 ASA version in regards to how routes for traffic for a VPN are handled. How can I adjust my config to accommodate this?
Sanitized Config:
ASA Version 9.1(7)15 ! terminal width 180 hostname 5516xa domain-name our.company.com enable password ***** encrypted xlate per-session deny tcp any4 any4 xlate per-session deny tcp any4 any6 xlate per-session deny tcp any6 any4 xlate per-session deny tcp any6 any6 xlate per-session deny udp any4 any4 eq domain xlate per-session deny udp any4 any6 eq domain xlate per-session deny udp any6 any4 eq domain xlate per-session deny udp any6 any6 eq domain passwd ***** encrypted names ip local pool vpnpool1 192.168.2.51-192.168.2.90 mask 255.255.255.0 ! interface Ethernet0/0 nameif inside security-level 100 ip address 192.168.2.98 255.255.255.0 ! interface Ethernet0/1 no nameif no security-level no ip address ! interface Ethernet0/2 nameif outside security-level 0 ip address 209.xxx.142.26 255.255.255.248 ! interface Ethernet0/3 description LAN/STATE Failover Interface ! interface Management0/0 management-only nameif management security-level 0 ip address 192.168.6.6 255.255.255.0 standby 192.168.6.7 ! boot system disk0:/asa917-15-k8.bin ftp mode passive clock timezone CST -6 clock summer-time CDT recurring dns server-group DefaultDNS domain-name our.company.com same-security-traffic permit inter-interface same-security-traffic permit intra-interface object network ourcompany-inside-isp subnet 192.168.2.0 255.255.255.0 object service ssh service tcp source eq ssh object network webschedule-server host 192.168.2.28 description Web server. object network webschedule-server-outside host 209.xxx.142.28 description The public (outside) address of the web schedule server. object service http service tcp source eq www object service https service tcp source eq https object network falcon host 192.168.2.14 description Falcon object network nat-ourcompany-at-customer host 192.168.5.2 description our addresses as they appear at customer thru tunnel. object network obj_any subnet 0.0.0.0 0.0.0.0 object network RA-DHCP-Pool range 192.168.2.51 192.168.2.90 description RemoteAccess DHCP Pool object network customerprd1 host 129.xxx.235.135 description Customer Server (prd1) object network customerprd2 host 129.xxx.235.134 description Customer Server (prd2) object network customertst host 129.xxx.235.132 description Customer Server (tst) object-group service traceroute udp description traceroute udp ports port-object range 33434 33534 object-group service allowed_outbound_services tcp port-object eq www port-object eq https port-object eq ssh port-object eq 465 port-object eq 587 port-object eq smtp object-group network group-remote-customer network-object object customerprd1 network-object object customerprd2 network-object object customertst object-group network group-inhouse-customer network-object object nat-ourcompany-at-customer object-group network DHCP_VPN_Users description DHCP_VPN_Users network-object object RA-DHCP-Pool network-object object webschedule-server network-object object falcon access-list group1_splitTunnelAcl standard permit 192.168.2.0 255.255.255.0 access-list group1_splitTunnelAcl standard permit host 129.xxx.235.132 access-list group1_splitTunnelAcl standard permit host 129.xxx.235.134 access-list group1_splitTunnelAcl standard permit host 129.xxx.235.135 access-list inside_access_in extended permit udp 192.168.2.0 255.255.255.0 any4 eq domain access-list inside_access_in extended permit icmp any4 any4 access-list inside_access_in extended permit udp any4 any4 object-group traceroute access-list inside_access_in extended permit ip 192.168.2.0 255.255.255.0 any4 access-list outside_210_cryptomap extended permit ip object nat-ourcompany-at-customer object-group group-remote-customer access-list outside_access_in extended permit icmp any4 any4 access-list outside_access_in extended permit udp any4 any4 object-group traceroute access-list outside_access_in extended permit tcp any4 object webschedule-server-outside eq https inactive access-list outside_access_in extended permit tcp any4 object webschedule-server-outside eq www inactive access-list outside_access_in extended permit tcp any4 object webschedule-server eq www access-list outside_access_in extended permit tcp any4 object webschedule-server eq https access-list outside_access_in extended permit udp any4 192.168.2.0 255.255.255.0 eq ntp access-list RA-ACL extended permit ip any4 any4 pager lines 50 mtu inside 1500 mtu outside 1500 mtu management 1500 failover failover lan unit primary failover lan interface failoverlink Ethernet0/3 failover replication http failover link failoverlink Ethernet0/3 failover interface ip failoverlink 10.1.10.1 255.255.255.0 standby 10.1.10.2 no monitor-interface management icmp unreachable rate-limit 1 burst-size 1 asdm image disk0:/asdm-762.bin asdm history enable arp timeout 14400 no arp permit-nonconnected nat (inside,any) source static any any destination static ourcompany-inside-isp ourcompany-inside-isp no-proxy-arp route-lookup nat (inside,outside) source static webschedule-server webschedule-server-outside service http http nat (inside,outside) source static webschedule-server interface service http http nat (inside,outside) source static webschedule-server webschedule-server-outside service https https nat (inside,outside) source static webschedule-server interface service https https nat (inside,inside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer nat (inside,outside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer nat (outside,outside) source dynamic ourcompany-inside-isp nat-ourcompany-at-customer destination static group-remote-customer group-remote-customer ! object network ourcompany-inside-isp nat (outside,outside) dynamic interface object network obj_any nat (inside,outside) dynamic interface access-group inside_access_in in interface inside access-group outside_access_in in interface outside route outside 0.0.0.0 0.0.0.0 209.xxx.142.25 1 timeout xlate 3:00:00 timeout pat-xlate 0:00:30 timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02 timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00 timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00 timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute timeout tcp-proxy-reassembly 0:01:00 timeout floating-conn 0:00:00 ldap attribute-map ldapmap2 map-name memberOf Group-Policy map-value memberOf CN=FullVPN,CN=Users,DC=,DC=us,DC=com group2 dynamic-access-policy-record DfltAccessPolicy aaa-server Radius protocol radius aaa-server Radius (inside) host 192.168.2.9 key ********************* aaa-server Radius (inside) host 10.0.1.128 key ********************* user-identity default-domain LOCAL aaa authentication ssh console Radius LOCAL aaa authentication http console Radius LOCAL aaa authentication serial console Radius LOCAL http server enable 4443 http 192.168.4.0 255.255.255.0 management http 192.168.2.0 255.255.255.0 inside no snmp-server location no snmp-server contact no snmp-server enable sysopt connection tcpmss 1300 crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac crypto ipsec ikev1 transform-set strong esp-3des esp-md5-hmac crypto ipsec ikev1 transform-set nah esp-aes-256 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac crypto ipsec ikev1 transform-set ts-memorial esp-aes-256 esp-sha-hmac crypto ipsec ikev1 transform-set transform-amzn esp-aes esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES256-SHA1_TRANS esp-aes-256 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES256-SHA1_TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES128-SHA1_TRANS esp-aes esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES128-SHA1_TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES256-SHA1 esp-aes-256 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport crypto ipsec security-association pmtu-aging infinite crypto dynamic-map outside_dyn_map 20 set ikev1 transform-set ESP-AES256-SHA1_TRANS ESP-AES128-SHA1_TRANS ESP-AES256-SHA1 crypto dynamic-map outside_dyn_map 30 set pfs crypto map outside_map 210 match address outside_210_cryptomap crypto map outside_map 210 set peer 129.xxx.230.17 crypto map outside_map 210 set ikev1 transform-set strong crypto map outside_map 210 set reverse-route crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map crypto map outside_map interface outside crypto ca trustpool policy crypto isakmp disconnect-notify crypto ikev1 enable outside crypto ikev1 ipsec-over-tcp port 10000 crypto ikev1 policy 20 authentication pre-share encryption 3des hash md5 group 2 lifetime 86400 crypto ikev1 policy 40 authentication pre-share encryption 3des hash sha group 2 lifetime 28800 crypto ikev1 policy 60 authentication pre-share encryption aes-256 hash sha group 5 lifetime 28800 crypto ikev1 policy 70 authentication pre-share encryption aes-256 hash sha group 2 lifetime 28800 crypto ikev1 policy 100 authentication rsa-sig encryption aes-256 hash sha group 5 lifetime 86400 telnet timeout 5 ssh scopy enable ssh stricthostkeycheck ssh 192.168.2.0 255.255.255.0 inside ssh 192.168.0.0 255.255.0.0 management ssh timeout 5 ssh version 2 ssh cipher encryption custom "aes128-ctr" ssh cipher integrity custom "hmac-sha1" ssh key-exchange group dh-group1-sha1 console timeout 5 management-access inside no vpn-addr-assign dhcp vpn-addr-assign local reuse-delay 1 dhcprelay timeout 60 threat-detection basic-threat threat-detection statistics host threat-detection statistics port threat-detection statistics protocol threat-detection statistics access-list no threat-detection statistics tcp-intercept ntp server 216.239.35.4 source outside prefer tftp-server inside 192.168.2.17 5516ax ssl trust-point _wildcard inside ssl trust-point _wildcard outside webvpn port 4443 enable outside dtls port 4443 anyconnect image disk0:/anyconnect-win-4.3.02039-k9.pkg 1 anyconnect image disk0:/anyconnect-macosx-i386-4.3.02039-k9.pkg 2 anyconnect profiles Default disk0:/default.xml anyconnect enable port-forward test1 4001 10.23.10.109 4001 tunnel-group-list enable cache disable group-policy DfltGrpPolicy attributes vpn-tunnel-protocol ikev1 l2tp-ipsec ssl-clientless group-policy group2 internal group-policy group2 attributes wins-server none dns-server value 192.168.2.17 192.168.2.14 dhcp-network-scope 192.168.2.98 vpn-idle-timeout none vpn-session-timeout none vpn-filter value RA-ACL vpn-tunnel-protocol ikev1 ikev2 l2tp-ipsec ssl-client password-storage disable split-tunnel-policy tunnelspecified split-tunnel-network-list value group1_splitTunnelAcl default-domain value our.company.com intercept-dhcp 255.255.255.255 enable webvpn html-content-filter none anyconnect keep-installer installed anyconnect ssl rekey time 30 anyconnect ssl rekey method ssl anyconnect dpd-interval client none anyconnect dpd-interval gateway none anyconnect profiles value Default type user anyconnect ask none default anyconnect tunnel-group DefaultL2LGroup ipsec-attributes ikev1 pre-shared-key ********** tunnel-group DefaultRAGroup general-attributes address-pool (inside) vpnpool1 address-pool vpnpool1 authentication-server-group Radius default-group-policy group2 tunnel-group DefaultRAGroup ipsec-attributes ikev1 pre-shared-key *********** isakmp keepalive disable tunnel-group DefaultRAGroup ppp-attributes authentication ms-chap-v2 tunnel-group DefaultWEBVPNGroup general-attributes authentication-server-group Radius default-group-policy group2 dhcp-server 192.168.2.98 tunnel-group group2 type remote-access tunnel-group group2 general-attributes address-pool vpnpool1 authentication-server-group Radius LOCAL default-group-policy group2 password-management tunnel-group group2 webvpn-attributes group-alias Group2 disable group-alias group2 enable tunnel-group group2 ipsec-attributes ikev1 pre-shared-key ****** tunnel-group 129.xxx.230.17 type ipsec-l2l tunnel-group 129.xxx.230.17 ipsec-attributes ikev1 pre-shared-key *********** isakmp keepalive disable ! class-map inspection_default match default-inspection-traffic ! ! policy-map type inspect dns preset_dns_map parameters message-length maximum 4096 policy-map global_policy class inspection_default inspect dns preset_dns_map inspect ftp inspect h323 h225 inspect h323 ras inspect rsh inspect rtsp inspect esmtp inspect sqlnet inspect skinny inspect sunrpc inspect xdmcp inspect sip inspect netbios inspect tftp inspect ip-options class class-default user-statistics accounting ! service-policy global_policy global smtp-server 192.168.2.17 prompt hostname context no call-home reporting anonymous hpm topN enable Cryptochecksum:7d5b78b68915ad22e68c090a72abb86a : end
Solved! Go to Solution.
05-09-2018 06:47 AM
I found the solution after recreating this in the lab:
no crypto map outside_map 210 set reverse-route
05-09-2018 06:47 AM
I found the solution after recreating this in the lab:
no crypto map outside_map 210 set reverse-route
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide