Unable to route specific traffic from HQ ASA to remote site ASA over IPSEC tunnel

[UZaman3]

I have set up a network with 2 ASA 5516-x's. We can call them HQ and Remote. I have configured an IPSec tunnel between both ASAs.  From both ASAs I can ping all nodes and devices over the IPSEC tunnel I created between them. At the Remote site there is an extranet connection that uses a Cisco Router (we can call this the CMS router). The CMS router has an IP address of 10.10.39.242. I am trying to get traffic from HQ (10.10.21.0/24 and 10.10.23.0/24), destined for 158.73.55.0 255.255.255.0 to go to the CMS router. From the remote LAN, I am able to get to the 158.73.55.0/24 network but from HQ I am unable to. I have attached a network diagram and can add configs as needed. Any help would be greatly appreciated.

[Richard Burts]

My first question is whether the ipsec tunnel is a traditional ipsec tunnel using crypto map etc rather than some other type of implementation.

My second question is whether network 158.73.55.0 is included in the access list used by the crypto map to identify interesting traffic to go over the site to site vpn.

My third question is whether there is any routing statement at HQ for network 158.73.55.0? If so what is the routing statement?

[UZaman3]

Thank you for getting back to me. It's a traditional ipsec tunnel using crypto maps. Here are the configs. I don't know if I am building the ACLs incorrectly or if I'm missing NAT somewhere. 

HQ:

ip local pool ANYCONNECT-POOL 10.10.23.5-10.10.23.250 mask 255.255.255.0

!
interface GigabitEthernet1/1
description EQUINIX UPLINK
nameif outside
security-level 0
ip address x.x.x.x x.x.x.x (WAN Subnet)
!
interface GigabitEthernet1/2
description LINK TO SG350
nameif inside
security-level 100
ip address 10.10.10.1 255.255.255.248

object-group network Local-Culpepper-Lan
description All Culpepper LAN IP Space
network-object 10.10.0.0 255.255.224.0

object-group network Remote-Columbia-Lan
description All Columbia LAN IP Space
network-object 10.10.32.0 255.255.248.0

object network CMS
subnet 158.73.55.0 255.255.255.0

object network OBJ-ANNYCONNECT-SUBNET
subnet 10.10.23.0 255.255.255.0

object-group service allow_internet_tcp tcp
description allow tcp ports for internet access
port-object eq www
port-object eq https
port-object eq domain

access-list VPN-Culpepper-Columbia extended permit ip object-group Local-Culpepper-Lan object-group Remote-Columbia-Lan
access-list VPN-Culpepper-Columbia extended permit tcp object-group Local-Culpepper-Lan gt 1023 object CMS object-group allow_internet_tcp

nat (inside,outside) source static Local-Culpepper-Lan Local-Culpepper-Lan destination static Remote-Columbia-Lan Remote-Columbia-Lan no-proxy-arp route-lookup
nat (any,outside) source static OBJ-ANNYCONNECT-SUBNET OBJ-ANNYCONNECT-SUBNET destination static Remote-Columbia-Lan Remote-Columbia-Lan no-proxy-arp route-lookup
nat (inside,outside) source static any any destination static OBJ-ANNYCONNECT-SUBNET OBJ-ANNYCONNECT-SUBNET no-proxy-arp route-lookup
!
object network obj_any
nat (any,outside) dynamic interface

route outside 0.0.0.0 0.0.0.0 x.x.x.x (ISP IP)
route inside 10.10.20.0 255.255.255.0 10.10.10.2 1
route inside 10.10.21.0 255.255.255.0 10.10.10.2 1
route inside 10.10.23.0 255.255.255.0 10.10.10.2 1
route inside 158.73.55.0 255.255.255.0 10.10.39.249 (Inside interface of Remote ASA)

crypto ipsec ikev1 transform-set ESP-AES-SHA esp-aes-256 esp-sha-hmac
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 10 match address VPN-Culpepper-Columbia
crypto map outside_map 10 set pfs group5
crypto map outside_map 10 set peer x.x.x.x (Outside interface IP Remote ASA)
crypto map outside_map 10 set ikev1 transform-set ESP-AES-SHA
crypto map outside_map 10 set security-association lifetime seconds 7200
crypto map outside_map interface outside

tunnel-group x.x.x.x type ipsec-l2l
tunnel-group x.x.x.x ipsec-attributes
ikev1 pre-shared-key *****

Remote ASA:

interface GigabitEthernet1/1
description WAN UPLINK
nameif outside
security-level 0
ip address x.x.x.x x.x.x.x (WAN Subnet)
!
interface GigabitEthernet1/2
description LINK TO LAN
nameif inside
security-level 100
ip address 10.10.39.249 255.255.255.248

object-group network Local-Columbia-Lan
description All Columbia LAN IP Space
network-object 10.10.32.0 255.255.248.0

object-group network Remote-Culpepper-Lan
description All Culpepper LAN IP Space
network-object 10.10.0.0 255.255.224.0

object network CMS
subnet 158.73.55.0 255.255.255.0

access-list VPN-Columbia-Culpepper extended permit ip object-group Local-Columbia-Lan object-group Remote-Culpepper-Lan

nat (inside,outside) source static Local-Columbia-Lan Local-Columbia-Lan destination static Remote-Culpepper-Lan Remote-Culpepper-Lan no-proxy-arp route-lookup
!
object network obj_any
nat (any,outside) dynamic interface

route inside 10.10.31.0 255.255.255.0 10.10.39.250 1
route inside 10.10.32.0 255.255.255.0 10.10.39.250 1
route inside 10.10.33.0 255.255.255.0 10.10.39.250 1
route inside 10.10.34.0 255.255.255.0 10.10.39.250 1
route inside 10.10.39.0 255.255.255.224 10.10.39.250 1
route inside 10.10.39.240 255.255.255.248 10.10.39.250 1
route inside 158.73.55.0 255.255.55.0 10.10.39.242 1
route inside 208.0.0.0 255.0.0.0 10.10.39.242 1

crypto ipsec ikev1 transform-set ESP-AES-SHA esp-aes-256 esp-sha-hmac
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 10 match address VPN-Columbia-Culpepper
crypto map outside_map 10 set pfs group5
crypto map outside_map 10 set peer x.x.x.x (Outside interface HQ ASA)
crypto map outside_map 10 set ikev1 transform-set ESP-AES-SHA
crypto map outside_map 10 set security-association lifetime seconds 7200
crypto map outside_map interface outside
crypto ca trustpool policy
crypto ikev1 enable outside
crypto ikev1 policy 10
authentication pre-share
encryption aes-256
hash sha
group 5
lifetime 28800

tunnel-group x.x.x.x type ipsec-l2l
tunnel-group x.x.x.x ipsec-attributes
ikev1 pre-shared-key *****

[Richard Burts]

Thanks for the additional information. There is much here to try to understand. The first thing that I notice is this:

route inside 158.73.55.0 255.255.255.0 10.10.39.249 (Inside interface of Remote ASA)

If the next hop is the inside interface of the remote ASA then why does the route point toward the inside interface. I would expect that it would point to the outside interface (similar to what the default route does).

[UZaman3]

That makes sense now that I think about it since the tunnel is built on the outside interface. Question I have now I guess is, does this traffic traverse the tunnel? And do I need to NAT it on the remote ASA? I changed the route on the HQ ASA to route outside 158.73.55.0 255.255.255.0 x.x.x.x (outside interface of remote ASA) but still no go.

[Richard Burts]

Is the x.x.x.x in the static route the same as the address specified in the remote peer statement for the vpn?

Since the source address for packets sent over the vpn is you private address network, I would believe that the remote ASA should translate the source address as it forwards the packets using CMS.

[UZaman3]

Yes, the route is pointing to the peer IP for the VPN. I ran a capture on the ASA and when I try to go to the site, i see traffic on the inside interface but nothing on the outside interface. I have an ACL that allows outbound https traffic and I know it works because all regular web traffic is working fine. If you look at Phase 9 of the packet-tracer (see attached document) its showing Type: NAT and Result: Drop. Could it be a NAT issue?

[Richard Burts]

This is good testing. I believe that you are quite right that this is an issue with nat. In looking at the config that you posted we find this for nat relative to what is going through the vpn.

nat (inside,outside) source static Local-Columbia-Lan Local-Columbia-Lan destination static Remote-Culpepper-Lan Remote-Culpepper-Lan no-proxy-arp route-lookup

This static translation is sometimes referred to as nat exemption because what it does is to not translate any traffic that is going through the vpn. It appears to work for LAN to LAN traffic, but not for LAN to CMS traffic. So the LAN to CMS traffic is being translated. I think you have choices about how to fix this:

- you could expand Remote-Culpepper-Lan to include the CMS subnet

- you could add another static nat for your LAN to CMS traffic to not be translated if going through the vpn.

[UZaman3]

Take a look at this packet capture output I grabbed after making some changes to natting. In Phase 10 the packet is being dropped. One question I do have is this, I have the webvpn (anyconnect profile) set up for split tunneling. How can I make sure the traffic (since its web traffic) is actually getting to the ASA. That could also be my problem

access-list outside-in extended permit tcp object OBJ-ANNYCONNECT-SUBNET gt 1023 object CMS object-group allow_internet_tcp
access-list outside-in extended permit ip object-group Local-Culpepper-Lan object-group Remote-Columbia-Lan
access-list outside-in extended permit tcp any object IIS_Prod_Server_80 eq www
access-list outside-in extended permit tcp any object IIS-Prod-Server eq https
access-list outside-in extended permit tcp any object SFTP-Server eq 11022
access-list inside-in extended permit tcp object OBJ-ANNYCONNECT-SUBNET gt 1023 object CMS object-group allow_internet_tcp
access-list inside-in extended permit tcp object-group Local-Culpepper-Lan any object-group allow_internet_tcp
access-list inside-in extended permit udp object-group Local-Culpepper-Lan any object-group allow_internet_udp
access-list inside-in extended permit tcp any gt 1023 object IIS-Prod-Server eq https
access-list inside-in extended permit tcp any object IIS_Prod_Server_80 eq www
access-list inside-in extended permit ip object-group Local-Culpepper-Lan object-group Remote-Columbia-Lan
access-list inside-in extended permit tcp any gt 1023 object SFTP-Server eq 11022
access-list inside-in extended permit tcp object-group Local-Culpepper-Lan object-group Office365_Exchange object-group Office365_Exchange_tcp
access-list global-access extended permit ip object-group CapitalBridge-ALL object-group CapitalBridge-ALL
access-list VPN-Culpepper-Columbia extended permit ip object-group Local-Culpepper-Lan object-group Remote-Columbia-Lan
access-list VPN-Culpepper-Columbia extended permit tcp object OBJ-ANNYCONNECT-SUBNET gt 1023 object CMS object-group allow_internet_tcp
access-list SPLIT-TUNNEL standard permit 10.10.0.0 255.255.0.0

nat (any,outside) source static OBJ-ANNYCONNECT-SUBNET OBJ-ANNYCONNECT-SUBNET destination static CMS CMS no-proxy-arp route-lookup

[UZaman3 · Accepted solution]

So figured it out, there were three things needed that I was doing incorrectly to make this work. 

One: Access-list to allow CMS traffic over split-tunnel (anyconnect)

Two: NAT exempt statement to ensure traffic doesn't get NAT'ted over IPSEC tunnel between sites

Three: Mirroring ACLs for Culpepper LAN traffic and CMS traffic on each endpoint firewall - I had this but I was trying to be as specific as possible by only using https ports for the ACL rule, once I changed it to IP instead of TCP as the protocol the traffic was allowed

Thank you for your help with troubleshooting this. It is greatly appreciated!

[Richard Burts]

Thanks for the update. Glad to know that you have got it working. Appreciate the details of what you had to fix to get it to work.