Re: FTD - Dual ISP and S2S VPN connections

AigarsK · ‎12-10-2023

Hi All,

I am working on concept of Dual ISP for some of the sites we have where they have FTDs as firewalls and server as Internet Edge.

At HQ I have placed 2x FTD in HA, each firewall has link to ISP01 and ISP02. I have created two static default routes and used IP SLA to track them, if ISP01 was to go down, SLA withdraws ISP01 route and ISP02 is used. This all great for purposes of Internet access from Inside network at HQ.

I am however having issues with S2S VPN to remote sites.

So if I use Policy Based S2S VPN, I would create one with ISP01 as its Local VPN Access Interface and all is great, VPN works, however if I was to fail over to ISP02 VPN goes down, and it is expected as source interface no longer has connectivity.

Problem here is that I cannot create second Policy Based S2S using ISP02 as Local VPN Access Interface, as the destination IP address for Remote IP Address is the same.

I had same issue when creating it with Route Based VTI interfaces.

At remote site is all fine, I can create two different VTI interfaces (each has its own subnet and will be matched against HQ site VTI interfaces) which are sourced from same Outside interface. I then proceed with creating two VTI S2S VPNs where specifying different VTI interfaces as source and Remote IP Address one for ISP01 and one for ISP02.For testing purposes I created two static routes for same HQ LAN to be located against different VTI interface at HQ and change metric on second one which can be later tracked using SLA.

However at HQ side this does not work as my remote site only has one Remote IP Address which would be the same and I get error: "Duplicate remote-peer IP-address is found, remote-peer IP address must be unique across all S2S VPN connection profiles"

So this effectively prevents me creating two VPN tunnels where one site has two ISP and remote site only one.

Worth to mention that I am running 7.0.5 at HQ, it does appear that later version have introduced option to select multiple "Local VPN Access Interfaces" when using Policy Based VPN.

So question is, what are my options? One of them appears to be upgrading and using Policy Based VPN, but are there other design options which would enable me to have ISP redundancy and still have means for VPN.

All this config is done on FDM without use of FMC.

Topology attached:

FTD01 and FTD02 are at the HQ site.

MHM Cisco World · ‎12-10-2023

MHM

AigarsK · ‎12-10-2023

Thanks,

I have this already working, issue is not with the Dual ISP as much it is running S2S VPN over Dual ISP, more specifically having VPN S2S work on Secondary ISP line when primary fails.

MHM Cisco World · ‎12-10-2023

So this effectively prevents me creating two VPN tunnels where one site has two ISP and remote site only one.

Worth to mention that I am running 7.0.5 at HQ, it does appear that later version have introduced option to select multiple "Local VPN Access Interfaces" when using Policy Based VPN

You already mention solution' why you not use two local interface and remote IP can use same?

MHM

tvotna · ‎12-10-2023

This is a FDM bug "CSCvt06355 FDM does not allow the configuration of an additional tunnel with the same Remote IP Addres" which is fixed in 7.1.

However, things are little bit more complicated. With SVTI the real problem is another bug: "CSCvo13642 ASA ENH: VTI: Add "tunnel route-via" CLI & feature support to mirror IOS", which, to the best of my knowledge, has not been fixed:

Symptom: ASA VTI feature routes ESP traffic out primary ISP interface even though "tunnel source interface backup-ISP" is configured on the tunnel interface. Conditions: This is a day-1 behavior borrowed from IOS-based routers. IOS routers have "tunnel route-via" CLI to change this behavior, but this option doesn't exist on ASA. Workaround: Configure correct static routes to tunnel destinations via primary and backup ISP interfaces. ESP traffic will be routed according to those routes. This requires two different outside interfaces on the tunnel peer (or requires different tunnel peers). Triangle topology (single tunnel destination for two tunnel interfaces) isn't currently supported.

This means you cannot have two tunnels active at the same time in triangle topology like yours, if there is a single IP address on the remote branch firewall. You can establish two IKEv2 UDP/500 connections if you make HQ firewall passive (responder-only):

crypto ipsec profile PROF
 set ikev2 ipsec-proposal PROP
 set pfs group24
 responder-only

In this case it will accept IKEv2 requests from the branch firewall, create UDP/500 connections and correctly respond to ISP01 interface for tunnel1 and to ISP02 interface for tunnel2. But there will be a problem with ESP traffic, because firewall actually creates two unidirectional ESP connections for each tunnel. Notice incorrect egress interface below. 3.3.3.1 is a branch firewall.

UDP isp1: 3.3.3.1/500 (3.3.3.1/500) NP Identity Ifc: 1.1.1.1/500 (1.1.1.1/500), flags - , idle 1m53s, uptime 2m3s, timeout 2m0s, bytes 1852
UDP isp2: 3.3.3.1/500 (3.3.3.1/500) NP Identity Ifc: 2.2.2.1/500 (2.2.2.1/500), flags - , idle 1m43s, uptime 2m3s, timeout 2m0s, bytes 1438
ESP isp1: 3.3.3.1/33344 (3.3.3.1/33344) NP Identity Ifc: 2.2.2.1/12200 (2.2.2.1/12200), , flags , idle 1s, uptime 1m41s, timeout 30s, bytes 7472
ESP isp1: 3.3.3.1/12777 (3.3.3.1/12777) NP Identity Ifc: 1.1.1.1/38669 (1.1.1.1/38669), , flags , idle 0s, uptime 1m53s, timeout 30s, bytes 8432
ESP isp1: 3.3.3.1/61760 (3.3.3.1/61760) NP Identity Ifc: 1.1.1.1/49939 (1.1.1.1/49939), , flags , idle 0s, uptime 1m53s, timeout 30s, bytes 8512
ESP isp2: 3.3.3.1/46260 (3.3.3.1/46260) NP Identity Ifc: 2.2.2.1/21574 (2.2.2.1/21574), , flags , idle 1s, uptime 1m41s, timeout 30s, bytes 7568
TCP tu1: 192.168.11.2/8604 (192.168.11.2/8604) NP Identity Ifc: 192.168.11.1/179 (192.168.11.1/179), flags UOB , idle 0s, uptime 1m53s, timeout 1h0m, bytes 872
TCP tu2: 192.168.12.2/53522 (192.168.12.2/53522) NP Identity Ifc: 192.168.12.1/179 (192.168.12.1/179), flags UOB , idle 1s, uptime 1m41s, timeout 1h0m, bytes 758

If there is a NAT after the firewall, NAT-T is negotiated and everything including ESP moves to UDP/4500, then this topology might work, although I didn't test this scenario.

So, create two outside interfaces on your branch firewall as a workaround, if possible. If not possible, use policy-based VPN, but it doesn't scale well, doesn't allow you to run dynamic routing over tunnels (like BGP or EIGRP as of 7.3 in case of SVTI) and is actually also complicated when it comes to connections going over VPN. Note that these connections will be torn down upon ISP switchover. SVTI might be advantageous here, as 7.1 should support traffic zones on tunnel interfaces and connections might be able to transparently migrate from one tunnel to another, although I didn't test. Other community members can comment.

AigarsK · ‎12-18-2023

Hi All,

Just wanted to updated you on the labing.

I think I managed to get it working, that is HQ with dual ISP and Remote site with 1 ISP.

I ended up migrating my lab devices all to be managed from FMC, this looks like a requirement to get over issue where I will have two VTI VPNs to the same one Public IP.

This is what I ended up doing at HQ:
- Created two interfaces, 1 per each ISP and gave them distinct Security Zones
- Added Static Default Routes for both ISPs and set ISP01 with lower AD to be selected as proffered
- Added SLA Monitor to ISP01 Static Default Route which monitors ISP01 gateway sourced from ISP01 interface (this can be changed to something upstream later on to detect ISP core routing issues)
- Created two VTI interfaces, each to originate from its own ISP interface (I did not specify Security Zone for them as doing so interfered with SLA Tracking config)
- Created two VTI VPNs, each to use its own VTI interface (which are sourced from its own ISP interface) destined to Remote Site with 1 ISP but two VTI interfaces sourced from same ISP interface.

Created NAT entries that would do PAT of either of ISP interfaces and created ACP policy to allow traffic between the local and remote subnet and did not specify Security Zone.

At Remote Site FTD:
- Created Static Default Route to its only ISP interface.
- Created two VTI Interfaces sourced from same ISP interface but on different subnets for each VTI

I also went about adding similar NAT and ACP rules to allow VPN connection.

There of course was need for routing information to be added, for lab I chose to just use static routing for now, I created at each ste Static entries for each sites LAn side to be accessible from VTI interface with next hop being set remote site's corresponding VTI interface IP.

Failover operates as per following:

ISP01 and ISP02 both working at HQ - two VTI VPN tunnels are established, clients use ISP01 for internet access as per Static Default Route, and uses VTI over ISP01 for VPN traffic, also due to static route.

ISP01 failed and ISP02 operational at HQ - SLA tracking withdraws Static Default Route using ISP01, this causes VTI VPN tunnel over ISP01 go down, this withdraws Static Route for remote LAN over this VTI interface. ISP02 Static Default Route is installed in routing table and used for Internet access. As primary VTI tunnel route is dead, one over ISP02 is used for VPN connection.

This technically accomplishes my operational goals, I will however need to carry on developing this as I have AnyConnect Client VPN requirements and some NATed services to account for.

Thanks all for your input on this

MHM Cisco World · ‎12-18-2023

sorry but why you not run one VTI with two Local interface
can I know why you need two VTI to same destination
thanks a lot
MHM

AigarsK · ‎12-18-2023

I did this purely to avoid the NAT issue tvotna explained.

Having Two TVI source from different at ISP interface will ensure that it would go down along with ISP outage, this allows for route to be withdrawn from routing table. I guess this would not be required if I was to use BGP, but static is enough for me in lab, plus in real live I only have 7 sites to cover, so having them static will ensure no added level of tshoot when routes were to disappear.

At remote site I did two VTI's as well, but source from same ISP interface as it does not have ISP redundancy, but for purposes to be able to terminate two distinct VPN tunnels and similarly have two static routes for remote LAN using different VTI for each one.

MHM Cisco World · ‎12-18-2023

there is no NAT in VTI
the NAT is need to override in case you use policy based VPN not route based VPN
also you need only ONE static route using the tunnel interface.

MHM

AigarsK · ‎12-18-2023

Indeed, I think I might have focused on two VTI solution as this is what I did when I was not using FMC.

I will give it a go and see if it would as to what you suggested.

Here is what I have topology wise and config wise for VPN.

tvotna · ‎12-18-2023

It seems there is still some misunderstanding here.

First, @MHM Cisco World , I believe you can have single VTI with two ISP interfaces only if one of the following is true: 1) the tunnel source of your VTI is a loopback interface and its IP address belongs to ISP-independent address space (do you have your own AS number?), or 2) ISP1 does not block traffic sourced from ISP2 address space and return traffic to ISP2 address can magically traverse ISP1 (never the case). Would you agree?

Regarding the test and the solution, I still don't believe (although this needs to be tested) that in your topology two tunnels can be fully operational at the same time. @AigarsK , to be on the safe side, you need to test dynamic routing over tunnels and also you need to block traffic sourced from the ISP1 address space inside the ISP2 cloud and vice versa, similar to what happens in real life. Without dynamic routing (e.g. BGP) running over tunnels you can only reliably detect local interface failure or next-hop failure (with IP SLA). But the failure can happen anywhere in the path, which can only be detected by dynamic routing. This in turn means that it is desirable to have two routing adjacencies active (e.g. two BGP sessions) simultaneously to minimize switchover time. And this means that two tunnels should be operational at the same time.

AigarsK · ‎12-18-2023

My Sincere apologies for giving false statement on both tunnels working.

I did not realize that my lab was actually on ISP side routing traffic for ISP02 asymmetrically which allowed for both tunnels to be up. Once I implemented ISP router ACL which discards IP which are not its own link to CE, I exhibited same issue you described.

Also, you are correct, I cannot create VTI without assigning no source interface.

I will test more tomorrow and make sure I do not have unexpected traffic patterns which can invalidate results.