ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

[emasters]

For some context, we have dual Firepower 2140s in HA managed by FMCv. Both FTD and FMCv are on 7.2.5.1 as of last week at the start of 2024. 

We currently utilize our FTDs as external firewalls and head-ends for both site-to-site VPNs and Remote Access VPNs. Our RA VPN users are currently utilizing IPsec. We have dual 1Gbps internet connections on a load-balancer that sits in front of the FTDs while the FTDs themselves have just a single outside connection that feeds into the LAN of the load-balancer. We also have a new 10Gbps connection as of the last month that we'd like to start utilizing. The cost of using 10Gbps on our load-balancer is going to be costly as it's about $70k worth in licensing. This started the research of seeing if the FTDs were capable of load-balancihng outbound traffic and that's where I came across ECMP and Policy Based Routing. As of 7.2, the FMC natively supports the configuration of both without the need of FlexConfig. Neat. 

Naturally, I started reading documentation on the requirements. For the most part, I met all the requirements execpt for two as noted by the documentation here.

• Following interfaces cannot be associated with an ECMP zone -> Interfaces in RA VPN configuration with SSL enabled
• Threat defense does not support ECMP with NAT in IPsec sessions—a standard IPsec virtual private network (VPN) tunnel does not work with NAT points in the delivery path of IPsec packets.

My first question/concern with the first point is are they referring to having SSL on the interface itself, or having SSL enabled in the RA VPN Group Policy? In other words, SSL enabled within the Access Interface configuration

Or.....the Group Policy

If it's referring to having SSL enabled on the actual interface itself and need to turn that off, then I suppose I'll need to come up with a gameplan regarding our VPN users as, If I recall, all RA VPN sessions utilize SSL initially. 

The second point I may just need clarification on it in general so I'm not misunderstanding the underlying concept.

The end goal here would be to have both ISPs terminating on the FTD with the ability to load-balance outbound internet traffic utilizing ECMP and PBR to avoid paying the licensing for the load-balancers. However, it seems that the VPNs may be my stopping point if I do not have an alternative solution.

I do have an active case with TAC opened about this, but figured I'd ask the question to the community and see if anyone has ever utilized ECMP/PBR with the FTD and some feedback. 

Thank you for your time. 

[MHM Cisco World]

check below
MHM

[emasters]

Care to elaborate on the first statement? ECMP is work but not for VPN. Not sure if I am following you.

Regarding your second statement, if I follow you, I think that's what I'm trying to achieve.

Would ISP1 and ISP2 not be considered "different next-hop?" Apologies if I'm not fully understanding what you're telling me. 

[MHM Cisco World]

Would ISP1 and ISP2 not be considered "different next-hop?" correct and it work 
https://www.cisco.com/c/en/us/td/docs/security/firepower/70/fdm/fptd-fdm-config-guide-700/fptd-fdm-routing.html

for VPN are we talk about S2S or RA ?

[emasters]

It would be both Remote Access and Site to Site VPNs. Not many compared to others I'm sure, but something to consider. 

In theory, I would also add another interface to the RA VPN configuration as well for redundancy. 

[MHM Cisco World]

MHM

[emasters]

This was from the documentation that was in my original post, which I do understand the concepts of it and what it accomplishes. Honestly, I think it would be a pretty easy solution to implement if I were not doing any VPNs of any kind on my 2140s. I like the idea of having dedicated firewalls for VPNs and another set for external access. Just need clarification on the "interfaces used for site-to-site VPN or remote access VPN connection" before I consider planning for a maintenance window to figure things out the hard way if I were to miss something. 

[MHM Cisco World]

later we discuss connect FW and use it as VPN
MHM

[MHM Cisco World]

So we Agree that in your case you need to use ECMP Zone  since you use two or more interface (in your case OUT1 and OUT2).

for VPN we have two cases 
1- VPN interface 
2- routing issue with VPN 

VPN types and point to consider 
1- VPN S2S (not VTI)
according to last ver. 7.4 you can use ECMP Zone interface config as S2S VPN 
so we will use two interface with one destination
https://www.cisco.com/c/en/us/support/docs/security-vpn/security-vpn/216709-configure-failover-for-ipsec-site-to-sit.html

here how FTD handle traffic, if you check the link above we use two ISP and config two default route with two different metric 
BUT in ECMP all route have same metric, here the trick according to Cisco the FTD route the traffic according to 
1- conn
2- RIB
so the traffic always will use one of interface and keep other as backup.
NOTE:-we must consider in this design that the VPN S2S Peer accept the traffic from two FTD VPN s2s IP 

2- s VTI 

this more easy, we can config one tunnel and make it use two FTD OUT interface, since it route-based it can use both route with same metric to send traffic to Peer 

3- RA SSL this according to last ver. 7.4 not support to use ECMP zone interface, so it totally not work 

4- RA IKEv2  same as point 1 VPN S2S (not VTI)

NOW 
this Firepower is work for ECMP (without PBR) and support RA IKEv2 still the issue of SSL VPN 

this depend on you now, you can add new FW connect it to DMZ of FPR2K and use it as VPN SSL RA server or you can try use RA IKEv2, but to be honest RA SSL VPN  is more simple and powerful and easy in install in user PC, I see alot of issue using RA IKEv2.


THIS NOT RECOMMEND IT ONLY IDEA I SHARE WITH YOU 
goodluck friend 
MHM

[tvotna]

@MHM Cisco World, don't confuse people with incorrect statements. The VPN-related limitations have been lifted in FTD 7.1 and ASA 9.17:

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/740/management-center-device-config-74/routing-ecmp.html#id_121522

Only SSL limitation remains.

Of course, all of this needs to be tested carefully.

[MHM Cisco World]

if you have something to add please share otherwise let us work, we share idea here. 
MHM

[tvotna]

So far as I remember, the system won't allow you enable webvpn on the interface configured as a member of ECMP traffic-zone. On ASA this corresponds to:

webvpn
 enable outside

You can test if this limitation still applies or ask TAC.

2nd limitation you mentioned is unclear. They probably mean that NAT-T is not supported on interfaces configured as members of ECMP traffic-zones. This is probably wrong. Typically, when you see such a note in the documentation written in bad English, it is added by Cisco TAC and doc team take no responsibility for it. There should probably be an internal doc bug which asked to add this note. TAC will be able to find it, if it exists, and shed some light.

So, if you use IKEv2 AnyConnect and not standards-based built-in IKEv2 client, ECMP becomes problematic, because AnyConnect uses SSL for profile and software updates. You can try to use PBR with path monitoring for load-balancing, although I'm not quite sure how to configure routing: in documentation they use "Send To: Egress Interface", but who provides next-hop IP address in this case? In CLI PBR with Path Monitoring corresponds to "set adaptive-interface {rtt | jitter | lost | mos} outside1 outside2", so it looks like we need static routes, but without ECMP traffic-zones it is not possible to add two equal-admin-distance static default routes via different interfaces... I'm puzzled.

Let us know what you learn from TAC please.

[emasters]

So far, not much of a response from TAC aside from the following:

"I hope this message finds you well. I did research and found that if we want to apply ECMP on an interface that has RAVPN configured, we will need to use ikev2 instead of SSL for the VPN. This is running version 7.1 or latest, on older version this is not possible. I confirmed this on our internal database and with my team."

Nothing new here. Just telling us what we already know per Cisco's documentation. 

[tvotna]

Thank you for the update. In my opinion this is a roadblock for the entire idea of configuring load-balancing for outbound traffic and AnyConnect RA VPN on the same box. It is possible to use only IKEv2 in AnyConnect, but this is very inconvenient as the client and profiles can no longer be auto-updated.

Also, I figured out that "PBR with path monitoring" won't work without ECMP. The problem is that this form of PBR uses "set interface" CLI instead of "set IP next-hop". PBR sets interface and after interface is determined FTD makes routing lookup to find valid next-hop on the chosen interface. This basically means that two default routes need to be configured which again requires ECMP (traffic-zones). So, the AnyConnect problem stays the same.

Finally, I have another concern. Do you use NAT/PAT on this ASA for outbound traffic? If you add 10G interface, will the PAT pool stay the same or there will be a new pool? What will happen in this case if ECMP is configured? In case of two different pools connections will be distributed randomly and NATed to different IP addresses, even if sender IP (user) is the same. User sessions may be rejected by Web sites which expect to see all connections for the same user coming from the same IP address. In case of a single pool and identical NAT configuration on both egress interfaces, same user should theoretically always be allocated same IP address from the round-robin PAT pool, but this needs to be carefully tested too.

[emasters]

You bring up very valid points. I'm finding out the frustrations of migrating users to IKEv2 from SSL as user profiles are not updating correctly as some users haven't connected to the VPN in quite some time. I learned the hardware that if you rename the XML profile, it does not rid of the existing profile. Yet, it just adds it to the list. I ran into an issue early on where even with a correctly configured IKEv2 profile, AnyConnect still wanted to latch on the SSL profile. I digress, though. In the end, if this doesn't all work I may just move users back to SSL and go back to the drawing board. 

Also, I would assume that if I have two public A records for both ISP interfaces, and the user resolves to just one, then I'd imagine the users VPN traffic would traverse that interface and that interface only for inbound/outbound. Load-balancing in this case wouldn't be a necessity per say, but it would be nice to have redundancy should ISP1 fail. But yes, I knew that from documentation that when configuring an ECMP zone, you configure two static routes. 

For your last concern, yes. We do use this FTD for NAT/PAT so it would be a new NAT pool that's created. We have a load-balancer that's in front of our FTD that load-balances our traffic based on HTTP/HTTPS and I've come across a handful of instances where the destination does not like user's sessions switching addresses. I've noticed this is more common in the finance world. Our accounting department had several issues where they would just get kicked out and this is why. So, policies had to be created to say if destination is to X, take this circuit, else, fallback to Y circuit. My goal would have been to re-create these rules using Policy Based Routing, but I also realized that you could use applications for your rules, but not DNS. Maybe I'm just not looking in the right place, but I see just destination IP and application when using PBR.

It's a tough situation overall. Sure, I can throw this in a lab using EVE-NG but even then, without actually having traffic flowing through it's tough to figure out if the changes you make are even working as intended. We are in healthcare, so budgets in general are pretty strict to begin with. 

I'm still working with my Cisco SE that's assigned to us and with TAC on coming up with the right solution. Perhaps this is the time for a possible re-design.