topic Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1 in Network Security

ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Thu, 11 Jan 2024 15:46:52 GMT

For some context, we have dual Firepower 2140s in HA managed by FMCv. Both FTD and FMCv are on 7.2.5.1 as of last week at the start of 2024.

We currently utilize our FTDs as external firewalls and head-ends for both site-to-site VPNs and Remote Access VPNs. Our RA VPN users are currently utilizing IPsec. We have dual 1Gbps internet connections on a load-balancer that sits in front of the FTDs while the FTDs themselves have just a single outside connection that feeds into the LAN of the load-balancer. We also have a new 10Gbps connection as of the last month that we'd like to start utilizing. The cost of using 10Gbps on our load-balancer is going to be costly as it's about $70k worth in licensing. This started the research of seeing if the FTDs were capable of load-balancihng outbound traffic and that's where I came across ECMP and Policy Based Routing. As of 7.2, the FMC natively supports the configuration of both without the need of FlexConfig. Neat.

Naturally, I started reading documentation on the requirements. For the most part, I met all the requirements execpt for two as noted by the documentation here.

Following interfaces cannot be associated with an ECMP zone -> Interfaces in RA VPN configuration with SSL enabled
Threat defense does not support ECMP with NAT in IPsec sessions—a standard IPsec virtual private network (VPN) tunnel does not work with NAT points in the delivery path of IPsec packets.

My first question/concern with the first point is are they referring to having SSL on the interface itself, or having SSL enabled in the RA VPN Group Policy? In other words, SSL enabled within the Access Interface configuration

Or.....the Group Policy

If it's referring to having SSL enabled on the actual interface itself and need to turn that off, then I suppose I'll need to come up with a gameplan regarding our VPN users as, If I recall, all RA VPN sessions utilize SSL initially.

The second point I may just need clarification on it in general so I'm not misunderstanding the underlying concept.

The end goal here would be to have both ISPs terminating on the FTD with the ability to load-balance outbound internet traffic utilizing ECMP and PBR to avoid paying the licensing for the load-balancers. However, it seems that the VPNs may be my stopping point if I do not have an alternative solution.

I do have an active case with TAC opened about this, but figured I'd ask the question to the community and see if anyone has ever utilized ECMP/PBR with the FTD and some feedback.

Thank you for your time.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Fri, 12 Jan 2024 00:39:03 GMT

check below
MHM

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Thu, 11 Jan 2024 16:25:54 GMT

Care to elaborate on the first statement? ECMP is work but not for VPN. Not sure if I am following you.

Regarding your second statement, if I follow you, I think that's what I'm trying to achieve.

Would ISP1 and ISP2 not be considered "different next-hop?" Apologies if I'm not fully understanding what you're telling me.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

tvotna — Thu, 11 Jan 2024 17:41:49 GMT

So far as I remember, the system won't allow you enable webvpn on the interface configured as a member of ECMP traffic-zone. On ASA this corresponds to:

webvpn
 enable outside

You can test if this limitation still applies or ask TAC.

2nd limitation you mentioned is unclear. They probably mean that NAT-T is not supported on interfaces configured as members of ECMP traffic-zones. This is probably wrong. Typically, when you see such a note in the documentation written in bad English, it is added by Cisco TAC and doc team take no responsibility for it. There should probably be an internal doc bug which asked to add this note. TAC will be able to find it, if it exists, and shed some light.

So, if you use IKEv2 AnyConnect and not standards-based built-in IKEv2 client, ECMP becomes problematic, because AnyConnect uses SSL for profile and software updates. You can try to use PBR with path monitoring for load-balancing, although I'm not quite sure how to configure routing: in documentation they use "Send To: Egress Interface", but who provides next-hop IP address in this case? In CLI PBR with Path Monitoring corresponds to "set adaptive-interface {rtt | jitter | lost | mos} outside1 outside2", so it looks like we need static routes, but without ECMP traffic-zones it is not possible to add two equal-admin-distance static default routes via different interfaces... I'm puzzled.

Let us know what you learn from TAC please.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Thu, 11 Jan 2024 17:55:10 GMT

Would ISP1 and ISP2 not be considered "different next-hop?" correct and it work
https://www.cisco.com/c/en/us/td/docs/security/firepower/70/fdm/fptd-fdm-config-guide-700/fptd-fdm-routing.html

for VPN are we talk about S2S or RA ?

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Thu, 11 Jan 2024 18:28:43 GMT

It would be both Remote Access and Site to Site VPNs. Not many compared to others I'm sure, but something to consider.

In theory, I would also add another interface to the RA VPN configuration as well for redundancy.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Thu, 11 Jan 2024 19:12:18 GMT

MHM

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Thu, 11 Jan 2024 19:07:28 GMT

This was from the documentation that was in my original post, which I do understand the concepts of it and what it accomplishes. Honestly, I think it would be a pretty easy solution to implement if I were not doing any VPNs of any kind on my 2140s. I like the idea of having dedicated firewalls for VPNs and another set for external access. Just need clarification on the "interfaces used for site-to-site VPN or remote access VPN connection" before I consider planning for a maintenance window to figure things out the hard way if I were to miss something.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Fri, 12 Jan 2024 00:34:22 GMT

later we discuss connect FW and use it as VPN
MHM

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

tvotna — Thu, 11 Jan 2024 19:23:19 GMT

@MHM Cisco World, don't confuse people with incorrect statements. The VPN-related limitations have been lifted in FTD 7.1 and ASA 9.17:

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/740/management-center-device-config-74/routing-ecmp.html#id_121522

Only SSL limitation remains.

Of course, all of this needs to be tested carefully.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Fri, 12 Jan 2024 01:01:43 GMT

So we Agree that in your case you need to use ECMP Zone since you use two or more interface (in your case OUT1 and OUT2).

for VPN we have two cases
1- VPN interface
2- routing issue with VPN

VPN types and point to consider
1- VPN S2S (not VTI)
according to last ver. 7.4 you can use ECMP Zone interface config as S2S VPN
so we will use two interface with one destination
https://www.cisco.com/c/en/us/support/docs/security-vpn/security-vpn/216709-configure-failover-for-ipsec-site-to-sit.html

here how FTD handle traffic, if you check the link above we use two ISP and config two default route with two different metric
BUT in ECMP all route have same metric, here the trick according to Cisco the FTD route the traffic according to
1- conn
2- RIB
so the traffic always will use one of interface and keep other as backup.
NOTE:-we must consider in this design that the VPN S2S Peer accept the traffic from two FTD VPN s2s IP

2- s VTI

this more easy, we can config one tunnel and make it use two FTD OUT interface, since it route-based it can use both route with same metric to send traffic to Peer

3- RA SSL this according to last ver. 7.4 not support to use ECMP zone interface, so it totally not work

4- RA IKEv2 same as point 1 VPN S2S (not VTI)

NOW
this Firepower is work for ECMP (without PBR) and support RA IKEv2 still the issue of SSL VPN

this depend on you now, you can add new FW connect it to DMZ of FPR2K and use it as VPN SSL RA server or you can try use RA IKEv2, but to be honest RA SSL VPN is more simple and powerful and easy in install in user PC, I see alot of issue using RA IKEv2.

THIS NOT RECOMMEND IT ONLY IDEA I SHARE WITH YOU
goodluck friend
MHM

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Fri, 12 Jan 2024 01:02:34 GMT

if you have something to add please share otherwise let us work, we share idea here.
MHM

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Mon, 15 Jan 2024 15:40:19 GMT

So far, not much of a response from TAC aside from the following:

"I hope this message finds you well. I did research and found that if we want to apply ECMP on an interface that has RAVPN configured, we will need to use ikev2 instead of SSL for the VPN. This is running version 7.1 or latest, on older version this is not possible. I confirmed this on our internal database and with my team."

Nothing new here. Just telling us what we already know per Cisco's documentation.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

tvotna — Mon, 15 Jan 2024 17:13:42 GMT

Thank you for the update. In my opinion this is a roadblock for the entire idea of configuring load-balancing for outbound traffic and AnyConnect RA VPN on the same box. It is possible to use only IKEv2 in AnyConnect, but this is very inconvenient as the client and profiles can no longer be auto-updated.

Also, I figured out that "PBR with path monitoring" won't work without ECMP. The problem is that this form of PBR uses "set interface" CLI instead of "set IP next-hop". PBR sets interface and after interface is determined FTD makes routing lookup to find valid next-hop on the chosen interface. This basically means that two default routes need to be configured which again requires ECMP (traffic-zones). So, the AnyConnect problem stays the same.

Finally, I have another concern. Do you use NAT/PAT on this ASA for outbound traffic? If you add 10G interface, will the PAT pool stay the same or there will be a new pool? What will happen in this case if ECMP is configured? In case of two different pools connections will be distributed randomly and NATed to different IP addresses, even if sender IP (user) is the same. User sessions may be rejected by Web sites which expect to see all connections for the same user coming from the same IP address. In case of a single pool and identical NAT configuration on both egress interfaces, same user should theoretically always be allocated same IP address from the round-robin PAT pool, but this needs to be carefully tested too.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Mon, 15 Jan 2024 17:31:18 GMT

You bring up very valid points. I'm finding out the frustrations of migrating users to IKEv2 from SSL as user profiles are not updating correctly as some users haven't connected to the VPN in quite some time. I learned the hardware that if you rename the XML profile, it does not rid of the existing profile. Yet, it just adds it to the list. I ran into an issue early on where even with a correctly configured IKEv2 profile, AnyConnect still wanted to latch on the SSL profile. I digress, though. In the end, if this doesn't all work I may just move users back to SSL and go back to the drawing board.

Also, I would assume that if I have two public A records for both ISP interfaces, and the user resolves to just one, then I'd imagine the users VPN traffic would traverse that interface and that interface only for inbound/outbound. Load-balancing in this case wouldn't be a necessity per say, but it would be nice to have redundancy should ISP1 fail. But yes, I knew that from documentation that when configuring an ECMP zone, you configure two static routes.

For your last concern, yes. We do use this FTD for NAT/PAT so it would be a new NAT pool that's created. We have a load-balancer that's in front of our FTD that load-balances our traffic based on HTTP/HTTPS and I've come across a handful of instances where the destination does not like user's sessions switching addresses. I've noticed this is more common in the finance world. Our accounting department had several issues where they would just get kicked out and this is why. So, policies had to be created to say if destination is to X, take this circuit, else, fallback to Y circuit. My goal would have been to re-create these rules using Policy Based Routing, but I also realized that you could use applications for your rules, but not DNS. Maybe I'm just not looking in the right place, but I see just destination IP and application when using PBR.

It's a tough situation overall. Sure, I can throw this in a lab using EVE-NG but even then, without actually having traffic flowing through it's tough to figure out if the changes you make are even working as intended. We are in healthcare, so budgets in general are pretty strict to begin with.

I'm still working with my Cisco SE that's assigned to us and with TAC on coming up with the right solution. Perhaps this is the time for a possible re-design.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

tvotna — Mon, 15 Jan 2024 20:12:55 GMT

Correct. Inbound VPN traffic will create connection on, say, outside2 interface of the firewall and reverse traffic will always take the same interface outside2. But if outside2 fails (doesn't matter if interface itself fails or connectivity is lost), AnyConnect will go into the "reconnecting" state until the user disconnects manually.

but I also realized that you could use applications for your rules, but not DNS

No, not really. An "application" in this context is a "DNS domain name" or "domain + subnet/protocol/port" combination. This is also called network-service group or NSG. This article explains the concept very well: https://secure.cisco.com/secure-firewall/docs/policy-based-routing-with-path-monitoring. So, this can be used to route certain "domain" traffic forcefully to specific outside interface.

It appears to me that at the current level of FTD software development it would be easier to use something like

route outside1 0.0.0.0 128.0.0.0 x.x.x.x
route outside2 128.0.0.0 128.0.0.0 y.y.y.y

avoiding all complications with ECMP, NAT, PBR, etc.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Tue, 16 Jan 2024 15:10:19 GMT

Quick update regarding my case with TAC:

"Hi team,
Regarding you questions, when configuring the RAVPN you can select the protocols that you are going to use for the tunnel and the interface under the tab "Advanced interfaces". On that tab we should disable the ssl protocol and only enable the ipsec protocol so you can use ECMP on the same interface where the VPN is being formed. Another change we need to perform to use ipsec is select the ipsec protocol on the xml profile as is shown on the screenshot."

The screenshot TAC attached. This is something I already have set for my users.

Which would mean this would involve disabling SSL at the interface level under Access Interfaces, which is what I was afraid of.

I think at this point I'm thinking this is a lost cause without this being a completely horrible experience for our end users.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

MHM Cisco World — Tue, 16 Jan 2024 16:03:54 GMT

I confuse here
you allow IKEv2 and SSL(+DTLS) in outside but that not meaning you use IKEv2 anyconnect.
if you decide to go with ECMP then SSL completely not work if you permit or not in outside.
and sure client will not happy using ikev2 instead of ssl
MHM

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

emasters — Tue, 23 Jan 2024 21:08:17 GMT

So, had my chat with some folks from Cisco that specialize in security. Disabling SSL for Remote Access VPN would seem to do more harm than good. It would ruin the experience and it was advised not to move forward with ECMP/PBR. However, we tossed around the idea of having a pair of firewalls that were dedicated for VPN and VPN only. This would include both Site to Site and Remote Access and would eliminate a few of the caveats regarding ECMP/PBR and would allow us to move forward. This should be able to hold us over until we're in a position to setup BGP.

I can follow up once we've moved forward and leave some feedback. That's where things stand at the moment.

Re: ECMP and PBR with dual FTD 2140s managed by FMCv on 7.2.5.1

tvotna — Mon, 12 Feb 2024 17:01:43 GMT

Ok @emasters , this is a move in the right direction.

As a side note, I've just accidentally found an answer to your second question about documentation saying "Threat defense does not support ECMP with NAT in IPsec sessions — a standard IPsec virtual private network (VPN) tunnel does not work with NAT points in the delivery path of IPsec packets". This is yet another software caveat:

CSCwf43850 ECMP + NAT for ipsec sessions support request for Firepower

Symptom: Incoming IPSEC packets dropped with Drop-reason: (bad-ipsec-prot) IPSEC not AH or ESP. Conditions: Equal-Cost Multi-Path (ECMP) over VPN and NAT Traversal (NAT-T, ESP UDP encapsulation). Workaround: Do not use NAT-T by avoiding NAT in the path. If not possible to avoid NAT-T do not use ECMP. Further Problem Description: This is a day one issue - the features combination never worked.

This bug was fixed in 9.18.4.8 and 9.20.2.