Route-Based (VTI) VPN with AWS in Active/Standby Firewalls
One thing I've learned about compatibility, even with standards, is that, between-brand documentation is almost always an after-thought. So, I've written this post in the hopes to help others get past the same issues I ran into between an FTD-based firewall and AWS when using route-based Site-to-Site VPNs.
Cisco wonderfully gifted users with the ability to meet AWS and the like with more modern route-based VPNs, where the tunnels stay up regardless of traffic, and you can use both tunnels as you see fit. AWS loves this, too!
But. The ideal architecture is that the on-premises appliances are active/active, and are routers, not Active/Passive firewalls. So, let me detail a few issues with this:
To AWS, the active/passive Cisco firewall is a single device. This means, there's only one path to AWS and vice-versa, but we need to ensure both AWS VPN tunnels are always UP.
AWS doesn't provide visibility into their BGP routers (VGW or TGW), unless you pay for support and an engineer looks at it for you.
AWS documentation doesn't detail how to work with their tunnels via a single active/passive HA firewall cluster.
Since the firewall HA cluster is a poor solution with asymmetric routing, we need a way to prepend AS paths on a per-neighbor basis, or deploy a MED attribute toward AWS. But, you have to have FMC to deploy this as this can't be done in CDO or FDM (oops.)
Problems
Unreliability - If you deploy this out of the box, asymmetric routing will cause problems.
Unreliability - When the VPN tunnel drops after n hours, BGP is interrupted and may not recover. Routes are flushed, and BGP sits idle, even after the VPN tunnels are reestablished.
My solution
There is a way to make this work and keep it reliable, but it takes some minor effort, and I have one major feature-request for Cisco in this area.
Key concepts:
The firewall cluster's active unit will peer with both AWS BGP peers.
The VPN tunnels will establish and remain up.
BGP requires the on-premises device to be forced as the BGP Master.
BGP on-premises must be forced to contact the peer upon any interruption, as AWS will only respond to the on-premises BGP peer.
AWS's BGP "router" is either a VGW (Virtual Private GateWay), or on an attachment on a TGW (Transit GateWay).
NOTE: All of this is based on the perspective that you do not have or use FMC or CDO. This is as if you have only the device and FDM/CLI access.
Step 1: Create an ACL and route map for inbound route-filtering. This is important to prevent AWS from advertising your on-premises routes back to your firewall from over the VPN.
Step 2: Set up your VTI route-based VPN. Each AWS VPN tunnel will require a separate Cisco VTI interface, so you'll need two. If you're migrating from a Policy-based VPN to VTI, be sure to eliminate any NoNAT rules for these, as the firewall will get confused between the tunnel interfaces and "outside" (or the interface where your tunnel interfaces are attached).
Step 3: Create your BGP AS. The firewall can only have a single BGP process, but it can peer with multiple AWS BGP neighbors. Use the "duplicate" function in the FDM's BGP configurator to add the other neighbor. You can do this for multiple sets of AWS Site-to-Site VPNs to other VPCs. They don't care, so long as you remember that the firewall will always use the same ASN. It's ok, it works.
TIP: When configuring BGP to multiple cloud environments, it's helpful to utilize the Global BGP settings for BGP parameters that affect all BGP sessions and to simplify configuration.
Step 4: Be sure to set up the following parameters for each BGP neighbor:
Step 5: Set up weight for your outbound routing decisions, since you're using a single BGP peer on the on-premises side. Do this only for your primary AWS BGP neighbor, not the second one in each pair of Site-to-Site VPN tunnels.
One final step
This is probably the most frustrating step in this procedure, because it isn't a resolution, but rather a band-aid for a major oversight or omission by Cisco for one of the most common requirements we have for connecting Cisco firewalls to AWS over route-based Site-to-Site VPNs: AS_Path Prepending. Now, I'm baffled as to why this was not only left out of the FDM configuration ability, but the command, router bgp is black-listed through the FlexConfig method in FDM. So, you cannot apply AS_Path Prepending in FDM to my knowledge.
Now, I won't detail exactly how to do this through the CLI (SSH) using the command system support diagnostic-cli, but you can Google it.
HINT: It's the method that creates a new expert mode Linux file and uses the show version serial number (not chassis serial number) inside the system support diagnostic-cli tool. Yes, this is a hack, but it's the only way to do this without FMC.
After you've made the change there to enable that back-door function, you may have to switch HA mode and back for it to actually work. Once it does, you should be able to perform config terminal in the system support diagnostic-cli tool.
1. Create a new route-map to set up your AS_PATH prepending, match your advertised subnets (toward AWS) in an ACL or prefix list, and use set as-path prepend asn asn asn. Be sure to set up a deny rule for anything else this route map should apply this AS_Path prepending to.
2. in config t mode, go into router bgp asn, address-family ipv4 unicast, and then enter the following for your route maps. Only do this for your secondary AWS BGP neighbor, not the first or primary one in each pair of Site-to-Site VPN tunnels. This allows the weight command in Step 5 above to prefer the first tunnel and this step to configure the second tunnel as least-preferred, causing symmetric routing.
Below is a BGP example of two route-based Site-to-Site VPNs, each with two tunnels:
router bgp 65000
timers bgp 10 30 30
address-family ipv4 unicast
neighbor 169.254.10.93 remote-as 64502
neighbor 169.254.10.93 ebgp-multihop 255
neighbor 169.254.10.93 transport connection-mode active
neighbor 169.254.10.93 activate
neighbor 169.254.10.93 weight 2000
neighbor 169.254.36.171 remote-as 64502
neighbor 169.254.36.171 ebgp-multihop 255
neighbor 169.254.36.171 transport connection-mode active
neighbor 169.254.36.171 activate
neighbor 169.254.36.171 weight 2000
neighbor 169.254.118.189 remote-as 64502
neighbor 169.254.118.189 ebgp-multihop 255
neighbor 169.254.118.189 transport connection-mode active
neighbor 169.254.118.189 activate
neighbor 169.254.118.189 route-map BGP_Outbound_AS-PATH-Prepend_RouteMap out
neighbor 169.254.112.209 remote-as 64502
neighbor 169.254.112.209 ebgp-multihop 255
neighbor 169.254.112.209 transport connection-mode active
neighbor 169.254.112.209 activate
neighbor 169.254.112.209 route-map BGP_Outbound_AS-PATH-Prepend_RouteMap out
network 10.2.1.0 mask 255.255.255.0
network 10.2.1.0 mask 255.255.255.0
distance bgp 10 30 30
distribute-list BGP_Inbound_Filter_ACL in
no auto-summary
no synchronization
exit-address-family
!
Note the ASNs, the placement of the route-maps and weight. Weight is for the first tunnel BGP peer (to influence your firewall's route decisions), and AS_Path prepending is for the second BGP peer (to influence AWS's route decisions).
NOTE: Due to configuration "state" information stored in the local device's database(?), any local commands applied using this workaround are lost if you deploy any new changes through FDM, or if the device reboots.
Feature Request!
For the sake of any customer who peers a Cisco FTD firewall with AWS using route-based VPNs and BGP, we need to use AS_Path Prepending to specific neighbors. It's a crime not to include that very basic BGP functionality in FDM.
NOTE: I have confirmed with AWS engineering that the AS_Path prepending does work.
The user is discussing a network setup where Cisco Firepower Threat Defense (FTD) based firewalls are used in Active/Standby mode to setup a Route-Based VPN with AWS. The user identifies some issues with this setup, including:
- AWS seeing the Active/Passive Cisco Firewalls as a single device resulting in a single path to AWS. - Asymmetric routing causing reliability issues. - BGP interruptions when the VPN tunnel drops, which may not recover even after reestablishment of the VPN tunnels.
The user suggests a solution that involves configuring the router for BGP (Border Gateway Protocol), having the active unit of the firewall cluster peer with both AWS BGP peers, and setting up the on-premises device as the BGP Master.
The user has also suggested some steps on how to implement this solution:
1. Create an ACL and route map for inbound route filtering. 2. Set up your VTI route-based VPN, each AWS VPN tunnel will require a separate Cisco VTI interface. 3. Create your BGP Autonomous System (AS). 4. Set up the following parameters for each BGP neighbor. 5. Set up weight for your outbound routing decisions, only for your primary AWS BGP neighbor.
The user indicates a need for AS_Path Prepending but has had issues with this functionality through the FDM. According to the user, commands applied using this workaround are lost if you deploy any new changes through FDM, or if the device reboots.
The user concludes by requesting this very basic BGP functionality be included in FDM and confirms with AWS engineering that the AS_Path prepending does work.
It is important to note that this solution is intended for those who do not have or use FMC or CDO, only the device and FDM/CLI access. It's not recommended to make changes through the CLI (SSH) using the command system support diagnostic-cli if you are not experienced. Misconfigurations can lead to network issues and potential downtime.
If you're not confident doing this, it is recommended to consult with an IT professional who is experienced in configuring Cisco firewalls and AWS Site-to-Site VPNs.
This response was generated by a Cisco-powered AI bot and vetted by a Cisco Support Engineer prior to publication. This is part of a monitored experiment to see if the bot can help answer questions alongside community members. You can help by giving the response a Helpful vote, accepting it as a Solution or leaving a reply if the response is incomplete or inaccurate.
Not sure what version you are running, but we have FTD/FDM version 7.2.5 and there is option to configure AS_PATH prepending and many other BGP related configurations using Smart CLI templates. You need to create route-map and prefix-list using Smart CLI objects and then use those in your BGP configuration. I was able to achieve the exact same thing that you have mentioned with Smart CLI objects.
I am not exactly sure when Cisco introduced this feature, here is the reference guide for ver 7.1 -
Good call on this. However, I should have been clear in this should be configured on a per-neighbor (peer) basis vs. per ASN. Yes, you can configure AS_Path prepending in the FlexConfig, but that works only if you're using multiple routers, one for each AWS VPN tunnel. IIRC, and based on the research efforts that TAC exhausted, there isn't a way to make the single FTD device provide AS_Path prepending per neighbor outside FMC. I have updated my original post to reflect that clarification. Thanks!
Have you been able to verify on the remote end neighbors/peers that the AS_Paths are indeed prepended? The Cisco documentation does not indicate this to be the case. I will admit, if this is the way they have allowed this to happen, Cisco certainly doesn't want us knowing about it.
RFC 1925
Learn, share, save
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
