ASA 9.7+ with failover, dual ISP links using BGP & /30 or /31 addressing

andrew.butterworth · ‎11-22-2018

I am after anyone who has deployed anything similar and what to expect regarding failover behaviour and times.

The scenario I am working with is a multi-site network, each site with two connections to the same ISP using either /30 or /31 IPv4 addressing. These will be brought into a L2 switch on two VLANs and then trunked up to two sub-interfaces on the HA pair of ASA's. I appreciate ASA HA works better with at least /29's on each interface so the standby can have an IPv4 address as well, however I don't have that luxury. In fact /31's will probably earn me some brownie points.

BGP will be used on each link and extended communities outbound on the ASA will be used to influence the inbound path for the public subnet sat on the inside of the ASA with one link primary and one secondary. To influence the outbound traffic from the ASA, weight or local preference will be set with a route-map or neighbor statement for the incoming default route the ISP will advertise to us, again giving us a primary and secondary path.

I am fairly confident that this will work and if it was a single ASA I don't see any issues apart from the obvious single point of failure. However if its a HA pair of ASA's I believe forwarding will still happen for the 1st 15-seconds outbound if there is a failover, however the BGP sessions will need to be reestablished and I am not too sure on how this will behave or how long it will take. There are only three prefixes advertised by the ASA (two P2P connected interfaces and the LAN segment), inbound we will just receive a default.

I have had a search but can't find exactly what I am proposing anywhere. If anyone has anything similar and would like to share experiences I'd appreciate it.

Cheers, Andy

Steven Williams · ‎11-24-2018

What ISP that a Business is dealing with is only giving out /30 or /31 that would be my first question. And Maybe you need to bring them in on a Layer 3 switch or router and run the NATs there and then build a RFC1918 network behind it and let the ASAs send all traffic out and let the router decide. I would be looking to my ISP for more address space or looking for a new provider.

andrew.butterworth · ‎11-25-2018

Hi Steven, thanks for the reply however that is not really what I was asking. The SP provides a /29 prefix, however they will route this down a /31 or /30 P2P link. If dual circuits are provided then eBGP must be used - there is no choice in this with the SP. This is a wires-only service so I need to terminate the circuits and eBGP on something. There is no budget for any new hardware so I am looking at what I have available to me. There is a HA pair of ASA5500-X series firewalls so with ASA 9.2(4)+ BGP is available and with 9.7+ /31 IPv4 addressing is available.

Therefore with what I have available I can terminate the circuits on a L2 switch (it will actually be a couple of VLANs on an existing DMZ switch that has four free interfaces) and then bring them into two sub-interfaces on the ASA's (four with HA) and then run eBGP. I know this will work. One advantage of terminating the P2P circuits on the ASA is the public prefix doesn't then need assigning to a physical interface and the whole eight addresses can be used with NAT.

My question was how will it work with ASA HA? I know monitoring is affected due to the standby not having an IPv4 address, however physical monitoring will still work (link up/down). The BGP sessions will need to be reestablished so I am guessing there might be some downtime? My understanding is the FIB continues for 15-seconds after switchover, however during this 15-seconds routing adjacencies need to be reform?

I am after anyone's experiences with running eBGP on a HA pair of ASA's and if possible on /30 or /31 circuits.

Steven Williams · ‎11-26-2018

Do you have a diagram that shows where these devices sit in relation to where the IPs will be assigned? Also are you wanting to run eBGP on the ASA's themselves?

andrew.butterworth · ‎11-26-2018

Hi Steven, I don't think I can explain it any clearer TBH..... Two Ethernet P2P links to the same ISP - one primary and one backup. Each link will be either a /30 or a /31 public subnet. These will come into a L2 switch with a VLAN dedicated to each link. A HA pair of ASA's will also link into this L2 switch and connect to each circuit. The ISP only supports eBGP. I therefore have to configure eBGP on the ASA with a peer at the end of each link. The ISP will announce a default downstream and the ASA will announce the /29 public prefix outbound. Route-maps will be used inbound and outbound to influence the primary and backup paths.
However this isn't the question as I know it will work... I want to know when ASA HA is thrown into the mix how will it behave?

Steven Williams · ‎11-26-2018

Well there are a few ways you can achieve that. You make sure the Active ASA is connected to only the primary active ISP link and connect the backup to the secondary standby ASA, then it would require a complete failover of the ASAs for traffic to shift. Do you even need a layer 2 switch? Unless you do then just connect the ISP links directly to the ASA.

But what dictates one of the ISP links as backup? Is it active, but you are using it as a backup based on routing? Is it not active until the primary fails because then you will need something like BFD I would imagine. Adding a secondary ASA is going to change this much. Now if you add a secondary ASA and do things like Active/Active - Multi-context then yes could cause some redesign, but just to add secondary you will be fine.

andrew.butterworth · ‎11-26-2018

Both links will be active. If we don't establish a BGP session with the ISP on both links then it will get flagged as an error and may lead to the ISP disabling the inactive peer. The routing decision will be based on the BGP best path which I will influence with route-maps inbound and outbound.
What is an uknown to me is ASA failover with /31 addressing and BGP in the mix.

Steven Williams · ‎11-26-2018

Ya that I am pretty sure wont work since you need Two addresses for the ASAs alone. One primary and one standby, unless you left them in a standalone mode and then used something downstream to determine routes based on IP SLA or PBR.

andrew.butterworth · ‎11-26-2018

Hi Steven, you are incorrect. Failover will work if the standby device doesn't have an IPv4 address. Monitoring becomes an issue, however physical link up/downs are still monitored.

https://community.cisco.com/t5/firewalls/single-ip-failover-pair/td-p/1375663
http://resources.intenseschool.com/asa-failover-series-insufficient-ip-addresses-for-standby-ip-address/

So back (again...) to my original question. How will this behave?

Steven Williams · ‎11-26-2018

Interesting. Never had to do that, but good to know. I still don't understand what answer you want. "How will this behave"

Again this depends on the network in my opinion, is the ASA the default gateway for your networks?
Is there a layer 3 core router/switch downstream that just has a default route pointed to an IP on the ASA?

I have never ran BGP on the ASA, nor would I ever, but I can assume that the secondary wouldn't be able to peer BGP with the ISP without an address...so there is that.

Rahul Govindan · ‎11-26-2018

Had the exact same setup for one of my customers using Firepwower running FTD's and 2 Comcast circuits. No WAN IP address for both circuits on the secondary Firewall. Failover was pretty seamless from what I saw. Since the standby maintains the same ip address and the failover process takes care of the initial traffic flow, there was no downtime noticed.

One effect that was seen was when both the circuits had the same cost/weight. On initial setup, the route would point to circuit-1. A failover scenario would cause the route to take the secondary circuit route to take over (established earlier that primary) in some cases. This really depended on which BGP session established first. I don't think this should affect our scenario since there is a weight set from the other end.

andrew.butterworth · ‎11-26-2018

Hi Rahul
With BGP there will be only one path in the table so with what you are describing (i.e. no influencing the paths with BGP attributes) then it will flip between the two with failover but will stick and BGP won't fail back to the preferred path. With what I am proposing it should sort itself out. We use extended communities to distinguish the primary path from the secondary and then the ISP (which is us anyway) sets local preference on the PE routers with a match of the community.
I think I need to get it up and running and test it.

Cheers
Andy