Multiple Multi-Homed Site BGP Implementation with Same ASN and Address Space at Both Sites with Primary Backhaul and L2L VPN for Failover Utilizing OSPF

[tom.neteng]

I'm attempting to achieve the following:

Create a fully automated failover and DR design with two sites, each of which has two ISPs for outbound Internet and two e-lines between them, utilizing shared IP space and the same ASN "loaned" to us from Comcast on both sides, for advertisement on another ISP, but with a backup static IP unique to each provider at each site (for a total of a minimum of 4 static IPs).  I don't know if this is possible.

Make branch office site, currently backhauled across e-lines, using primary site's firewall, backhaul all traffic except for static IP VPN across the e-lines, but in the event of a loss of either:

2x e-line

HA firewall pair

2x ISP

reverse the traffic and utilize the outbound Internet locally and a site to site VPN built between the firewalls at either site over the Internet for the internal traffic that would normally pass between the two sites over Metro-E (which are p2p with VTI overlay currently running EIGRP).

So:

SITE 1

FW ISP1 VPN ON STATIC IP 10.12.34.1/30-----------------VPN ON STATIC IP 10.54.32.1/30 ISP1

       \                                                                                                             /

ASN2345 10.23.45.1/24 PRIMARY > > > > > < < < < < ASN2345 10.23.45.1/24 DR BACKUP

       /                                                                                                             \

ISP2 VPN ON STATIC IP 172.16.2.1/30---------------------VPN ON STATIC IP 172.16.3.1/30 ISP2

/\                                                                                                                    /\

R1---------------- -----------------------------------------------R2

/\                                                                                                                    /\

CORE                                                                                                           CORE

The current routing protocol is EIGRP, which I am going to run alongside OSPF and iBGP until decommissioned, as currently the firewalls have static routes on them as they don't support EIGRP.  Then, I would like the firewalls to peer with multiple ISPs on either side.  So would I just use the same BGP ASN on both firewalls at either site, using iBGP between each firewall over the e-lines or would it suffice to run eBGP on the firewalls with the two providers on either side (which are the same on both sides) and simply run OSPF across the e-lines and the tunnels? What, if any redistribution of routes will I have to configure between eBGP and OSPF?

I'm not certain how I could advertise the same address space using the same ASN and have connectivity between the two sites over the Internet without a secondary assigned static IP.  In my diagram, I supposed that I have both BGP and a secondary static IP that I can terminate a VPN on.

Is this possible and an effective design or should I break each site into a separate BGP ASN with non-duplicated address space and somehow have the DR come up with an ASN or address space that otherwise wouldn't be advertised--or advertised at a lower metric if the primary site hasn't had a complete loss of either firewalls or both ISPs?

Tom

[Philip D'Ath]

I want to verify one thing you said - this is for outbound Internet access (aka web browsing) only?

You don't even need "loaned" address space for this, as you can simply NAT to each providers address space.

[tom.neteng]

No.  This is also for site-to-site connectivity across the Internet in the event of a dual private line failure, in addition to Internet failover (move default route from fw to router so it gets backhauled in the opposite direction) over the e-lines in the event of a multiple firewall or multiple ISP failure on either side.

Tom

[Francesco Molino]

Hi

This isn’t clear enough for me, sorry :-)

First you want to use BGP with your ISP and you said you get an ASN “loaned” to advertise your network. You’re talking about 4 IPs. 

  —> What do you want to advertise to your ISP?

  —> I’ll talk about advertisement to internet after your answers but who gave you your public subnet (ISP or yours)?

  —> Why do you want to advertise your subnet? Is it for inbound traffic?

Can you share a design of what you want to implement?

Between your 2 sites, you have 2 E-lines and a L2L VPN?

Just to clarify also, why do you want to use OSPF over tunnels and not BGP?

To answer remaining questions, I’ll wait for your answer.

Anyway, you can advertise same subnets from 2 routers (1 on each site). But again, the same question is it for inbound traffic? Do you want inbound load-balancing..

[tom.neteng]

---

@Francesco Molino wrote:

Hi

This isn’t clear enough for me, sorry :-)

>Let me try to clarify.

First you want to use BGP with your ISP and you said you get an ASN “loaned” to advertise your network. You’re talking about 4 IPs.

>I originally thought that you had to have provider independent space to advertise through multiple ISPs.  The engineer from Comcast is telling me that they will let us advertise the routes to their subnet with a different provider.

>In order to provide for failover to the DR in the event of a Primary Site Failure (Site 1) which would correspond with a dual ISP failure at that site or region, a duplicate /24 advertised simultaneously with the same ASN is to be advertised on the DR site (which would not be active).

>Basically what I read in the following post:

>https://supportforums.cisco.com/t5/wan-routing-and-switching/multi-site-bgp-with-separate-isps/td-p/2482822

>"Enable iBGP between Internet routers on each site (really internal routers and external firewalls--Tom).  Either use AS Path prepending or BGP Conditional Advertisement to have each site make inbound traffic favor the site hosting our services."

>Normal Operations

>SITE 1

>Primary BGP Route Internet External for both SITES 1 and 2.

>SITE 2

>Backup BGP Route Internet External in the event of dual ISP or FW failure at SITE 1.

>SITE 1 TO SITE 2

>Primary communications between SITES 1 and 2 via redundant e-lines from multiple providers.

>Backup communications (in the event of dual e-line failure) out each respective ISP through Internet L2L tunnel.

>Due to the fact that I would be advertising the same address space and same asn across mutliple sites, I would anticipate that an additional address not advertised via BGP and statically assigned by the ISP would be required as a sort of "secondary" permanent IP that could be assigned, one per ISP, per side, for a total of an additional four IPs, which would be used for VPN termination across the Internet from SITE 1 to SITE 2.

What do you want to advertise to your ISP?

>The duplicate address space primarily used by SITE 1 for normal operations, so that in the event of a SITE 1 dual FW or ISP failure, all routes would be advertised via BGP to SITE 2 so that no external DNS or GSLB needs to be used within the organization and can remain hosted.

I’ll talk about advertisement to internet after your answers but who gave you your public subnet (ISP or yours)?

>ISP.  The engineer from Comcast said we can advertise their address space through another ISP.  I didn't think that was possible.  I originally thought we needed provider independent space and the waiting list for IPv4 on ARIN is over a year right now, but after enquiring I was told that was possible.  I'm not certain how and if we could indeed use the same ASN or whether we would need to establish a private ASN with the other provider.  I am waiting to hear back from that engineer, who I e-mailed today.

Why do you want to advertise your subnet? Is it for inbound traffic?

>Inbound traffic including to on-prem services.

Can you share a design of what you want to implement?

>I need to get Visio installed on my new laptop.  I will work on it and update the thread with an attachment.

Between your 2 sites, you have 2 E-lines and a L2L VPN?

 >Yes.  That is the new design.  Two e-lines connecting a router on either side for redundancy between two providers.  And a tunnel from ISP1 at SITE 1 to ISP1 at SITE 2 (will do multiple peers if possible and add ISP2 at SITE 2 if supported) and a second tunnel from ISP2 at SITE 1 to ISP2 at SITE 2 (will add ISP1 as a second peer if possible at SITE 2).  The routers sit inside the firewall on either side.

Just to clarify also, why do you want to use OSPF over tunnels and not BGP?

> I thought that I could complete a circular topology of all internal network equipment with the same routing protocol and only allow the default outbound routes and advertisement of the ASN to the ISPs to be handled by BGP, with iBGP running on only the routers and the FWs inside interface.  I don't know what the best design would be in this circumstance.  Feel free to suggest another type of implementation.

To answer remaining questions, I’ll wait for your answer.

Anyway, you can advertise same subnets from 2 routers (1 on each site). But again, the same question is it for inbound traffic? Do you want inbound load-balancing...

>Load balancing would be nice to have as well, but the primary focus of the design was simply for failover so that Internet traffic inbound to the primary site SITE 1 could be redirected to the two ISPs at SITE 2 in the event of a daul FW or ISP failure at SITE 1.

>Tom

[tom.neteng]

Network diagram attached.

[Francesco Molino]

Thanks for the design it helps to understand better.

I see that your 2 IPSEC tunnels, 1 between firewalls and 1 between routers.

If your ISP allows you to advertise your public subnet then you can do it but you need to validate with your second isp they're going to accept the advertisement.

Now, what type of subnet you have that you're going to advertise. To do load-balancing or simply redundancy using as-path over the internet, you can't announce a subnet smaller than /24.

In terms of routing protocol, I will use iBGP between your firewall and internally and eBGP (for sure) with your ISP. You won't need to play with AD if you use only iBGP compared to using iBGP and OSPF where you'll need to be careful about your preferred routes as AD is different.

For outbound as well, you'll do active/standby to not have asymetric traffic unless you can manage it with your firewalls.

I'll review it tonight when home to see if I missed a question as I'm in the traffic right now :-)