Showing results for 
Search instead for 
Did you mean: 

Multi-Site BGP with Separate ISPs

Hello, I am beginning to work on a BGP implementation and have some flexibility in how to architect things, but I am looking for some recommendations.

Current Architecture:

    Three physically separate sites all connected by 1Gbps layer 2 MPLS
    Each site is has its own ISP with a /24 of public address space from the respecitve provider
    Each site  has a /11 in private network address space for LAN subnets and hosts
    Each site runs OSPF as IGP with static routes to each respective /11 of the other two sites
    Unique services hosted on each site

We will be reducing our ISP count down to two and routing Internet-bound traffic from the site without direct Internet access to one or both of the other sites. We currently do this now if there is an ISP outage and we can change the external DNS of some critical services to come in via a backup IP on another site and route over to the site in trouble.

Option 1

Use one site as a "headquarters" and have both ISPs terminate physically on the headquarters site. We have stateful firewalls and perform NATing so we would want to place a router capbable of handling BGP routing tables (assuming we get full routing tables from ISPs) between our firewalls and the ISPs. We have been approved of a /24 and ASN from ARIN so I believe we could advertise this prefix and ASN to both ISP BGP peer routers. Return traffic to our edge router would then all go to our firewall eliminating asymmetric routing problems due to stateful inspection.

Things would be simplified due to the fact that all LAN hosts, regardless of site, would have a default route to our firewall which in turn hands off traffic to our premise router which is peered to each ISP BGP peer. No routing blackholes, but a disaster on the headquarters site would bring down Internet for all sites. This site is on generator power and one ISP is via directional microwave so we have some safeties built in, but accidents happen.


Option 2

Use two sites for Internet access. Enable iBGP between Internet routers on each site. Either use AS Path prepending or BGP Conditional Advertisement to have each site make inbound traffic favor the site hosting our services. Even if we host some services out of the second site with Internet access we could use PBR to ensure these servers always send their return Internet traffic back across the MPLS to the headquarters site.

This would function more like a backup path for inbound access to hosted services, while giving clients direct Internet access. I would need to use IP SLA to make sure the site with failed Internet would use a backup default route to the other Internet connected site so traffic is not blackholed.


This is my my first real-world BGP setup, everything prior was just Cisco-cert oriented and did not touch much on multi-site multihoming. Recommendations, corrections, or general advise would be much appreciated. I can post a diagram too, if that's helpful. Thanks!


This would depend on where

This would depend on where your critical data is and whether you're hosting.


If all you have is the single MPLS and you're only allowed to have a total of 2 Internet connections, I would separate the two connections across two sites so that you can create a back VPN tunnel in case the MPLS goes down.  (our MPLS connection has gone down multiple times in the last couple months)


Then, you could use AS path prepending and local preference to favor one internet link.


That is exactly what we do.

That is exactly what we do. Site to Site VPN from site A to Site B in the event of MPLS failure and each side has a floating static route in the event of MPLS failure, traffic will flow over the site to site VPN. Not good that your MPLS connection is going down that much. Might want to have a look into your SLA for that.


I am having similar design

I am having similar design requirements, except I don't know how to go about all of this.

We have two data centers, both of them are active. Each site has one ISP, and the ISPs are from different provider. The two sites have a WAN connection in between, no Layer 2. So I think we will have to get some kind of L2TPv3 setup so we can span the subnets across two sites. Our firewalls from both sites (one from each) will have to be in a cluster so the public facing interfaces will have VIP?

We have a /23 IPv4 address space assigned by ARIN with our own ASN.


1. Active/Active configuration from both data centers.

2. Failover capability for both sites. If resources from one site is unavailable, those resources need to be available on the other site (we have replication in place that is currently replicating the data/servers, etc between two sites).

I think the first step for us was to obtain a /23 IPv4 address space with our own ASN since we do not want to use the ISP assigned IPv4 address space.

Do I have to advertise the entire /23 from both sites?

Do the firewalls from both sites have to be in the same address space, the /23 for the public facing interfaces at least?

Our web servers are placed in a DMZ, so does the DMZ have to be spanned across the two sites?

I have attached a pdf of a layout.

Thanks in advance!




Re: I am having similar design

Look at . Its some similar case trying to solve it with Conditional BGP Routes
VIP Mentor

Re: I am having similar design



post the configs of your router and indicate which route you want to be conditional...