Hello, I am beginning to work on a BGP implementation and have some flexibility in how to architect things, but I am looking for some recommendations.
Current Architecture:
Three physically separate sites all connected by 1Gbps layer 2 MPLS
Each site is has its own ISP with a /24 of public address space from the respecitve provider
Each site has a /11 in private network address space for LAN subnets and hosts
Each site runs OSPF as IGP with static routes to each respective /11 of the other two sites
Unique services hosted on each site
We will be reducing our ISP count down to two and routing Internet-bound traffic from the site without direct Internet access to one or both of the other sites. We currently do this now if there is an ISP outage and we can change the external DNS of some critical services to come in via a backup IP on another site and route over to the site in trouble.
Option 1
Use one site as a "headquarters" and have both ISPs terminate physically on the headquarters site. We have stateful firewalls and perform NATing so we would want to place a router capbable of handling BGP routing tables (assuming we get full routing tables from ISPs) between our firewalls and the ISPs. We have been approved of a /24 and ASN from ARIN so I believe we could advertise this prefix and ASN to both ISP BGP peer routers. Return traffic to our edge router would then all go to our firewall eliminating asymmetric routing problems due to stateful inspection.
Things would be simplified due to the fact that all LAN hosts, regardless of site, would have a default route to our firewall which in turn hands off traffic to our premise router which is peered to each ISP BGP peer. No routing blackholes, but a disaster on the headquarters site would bring down Internet for all sites. This site is on generator power and one ISP is via directional microwave so we have some safeties built in, but accidents happen.
Option 2
Use two sites for Internet access. Enable iBGP between Internet routers on each site. Either use AS Path prepending or BGP Conditional Advertisement to have each site make inbound traffic favor the site hosting our services. Even if we host some services out of the second site with Internet access we could use PBR to ensure these servers always send their return Internet traffic back across the MPLS to the headquarters site.
This would function more like a backup path for inbound access to hosted services, while giving clients direct Internet access. I would need to use IP SLA to make sure the site with failed Internet would use a backup default route to the other Internet connected site so traffic is not blackholed.
This is my my first real-world BGP setup, everything prior was just Cisco-cert oriented and did not touch much on multi-site multihoming. Recommendations, corrections, or general advise would be much appreciated. I can post a diagram too, if that's helpful. Thanks!
