Multihoming, BGP - 1 or 2 routers? Pros? Cons?

[brettp]

So, I've been tasked with re-designing our internet connections for "automatic failover." To me, it's a daunting project and I came up with two designs. In theory, it easy, but when considering nuances of the network -- VPNs, ppp links, etc. it turns into a headache. One option is to have both providers come into one active router (using SLA to monitor some internet IP and remove/add default route. I would also have a second standby router in this case.) Or, have one provider come into one router and the other into another, using HSRP on the "inside" interface to switch over when one ISP goes down (using SLA to monitor some internet IP.) Load balancing is not important in any case. Are there any pros or cons for either of these designs? I know there's no answer set in stone, but I'm wondering if any one has anything to add from past experience. I have zero experience.

[Richard Burts]

I am slightly confused. The title of this post indicates that you are using BGP on connections to two ISP. Then the post talks about using SLA to monitor and to add/delete default routes. Why would you want to use SLA when you are using BGP? With BGP if the provider is working ok then it advertises its default route and if the provider is not working then it is not advertising its default route. Why do you want SLA involved in this?

As I see it here are some pro and con of 1 router and 2 router solutions.

1 router pro: may be less expensive since you need a single platform (of course if you want a warm spare/cold spare then the cost advantage goes away). may be more simple to effect failover since a single device is making and implementing the failover decision. may be more simple to configure since there is no requirement for IBGP.

1 router con: this is a significant single point of failure. If the single router fails then you lose Internet connectivity and there may be challenges in physically making the changeover and there may be challenges in keeping the config of the standby router in sync with the active router.

2 router pro: eliminates the single point of failure. failover is automatic and dynamic and there is no requirement for physical intervention. keeping configuration in sync is easier.

2 router con: may be more expensive since 2 platforms are required. configuration may be somewhat more complex since there is requirement for an IBGP session in addition to the 2 EBGP sessions.

HTH

Rick

[brettp]

Thanks for the reply, Rick. To clarify, I would use SLA because in my experience, when the “internet goes down,” the outage is usually in the providers network and not the actual link between our router and theirs. That interface generally remains up when our internet is really down. Thus, if I ping 8.8.8.8 or something, that’s a better gauge of the status of that connection. And maybe I wasn’t clear… I would be using two routers regardless… My real question is, would it be better to have both ISP connections connect to one router (that is, two ISP connections on the active router and two ISP connections on the standby router) OR one ISP connection per router (the active router has one ISP connected to it, the standby router has one ISP connected to it.) Just wondering if there is a better option to go with. I would imagine two ISPs on router is easier. Thanks again!

[Richard Burts]

Thanks for the additional information, especially for the clarification that you plan to use two routers no matter which option you utilize. I am a bit puzzled about having two routers and connections from both ISP to both routers. Are you suggesting that both routers would be actively connected to both ISP and to your network all the time? That would certainly provide a high degree of redundancy but would be more complex to configure and to operate and would seem to be more expensive.

If you are planning to have both routers active on your network then my suggestion would be to configure one ISP per router.

HTH

Rick

[Richard Burts]

Jon

I feel like I am not on the same page as you (and perhaps not the same page as the original poster). I thought that the focus on the original question was about failing over from one provider to the other. And so the question was about which default route to use. And I do not see how SLA comes into play when we are evaluating dynamic routes. Are we perhaps suggesting running BGP but not accepting either default route, and using a static default route to one provider, a floating static default to the other provider, and SLA tracking of the static default route to achieve failover?

I see that you are including HSRP in your suggestion. Does that indicate that you are advocating for one ISP connection per router, that the HSRP primary is to the preferred ISP and the HSRP is to the backup ISP?

HTH

Rick

[Jon Marshall]

I think the focus is on failing over as far as I can tell. 

We are talking about BGP dynamic routes but it depends on how the ISP generates that default route. 

So as an example two routers R1 and R2 peering to two different ISPs, one per router. R1 is the HSRP active as well as being the primary for internet connectivity. Both ISPs send a default route to their respective EBGP peer. 

Now R1's EBGP peer stays up but a router further upstream in the ISP network fails and this effectively cuts you off from the internet. 

If the ISP is generating the BGP default route on the EBGP peer and is not checking connectivity to the rest of their network you still receive the default route from the EBGP peer which is still up and as R1 is the HSRP active router all outbound traffic to the internet still goes via R1's ISP and is dropped because of the upstream failure. 

If you use IP SLA with HSRP and ping an IP within the ISP network that guarantees you cannot be isolated then if the ping fails you know you have lost connectivity and can switch to the other router which has it's own default route from it's own ISP. 

This is not theoretical, I have seen it happen. 

Jon

[brettp]

Bingo… Yes, this is exactly what I am trying to say… but I was wondering if this route is better than having two ISP connect to one router and simply use a floating default route.

[Jon Marshall]

But that would not solve the issue I was talking about ie. a floating static default route would only be used if the BGP default route was removed from the routing table but if the ISP still advertises it you will never use the floating static. 

Jon

[brettp]

1 router with two ISP, a floating default route would work fine using SLA to track. If using 2 routers with one ISP each, HRSP with SLA to track.

[Jon Marshall]

Sorry, missed the part about using IP SLA with the floating static. 

Yes, that would work fine so it comes down to complexity and redundancy. 

If you are talking about two routers with an ISP each vs two routers each with a connection to both ISPs I would choose the two routers with an ISP each to keep it simple. 

You are configuring IP SLA anyway just for HSRP instead of tracking a route. 

Jon

[Richard Burts]

This is becoming an interesting discussion. 

Jon

Would it be an overstatement to suggest that in your suggestion we do not care what the ISP advertises, and that we will depend on SLA and HSRP to choose which outbound router to use. And therefore that the outbound router does not really need the ISP default route and could just as easily have a static default route of its own?

There was mention of BGP advertising the address space to the ISP. We do not know what the address space is like. Do they use provider independent addressing? Or is their address space assigned by one (or both) ISP? Perhaps the original provider can provide some clarification about this?

HTH

Rick

[Jon Marshall]

Rick 

Yes, technically you could just use a static route and track it rather than use the BGP provided route.  

In the example I was discussing BGP was needed for the advertisement of provider independent addressing to both ISPs and for influencing with path to use for inbound traffic. 

If the addressing is not provider independent then as you say this would have a big effect on what the final solution would be. 

Jon

[Richard Burts]

Jon

Thanks. Now we are on the same page. Certainly agree with the need to understand their address space and how it could be advertised.

HTH

Rick

[brettp]

You’ve brought up a good point with the provider independent IP addressing. Currently, we’re using IPs provided by our ISP. Ultimately, I would want to get a block of our own IPs which seems like it’s becoming a headache due to the fact there are no IPs to be had from ARIN. Of course, my director asked what about keeping our existing ISP provided IPs (not considering the headache of changing IPs if we change providers.) What would be so dramatically different if we kept our ISP IPs, got their permission, and shared them with the other ISP? Keeping the same typology I posted in that diagram. I’d imagine it could get a bit dodgy if the backup provider had summarized routes back to our active ISP.

[Jon Marshall]

The main issue is the ISP that owns the IPs will be advertising a summary to the rest of the internet which includes your IPs. 

If you could get the other ISP to advertise those IPs you would also need the ISP who owned them to advertise the specific block otherwise all traffic would be coming in via the backup ISP as they would be advertising the more specific route and that will always override any BGP attributes. 

You really need to talk your ISPs. 

I know that in Europe you can still buy IPv4 blocks although they are getting more expensive. 

Jon