How would you approach this BGP Multihoming design? Please critique my plan

[JustinSandersHylife]

I'm about complete on a bgp multihoming implementation and would appreciate some ideas on the last piece before completion. Basically we have two ISP's offering drastically different speeds ( due to remote location, there is not much choice). Given this limitation the requirement is to send all traffic IN|OUT of ISP A, and use ISP B as standby backup. We have registered our own AS and IP block with ARIN. 

See attached topology. 

The two routers will be an HSRP group. Static route pointing to primary router from the firewall. So I have two options and a debate between a consultant and myself. Assume all other BGP route maps, prefix lists etc.. are configured correctly to route primarily to and from A.

Option 1: Each router establishes a BGP session to each ISP. (Router 1 - ISP A, Router 2- ISP B) . HSRP tracks some sort of SLA, and shuts down interface on router 1 if that SLA is not met. Router 2 takes over as primary router and forwards accordingly. Router 1 preempts when condition clears. 

Option 2: Each router establishes a BGP session  to BOTH ISPs. HSRP will not track based on SLA but will only take effect if Router 1 completely fails. Else if there is a problem with ISP A, router 1 will forward to ISP B after BGP session to ISP A dies.

Given that the main requirement is to minimize any downtime what are the benefits or drawbacks of either scenario and which would you choose?  Does the bgp router id have any effect on this decision? (I.E, the router ID is either going to be an address that will change under certain conditions, or an address that isnt actually assigned to an interface.)  

[paul driver · Accepted solution]

Hello
Looking at your topology, possibly applying something like this would be applicable: :

1) create Ebgp peering to each isp 1 and 2 from each respective rtr, advertised local networks to each isp

2) create ibgp peering between rtr1 and rt2 and specify next-hop-self command to between each neighbour

3) apply as-path prepending to rtr2 and  isp2 so to manipulate incoming traffic to take isp1 and rtr1 as preferred path.

4) apply local-preference lower than 100 to rtr 2 so outgoing traffic will take rtr-isp1 path ( default is 100 - higher value take precedence)

5) apply as-path filtering outbound towards each isp on local routes only  (as-path access-list x permit ^$) so negate your rts from becoming transit path between the two isp's

6) apply hrsp and  ip sla tracking on both rtr1 and 2 to monitor the reachability of each its respective isp connecting interface then apply standby 1 track 1 decrement 60 ( change numbering to whatever your wish) to each hrsp interface and give hrsp priority to rtr1and isp 1( higher value more preferred)

7) Give the Fw a default route to the vip ip address of the hrsp from Rtr1 and Rtr2

res

Paul

 

[JustinSandersHylife]

Hi Paul, 

Thanks for your input. I have already completed all of the steps above with the exception of the HSRP group and the iBGP peering. 

It seems like your plan is mostly the same as mine but for some reason or consultant (who has much more experience than I) seems to think I am better off having both routers peer to both bgp neighbors and HSRP only kicks in when there is a total router failure. 

Just wondering if anyone else agrees with this? All of the documentation I have found everywhere has each router peering to 1 ISP only.

Wasn't planning on doing the iBGP part. What benefit do I get from that in this scenario? 

[paul driver]

Hello

Do you have dual links to each isp?
Have you queried his/her reasoning behind this idea? -

res

Paul

[JustinSandersHylife]

Consultant hasn't given me a valid reason other than it is easier and there is little benefit to do it any other way. I disagree. 

Thanks all for your input. This (WAN Stuff) is all relatively new to me and I appreciate the help. 

[Jon Marshall · Accepted solution]

Like Paul I am not seeing why you need to peer with both ISPs from each router.

Unlike Paul (sorry Paul :-)), I also can't see what you gain from peering your own routers with BGP. Because you are only using the second ISP as a backup then you don't need to influence which path your outbound traffic takes per route so it seems unnecessary. 

In addition if you didn't run it then you would not need to worry about becoming a transit path for both ISPs because your routers are not exchanging BGP routes.

That said if you think at a later date you may want to use both links then it may be worth setting it up.

You do obviously need prepending when you advertise your local subnets to both ISPs to influence inbound traffic.

My general motto is to keep things as simple as possible and only complicate it when it needs to be.

However I may be missing something because Paul is both very knowledgeable and very experienced and he may well be seeing something I am not.

I would also be interested to hear what the consultants thoughts are on why it is better to have both routers peering to both ISPs.

Jon

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

You note your main requirement is to minimize any downtime, yet your one switch a single point of failure?

Is the firewall a single point of failure?  Does firewall only have single link to switch?  Does firewall support any dynamic routing protocol?

How drastic are the bandwidth differences between your two ISPs?  Do plan on any outbound QoS?  If so, how QoS rich is the switch?

Do your routers support mHSRP?  Do they support OER or PfR?

[Jon Marshall]

Joseph

QOS/mHSRP/OER/PfR - and here was me saying keep it simple :-)

And yes I am just joking, all good questions.

Jon

[Joseph W. Doherty]

Jon, even if you weren't joking, I often find more advanced technical solutions simpler than what needs to be done without them.  (Of course, there's is the learning issue!)

For example, ever work with someone who doesn't know how to use a dynamic routing protocol, but they try to do what dynamic routing does with static routing?  I.e. if we just adjust this static metric here, and use HSRP here and use interface tracking here, and . . .  Then, see there's one done, we now only have another 99 routers to configure.  (laugh)

[Jon Marshall]

Joseph

I'll rephrase my answer as it didn't really get the point across.

When I talk about simple I am not referring to the technology used but the solution. Your rather extreme example of static routing illustrates this perfectly so we are in no disagreement here.

The OPs original request was to use the second link as a backup and how to do that in relation to BGP which was what my answer was directed to.

I think I just need to remember humour is a tricky thing on forums and stick to the technical stuff :-)

Jon

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Jon,

"I'll rephrase my answer as it didn't really get the point across."

Perhaps true for us both.

"When I talk about simple I am not referring to the technology used but the solution."

Actually, so am I.

With my example of static routing vs. dynamic routing, my point was, dynamic routing might be considered more complex than static routing, technically, yet its additional complexity might actually make the overall solution less complex.

So, likewise, perhaps some additionally technical complexity, e.g. OER, mHSRP, etc., in the technical approach might actually make the overall solution actually simpler.

If that wasn't clear, the fault is mine.

"The OPs original request was to use the second link as a backup and how to do that in relation to BGP which was what my answer was directed to."

I also agree about OP's original request, and there's nothing wrong in directing an answer to the OP's actual question.  Most OP's are looking for just that.

However, I sometimes look to see if something might be suggested that's even more suited to what the OP's problem is.  Sometimes I'm able to suggest an alternative approach that hadn't even been considered.

In this case, I wondered whether both ISPs might be used actively, and whether that had been even considered.  It may have been, but often when working with BGP, it isn't because it's often difficult enough to even use ECMP with BGP let alone unequal cost.

"I think I just need to remember humour is a tricky thing on forums and stick to the technical stuff "

Regarding humour being tricky, I didn't take any offense from anything you wrote, and I hope you haven't taken offense from anything I've written.

Sometimes convening anything, with clarity, is also tricky.

[JustinSandersHylife]

I appreciate your feedback. 

Firewalls will be clustered, switch will be stacked. I just didnt draw that into the topology as it wasnt really part of my question. 

Bandwidth from one ISP is 100mbps The other link is limited to 10mbps. We don't have much choice in providers in this area. 

Worth mentioning the 10mpbs link was adequate but slow for the most part so it's a viable fall back option but useless for load balancing.  

This is all being used with existing hardware with no budget in the near future ( less than one year away)

QoS is something that would be nice to have. I doubt a 2960 running lanbase will have much to offer and I havent really looked into it fully yet. 

Until this post I hadn't heard of mHSRP, OER, or Pfr so I can't even comment on that.  

Like Jon mentioned, it would be ideal to keep this simple as I can't be the only one in the organization that knows how to support it and there isn't a lot of talent pool available if I am no longer here.  Unless these technologies will provide a significant improvement, they probably shouldn't be considered.  

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

"I just didnt draw that into the topology as it wasnt really part of my question."

Indeed, but those of us hoping to help having incomplete information precludes giving the best answer.  Also, having a better understanding of the situation sometimes allows offering a suggestion that might have been totally overlooked.

"Like Jon mentioned, it would be ideal to keep this simple as I can't be the only one in the organization that knows how to support it and there isn't a lot of talent pool available if I am no longer here.  Unless these technologies will provide a significant improvement, they probably shouldn't be considered. "

Well as Jon also noted, he was joking, but "simplicity" and support are important points.

"Simplicity" is somewhat in the eye of the beholder.  I personally like the simplest solution, but the simplest solution that meets all business goals.  Goals can be conflicting.  In your case, perhaps providing the best performance vs. supporting it.  I cannot say which is really more important to your business.  Often support of, or implementation of, unknown technologies can bias the person designing a system.  In such cases, I recommend trying to analyze the impact of new learning, and supporting it, and its advantages to the business, and then presenting that to management and let them decide what they want.  (Personally I don't believe new technologies should be used just because they can be, but only if there's a benefit to the business, and even then, the benefit needs to be compared to cost.)

"Bandwidth from one ISP is 100mbps The other link is limited to 10mbps. We don't have much choice in providers in this area. 

Worth mentioning the 10mpbs link was adequate but slow for the most part so it's a viable fall back option but useless for load balancing."

Ah!  Well the reason I asked, OER/PfR can do dynamic load balancing and can manage and monitor performance across different bandwidth links.  (Once I even used it to control a T3 and T1.)  However, the reason I asked, was if bandwidths are not too far apart, I would initially set traffic 50/50 and let OER/PfR rebalance as needed.  With 100 to 10, I would start with sending everything to the 100 link, and let OER/PfR use the 10 link, if it saw an advantage in doing so.

(BTW, mHSRP is nothing more than HSRP with different virtual IPs per interface.  It would allow your firewall to have a static route to different virtual IPs, allowing static route load balancing.  Yet, if one of your routers failed, the remaining router would pick up the other router's virtual IP.)

The reason I asked about QoS on the switch, if both router's have their own session to each ISP, depending how the firewall sends them traffic, each could send traffic to the same ISP, creating possible congestion on the switch port.

The reason I asked about the switch itself, and firewall links, depending on actual firewall, switch (stack) and router interconnections, different failures will have different impacts which can impact the logical design, i.e. your peering questions.

Anyway, so the firewall is a cluster.  Each unit has a link to a different switch stack member?

Each router only has a single link to the stack?  If so, each router links to different stack member?

Just one link to each WAN provider?  Also each on different stack member?