Multihoming with BGP

[pschwalger]

Hi,

I currently have a single internet router which is multi-homed to 2 x ISP's.

Eventually we plan to add a second router for greater resilience etc, but just for now we have a "warm spare" router which we can use in the event of a failure.

Currently, we have 2 x fibre connections to ISP A - These are treated as Primary & Secondary, so we are not load sharing accross these links.

The connection to ISP B is configured as a secondary connection which was mainly due to bandwidth constraints; consequently we are prepending our our AS to ISP B.

We are also learning national and default routes via both ISP A & ISP B. Due to the bandwidth constraints of the ISP B connection. I have configured local-preference to influence my outgoing traffic to route out via ISP A.

Due to a recent outage affecting ISP A (caused by ISP A configuration changes) I had to shut down the BGP neighbour connections to ISP A and force the traffic over to ISP B. As a result of this outage we have subsequently increased our bandwith cap via ISP B and I am now wondering whether the Primary (ISP A) and Secondary (ISP B) is still the best operating mode.

To enhance resilience I am wondering whether I would be better off removing the prepending (via ISP B) and also removing or making the local-preference value for ISP B the same as the connection to ISP A (Primary neighbor) and letting BGP do its thing.

With the recent incident which affected the connectivity via ISP A I had to shutdown the connections (Primary & Secondary) to force the incoming and outgoing traffic to use ISP B and I would prefer that the BGP routing to be more dynamic and not needing to intervene manually.

I would appreciate anyones comments that is routing via 2 x ISP's in this mode and any other suggestions as well as the Pro's or Cons of changing from treating ISP B as a secondary connection.

thanks, Peter. 

[JohnTylerPearce]

Currently, we have 2 x fibre connections to ISP A - These are treated as Primary & Secondary, so we are not load sharing accross these links.

- Are you saying that from your router, there are two links going to ISP-A and a link going to ISP-B?

- If the above is correct, is the secondary link not used, unless the primary link is down?

Also do you have Provider Independent address space (sounds like you do)?

If your firewall is behind your router, before the ISPs come in, then you shouldn't have any weird issues with asymetric routing when it comes to firewalls.

What are the access speeds of the link to ISP-A and ISP-B?

You can use both at the same time by changing some BGP Path-Attributes, and then you may have to tune your AS_PATH prepending to make sure it comes back which ever way you prefer, or either way depending on speed.

When you had to 'shutdown' the connection to ISP-A to reroute traffic to ISP-B, how did this actually occur? Are you running IP SLA by any chance and how are your static routes configured?

Sorry for all the questions.

[pschwalger]

Hi John, thanks for your questions.

Yes, you are correct I have 2 x connections to ISP A and 1 x connection to ISP B and ISP B would only be used currently if both connections to ISP A are down.

ISP A made some updating of policers on the interfaces connecting to our router (without any prior notification) and effectively applied the wrong policy, which resulted in bandwidth starvation and packet loss.

I did some troubleshooting and had to manually to shutdown both ISP A's BGP neighbors which restored connectivity via ISP B. I had to do this as the BGP Peers to ISP A were both still up.

Yes, we do also have our own ISP independent network blocks and yes our firewall is behind our router so any asymetric routing would be confined to the internet router.

Previously our bandwidth capacity was greater through ISP A but since the last network event we have now increased the capacity via ISP B. The connection is effectively Gigabit to both ISP's with bandwith capacity assigned for National and International traffic.  For the purpose of this discussion lets say that speed & capacity via ISP A & ISP B are the same or at least very similar.

I am advertising a few Class C networks using an outgoing policy route-map that filters my local routes to both ISP's. I advertise the same routes to both ISP's and I have another route-map to prepend the routes via ISP B to effectively make it the "secondary ISP"..

As the bandwidth & speed of the links to both ISP A & ISB B are now effectively equal I am wondering whether I should still care about which ISP I currently manipulate the outgoing traffic via (local-preference) and which ISP the incoming traffic might take (i.e. would you recommend removing the pre-pending to ISP B)?

As you say I am interested in using both ISP's at the same time, possibly by allowing BGP to determine the best path or via configuration if you are referring to load sharing?

I still realise the majority of my traffic may still prefer to use one ISP over the other but I am keen to know what others have done and what their experiences are and what recommendations they might have for this scenario.

My knowledge iof BGP is a bit limited and I realise I should talk to my ISP's before I make any drastic changes at my end. any configuration examples would also be appreciated.

thanks, Peter.

[JohnTylerPearce]

Pschwalger,

   Thanks for the quick response. I think it would be a good idea to utilize both links in this scenario. I would start by

   running some sample traceroutes from each router to maybe 8.8.8.8 and various other IP's, and get a somewhat

   decent understanding, of paths that traffic takes as well as hops. (I realize this cannot be the most accurate way

   in the world, but you get an idea of the path that traffic takes from each ISP).

   I would then do a 'show ip bgp'  and look at the AS hops and get a good understand of the AS_HOPS for different

   networks on each ISP (You should see a * and a *> for each).  I'm not saying do this for every single prefix advertised to

   you from the Internet as that would be insane.

  As far as BGP Path Attributes, you could advertise out a /25 of one class C on ISP-A with AS_Prepend of three

  AS_PATHs, and then the other /25 of that Class C on ISP-B prepending three AS_PATHs. That way hopefully

  getting some sort of inbound load-balancing going on. Granted it depends on which Internet routers see what, as

  a lot only accept /21 and /22.

  For outgoing, you could configure the same /25 scenario, but inclue LOCAL_PREFERENCE (and or weight values if

  you have a Cisco router which I assume you do).

  Another way would be, let's say if you have two Class C networks, such as 100.100.100.0/24 and 200.200.200.0/24.

  You could do AS_PATH prepending on 100.100.100.0/24 on ISP-A and 200.200.200.0/24 on ISP-B, and the same with

  LOCAL_PREFERENCE

You could also just let them fight it out, but that might no get you the best results.

I hope this helps to maybe figure out a way that works best for you. I'll look at this thread in the morning, I need to get

some sleep.