BGP load balancing!

[kawin]

Okay BGP gurus, here's one for you:

Currently, we have 4 T-1's going into our 7206VXR Router. We just turned 2 of the 4 up this past week.

The new 2 T-1's go to one ISP, while the 2 others go to 2 separate ISP's. We run BGP between all three ISP's. The new ISP is providing us with full routing tables for the 2 T-1's. (We have 512MB of memory, plenty for this scenario). Another ISP is announcing just their routes to us, while the last ISP (which we have problems constantly with) is announcing just their routes as well. I believe we'll eventually want the other 2 ISP's to announce to us full routing tables as well, which would be ideal.

Here's the problem:

The configuration of BGP is exactly the same for the two T-1's from the new ISP, so apparently most/all traffic is going to one T-1 since they're distinguished only by IP address. The ISP provided us with a configuration to help load balance/load share the two T-1's, but they do so by spliting up our IP blocks equally. This assumes that half the traffic will go to the fist half of the block, and so on.

The other option is to mess with the attributes for BGP, which I'm quite taken aback with, since many documents says tweaking the BGP attributes is very risky.

The larger problem is load balancing among all four T-1's.

Any help would be greatly appreciated!!!

Thanks,

Kawin

[Gilles Dufour]

if you have 2 T1 from 1 of your router to 1 router of your isp (sth like R1 ===== R2), configure a loopback address on both router and establsih a bgp session between the loopback.

Then, make sure R1 and R2 knows each other loopback over the 2 T1 (you can use 2 static routes - 1 for each T1).

This is the best way to achieve loadbalancing for this scenario.

If your 2 T1 are going to different routers at your ISP, then you won't be able to control incoming traffic.

For outgoing traffic you can use the command max-path 2 to loadbalance over the 2 T1.

[kawin]

Hi,

Thanks for the reply. Unfortunately, our 2 T-1's going to different routers at the one ISP. There is no way to change the attributes on BGP to control both incoming and outgoing traffic?

Thanks,

Kawin

[kawin]

Oh another question for you:

When I issue the command max-path 2 to loadbalance over the 2 T-1's, is it possible to apply this to my overall bandwidth which includes all 4 T-1's?

[chen__nan]

If you issue the command max-path 2, for a single prefix/route, bgp will select 2 best route in the bgp table inject to the routing forward table, certainly, if you recieve this prefix/route from all 4 bgp neighbors, it's really possible.

you can use local preference to control your outbound traffic, for inbound traffic, you should negotiate with your isp, maybe they permit you use MED to control your inbound traffic policy.

If you have some address block really want to be accessed by outside use a particular link, you can advertise a more specific route out of this bgp session, they will choose this path to get your address. But this is not a recommended way because this maybe conflict with your isp's policy.

[velimirmkd]

Hi,

First of all you have to differentiate between incoming and outgoing traffic.

For the outgoing traffic you can use the max-path 2 as somebody said in a previous reply, and that is pretty easy to control since it is traffic that originates from your side.

For outgoing traffic is a little bit trickier in your situation.

You can use MED as someone suggested but MED is only indicator/sugestion to the neighboring router which route you preffer, and as previously mentioned it will take cooperation from your ISP. Anothier way is to play with what you advertise to whom. For example if you have lets say to IP ranges you can advertise one of the ranges to one neighbor and the other to the other neighbor. This way you'll load balance between the two connections but only with specific traffic for that connection. But I'm not sure if this is what you want to achieve. Another thing you have to check is to make sure that your ISP has set same prefference on both routes to you, since you are connected to different routers on the ISP side assuming that the ISP has one AS to which you are connected.

But the easyest way to fix this would be to ask the ISP to put both your T1's on the same router and to use the loopback interface on both sides.

Hope this helps

Velimir

[kawin]

Hi Velimir,

Here's a copy of my bgp config without IP addresses:

router bgp (my AS)

no synchronization

bgp log-neighbor-changes

network x.x.x.x mask 255.255.255.0

network y.y.y.y mask 255.255.255.0

network z.z.z.z mask 255.255.255.0

neighbor a.a.a.a remote-as (ISP#1 AS)

neighbor a.a.a.a version 4

neighbor a.a.a.b remote-as (ISP#1 AS)

neighbor a.a.a.b version 4

neighbor c.c.c.c remote-as (ISP#2 AS)

neighbor c.c.c.c ebgp-multihop 5

neighbor c.c.c.c update-source Serial1/1:1

neighbor c.c.c.c version 4

neighbor c.c.c.c distribute-list 1 out

neighbor c.c.c.c route-map (routemap) in

neighbor c.c.c.c route-map (routemap2) out

neighbor d.d.d.d remote-as (ISP#3)

neighbor d.d.d.d version 4

neighbor d.d.d.d distribute-list 1 out

Now can I issue a "maximum-paths 4" to load balance among all 4 lines for outgoing traffic?

Notice in my config that for ISP#1 I have neighbor statment as a.a.a.a and a.a.a.b, thus each T-1 line for the ISP#1 is going to a different router on their end.

[velimirmkd]

Hi,

First of all lets try to answer your question about the max-path 4. To my knowledge in your case it wont do full load balancing since you are using distribute list on some of the peerings. That means that you are letting those systems to know only some information about your network. You are probably also filtering or adding weight with your route-maps to your peer c.c.c.c. All this will influence the load balancing when used withmax-path. I'm not sure but be carefull with that.

Second, looks like you are doing full exchange with your ISP#1. If you want to do load balancing between all four peers, I think that you'll have to do some modifications on your BGP settings, or you can do separate load balancing with ISP#1 and separate load balancing with ISP#2 and ISP#3.

Lastly, find out how your network is seen from the outside, do a BGP scan, here is a link to help you with that http://nitrous.digex.net/ this will give you some idea of how your network is seen/reached from the outside. Now if you see that the best path to some of your network is though ISP#1 and the second best is through your ISP#2 but with more hops for example, in that case you'll have to use prepend.

Since BGP best path selection uses number of hops as second criteria changing the max-path or local pref or MED will not do the job.

These are just some consideratoins you should have in mind

Hope this helps, correct me please if I'm wrong

Velimir

[nuno.morais]

Hi.

Concerning your 2 T1s connecting to the new ISP, if the 2 T1s are connecting to the same ISP router, you may configure the peering using the loopback addresses (don't forget to use ebgp-multihop and any static routes or dynamic routing to let your router know how to reach the ISP's router loopback) load balancing will be achieved not by BGP itself, but via the forwarding process. It may vary, according to the forwarding process you area using. As an example, if you are using CEF, you should enable per-packet load-sharing to have a fair load balancing between the 2 circuits.

Regarding the other 2 peerings, as someone already wrote, it is tricky... You can use local-preference to influence your outbounf traffic and/or agree with your ISPs some MED or BGP community policy to influence inbound traffic.

I hope it helps.

Rgds.

Nuno Morais

[kawin]

Unfortunately, my 2 T-1s are going to separate routers to the new ISP. I'm going to find out if they can put the T-1's on one router. That way, I can implement your advice.

Also, should i use the max paths 4 command to balance between all 4 of my T-1's?

I don't have CEF, but I've been reading up on it, is it hard to implement? Any consequences regarding implementing this layer 3 switching technology?

Thanks for all the replies!

Kawin

[ruwhite]

If you're running a VXR, you're probably already running CEF. It should be on by defualt.

:-)

Russ

[kawin]

Hey,

It's not on, just checked. If it's on by default, looks like the original admin turned it off.

Should I turn it on?

[nuno.morais]

Hi.

Max-Paths only have a word on the BGP selection algorithm if reaches a certain point. Please check the following link:

http://www.cisco.com/warp/customer/459/25.shtml

What I mean is, there are a lot of things that BGP checks before deciding based on max-paths if they are configured. BGP by default doesn' to load balancing.

Regards,

Nuno Morais

[Gilles Dufour]

the max-path command only works for route that you learned from 1 ISP over multiple link.

So, in your case only 2 T1.

There is a new feature called eibgp multipath but not sure if it is available yet and if it could do anything good in your case.

For the outgoing traffic, the only thing you can do is control what routes you advertise and the as-path (you can use as-path prepend to make a route look worst).

I really suggest you enable CEF. Lot of features require CEF to be enable nowadays.

To enable CEF :

ip cef ! global command

interface XXXXXX

ip route-cache cef

!

[kawin]

Hi,

So what you're saying is that if I put in the command

"maximum-paths 2" into my bgp config, then the router will balance outgoing traffic through the 2 T-1's going to the same ISP? The command will not affect the other 2 separate T-1s?

As for CEF, unfortunately, it will only work if the ISP implements CEF on their end. I already called our new ISP with the 2 T-1's, and they do not use it, and will not turn it on for our sake.

Thanks!

Kawin