Using OSPF and BGP to dynamically choose egress points

[SlipperyPete]

Hi,

We have private WAN that has two connections to a larger ISP, one on the east coast, one on the west. Right now the two ISP connections are routed indpendently, with simple static routes pointing out, depending on which traffic takes which route. I'd like to take advantage of the two egress points and have traffic take the closest egress point, but dynamically fail over to the other egress point if one fails.

Without going into too many specifics, I think the way to do this is to run BGP on the edge and an IGP on the inside (such as OSPF). BGP would see the ISP as a single neighboring AS, with two separate paths to reach it. But how would you get OSPF to distinguish between the two? For example, networks on the east coast would probably have a default route pointing out the east coast ISP POP, so that any traffic not destined for internal OSPF networks would be routed out towards the closest ISP. Networks on the west coast would have the same, presumably pointing towards the west coast POP. But if one POP failed, and the BGP edge routers detected it and knew to fail over to the alternate path, how would internal OSPF routers learn about that and begin sending traffic to the remaining ISP POP?

Any guidance is appreciated. Thanks.

[lgijssel]

Your thoughts are correct. This would indeed require BGP.

You must establish full BGP routing with both ISP's, please note that this requires a lot of memory in your edge routers.

Your routers should run iBGP between eachother to exchange the best routes and for failover.

Both routers should also inject a default route into OSPF.

Regardless of where a packets arrives, it will be sent to the BGP router with the shortest path to the destination.

This is because the full routing table always has a more specific route than ospf default (0.0.0.0).

When one ISP becomes unavailable, the BGP routes will drop from the routing table and all traffic will be sent to the remaining ISP.

This is the scenario in a nutshell, hope this clarifies things for you.

regards,

Leo

[milan.kulik]

Hi Leo,

I'm afraid your solution would require to redistribute the full BGP routing to OSPF to make it working.

IMHO, full BGP routing is not necessary in this case.

It would be enough to ask the ISP to advertise the default route (nothing else) via eBGP  to both the customer edge routers.

And those routers would redistribute the default route from BGP to OSPF with type 1 metric.

This way, each client PC would take the closest egress point.

And of course in a case of one BGP session failure, all traffic would take the other router.

What is not clear to me is the incoming traffic though.

Is each cost NATing to other public IP range?

HTH,

Milan

[lgijssel]

Hi Milan,

No, I think it will not. A default route in ospf would be sufficient.

When a packet hits one of the BGP routers, there will be a more specific route pointing either to the local ISP or to the iBGP neighbor. Which one is primarily decided by means of as-path length. When these are equal, the packet will be sent to the local eBGP peer.

This is in fact a better solution than having a default route from the ISP because then you would have two equal cost routes to the Internet. I have assumed two different public subnets for the edge routers but CEF will normally ensure that sessions are routed over the same internal path. This also resolves the issue regarding the return traffic.

regards,

Leo

[milan.kulik]

Hi Leo,

"When a packet hits one of the BGP routers, there will be a more specific route pointing either to the local ISP or to the iBGP neighbor."

What happens when the more specific route points to the iBGP neighbor?

The router will send it to the iBGP neighbor on the other coast. I.e., it will forward it back to the customer network. In a case the full BGP table is not redistributed to OSPF, the packet (with the Internet destination still in the header) will be routed according to the default route inside the customer network and return back to the BGP router.

(See the well-known BGP synchronisation issue.)

You could survive this problem by building some tunnels through the customer network, but that would make the solution too complicated, I think.

There will always be two equal cost routes to the Internet (even with your design).

But metric type 1 would ensure the router inside the customer network would always use the closest egress point to the Internet (as the cost of the path toward the egress point would be added to the metric).

Hope to be clear now,

Milan

[lgijssel]

Hi Milan,

Obviously my solution (nutshell) requires the two iBGP routers to be peered which also means they can redirect packets to eachother. The question states that there are two ISP's. One on each coast.

This does not necessarily imply the two BGP routers are also far from eachother. In fact I have assumed a direct connection of 100Mb or faster.

My point is that the injection of a default route into the ospf domain in this scenario is sufficient with BGP making the final routing decision based on as-path length. I am convinced this is a vialble solution.

The only issues I see are concerning the possible asymmetric routing which may occur when traffic is rerouted internally. The return traffich will then come from the other Internet router and may follow a different path internally.

Whether this occurs or not will depend on the internal topology which can also be tweaked to make whatever scenario work. They can hire me as a consultant to work this out!

When assuming indirect peering between the iBGP peers (like you have done), the scenario indeed becomes a bit more complex but still solvable.

Nice to hear from you anyway, you are also around for quite some time already.

regards,

Leo

[milan.kulik]

Hi Leo,

funny how one problem description can be understood different ways :-)

Let's continue when more details are known.

BR,

Milan

[SlipperyPete]

Hi Milan and Leo,

Sorry for the late reply, but I've been busy with some other stuff and couldn't get back to this thread.

Thanks  for the good suggestions. From a high-level perspective, I'm following  what you guys are suggesting: I think a default route, injected from BGP  into OSPF, would work in our case, so our edge routers wouldn't have to  support the entire Internet routing table. Local routes would be routed  internally, via OSPF, and any external traffic would follow the  shortest path to the nearest ISP POP.

There are a lot more details that I'll have to delve  into as this gets planned out, but I'll go forward with this rough  framework. Thanks again for the advice.

Pat

[SlipperyPete]

Actually, I did have a follow up question, now that I think about it.

I understand that there's almost always an "edge" router running BGP, and that edge router is an eBGP neighbor with the ISP's edge router. But do you have to have separate physical routers as your edge router and your IGP border router? Why couldn't you have a single router, running OSPF on one interface, and BGP on the other interface, and redistributing right on that single router?

Thanks,

Pat

[milan.kulik]

Hi Pat,

sure you can (and have to in fact) redistribute on one physical router.

But I just realised another problem in the design:

According to

https://supportforums.cisco.com/message/3413260#3413260

"A default route cannot be redistributed into Cisco's implementation of the OSPF protocol."

That thread is talking about RIP/OSPF redistribution, but I'm afraid this is also a problem of BGP/OSPF redistribution.

So you would nee to use

router ospf x

default-information originate ...

command as explained in

http://www.cisco.com/en/US/customer/tech/tk365/technologies_white_paper09186a0080094e9e.shtml#t38

and possibly try to change the metric-type to 1 for the default route.

HTH,

Milan