BGP to OSPF redistribution with redundancy - A Problem?

[eiwanski]

I have an issue I would like to get some opinions or help on. We have a WAN that has primarily 2 ways to connect to each site. The primary is an MPLS VPN, but has 2 connections at 2 different locations (I'll call these R1 and R2) to the same VPN cloud for redundancy (essentially the first way for backup, but keeping the high speed network). The backup is a basic frame-relay low speed connection (I'll call this R3). Seperate from all of this, I have redundant internet connectivity that my 0.0.0.0 route follows. There's the simple background, here is my question:

I run OSPF for my IGP everywhere at our main central campus. For each remote site, I run their own instance of OSPF internally to them, in case that business unit is no longer with the parent company they are their own contained unit. The MPLS VPN has a BGP network in the middle.

So we have at the 2 different central sites (1 primary, 1 at a separate location for redundancy) we have an eBGP peering session with the MPLS provider. In order for our networks at the campus sites to take the correct path, I must inject these routes from the WAN into my OSPF cloud from either the primary if it is up, but also from my backup in case the primary goes down. I do this with redistribute commands from the eBGP received routes to go into my OSPF cloud.

The problem: Since I have 2 eBGP peering sessions, 1 at the primary site, and 1 at the redundant site, both will have a 'redistribute' statement from BGP into OSPF, just one goes through a route-map to set the metric much higher. Overall, this seems to work, Say a route from an external site goes away, gets deleted off the primary, then at some short interval later gets deleted from the secondary, it then goes away in my IGP (OSPF) and everyone is happy. But lets say it comes back, first at the secondary, and it then redistributes immediately into OSPF. When the primary gets the route back in BGP, it also tries to redistribute into OSPF, but it already SEES the route in OSPF and then doesn't seem to inject it! (like maybe a withdraw?) Is BGP doing this because it already sees the route in the IGP? Is redistribution doing this because it also sees it in the IGP?

In a nutshell, I need 2 different locations where a default route is not followed to receive routes via BGP (the same ones), and then inject these routes with different metrics (primary and backup) into our IGP, whether or not it already exists (let me worry about fat fingering the metric and having an equal cost here).

Any way to do this?

I will try to post a text pic in a reply, just didn't want to run out of space.

Thanks!!!!

[eiwanski]

Well the text pic looked fine with non-proportional font, but after post, it looked all messed up without the proper spacing. Let me know if you would like a pic, I can whip up and attach a Visio if desired.

Thanks again!!!

[vcjones]

My initial reaction is that you have a problem with administrative distance (iBGP is 200, OSPF is 110, and eBGP is 20). See http://www.cisco.com/en/US/tech/tk365/tk207/technologies_tech_note09186a0080094823.shtml

.

You also don't seem to have BGP correctly configured. An iBGP peering is normally required between all routers running BGP within the same ASN. Running iBGP, you could enforce your route preference within BGP by applying appropriate metrics (such as exit discriminator) to the eBGP peers. This would allow iBGP to distribute knowledge of which exit path should be followed rather than depending upon route redistribution into OSPF.

If all else fails, keep in mind that longer prefix trumps all route metrics, including administrative distance, so you could split the prefered path into two routes, each of half the subnet. That way, there would be no conflict with route redistribution, and all packets would follow the desired path. Setting this up if you don't have control of what advertisements are sent to your eBGP routers would be tricky and could require some fancy footwork with the route maps.

Good luck and have fun!

Vincent C Jones

www.networkingunlimited.com

[eiwanski]

Actually I do have an iBGP session between the 2 routers, but the key issue here is that I must be able to inject these 'external' routes into our OSPF cloud, as they are really our other business units, and from 2 different locations for automatic failover. The BGP sessions are working just fine, I have MEDs and Local Prefs set perfectly.

The issue really is when I need to take the BGP routes and put them into my IGP (OSPF) so that my central network knows how to get to these remote sites (either over the primary, or over the redundant). Since I would like to keep the cost incrementing, I've injected them with a type of E1 and at the primary site a low cost, and the backup site with a high cost. I don't know why that sometimes one of the sites does not take the BGP route (which is clearly there and working 100% as expected) and redistribute it into the IGP (OSPF), other than it is possibly seeing the route already being injected into OSPF from a different location (it is already there and in the routing table) so it does not inject it from the other location. This of course does not work when I have a failure and the network just doesn't know to send the traffic to the other router, even though all of the routes are actually there.

Does that make more sense?

Thanks!

Ed

[vcjones]

I'm too busy with paying work to try this out in the lab, but here is what I think is happening:

You are configured to learn the route in eBGP with the same metrics at both sites.

Your preferred site learns the route via iBGP and via OSPF.

Your preferred site then learns the route via eBGP.

BGP at your preferred site compares the iBGP advertised cost with the eBGP learned cost and sees they are equal.

BGP keeps the iBGP route as the preferred route because it was up first.

BGP declines to redistribute the route to OSPF because it is not the originator of the "best"route.

To fix this, you need to make sure that BGP on both systems see the preferred route as the BGP route with the best metric (keeping in mind that BGP has a long hierarchy of metrics, and moves down the hierarchy only if all higher metrics are tied).

Keep in mind that BGP has no idea of how OSPF metrics are done, and consequently has no idea if the route it holds is better than the route already in OSPF. Ditto for OSPF with regard to BGP metrics. Both have to agree independently that the preferred route is preferred.

A debug bgp events and debug ospf events (or whatever the equivalent debugs are for each) while bringing up the preferred BGP route might give you a better indication of where the blockage is. Depending on your connectivity, split-horizon could also do you in.

Good luck and good hunting!

Vincent C Jones

www.networkingunlimited.com

[eiwanski]

Just figured I'd post a followup for anyone interested since I think I licked the problem:

The exact scenario was that we run an MPLS net to our WAN, and they only support BGP as a routing protocol for the sites. We have 2 central sites (at different locations), that are well connected and using OSPF as an IGP. We have 20+ external sites that each inject 5-10 routes into the BGP cloud. The provider is running as-override, and route reflectors in the cloud. Everything in the pure BGP sense of the network works great, but we must inject these 'external' routes into our OSPF cloud so that we can get to them. This meant BGP<->OSPF redistribution at both the primary and secondary sites. Well, when say the backup router would get the BGP route first, and then redistribute it into our OSPF cloud, the primary router would then get that route via OSPF first. Then, quickly after, it would get the same route via BGP. Since there was already an IGP route learned, the BGP route was there, but it was not selected as the 'best' route, the IGP one was selected, and therefore it would never redistribute the BGP learned route into the OSPF cloud (we really want 2 OSPF routes in our internal IGP cloud, the primary route with a low cost, and the backup route at a very high cost). So we would maybe only get the backup route in our IGP, but never the primary again, until we soft cleared after the IGP route went away.

Solution: Set the BGP weights of the BGP routes (yes, be sure you filter the incoming BGP routes appropriately) to override your IGP weights. These default typically to 32768, anything over that number should work, they are only local to the router though, so make sure to set them at all your entry points where you want this to happen. Then a route learned from BGP will override the IGP route, get selected as the best, and then be redistributed into your IGP. Somewhat of a head scratcher, but very easy to fix once you trace exactly what is going on.

Hope this might help, but of course YMMV.

Ed