Help to Prefer MPLS route over backup wireless bridge

[Daniel Hanson]

We have 2 sites (A and B) that each have their own MPLS connection through BGP. These are redistributed to OSPF, which then connects to the core switch at each respective site. We have recently installed a wireless bridge connecting site A and B that we would like to use as a "backup" connection, should the MPLS at one site fail.

Since we have no additional ports available on our routers, we have to connect the wireless systems to the core switch. Also, the idea is to establish redundancy, so terminating to the same endpoint (the routers) would create a single point of failure. We used a unique SVI (because one of the bridges terminates in a remote switch, Site A - SA1) to route traffic up to the core for each bridge connection. This also jives with our core redundancy at site A, which we'll eventually be utilizing for S2S VPN redundancy through a firewall/internet link, but that's a future problem.

We run OSPF currently between the cores (on the diagram as "SC#") and the routers (RW#). Site A uses OSPF area 0, and site B uses OSPF area 36. We have BGP to the MPLS, with both sites in the same AS.

Here's the rub: We want the primary connection to be the MPLS, and the wireless bridge to act as a backup. Since the bridge's network will currently be shared through OSPF as an inter-area route, it is still preferred over the E2 route learned from MPLS. How do we manipulate the routing to prefer the E2 route?

Things I've looking into, and why I'm concerned about them not really resolving the issue:

1. OSPF virtual-links. The problem here is that there is no intermediary area or router that the sites need to communicate through. Also, this doesn't necessarily resolve our problem, as the route (as I understand it) will still be preferred over the E2 routes from MPLS.

2. OSPF sham-links. My understanding of this functionality is extremely limited, but my belief is that this doesn't apply to our situation since we do BGP to the MPLS, not OSPF. If my understanding of this is incorrect, this might be exactly what we're looking for, I just need clarification.

3. Creating an additional OSPF process to redistribute between the existing OSPF instances, and manipulate the routes as they redistribute. I'm simply not comfortable with this (wouldn't these routes be inter-area routes anyway?) so any clarification on how we could use this would be appreciated. I've yet to find a clear example with any similarity to our issue.

4. Bring BGP down to the core switches and modify AS-PATH or local preference to manipulate routing. This causes additional routing calculations to occur on the core switches, which is not optimal. This would solve the problem, though, with the risk of over-tasking the core CPU's.

Please let me know if I'm overlooking something, or if there is a recommended method for this.

[Jon Marshall]

Ignore last post which I edited.

A lot of it still applies but not all.

So for all the sites using same AS under the CE BGP configuration -

"neighbor <PE IP address> allowas-in 1"

which means your CE router will accept routes with one occurrence of their own AS number in the path.

It can lead to routing loops in some scenarios but I have used it in a similar setup to yours and it worked fine.

In terms of advertising out each other's subnets if you got my last post about the issues you can face it is still relevant but it's not clear before that how you intend to advertise those subnets out.

So site A will receive BGP routes for site B from MPLS and it will also receive the same routes via OSPF from the backup link.

If you use OSPF to BGP redistribution on the CE devices then it depends on which comes up first ie. if the OSPF routes for site B are installed in the routing table on the site A CE device, before the MPLS routes for site B are received by the site A CE router,  they might well be redistributed into BGP in which case they will take preference because of the weight issue I was describing in my last post.

So site A will use the backup link to get to site B.

It all depends on the order of things.

If however site A receives the BGP routes from the MPLS router first it will use those but it won't then advertise site B's subnets to MPLS because the OSPF routes are not installed in the routing table and so not redistributed into BGP

So you won't  be able to advertise site B's subnets from site A until site B loses it's MPLS connection and stops advertising it's routes to site A.

Which is fine just means there will be a delay before site A can advertise them out.

Regardless of the above ie.which order things happen failing over between the MPLS and the backup link and then back again would definitely create problems because of the weight of the BGP routes.

Not trying to confuse the issue and not hopefully not telling you something you already know but this sort of redistribution using a backup link has these issues.

The solution to all the above is to make sure site A always uses the MPLS routes for site B when they are available by increasing the weight of those routes to be > 32768 assuming you use OSPF internally.

Obviously the same logic applies at site B in terms of weight manipulation.

Jon

[Jon Marshall]

Dan

Last post before Iog off.

Apologies for the confusion with the last two posts.

When you do IGP to BGP redistributon with a backup link like this it can cause the issues I was describing and we have seen this a number of times on these forums.

Like I say weight is the solution if you do use OSPF internally.

In terms of using BGP internally I haven't really given it much thought at the moment but if you want to discuss it  just let me know.

Jon

[Jon Marshall]

Dan

Just to let you know.

I had some spare time over the weekend so did a lab with a site C as well.

I came up with a solution using OSPF internally and BGP to MPLS that would allow you to advertise out both site's subnets (A and B) from each site and you would not have to do any BGP route modification, just some route filtering and redistribution.

The configuration needed is actually quite minimal.

But it relies on summarisation.

So lets say site A has this range -

192.168.0.0/24 ->  192.168.3.0/24 which can be summarised as 192.168.0.0/22.

If you could advertise these ranges as 192.168.0.0/23 and 192.168.2.0/23 to MPLS but as 192.168.0.0/22 across the backup link then it would work.

Basically you need to make sure whatever you advertise to MPLS is more specific than what you advertise over the backup link.

I ran a few tests and it worked but then the site was down so I haven't run full tests but I could do today if you thought it was worth it.

No problem if you have decided on something else etc., just thought I'd mention it as a possible option.

Jon

[Daniel Hanson]

Hi Jon,

Thank you for spending some time on this over the weekend! I didn't get a chance to dig into this as much as I had wanted to over the break, but we're back in the saddle now.

We advertise route summaries for 3 non-contiguous private networks (per site) through MPLS.

So, as an example, let's say the sites have the following private LAN networks:

Site A: 10.1/16, 192.168.1/24, 172.16.1/24

Site B: 10.2/16, 192.168.2/24, 172.16.2/24

Site C: 10.3/16, 192.168.2/24, 172.16.3/24

*= In addition to these, the only other routes advertised through BGP are those shared with the MPLS PE. Kind of simple stuff, but not to be over-looked.

Assuming I can get OSPF to share the same summarized routes, I'd like to be able to do this while still maintaining the same BGP route statements we currently have in place.

Sorry to make this harder!

[Jon Marshall]

Basically you can't.

The problem is that you want to advertise out the same subnets from both sites.

So looking at it from site A's perspective (same applies to B).

Site A advertises it's subnets to MPLS and site B advertises it's subnets.

Site A receives site B's routes from BGP and you want it to use those routes as the primary path.

So it can't then advertise those same routes back to it's PE for backup.

It can only advertise those routes when site B stops advertising them so you have introduced a further delay ie.

when site B goes down the PE connected to site B has to wait for it's BGP session to time out and then it withdraws the routes and notifies the other PEs.

This would have to happen whatever solution you come up using BGP.

Now if you were able to advertise out site B's subnets from site A when everything was up that would be the only delay because every other site would then be routed to site A for B's subnets.

But because site A hasn't been advertising B's routes because it couldn't until B stopped advertising them you now have another delay while the routes are advertised and propagated to all the PEs and CEs in other sites.

I can't say how long that extra delay would take but it is there.

If however you could do different summaries as in my last post then site A could receive site B's more specific summary routes and still be advertising out site B's less specific summary route because they are different routes.

Then each remote site has the more specific routes and the less specific summary as do the PEs in the MPLS network.

Like I say it's up to you and perhaps there are other solutions you could use but if your current setup does not work with a suitable design you need to think about modifying the setup ie. we are really only talking about a few summary route changes.

One last thing to be aware of and this applies to most solutions as far as I can see.

Both site A and B need to have routes via the backup link for the remote sites as well via MPLS.

This is because if site B is down for example and traffic comes in via A and over the backup link B must have routes back to the source IPs which are from the remote sites.

So if you are thinking still of using OSPF multiple processes be aware however many E2's you have now for remote sites you will double that number on your core switches because the other OSPF process also needs them.

Jon

[Daniel Hanson]

yeah, I kind of stopped in the middle of my thought with that last post (had to run to a meeting), but yeah, I understand Site A will need to advertise routes for Site B's subnets. I apologize for sounding silly there for a second, and confusing this.

[Jon Marshall]

No problem.

The issue you have is that this type of setup where each site advertises it's own and the other sites subnets works if the direct link between the sites is the one used for the sites to communicate with each other.

But you don't want that, you want site A and B to use the MPLS link if it is up which means you can't then advertise the same subnets from both sites because of what I described in my previous post.

Something is going to need to change with your current setup if you want to advertise both sites subnets from both sites at the same time.

Jon

[Jon Marshall]

Dan

One thing I didn't do was filter when I redistributed between OSPF processes.

So a site receives the other sites subnets via BGP redistributes into the current OSPF process and then I redistributed into the new OSPF process.

I forgot to filter the remote site's subnets so they get advertised back to the other site via the backup link.

Still worked but created quite a large OSPF database :-)

Let me do the filtering and I'll get back to you.

Jon

[Jon Marshall]

Dan

Added the filtering and it definitely reduced the OSPF databases on each switch.

You can filter on the prefixes or because BGP adds a tag to the redistributed routes you could filter on those ie. don't redistribute any routes with the BGP AS as the tag.

So it still works but I am just trying to work out why it prefers the MPLS network every time until I shut down the connection.

It's what you want but it doesn't necessarily mean it will happen.

The metrics are the same, the only real differences being the advertising router id and the tag with the redistributed BGP routes.

The path selection for E2's is a bit more complicated than most explanations suggest so I need to make sure if you do need to alter the metrics it is an easy thing to do otherwise it is pointless.

One last point for now is that obviously your OSPF database is larger on each core switch now so you need to factor that in to any decision you make.

You may decide not to use this but it would be good to know it would work if needed.

Jon