Help to Prefer MPLS route over backup wireless bridge

[Daniel Hanson]

We have 2 sites (A and B) that each have their own MPLS connection through BGP. These are redistributed to OSPF, which then connects to the core switch at each respective site. We have recently installed a wireless bridge connecting site A and B that we would like to use as a "backup" connection, should the MPLS at one site fail.

Since we have no additional ports available on our routers, we have to connect the wireless systems to the core switch. Also, the idea is to establish redundancy, so terminating to the same endpoint (the routers) would create a single point of failure. We used a unique SVI (because one of the bridges terminates in a remote switch, Site A - SA1) to route traffic up to the core for each bridge connection. This also jives with our core redundancy at site A, which we'll eventually be utilizing for S2S VPN redundancy through a firewall/internet link, but that's a future problem.

We run OSPF currently between the cores (on the diagram as "SC#") and the routers (RW#). Site A uses OSPF area 0, and site B uses OSPF area 36. We have BGP to the MPLS, with both sites in the same AS.

Here's the rub: We want the primary connection to be the MPLS, and the wireless bridge to act as a backup. Since the bridge's network will currently be shared through OSPF as an inter-area route, it is still preferred over the E2 route learned from MPLS. How do we manipulate the routing to prefer the E2 route?

Things I've looking into, and why I'm concerned about them not really resolving the issue:

1. OSPF virtual-links. The problem here is that there is no intermediary area or router that the sites need to communicate through. Also, this doesn't necessarily resolve our problem, as the route (as I understand it) will still be preferred over the E2 routes from MPLS.

2. OSPF sham-links. My understanding of this functionality is extremely limited, but my belief is that this doesn't apply to our situation since we do BGP to the MPLS, not OSPF. If my understanding of this is incorrect, this might be exactly what we're looking for, I just need clarification.

3. Creating an additional OSPF process to redistribute between the existing OSPF instances, and manipulate the routes as they redistribute. I'm simply not comfortable with this (wouldn't these routes be inter-area routes anyway?) so any clarification on how we could use this would be appreciated. I've yet to find a clear example with any similarity to our issue.

4. Bring BGP down to the core switches and modify AS-PATH or local preference to manipulate routing. This causes additional routing calculations to occur on the core switches, which is not optimal. This would solve the problem, though, with the risk of over-tasking the core CPU's.

Please let me know if I'm overlooking something, or if there is a recommended method for this.

[Jon Marshall]

This may not be a solution for you but sometimes the simplest answers are the best.

What about a floating static route on each core switch pointing to the wireless connection.

The static could be a default route or if that is in use you could use a summary route assuming each site can be summarised.

If the MPLS connection goes down the BGP routes are no longer received and redistributed into OSPF so the floating static takes over.

And if the link comes back up the OSPF routes take over again.

I am not a fan of statics everywhere but sometimes they can be quite useful.

Jon

[Daniel Hanson]

Hi Jon,

Thank you for your input. I hadn't considered static routes, mostly due to the fact that I have two cores at one of the sites, and we're planning a tertiary backup connection, which would further complicate the static configuration. I'm against this option, but I will add it to the possibilities.

Very Respectfully,

Dan Hanson

[Jon Marshall]

Dan

No problem, just a suggestion.

One other thought that I suspect you won't be a fan of :-) is with option 3) you seem to be suggesting you would be happier to run an additional OSPF process on each switch rather than extend BGP to the core.

I don't know whether option 3 would work, would need to lab it up and try out a few things but if the devices are all Cisco you could run EIGRP on the backup link.

If you did a redistribute connected on the core switches then the EIGRP routes would be external ie. AD 170 and so your OSPF routes would be preferred.

If you then added a tertiary backup connection you could run EIGRP on there as well and use delay to manipulate which of the backup links took preference.

Again I should stress I am not a fan of overcomplicating things and it would mean running another routing protocol (if supported) but just thought I'd mention it as another option.

Jon

[Daniel Hanson]

Another good option...but I can't deploy EIGRP purely because we want to stay open to additional hardware options. I know EIGRP is now an open standard, but only in a stub configuration. If we end up implementing something with the Cisco proprietary commands, it locks us in to their hardware. That's the reason we run OSPF as our IGP.

Believe me, if it was strictly my choice, I'd go EIGRP (way easier to manipulate than OSPF) but there's a pretty solid reason not to.

Thanks again for another really solid idea, but sadly I can't go that route either.

[Jon Marshall]

Again no problem.

With option 3) what were you thinking of ie. creating a new OSPF process on the core switches and redistributing the existing OSPF process into it ?

If you redistribute they should be external routes but I'm not totally convinced it would work.  

I do have an MPLS lab scenario setup so I could test this if you provided more details.

As I understand it the wireless connection is simply terminated on each core switch with an SVI in the same IP subnet.

Is that how it is setup ?

So in effect it is just like having a direct L3 P2P connection direct between your core switches ?

Jon

[Daniel Hanson]

So here's the theory:

1. Yes, we have SVI's on each core in the same subnet, and the wireless bridge exists in that VLAN.

2. The network between them is not shared by any of the summary-routes that are being used to advertise each particular site, so this network can be isolated into it's own range. for example: if Site A is 10.10.0.0/16 and B is 10.20.0.0/16, the SVI is 192.168.30.0/29.

3. With an additional OSPF instance, my understanding is that it would consider all routes redistributed into this process as external routes, and we could manipulate them accordingly. With an appropriate cost manipulation, you might be able to make the route less preferred over the E2 route learned via the MPLS path. I'm not 100% sure this would work, but it's simply an idea.

I unfortunately don't have a lab to test this, so if you have the time/patience to play with this, your results would be greatly appreciated.

If you need any additional details, please let me know what kind of information you need, and I'll supply anything I can.

[Jon Marshall · Accepted solution]

Dan

Okay, did a quick test.

Just so we are using the same setup -

1) I assumed you were not redistributing OSPF into BGP only BGP into OSPF otherwise you would get some very funny results and you would need route filtering

2) the SVI connection - both ends needs to be in the same area for the adjacency to form so I put them both in area 0.

I used loopbacks on the switches to emulate local vlans/IP subnets and I only used one L3 switch at each site.

So I removed the network statement from the OSPF process for the SVI subnet and created a new OSPF process on each switch and then added the SVI subnet to that process.

Then I simply redistributed one way from the current OSPF process to the new one.

It worked fine and actually it preferred the MPLS E2's without any modification of metrics etc.

I then shut the interface of one of the CE routers connecting to the MPLS network and once BGP worked itself out it failed over to the backup link.

Then brought it back up again and it failed back, again reliant on BGP more than anything else.

With each test I pinged between loopbacks on the switches and they worked.

One thing to note was that because I only did one way redistribution the CE routers never get the routes via the backup links which I didn't think would be a problem because it is the switches the traffic hits first and it should be routed from there.

Finally all of this was done using an emulator not real kit but the routing aspects of the lab are pretty reliable.

I can't guarantee the solution and you may need to play around with metrics even though I didn't have to and your setup is probably more complicated than what I used but initial results look promising.

I can save the lab so if there is anything else you want me to test or try other things only too happy to try them out.

Edit - my tests were rather basic so if there are other failover scenarios you want me to test just let me know.

Jon

[Daniel Hanson]

We redistribute both ways (not my design!) and use route-maps to filter advertisements out per site. I'm not certain how this will affect the overall design, or simply adds the complexity of the route filtering to ensure cleanliness in advertisements.

Are you using GNS3 as the emulator?

Also, just wanted to clarify that we're on the same page for the design: Both sites have their own legs into the MPLS, and we want both sites to prefer that connection over the bridge unless one or the other MPLS connection is down. For example: Should Site A drop MPLS, we'd want it to fail over to the bridge, and use Site B's MPLS as the preferred route for the supernets at Site A. Is that how yours is working? Theoretically, your MPLS PE should have both routes available, and simply prefer Site A's shared routes for Site A's supernets, and the inverse for Site B's routes.

If that's the case, then the question becomes:

Which would make this more complicated or process consuming? an additional OSPF process or bringing BGP to the core?

Your help is greatly appreciated, sir, and I'm truly appreciative of your time and efforts thus far. =)

 This design has been a thorn in my side for some time now, and in lieu of a really complicated PBR configuration, I couldn't get a viable script prepared to implement this. There's gonna be a lot of guessing when this gets implemented, but anything I can do to reduce that is HUGE in building confidence in the change.

[Jon Marshall · Accepted solution]

Okay, that complicates it but doesn't necessarily mean it won't work.

So when you redistribute into BGP you only redistribute the local subnets making sure any subnets received from the backup link are not then advertised back to the MPLS cloud.

That shouldn't be a problem because I was only redistributing one way ie from the main OSPF process to the new one.

In fact that means that your CE router never sees any of the routes via the backup link (which it does currently) because the CE router is only in the current OSPF process and not the new one.

Not that I am recommending removing the filtering.

You definitely need to filter when you redistribute from the current OSPF process to the new one but it is relatively easy to do.

I'm not sure of the additional overhead of another process compared to extending BGP to the core and I'm not sure at the moment which fits better with the tertiary connection you mentioned.

What I may do over the weekend is get the lab up again and check exactly why it is preferring one path over the other and make sure you can influence it otherwise like I say it is not really a solution for you.

And obviously bear in mind I literally had a switch and router per site and that was it in addition to an MPLS network between the sites so you probably have more complexity in your network.

Basically it's just another option for you and it does seem to work but like I say I was just shutting down an MPLS connection and it was all quite basic so no guarantees.

The emulator is not GNS3 unfortunately which I used to have a while back.

I get access to a training site Cisco have due to my participation on here and it has an emulator you can use.

Jon

[Jon Marshall]

Just seen the additional part of your post.

Not sure I follow the bit about if site A loses it's connection you want it to use the backup link but use site B's MPLS as the preferred link to site A's supernets.

If site A loses it's connection then it is no longer advertising it's networks to site B so site B won't see a path back via MPLS.

Site B will also have to use the backup link.

Can you clarify what you want to happen ?

Edit - in terms of BGP vs OSPF can't really say in terms of performance to be honest as I have never run BGP on L3 switches before although I know people do it.

So it is a factor but I think the more important consideration is which one fits the design better especially when you consider the third link.

You have already said OSPF is not as flexible as other routing protocols for some things (and I agree) whereas BGP is designed to give you flexibility over which paths traffic takes, that's it's main strength.

It is slower to converge but you are quite reliant on it anyway because of your MPLS connection.

Difficult to give a definitive answer as to which to choose without understanding the whole network and how you want everything to work when all the links are in place.

Perhaps others could comment or you could start a new thread on the specific topic of BGP vs OSPF resource usage and link back to this thread to get other inputs.

There are a lot or experienced people on these forums and I'm sure someone would be able to help with that.

Jon

[Daniel Hanson]

Ok, so my bit about the Site A vs. Site B advertisements...Let's introduce site C.

Site A, B and C are all on the same MPLS. C knows the best path to 10.10/16 is through site A, and 10.20/16 is through site B.

If Site A's MPLS goes down, site C still needs to know that 10.10/16 is available through site B, and vice versa for 10.20/16.

To verify this, we'd need to check the received routes at the PE and advertised routes at the CE. We'd need to know the PE is forwarding the routes to basically say "here's two paths to 10.10/16, and Site A is the better metric."

I think we might end up just doing the BGP option, especially due to the added complexity we're looking to introduce within the next few months.

Again, really, I appreciate your input and efforts related to this. Glad to have your help!

[Jon Marshall]

Okay, I assumed you only wanted to use the backup link for these two sites, I didn't realise that you also wanted to use it for other sites as well.

Definitely the way I set it up would not work because I did not know you wanted sites A and B to advertise out each other subnets as well so I only redistributed from the main OSPF process to the new one which meant the CE routers only had the other sites subnets via BGP not via OSPF from the backup link.

So are you currently advertising site As from B and the other way round how are you manipulating BGP so that the direct link is used if the site is up ?

In theory you could still do this with OSPF but you would need to redistribute both ways between the OSPF processes so that the CE routers got the other site's subnets via OSPF if the MPLS link was down.

I say in theory because it would need testing and probably a lot of route filtering but if you are going to go with BGP I'll leave it as it is.

It has already got quite complicated with just the one backup link so that may be a better choice.

If there is anything else you want me to test re. the OSPF setup or have any more queries about any of the design just let me know and if I can help out I'm happy to.

Jon

[Daniel Hanson]

Right now, the bridge is up, but not routing any traffic, because I couldn't decide how to implement a routing protocol without breaking everything. As it stands right now, Sites A and B only know about each other over the MPLS. The idea is to introduce the bridge so both sites have kind of this triangle of "backup" paths should either site's MPLS fail. Since site C exists (and so does D, E, etc.) we'd need A and B to be able to advertise the routes (as less preferred, obviously) for the respective neighbor across the MPLS for all of our other sites. This will be the only pair of facilities to have this design, as the other sites are out of state.

Which means some wonderful redistribution and route manipulation! Hence my dilemma on how to approach this.

a) OSPF might work, and would keep it as an appropriate IGP. Figuring out the route manipulation (especially with regards to the redistribution) would be/is a pain.

b) BGP would work, but is robust and might lead to CPU utilization problems. Also, slower convergence than OSPF, so sub-optimal for a failover without tweaking timers. Another problem is the fact that all of our sites share the same AS, so something funny might have to happen in order for us to make this work.

Thanks again for all your help! I'll start planning the BGP configs.

[Jon Marshall]

-