MPLS/DMVPN/OER Design question

[manish arora]

Hello Experts,

I have design question with following requirements & need your expert advice on it :-

Current : 100+ sites connected using MPLS with static routes being redistributed into MP-iBGP on the PE's.

My thinking to make it better & redundant :-

1> MPLS + DMvpn.

2> Making use of both links using OER or some other vendor equipment like ipanema etc.

Questions :-

1> My edge devices are Cisco , but with 100 + sites ISP will most likely be Mixed devices , what routing protocols should use between ISP(mpls provider) & my devices ( OSPF or bgp-4 ) ?

2> I would want to have load balancing between mpls + dmvpn using OER in order to avoid jitter or delay or issues with flow based load balancing , is this a good idea ? any design inputs on that ?

plus anything that I should look into.

Thank you

Manish

[manish arora]

anyone ... Peter , Jon .....????

[Marwan ALshawi]

Hi Manish,

while you waiting for Peter and Jon do you have diagram to share for better understanding to your topology

from what i can understand your ISP will be providing a CE router on site to injected your routing in to MPLs and you have a router back to back to the ISP router that you want to run either OSPF or BGP ? if this is correct then ar eyou sure the router from the ISP will be L3 or just a L2 ( bridging ) device to be provided by the ISP

if you have two separte links per site over two separate ISPs i thin using PfR ( aka OER ) can be useful in terms of intelligent routing ( but mre complexity to your config and design )

once you get back with the answers to the above we can discuss in more details

HTH

[Peter Paluch]

Hello Manish,

To be honest, I do not feel myself adequately competent to answer your question. What follows are my open thoughts but please take them with a grain of salt.

1> My edge devices are Cisco , but with 100 + sites ISP will most  likely be Mixed devices , what routing protocols should use between  ISP(mpls provider) & my devices ( OSPF or bgp-4 ) ?

ISPs are usually more happy with BGP instead of running an IGP towards a customer, even if the IGP runs within a VRF. Also, the BGP gives you great control about what networks you advertise and accept. The BGP would therefore seem a natural choice.

Then again, if you are planning to redistribute the BGP on your branches to an IGP then you have to consider the fact that after the redistribution, all the routes will be considered as external in the particular IGP (OSPF, EIGRP). If you are not planning any backdoor links, backup interconnections, etc. then that may be fine but otherwise, you would run into problems if your IGP exchanged internal routes over your backup connections and received the same routes redistributed from BGP. Both OSPF and EIGRP trust the internal information more than the external, resulting in all your traffic going through the backup connection and avoiding the MPLS VPN cloud. In such a situation, you would be better running OSPF or EIGRP towards your ISP, and asking the ISP to use appropriate means (OSPF Sham Link, EIGRP Site of Origin) to make sure that the routes learned via the MPLS VPN cloud are preferred to those learned via backup links.

2> I would want to have load balancing between mpls + dmvpn using OER  in order to avoid jitter or delay or issues with flow based load  balancing , is this a good idea ? any design inputs on that ? 

I apologize - I do not have practical experiences with OER. However, if you were planning to use OER then as far as I understand, the controlled nodes must run BGP. The question is, how would you want to integrate it with your MPLS VPN provider - obviously, you can not control his PEs, and he might not be happy about periodic additions and withdrawals of routes from your CEs should they be OER-controlled.

The question also is whether the flow-based load balancing (or destination based load balancing) is something you want to avoid. I understand that you are concenred with "jitter or delay or issues with flow based load balancing". My point is that if you do not require some kind of inter-flow synchronization or correlation then I see no outstanding issues about that. Perhaps you could describe in more detail what is your thought on this.

Jon, Giuseppe, Paolo, Edison and others - would you guys mind sharing your expertise here?

Best regards,

Peter

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

The original versions of OER/PfR did only support BGP (or statics); the later PIRO variants are supposed to support any routing protocol (or so I believe, haven't work with OER/PfR not using BGP or statics).

OER/PfR works best at changing routes for egress, PfR (I believe, but again haven't used this option) can also try to control ingress.  A provider shouldn't be troubled by you changing your egress routing, assuming you don't accidentally leak the changes, ingress might, as Peter notes, get their attention.  If I recall correctly, one of the ingress options is to send BGP community strings, so if you want to impact ingress routing you'll likely want cooperation from your provider.

The OER variant is an all or nothing for traffic to the same prefix.  PfR also supports (again if I recall correctly, and haven't worked with it) PBR so that you can control different traffic types to the same prefix (e.g. VoIP vs. bulk).

When I used OER, my goals were dynamic load sharing, and auto routing around brown outs or black holes in the providers' cloud.  This I thought it did well.  Unsure how well it will work if you're worried about jitter or short term delay.  Except for black holes, OER/PfR short term computations, I recall (?), work down to 30 seconds periods; longer term 5 minutes.  What OER has to contend with is to not needlessly slosh traffic about or too quickly react to performance changes made by OER changes.  In other words, don't count on OER/PfR immediately trying to remediate a sudden spike in jitter or latency although it should respond to longer term trends.

Personally, I believe OER/PfR is the next evolution in routing, but, somewhat like advanced QoS, can muck things up if you don't take the time to really understand it.

[Jon Marshall]

Manish

I too don't have a lot of experience with either DMVPN or OER (thankfully Joseph who has used OEF/Pfr has responded as well).  I'll add what i can.

It's not clear exactly what connections you are going to have at each site. You talk of using DMVPN, so are you planning to have a backup ADSL connection at each site as well as the main MPLS connection ?

If so, one very good  point made by Peter should be factored in. If you use BGP on the main MPLS and an IGP internally then you need to redistribute BGP into that IGP. Using EIGRP as an example.

EIGRP internal routes = AD 90

EIGRP external routes = AD 170.

Each site has EIGRP running and these networks are advertised into BGP at each site, either with redistribution or if you have summarised to only a few supernets per sites with the aggregate address command. Or even using network statements.

When those routes are received at a site via BGP they are redistributed into the local EIGRP AS. Those routes will have an AD of 170. If you also then have a second link using DMVPN and exchanging EIGRP routes these will be seen as EIGRP internal routes and will have an AD of 90 so these routes will be the preferred routes.

Don't underestimate how much of a pain this can be because you now have extra config to add to make the MPLS link the preferred link or even to use both links simultaneously. You can mitigate it by advertising a summarised address via DMVPN and the more specific addresses via BGP but this is dependant on how you have done your addressing.

Perhaps if you could be more specific about the connectivity at each site we could be of more help.

Jon

[Marwan ALshawi]

just to add

OER/PfR first was only working with BGP and egress direction policies now you can have it with EIGRP too

you can use it with BGP to effect ingress routes, also this can be controlled per prefix, or even traffic type like VOIP, Telnet ..etc

the main idea PfR uses polices if the route through one link considered out of policy OOP then the MR will look into other available exit linkes in the BRs to route the traffic using attributed pre configured like Local Pref with BGP for example

stat routes can be used with OER/PfR as well

bellow osme documents might help you

OER/PfR

https://supportforums.cisco.com/docs/DOC-8353

DMVPN

https://supportforums.cisco.com/docs/DOC-8356

HTH

[manish arora]

Thank you everyone for giving their inputs on this topic. This is what I was hoping to get from experienced guys like you , issue or what could go wrong etc from you all.

The current topology is very simple with just one link from MPLS provider with static routes. The future plan is to have redudancy with DMvpn ( using a broadband connection ) but with requirement of using both links at a same time with efficiency & required performance.

I will be working on setting up mock environments or labs to test different options with or without OER , with or without other vendors equipments (ipanema ans) claming to provide  wan selection based on performance etc for next few weeks and will get back to you guys.

Thanks again everyone , much appreciated.

Manish

[manish arora]

Hello Everyone,

I am stuck with routing table's route  preference issue. I have a router learning route to say 3.3.3.0/24 from ebgp ( mpls ) and same route from EIGRP ( dmvpn ) , I have changed the AD for EIGRP to 20 to match the AD of the ebgp route. But the router is only istalling route from eBGP and not EIGRP even if they have same AD. If i reduce the AD further for EIGRP to 10 then it install EIGRP route.

Is there a way to have both of the routes installed given I keep the AD same for both protocols.

Thanks

Manish

[Peter Paluch]

Hello Manish,

This might be a problem. To my best knowledge, the behavior of two different routing protocols that have been given the same AD is not well defined. Either the first one wins - i.e. the first one that offers a route to the routing table will have it installed, or the routing table will contain the route with the lower metric. With BGP, the "metric" is actually the value of the MED attribute, and if not set, Cisco by default considers it to be 0.

An idea: try increasing the value of the MED attribute of the networks inside your BGP process (but do it in a sensible way to not wreak havoc with your routing) so that their "metric" is higher than in EIGRP. Perhaps this will allow the EIGRP routes be preferred.

If this does not help then I am afraid there is no workaround. I do not believe that Cisco ever intended to support a scenario where two uncomparable protocols provide routes with the same AD to the routing table.

Best regards,

Peter

[manish arora]

Thanks Peter. Looks like I have to re-think the topology as I need routes from eBGP( MPLS) & EIGRP ( DMvpn ) installed at the same time in routing table so that I can use both links efficiently using OER.

Thanks Again for your Help

Manish

[Peter Paluch]

Manish,

You are welcome. Just curious: have you tried the MED trick?

Best regards,

Peter

[manish arora]

Not yet , I will try that latter in the Day and will let you know.

Thanks

Manish

[manish arora]

Peter,

By changing MED for a route equal to the route learned from EIGRP didn't work either.  It still makes EIGRP's route as FD inaccessible but as soon as I reduce the AD for eigrp it installs the route compared to eBGP route.

I will try some other topology now and will update you.

Thanks for your help.

Manish

[Marwan ALshawi]

Hi Manish,

why you don't run eBGP over the MPLS and the DMVPN in this case you only need to enable eBGP multipathing

however you need to conisder the bellow

- as long as you are runing the DMVPN privately not with any ISP PE then you could use the same AS number but also you need to use AS path same as the one learned form the MPLS inorder to qualify for eBGP multipathing

- if you are going to use differnt AS number over the DMVP ( with equal AS path i mean number of ASNs in the path ) then you need to use a BGP hidden command to enable eBGP multi pathing over two differnt ASNs

- you could disable AS path comparison in the BGP so when you look into a route with differnt number of ASNs in the path bgp can stil consider it for multipathing using the bellow commend

bgp bestpath as-path ignore

keep in mind any of the above suggestion need to be tested first, and also you need to make sure the route is not being advertised between MPLS BGP and DMVPN BGP you can do it in the hug router using filtering with bgp AS path or community strings

HTH

if helpful Rate