Solved: MPLS BGP Failover

du4blake1 · ‎05-28-2015

so im a network admin for a fairly large credit union (cisco routers/HP procurve switches), previously the assit network admin for a very large bank. we are on a mpls/bgp network here at the credit union. we have two data centers, our main site & our DR site. Separate ISPs for internet at each site, both on the same AS MPLS NT to get to the internet from each site, as we would normally see. ive been at this company for about a year and a half and we decided to move our data center to a new location for a few different reasons. the orginal configs were done by a contractor long before i was hired. honestly, looking at the configs, i see A LOT of odd discrepancies. im a valid CCENT & CCNA R&S, but ive been working with bgp and mpls networks for a few years. im certainly not at a CCNP level just yet but once my CCNA is close to expired i plan on tackling that. so i just have a few ?s on how to set up the new configs on our MPLS router at the new DR datacenter. we currently track routes with IP SLAs at the main site, and if those tracks go down the routes get pulled from the routing table after a set time and everyone then fails over to the DR site. it works as intended, some what. the DR site doesnt use the main site for internet, it finds its route to its ISP for internet to be better and i know why. on the static route, it doesnt have any AD value after the route - ip route 0.0.0.0 0.0.0.0 192.168.*.* thats all. so that route is preferred over BPG im thinking. heres the config for the DR site.

router bgp 6****
no synchronization
bgp log-neighbor-changes
network 0.0.0.0
network 192.168.*.* mask 255.255.255.128
redistribute static route-map Redistribute_Static_Into_BGP
neighbor 12.*.*.* remote-as 7***
neighbor 12.*.*.* soft-reconfiguration inbound
neighbor 12.*.*.* weight 100 (weird as the route map is zeroing it out, plus rather have RMs handling this at redistribution)
neighbor 12.8*.*.* route-map SETMED out (dont need)
default-information originate
no auto-summary
!
ip forward-protocol nd
ip route 0.0.0.0 0.0.0.0 192.168.*.*
ip route 10.*.1.160 255.255.255.224 192.168.*.126 100 (the 10.*.*.* are routes to our data processor, have a router for them at each site for FO)
ip route 10.*.*.0 255.255.255.0 192.168.*.126 100
ip route 10.*.*.175 255.255.255.255 192.168.*.126 100
!

!
ip prefix-list DEFAULT seq 5 permit 0.0.0.0/0 (is this needed? im thinking no, no route maps about prefixes)
!
ip prefix-list WDS seq 5 permit 192.168.*.0/25 (is this needed? thinking no)
logging trap notifications
logging 192.168.*.*
access-list 10 (want to just use 10 for all ACLS to make it easier) permit any
access-list 20 permit 0.0.0.0 255.255.255.128 (get rid of this and the above ACL and replace both with just permit 0.0.0.0)
access-list 30 permit 10.*.*.175
access-list 30 permit 10.*.*.160 0.0.0.31
access-list 30 permit 10.*.*.0 0.0.0.255
no cdp run

route-map SETMED permit 10 (could get rid of this whole bit as local preference could be used instead of this)
match ip address 10
set metric 50
set as-path prepend 6**** 6****
!
route-map Redistribute_Static_Into_BGP permit 10
match ip address 30
set weight 0
!
route-map Redistribute_Static_Into_BGP deny 20 (implicit deny all, shouldnt need this route map?)

here's my updated config for the new DR site.

ROUTE_MAP

access-list 10 permit 0.0.0.0
access-list 10 permit 10.*.*.175
access-list 10 permit 10.*.*.160 0.0.0.31
access-list 10 permit 10.*.*.0 0.0.0.255

route-map Redistribute_Static_Into_BGP permit 10
match ip address 10
set weight 0 (zero out the weight as local distribute routes get a weight of 32768)
set local-preference 70 (set LP lower than default so main site is the preferred choice here and among all peers)

BGP

router bgp 6****
bgp log-neighbor-changes
network 192.168.*.0
redistribute static route-map Redistribute_Static_Into_BGP
neighbor 12.*.*.* remote-as 7***
neighbor 12.*.*.* soft-reconfiguration inbound
default-information originate

IP ROUTES STATIC

ip route 0.0.0.0 0.0.0.0 192.168.*.* 100
ip route 10.*.*.160 255.255.255.224 192.168.*.* 100
ip route 10.*.*.0 255.255.255.0 192.168.*.* 100
ip route 10.*.*.175 255.255.255.255 192.168.*.* 100

i would like the config as clean as possible and route maps handling most attributes. any concerns or suggestions would be highly appreciated on my part. thanks for your time.

regards,

blake

Jon Marshall · ‎06-03-2015

Blake

Had a read of that thread and it is not what you are trying to do.

The difference is the OP in that thread was talking about if from the PE end not the CE end ie. he was the SP for the MPLS service whereas you are a end user of the MPLS service.

He was talking about setting local preference on a PE to influence the route selection.

But if you want to express a preference to your SP as to which link to use ie. main DC or DR you can't do it using local preference, you do it with MED or use AS path prepending.

I would recommend you try this in GNS ie. simply set up two routers and make them EBGP peers.

Then advertise a loopback network from one router to the other and try setting the local preference in the route map.

What you will see is that is makes no difference to the local preference on the other router ie. it is not passed across. This is because local preference is used to determine which outbound path to use from your AS so you would set it on inbound routes into your AS.

Then instead of local preference use MED ie. set a metric and do the same. This time you will see that metric displayed for the route on the other router. This is because MED is used to determine which inbound path to your AS to use and therefore you need to be able to tell the neighboring AS this information.

So you cannot use local preference the way you are proposing to influence which path traffic takes to get to the DC subnets ie. via the main DC or the DR site because it won't work.

Hope that makes sense, please come back if you want to discuss more.

Jon

View solution in original post

Jon Marshall · ‎06-03-2015

No problem, glad to have helped.

If you set it up as is in GNS3 and run some connectivity tests and then make your changes and retest that should give you some confidence in what you are doing.

I am still not entirely clear on some of the other configuration so i'll give it another look when I get the chance.

But yes, by all means post the new configurations and I'd be happy to look them over for you.

Jon

View solution in original post

du4blake1 · ‎06-02-2015

anybody?

Jon Marshall · ‎06-02-2015

Blake

Probably no one has answered because it really isn't clear how this is meant to work.

Some comments I can make -

1) the "neighbor 12.*.*.* weight 100" command sets all routes received from the PE to that weight.

This is not the same as setting them in the route map for static redistribution because that is for local routes.

2) You say you can get rid of the MED by using local preference but they are two completely different things.

MED affects traffic coming into the DR whereas local pref affects traffic going out of the DR.

It looks to me at first glance that the DR is advertising out the same IP subnets as the main DC but is using both MED and path prepending which seems like overkill as you usually only need one or the other as they are both used to influence traffic inbound.

This would make sense if the DC's were linked to each other via another connection so you could use the DR site to get to the DC if the DC's MPLS link failed or if you were using the same IP subnets in each site.

If that was the case and you removed that config then your DR would be advertising the same subnets to the MPLS cloud as your main DC without any preference as to which to use and it would be a lottery as to which DC your remote sites used.

We could probably make more sense of this by understanding more about the setup ie. are the two DC sites interconnected by a link other than the MPLS one.

The static routes are not clear as to what all of them do ie. why does each site have static routes to the data processor subnets.

I wouldn't be making any changes until you understand exactly what is happening and certainly don't be changing the MED and/or local pref configuration until you know it is safe to do so.

Jon

du4blake1 · ‎06-02-2015

Hi Jon,

thanks for taking the time out of your day to give me a hand. this is my first question ever posted here, prob why things arent as clear as they need to be. so to answer your questions:

1. that makes sense thanks for that comment.

2. the reason i would like to get rid of MED and prepending the AS path is bc local preference is the 2nd criteria checked, first is weight - but weight is only used on the router its set on. local pref will be seen and can be used by every router in the local AS as long as weight is zeroed out or equal( which seems to be the right way to go here.) also we are talking about a DR site and main site that are in the same local AS as well. prepending the AS path would be used if we have another, to multiple different AS, but we only have the same local AS btw all sites. i think thats why you were thinking we may have had another link btw the DR site and main site bc of the prepended AS path, which is typically used if there are more than one AS. (by the way, i knw that local pref is different than MED, im a bit green, but i understand the attributes of BGP) and then aslo, local pref is preferred by me bc i want to affect the traffic and bgp attributes leaving the DR datacenter with the route maps. so if i set the local pref to 70 at the DR datacenter through a route map, it will be less preferred to the other branches bc the main site is using the default 100. so the main will win that battle among peers. of course, unless he main site is down, then 70 local pref will be preferred as both sites have weights of zero. and i totally agree, med and prepending the AS path is an overkill no need for it in our environment! hope that clears up what is going on inside of my head and why i want to use local pref.

"""This would make sense if the DC's were linked to each other via another connection so you could use the DR site to get to the DC if the DC's MPLS link failed or if you were using the same IP subnets in each site.

If that was the case and you removed that config then your DR would be advertising the same subnets to the MPLS cloud as your main DC without any preference as to which to use and it would be a lottery as to which DC your remote sites used""""

main site and DR site are ONLY connected through the one MPLS circuit on the same local AS at the moment. the part where you said if i remove that pat of the config it would be a lottery n which dc would be chosen for path selection. but what i was saying, was remove that part of the config and replace with local pref lower than default on the main datacenter mpls router. so there would be a preference. i didnt mean just remove that part of the config and that was it. it was remove and replace with local pref. hope that clears up what i was trying to get at in the original post.

The static routes are not clear as to what all of them do ie. why does each site have static routes to the data processor subnets.

hmm, not sure we are on the same page with this one prob the original post was very confusing on my part and im sorry about that, was my first post. the static routes to the data processor are only at the two datacenters. so that if the main site goes down we have a backup router for our data processor at the DR datacenter, which those routes are redistributed into the MPLS cloud so the branches can get those routes. i think this line is where you thought i meant maybe ALL sites have the routes, but i really meant the 2 DCs - ip route 10.*.1.160 255.255.255.224 192.168.*.126 100 (the 10.*.*.* are routes to our data processor, have a router for them at each site for FO)

I wouldn't be making any changes until you understand exactly what is happening and certainly don't be changing the MED and/or local pref configuration until you know it is safe to do so.

after, i explained myself in a bit more detail and cleared up a few things. i think its ok to take my plan of action? using IP SLA to track routes, and using floating static routes & using route maps to zero out weight and set local pref seems like a pretty common practice in failing over when using a mpls vpn and bgp all within the same local AS.

and again, thanks Jon for taking the time out to help me. much appreciated! looking forward to hearing back from you!

regards,

blake

Jon Marshall · ‎06-03-2015

Blake

so if i set the local pref to 70 at the DR datacenter through a route map, it will be less preferred to the other branches bc the main site is using the default 100.

I may be misunderstanding the above but no it won't.

Local preference does not affect what other sites uses, it affects what the DR uses ie.

if the DR has multiple routes to a destination prefix and you want to favour one over the other then you would use local preference.

if you want to advertise the same subnet(s) from the main DC and the DR site to the MPLS cloud and you want the remote sites to use the main DC as long as it is up then you use MED or AS path prepending.

I think the confusion is coming because you are thinking of outbound in terms of route updates but it is outbound in terms of traffic ie.

local preference affects traffic going outbound from the DR.

MED and AS path prepending affect traffic inbound to the DR.

As for the other points in the post thanks for clearing up the static part, I thought that may be what you meant ie. only at the DCs but wanted to be sure.

I still wouldn't make changes yet until we have cleared up the local preference bit at least.

Does what I say make sense ?

Jon

du4blake1 · ‎06-03-2015

good morning Jon,

this is a post ive looked at a while back off the cisco site. this idea here led me to believe i could use local pref. check it out.

Initially we have the BGP external route dominating and static with high distance is suppressed.

I shut the external peer on PE-1.

1) BGP external route disappears from the backbone

2) static gets into vrf blue table on PE-2

3) static gets redistributed into BGP (weight 0, local_pref 90) on PE-2

4) local_pref 90 BGP route reaches PE-1 and is installed there

I restore the external peer on PE-1.

1) BGP external route reaches PE-1

2) local_pref 100 wins the battle on PE-1

3) PE-1 advertises local_pref 100 route to PE-2

4) local_pref 100 route wins the battle with the local_pref 90 route on PE-2 (both have a weight of 0)

5) local_pref 100 route is installed in vrf blue table (distance 200 wins the battle with the static)

6) static is back to were it was before the failure

7) since static is out of the vrf blue table, redistribution of the static into BGP on PE-2 stops

- See more at: https://supportforums.cisco.com/discussion/10983116/stripping-weight-32768-bgp-redistribution-danger#sthash.f6xE8Bkp.dpuf

Local preference does not affect what other sites uses, it affects what the DR uses ie

the above post i used from another discussion seems to say that when we redistribute those static routes into bgp and set the local pref through route maps. that those changes are seen at the other datacenter and are used to for best path selection.

3) static gets redistributed into BGP (weight 0, local_pref 90) on PE-2

4) local_pref 90 BGP route reaches PE-1 and is installed there

still not making any changes per your suggestion, i fly out there at the end of this month. prob just need to set this up in GNS3 to play with it.

thanks again for your time,

blake

du4blake1 · ‎06-03-2015

again, im not saying any of this is right. i just wanted to post that bit from the other forum so you see where im getting the idea about local pref. it could still be that im misunderstanding something still (which is most likely the case.) just wanted to through that other post out there for you to see, and what you think about it.

blake

Jon Marshall · ‎06-03-2015

Blake

No problem.

I am a bit busy at the moment but I will have a good read of that link later and get back to you.

It may be that I have made a mistake (it happens quite often :-)).

Jon

du4blake1 · ‎06-03-2015

hey, no worries Jon. if networking didnt throw odd balls here and there it wouldnt be super interesting and fun, and everyone would be doing it! surely keeps us on our toes!

and again, thanks for all your time and knowledge on the subject. i know we are all busy and any spare time is a blessing to help each other out. im a young guy only 27, but been in IT since 19, only in networking a few years. so im really just trying to learn as much as possible. im not the guy who just settles for put these configs here and thats it. i really need to understand what im doing and why it is im doing it. ive always been that way.

look forward to hearing back from you.

blake

Jon Marshall · ‎06-03-2015