Advertise two different Data center summary routes into same BGP MPLS Cloud.

[Mike Hogenauer]

Afternoon –

I’m looking for the best way to advertise two separate datacenters subnets into our private BGP / MPLS cloud. We have two main datacenters one advertises the summary router of 10.1.0.0/16. The other datacenter advertises the subnet 10.2.0.0/16. I would like to advertise each datacenters summary route into the cloud for redundancy and failover. Can is just advertise the subnet and control the weighting of each route when I redistribute into my IGP or should I route-map the subnet and weight it into BGP from each WAN router. Just looking for the best practice / best way to accomplish – I’m a newbie to BGP.

Example:

Datacenter 1 – summary 10.1.0.0/16

-----------------------------------------------------------

router bgp XXXX

 no synchronization

 bgp log-neighbor-changes

 network 0.0.0.0

 network 10.1.0.0 mask 255.255.0.0

network 10.2.0.0 mask 255.255.0.0

no auto-summary

Example:

Datacenter 2 – summary 10.2.0.0/16

-----------------------------------------------------------

router bgp XXXX

 no synchronization

 bgp log-neighbor-changes

 network 0.0.0.0

 network 10.2.0.0 mask 255.255.0.0

network 10.1.0.0 mask 255.255.0.0

no auto-summary

Thanks in advance –

Mike 

[Jon Marshall]

Mike

Your question is asking how to achieve failover between DCs by advertising out each other's subnets but then you talk about weight and IGP metrics but these would only influence outbound traffic from the DCs not inbound traffic.

So I'm not sure which you want ?

Assuming you mean you want to influence inbound traffic to the DCs in case of a DC link failure then yes you can do this with BGP.

What you do is have each DC advertise it's own and the other DCs subnets but you make sure that if both links are up each DC is used for it's own subnets.

There are two main tools for this -

1) AS path prepending. The AS path length (in terms of the number of ASs in the path) is one of the criteria used in the BGP best path selection. So what you would do is -

DC1 advertises it's own subnets as they are and advertises DC2's subnets but you modify the route advertisement by adding multiple entries of the AS in the path.

DC2 does the same in reverse.

So the route with the shorter path will be used unless the DC fails then the route with the longer AS path will be used

2) MED. This is the other way of doing it.  Here you simply increase the metric of routes you don't want to use unless their a failure. The lower the MED value the better. So -

DC1 advertises it's own routes as normal but gives a higher metric to DC2's subnets and DC2 does the same in reverse.

MED only works on the neighbouring AS ie. it doesn't go beyond that so it's not particularly useful for internet traffic for example where there are multiple ASs to go through but with MPLS there is only the SP AS.

If you are using different AS numbers at each DC then your SP would have to use the "bgp always-compare-med" command which they may or may not want to.

I have used MED to do exactly what you are trying to do but we used the same AS at every site so there was no extra configuration needed by the SP.

AS prepending has none of those restrictions.

So I would have a quick chat with your SP and tell them what you want to do and see which they recommend as they may have local policies that override your configuration.

If I have misunderstood what you are trying to do let me know and if after talking to your SP you need help with the configuration then please come back.

Jon

[Mike Hogenauer]

Hi Jon, 

I think the Path prepend might be my best option. yes,I am looking for DC redundancy from our remote offices and I want to advertise both subnets in case of a failure into one of the DC's. So based on your recommendation, i put together the following config. Thoughts??? 

-----------------------------------------------------------
 DC 1: 10.1.0.0/16
-----------------------------------------------------------

router bgp 65103
 no synchronization
 bgp log-neighbor-changes
 network 10.1.0.0 mask 255.255.0.0
 neighbor X>X>X>X route-map AS-65115-OUTGOING out
 no auto-summary

access-list 40 permit 10.1.0.0 0.0.255.255

access-list 50 permit 10.2.0.0 0.0.255.255

route-map AS-65115-OUTGOING permit 10
 match ip address 50
 set as-path prepend 65115 65115
!
route-map AS-65115-OUTGOING permit 20
 match ip address 40
!

-----------------------------------------------------------
DC 2: 10.2.0.0/16
-----------------------------------------------------------

router bgp 65115
 no synchronization
 bgp log-neighbor-changes
 network 10.2.0.0 mask 255.255.0.0 
 neighbor X>X>X>X route-map AS-65103-OUTGOING out
 no auto-summary
!

access-list 40 permit 10.2.0.0 0.0.255.255

access-list 50 permit 10.1.0.0 0.0.255.255

route-map AS-65103-OUTGOING permit 10
 match ip address 50
 set as-path prepend 65103 65103
!
route-map AS-65103-OUTGOING permit 20
 match ip address 40
!
-----------------------------------------------------------

Thanks again for your response. 

Mike 

[Jon Marshall]

Mike

Few things -

1) you don't need an acl for the DC's own subnets but you do need an empty route map permit statement to allow them to be advertised.

So your route map config is fine you just don't need acl 40 in the second permit statement but you do need the statement itself.

2) your prepending the wrong AS number. DC1 should prepend it's own AS number to DC2's subnet and DC2 should prepend it's own AS to DC1.

3) DC1 still needs a network statement under the BGP configuration for DC2's subnet and the same for DC2 in reverse otherwise the route will not be advertised.

I am assuming your DCs are interconnected via another link ie. not MPLS otherwise you couldn't be doing the failover.

For BGP to advertise a route matching the network statement under the BGP configuration there must be an exact match in the IP routing table.

So DC1 has to have an entry in the IP routing table for both 10.1.0.0 255.255.0.0 and 10.2.0.0 255.255.0.0 and so does DC2.

I don't know how you have setup that up in terms of making sure they are in the routing table so if they aren't perhaps you can clarify what is in each of the DC's routing tables.

After you have done all of the above you will need to do a soft reset on the BGP session for your configuration to take effect.

Anything not clear or you have any more questions please feel free to come back.

Jon

[Mike Hogenauer]

Hi Jon, 

Ok - I think I see what you're saying. Btw - each DC is connected via two separate WAN links so yes, there is connectivity between the two over a private WAN. On each CE BGP router I have a route to NULL0 for each DC's Summary address. I do understand that if it's not in the routing table BGP will not advertise the prefix. My IGP ie EIGRP - on each WAN Facing link between the DC's i have a summary route that advertises 10.1.x.x and 10.2.x.x respectively, so the inside interface is participating in EIGRP and the routing table does know about each others' subnet. However i still ahve the summary route to Null0 still in place. 

So, based on your recommendation - I changes the config.

Look good? 

-----------------------------------------------------------
 DC 1:
-----------------------------------------------------------

router bgp 65103
 no synchronization
 bgp log-neighbor-changes
 network 10.1.0.0 mask 255.255.0.0
 network 10.2.0.0 mask 255.255.0.0 
 neighbor X>X>X>X route-map AS-65103-OUTGOING out
 no auto-summary

access-list 50 permit 10.2.0.0 0.0.255.255

route-map AS-65103-OUTGOING permit 10
 match ip address 50
 set as-path prepend 65103 65103 
!
route-map AS-65103-OUTGOING permit 20

!

-----------------------------------------------------------
DC 2:
-----------------------------------------------------------

router bgp 65115
 no synchronization
 bgp log-neighbor-changes
 network 10.2.0.0 mask 255.255.0.0 
 network 10.1.0.0 mask 255.255.0.0
 neighbor X>X>X>X route-map AS-65115-OUTGOING out
 no auto-summary
!

access-list 50 permit 10.1.0.0 0.0.255.255

route-map AS-65115-OUTGOING permit 10
 match ip address 50
 set as-path prepend 65115 65115
!
route-map AS-65115-OUTGOING permit 20

!
-----------------------------------------------------------

Once again thanks for the quick responses. I'm taking my CCIE written this Friday - I think I might want to brush up on my BGP this week.  :) 

Regards, 

Mike 

[Jon Marshall]

Mike

Looks fine now.

I suspected you may be using null0 routes to advertise them out so that makes sense.

Don't forget the soft reset for it to take effect and then obviously run some traceroutes just to make sure everything is working as it should be.

Good luck with the written CCIE , not something I've ever done so you have my respect for going down that path :-)

Jon

[Mike Hogenauer]

Yes, I can wait until tomorrow to implement. I was going to lab this up as well tonight whit GNS3. What are you concerned about? 

Thanks,

Mike 

[Jon Marshall]

Mike

It's the AS paths and how it will work.

Each DC advertises to the MPLS network which means each DC also receives advertisements via MPLS for it's own subnets ie. DC2 will receive a route advertisement for 10.2.0.0 255.255.0.0 from the MPLS PE because DC1 advertised it out.

And the same in reverse.

If, as you originally planned, advertise out DC2's subnets from DC1 using DC2's AS to prepend then DC2 will drop the route advertisement from it's PE because it sees it's own AS in the path.

But because you are advertising DC2's subnets from DC1 with DC1's own AS it won't, it will accept the route.

Now you also have a network statement on DC2 for that network under BGP so that should take preference because the route received from the PE has a weight of 0 whereas your locally generated route with the network statement has a weight of 32768 so it takes preference and everything should work fine.

The idea behind using the DCs own AS was that it is generally what you should do ie use your own AS although it's not so important on a private MPLS network.

I just want to make sure there are no conditions where the DC routers think the best path to their own network is via the MPLS cloud.

I can't see how there would be but just wanted to double check that's all.

Jon

[Mike Hogenauer]

Now you've got me confused, I'm just about to start my lab right now.

I think it will work based on the last config I sent back. DC1 advertises both subnets but it path prepends DC2's subnet with DC1's AS number. I agree - when DC1 or DC2 for that matter see their own subnet it wont drop it because it's coming from a different AS number. 

I'll let you know if I find anything weird as well. 

Thanks Again. 

Mike 

[Jon Marshall]

Mike

I did a bit of testing and it worked most of the time using the above configuration.

I did however get one time where I restarted all the BGP neighborships and one of the DCs thought the best route to it's own subnets where via the MPLS cloud so it wasn't advertising it's own network.

This doesn't cause a loop it just means everything goes via the other DC.

I only got it when I restarted everything so I think it was a race condition.

So I tried prepending with the other DCs subnet but for some reason the DC routers were not dropping the router even though it's own AS was in the path.

I suspect that was an issue with the lab as it is a emulator.

So instead I simply filtered the incoming routes to each DC ie. deny it's own network and allow everything else.

This worked fine and produced consistent results.

So I think having each DC filtering it's own network from the MPLS network is probably the safest bet. Like I say even without filtering I only managed to get it stuck once and the rest of the time it worked fine but perhaps it would be safer to do that.

The prepending should have worked because BGP will drop a route with it's own AS in the path. You shouldn't use an AS that doesn't belong to you but they do because it is all a private MPLS network so you may want to use that ie. your original configuration.

I can't see why it wouldn't work but I never got to do failover testing with it because my CE routers weren't dropping the route but I could have missed something else as it's late here and it's been a long day :-)

So either that or simply filtering out the DC's own route advertisement should work fine.

If you managed to get out around to labbing it up then let me know how you got on.

Jon

[Jon Marshall]

Mike

Can you just hold off implementing this until I test something.

It will work but I just want to make sure of something to do with the AS path.

Jon