Two datacenter setup (BGP + OSPF)

Stefano Sasso · ‎09-22-2011

Hello folks,

for a customer of us we are planning BGP connectivity for 2 datacenter. The 2 DC are connected via a L2 link, and each datacenter "buys" transit from 2 ISP.

Each DC has two edge router, speaking eBGP with ISP and iBGP between them. There is no iBGP between the 2 Data centers.

Each DC has a stacked L3 switch acting as "core", and linked to the other DC core.

The customer is AS (let's call it AS100), and has a PA, let's say 100.100.0.0/20. So, DC1 will use 100.100.0.0/22 and DC2 will use 100.100.4.0/22.

The OSPF area 0 includes all the two DC.

DC1Core redistributes (via OSPF) 100.100.0.0/22 (its Null0), and DC2Core redistributes 100.100.4.0/22.

DC1 BGP has:

network 100.100.0.0 mask 255.255.252.0

network 100.100.4.0 mask 255.255.252.0

neighbor ISP route-map as-prepend out

ip prefix-list dc2 permit 100.100.4.0/22 le 24

route-map as-prepend permit 10

match ip address prefix-list dc2

set as-path prepend 100 100 100

route-map as-prepend permit 20

DC2 BGP has:

network 100.100.0.0 mask 255.255.252.0

network 100.100.4.0 mask 255.255.252.0

neighbor ISP route-map as-prepend out

ip prefix-list dc1 permit 100.100.0.0/22 le 24

route-map as-prepend permit 10

match ip address prefix-list dc1

set as-path prepend 100 100 100

route-map as-prepend permit 20

Doing so, DC1 routers will announce the two networks, but DC2network with as prepending.

DC1 routers and DC2 routers don't have the aggregate as Null0: in this way the network will be announced only if received via OSPF. So, if there is no link between DC1 and DC2, DC1 won't announce DC2 network (and vice-versa).

In my opinion there is no need for iBGP between the 2 Data centers, because no transit is provided.

Is this setup correct?

I'm also thinking about using 2 distinct OSPF areas, one per datacenter. How would you do?

This is my first "Two datacenter" setup In the past I "developed" only single datacenter scenarios

ameya_oke · ‎09-22-2011

Hi Stefano,

Let me mention that setup designed is impressive and highly redundant.

I would prefer to keep the core L3 switches of both DCs in Area 0 rather than keeping the whole DC in Area 0.

(But again, it is an individual’s choice, only when we scrutinize the implications of both a decision can be reached upon)

1. From your statement “each datacenter "buys" transit from 2 ISP”, I believe you mean to say for DC2 to reach DC1, direct L2 connectivity is not used and traffic flows via ISPs. Please confirm if right.

2. Also if that is the case IBGP is not required.

having said that, still I would suggest you to design your Edge router setup to involve IBGP as it would give you even more redundancy.

The reason why I say this is , due to some issue in BGP in DC2, if you fail to advertise 100.100.4.0/22 to ISP3 and 4, atleast with IBGP between DC 1 and 2, you will have a tertiary redundant path to reach 100.100.4.0/22 from outside world

Mind you, this is an IBGP fan’s opinion :-D

Conclusion: Your opinion is correct considering your level of redundancy and “transit” clause you mentioned in requirement. But IBGP would bring even more redundancy.

Please rate the helpful posts

Ameya

lgijssel · ‎09-22-2011

Hi,

Your design looks good and is likely to work but you do you have a clear view of what you want to achieve regarding redundancy? The prepended advertisement is in fact only a protection against dual provider failure (on one DC)!?

In exchange for this feature (which may not be required often) you are introducing extra complexity in the solution.

There is alway a trade-off between redundancy and simplicity. I am not stating that I dislike complex designs but from experience I have learned that straightforward designs are often more reliable or at least easier to maintain/troubleshoot.

Take this into consideration when working out your final solution.

Regarding OSPF: With this amount of routers and routes, I do not see a need to split up in multiple area's.

If you are considering that, please check on the recommendations for the maximum size of opsf area's, for example in the ospf design guide.

http://www.cisco.com/en/US/partner/tech/tk365/technologies_white_paper09186a0080094e9e.shtml

regards,

Leo

Stefano Sasso · ‎09-22-2011

Leo,

thank you for your opinion.

Your design looks good and is likely to work but you do you have a clear view of what you want to achieve regarding redundancy? The prepended advertisement is in fact only a protection against dual provider failure (on one DC)!?

In exchange for this feature (which may not be required often) you are introducing extra complexity in the solution.

In the first phase of the project the customer will use only one provider for DataCenter 2. So, if that ISP3 fails, I'll use DC1 to reach DC2.

As the project become complete (and ISP4 will be added), I can simply remove DC2 network announce from DC1 edge routers. Would you recommend to do so?

thanks,

stefano

lgijssel · ‎09-22-2011

Stefano Sasso wrote:

Leo,

thank you for your opinion.

Your design looks good and is likely to work but you do you have a clear view of what you want to achieve regarding redundancy? The prepended advertisement is in fact only a protection against dual provider failure (on one DC)!?
In exchange for this feature (which may not be required often) you are introducing extra complexity in the solution.

In the first phase of the project the customer will use only one provider for DataCenter 2. So, if that ISP3 fails, I'll use DC1 to reach DC2.

As the project become complete (and ISP4 will be added), I can simply remove DC2 network announce from DC1 edge routers. Would you recommend to do so?

thanks,

stefano

Peering with two providers is generally regarded as sufficiently redundant so for the sake of simplicity it's certainly worth considering.

ameya_oke · ‎09-22-2011

Hi,

For your statement,

"I'm also thinking about using 2 distinct OSPF areas, one per datacenter. How would you do?"

Yes, you can and it would be a good thing, route flaps in DC1 wont affect routing process in DC2 and vice versa.

Also below can be done in case individual areas have large number of subnets.

1. Core Swithces - Area 0

2. DC1 subnets in Area 100

3. DC1 Edge router's subnets in Area 101

4. DC2 subnets in Area 200

5. DC2 Edge router's subnets in Area 201

Ameya

ameya_oke · ‎09-22-2011

Hi,

As is mentioned in orignal post IBGP is not required considering the level of redundancy used.

I completly agree with lgijssel that simplicity is key to ease in management.

Ameya

Stefano Sasso · ‎09-22-2011

Ameya,

thanks for your reply.

1. From your statement “each datacenter "buys" transit from 2 ISP”, I believe you mean to say for DC2 to reach DC1, direct L2 connectivity is not used and traffic flows via ISPs. Please confirm if right.

L2 connectivity is used first; if L2 connectivity fails, "internet routes" will be used.

The L3 core switch is the default for firewalls.

thanks again

ameya_oke · ‎09-22-2011

Hi Stefano,

Thanks for clarification.

I believe you have configured ether-channel on L2 uplink between core swithces.

HTH

Ameya

Stefano Sasso · ‎09-22-2011

Hi Ameya,

I believe you have configured ether-channel on L2 uplink between core swithces.

at the moment ether-channel on L2 uplink is not possible

the link is a mpls pseudowire between DC1 (Italy) and DC2 (Germany)

in the future there will be another pseudowire with another transit provider.

DC1 will reach DC2 through internet if the L2 uplink fails. The customer knows this.

bye,

stefano

Marwan ALshawi · ‎09-23-2011

First of all I totally agree that it's better to keep it simple always

About your initial design it looks ok but you need to consider some points

First why you put the public range in a static route to null0 ? Where these IP ranges located ? Iam assuming those ip located behind the firewalls and might be nated on the firewalls

If this the case then I would suggest you to use the bellow design

Ebgp from each edge router to ISP with with the AS prepending as you did exactly

Ibgp between those edge routers in each dc

From the edge to the core use ospf area 0 and the link between the tow DCs in area 0 as well this will be your core igp

Now from bgp to ospf you can advertise/ redistribute based on the routing needs

In the core switched in each DC create static route point to the relevant firewall outside interface for the public ip range assuming the firewall nat these ip range back to the LAN

Redistribute this static route in each core swtch to ospf

In the edge routers redistribute the ospf route of this public range into bgp

Be care careful when you redistribute between ospf and bgp in both edge routers as you might get route looping due to mutual redistribution you may need to use route tagging and filter the ospf route fro

Being injected back from bgp to ospf with less AD

Hope this help

If helpful rate

Sent from Cisco Technical Support iPhone App

Stefano Sasso · ‎09-26-2011

Hi marwanshawi,

thank you for your reply.

Having the whole range in a static route to null0 allows me to announce that specific range via eBGP "network a.b.c.d mask e.f.g.h".

I have it on the core switch instead of on the edge just to prevent the announcement of the network by the edge if all links with the core were not working.

About redistributing bgp onto ospf and vice-versa, I don't think it's a good idea.

The network has to be as more scalable as possible.

I followed lots of seminar about BGP, and read lots of documentation, and every of them discourages injecting bgp routes into a igp protocol to achieve scalability.

bye,

stefano

Marwan ALshawi · ‎09-26-2011

You right redistributing bgp into igp is not always recommended especially for larg routing tables

However you might need to do selectively redistribution if you as i am not fully aware about your network so it is all depends if the Internet even through the same routers then just injecting default route will be fine

About null 0 if you can get it through igp up to the bgp its better to avoid black holing the traffic

Hope this help