eBGP/iBGP between 2 physical locations. Need experts advices.

[tinhnho123]

Hi guys,

I currently have 2 routers doing eBGP to advertise our public subnet /23 to the internet at Data Center 1 (DC1). There is an iBGP link between these routers, DC1_ASR1k_A and DCAS1k_B. The DC1's topology below is working well today.

Due to network growing and required redundant, I'm looking to expand our BGP network to 2nd data center (DC2) which located about 500 miles away. I'd like to setup 2 routers which would advertise our public subnet /23 to the internet at DC2 but standby (prepend). They'd only active when 2 routers at DC1 are offline. Below is my future topology for both DC1 and DC2:

There will be 2 darkfiber between 2 data centers for iBGP. 

Has anyone done similar thing like this before? Appreciate if you could list Cons and Pros. Any other suggestions?

[chrihussey]

Hello, 

It looks like a pretty standard setup and should work well. With the data centers 500 miles apart I assume any dark fiber provider will have to regenerate the signal at points. Obviously be sure the dark fibers are diversely routed and even use different providers if that works too. I'm sure you know this but I'll say it anyway, you'll need to have each router IBGP peer with the other three.

Anyway, hope this helps and good luck.

[tinhnho123]

Thanks. We just got a couple of new dark fiber quotes, which are super expensive. In the past, yes, the provider would regenerate the signal at points for dark fiber for our other stuff. The second option is using P2P links between 2 DCs. The latency of the P2P links is between 35 and 36ms. Do you think 35-36ms latency with P2P links would work in this case?

[chrihussey]

35-36ms latency for a 500 mile P2P link does sound a bit high. I'd expect 10-15ms...20 tops. Tough to say if it would have an impact, but definitely something to consider. On the other hand I'm sure there are instances where there are data centers at greater distances or even internationally that operate with that kind of latency.

[balaji.bandi]

that is standard, how you configure is best achieve based on the config

check below some example :

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13762-40.html

[tinhnho123]

Thanks for the links.

[Cristian Matei]

Hi,

   Few questions:

1. Are you looking for both ingress and egress traffic to go via DC1 and only if both ISP's of DC1 are down, ingress/egress to failover to DC2?

2. What is order of preference for ingress/egress in terms of ISP's?

3. Do you have same ISP's on both DC's as you've shown on the diagram? Like ISP_A is same on both DC's and ISP_B is same on both DC's?

Best,

Cristian.

[tinhnho123]

Hello,

 1. Yes.

 2. I'm not sure that I understand this question. Can you please clarify more?

3. Yes, we have same both ISPs on both DCs.

[Cristian Matei]

Hi,

  Since first question was answered, second question now becomes: which is your primary ISP, A or B? Be aware that ingress traffic will come through both A and B (you can only control ingress traffic coming from outside A's BGP ASN ad B's BGP ASN), however you can control primary ISP for all egress traffic (although you may also want to route for A's BGP ASN via A, for B's BGP ASN via B and for all other Internet prefixes via A or B). So what's your preference for egress traffic?

Best,

Cristian.

[tinhnho123]

Currently, at DC1, ISP A is the primary ISP. Ingress traffic has come to ISPs A and B but ISP A gets more ingress traffic. more than 80% of egress traffic prefers ISP A. I don't have any AS prepend either router A nor is B of DC1. I'm advertising our subnet 123.123.123.0/23 to the internet (my real ip is not 123 =)). The current HSRP priority of DC1 of router A is 160 and router B is 140. Router A has IP 123.123.123.2, and router B has IP 123.123.123.3. The HSRP VIP is 123.123.123.1

When two routers of DC2 are ready (future), I'm planning to have minimum 3 AS prepend both routers' eBGP of DC2. The HSRP of DC2's Router A is 120 and router B is 100. Also DC2's router A has IP 123.123.123.4 and router B has IP 123.123.123.5. Both are using 123.123.123.1 as the VIP for HSRP. With this setup, when 4 routers are online. I'm preferring egress traffic will be going to DC1's router A. Any opinions regarding this setup?

[MHM Cisco World]

I am until now dont know how server connect to both DC routers but check this link 

https://www.cisco.com/c/en/us/support/docs/ip/border-gateway-protocol-bgp/13768-hsrp-bgp.html

MHM

[Cristian Matei · Accepted solution]

Hi,

    While there are many options, I always tend to use the simplest one which does the job, as simplicity means efficient overall network functionality, network stability and feasibility when operating it and performing investigations. Here's how I see it progressing (assuming you don't want egress traffic load-balancing):

1. use local preference to set ASN wide egress traffic preference, use inbound route-map on your eBGP peerings with ISP's; e.g local preference value of 2000 for ISP_A on DC1, value of 1500 for ISP_B on DC1, 1000 for ISP_A on DC2 and 500 for ISP_B on DC_2
2. due to you not being able to control speed of BGP route propagation for you public subnet and any penalties that ISP_A or ISP_B might run today or in near future without notifying you (assuming DC1 is down / both ISP's are down), it's better to advertise two /24's from DC1 to both ISP's and one /23 from DC2 to control ingress traffic; for DC1, if you want ISP_A to be primary, just advertise both /24's, this way ISP_B becomes secondary by using AS-PATH prepend; for DC2 you perform AS_PATH prepending based on ISP preference for the advertised /23; because you advertise two /24's from DC1 and one /23 from DC2, most specific routing will win and all ingress traffic will come through DC1
3. Additionally, if internal / IGP routing is properly done, you technically don't need any of the IBGP peerings; these will just add complicity and unnecessary overhead.

Best,

Cristian.

[tinhnho123]

Thanks so much for your advice. I've been working on a lab to simulate it in the last few days. We'll see how it works out before order equipment/services.  

[MHM Cisco World]

please share how server connect to both DC1/2 routers 

MHM