BGP Design

[NetworkGuy!]

Hi

I have the present scenario and would like to ask for any suggestions / best practises

There is two buildings and 1 ISP Circuit/ routers in each building. I would like to do BGP multihoming - I want building A to be primary and building B to be secondary. both buildings are connected to each other by internal fibres

I want all traffic to go through Building 1 (if things are well) but if the fibre breaks, and if router in Building 1 goes down, then all traffic through Building 2. 

But if the fibre breaks, then I want all local traffic to go through local routers - is this possible? ISP prepared to do bgp config as my design

1/ would cross peering between the circuits help? (like R1 has BGP peer to ISP Circuit 1 and 2 and R2 has BGP peer to ISP circuit 1 and 2)? 

2. or can i design as Active/Active - how will incoming traffic behave? no issue with public address - I have many public ip address available

3. any other design recommendations even if its something which is out of the box! ?

[chrihussey]

Hello,

Suggest the following:

1- Dedicated public IP space to each building.

2- Have both routers run IBGP across the fiber where building 1 advertises its public IP space to building 2 and vice versa.

3- Both buildings advertise both IP spaces to each provider.

4- Using local preference have building 1 favored for outbound traffic and as-path prepend from building 2 to influence return traffic to use building 1.

5- If the fiber breaks between buildings then each building will only advertise the address space assigned to that building so nothing will get black holed.

6 - If one of the building routers fail, then the other will take over.

There are probably some intricacies, but hope this conveys the general concept and is of some help.

Regards

[NetworkGuy!]

thanks for your input, I should have explained better! 

Yes the plan is to the use iBGP and local preference on R1 and AS Pre-Pend on R2 but what would happen in case of fibre break? - how will the data pass through? wont will happen in case of split brain scenario? how will traffic pass by? what config is needed on ISP side? or what i tell them?

[chrihussey]

If the fiber breaks and connectivity is lost between the two buildings then building 1 will only advertise its network, stop advertising building 2s to the Internet and vice versa. To further explain, building 1 has a public network known locally to advertise to the ISP, it learns about building 2's network via an IGP and also advertises it. If the fiber gets cut between the buildings, building 1 no longer is getting the building 2 route and stops advertising it.

You of course need at least two /24 at minimum to accomplish this as most ISPs will not accept anything less.

Finally, provided the ISP isn't putting any restrictions you should be able to accomplish this from your network. This puts everything in your control.

Regards

[NetworkGuy!]

only trouble with that is i dont have two /24.

I can only have a single /25 - what will be best solution using this?

[chrihussey]

Ahhh, thought you had more based on your earlier post. I assume then that you got this space from the ISP as it is only a /25. If it is the ISP's space, then perhaps they will let you advertise a /26?

[NetworkGuy!]

No, its our own address space (/24 but using amongst multiple sites) 

ISP happy to advertise /25

[Julio E. Moisa]

Hi

You could create an iBGP peering between R1 (primary site) and R2 (secondary site) using the directly connected IPs or using loopbacks (in this case you require a IGP as well).

Once you have the iBGP peering, you can use BGP attributes to move the traffic and prefer a path over other. The basic configuration could be something like:

R1 (Primary site)

int g0/0

description TO-ISP1

ip address 190.x.x.2 x.x.x.x

no shut

int g0/1

description TO-R2-SITE2

ip address 10.0.0.1 255.255.255.252

no shutdown

route-map INBOUND permit 5

set local-preference 10000

route-map OUTBOUND permit 5

set as-path prepend 100 100 

router bgp 100

no sync

no auto

neighbor 190.x.x.1 remote 6500

neighbor 190.x.x.1 route-map INBOUND in

neighbor 190.x.x.1 route-map OUTBOUND out

neighbor 10.0.0.2 remote 100

neighbor 10.0.0.2 next-hop-self

R2 (Secondary site)

int g0/0

description TO-ISP2

ip address 200.x.x.2 x.x.x.x

no shut

int g0/1

description TO-R2-SITE2

ip address 10.0.0.2 255.255.255.252

no shutdown

route-map INBOUND permit 5

set local-preference 5000

route-map OUTBOUND permit 5

set as-path prepend 100 100 100 100

router bgp 100

no sync

no auto

neighbor 200.x.x.1 remote 6600

neighbor 200.x.x.1 route-map INBOUND in

neighbor 200.x.x.1 route-map OUTBOUND out

neighbor 10.0.0.1 remote 100

neighbor 10.0.0.1 next-hop-self

Also take in consideration you must avoid that your network becomes a transit network using: ip as-path access-list 1 permit ^$ going out direction. You can filter the traffic using Prefix List in order to avoid extra CPU utilization and by security reasons. You can also apply load balance if it is required. 

Hope it is useful

:-)

[NetworkGuy!]

thanks for your input, I should have explained better! 

Yes the plan is to the use iBGP and local preference on R1 and AS Pre-Pend on R2

main questions:

what would happen in case of fibre break?

wont will happen in case of split brain scenario?

how will traffic pass by? what config is needed on ISP side?

"Network becomes a transit network" - I dont understand this?

Also outside of box, will active/active work in this scenario? also what will happen if cross bgp peers?

[Julio E. Moisa]

Hi,

what would happen in case of fibre break?

Once the fiber is down, the private traffic will prefer the path with higher local preference (I assume both Edge routers are connected to the Private network via iBGP or redistributing the BGP learnt prefixes into a IGP like OSPF, EIGRP, RIP, etc. with different metrics. 

wont will happen in case of split brain scenario? 

I dont understand the question but I think it is related to the previous one.

how will traffic pass by? what config is needed on ISP side?

No changes on the ISP are required. 

"Network becomes a transit network" - I dont understand this?

Transit network is term used to indicate all Internet traffic is passing through your routers, Usually it happens when the BGP routers are connected to Internet and they are receiving all the prefixes, so it could not be desired. 

Also outside of box, will active/active work in this scenario? 

No, it will not work in active active, it will be Active Standby.

also what will happen if cross bgp peers?

Could you please provide more details?

My suggestion is use explicit parameters, I mean configure manually Local preferences and AS Path preprend values like I shared previously. 

Primary router: Higher LP and lowest AS Path

Secondary router: Lowest LP and Higher AS Path

It will help you to acchieve symetric traffic, also you could configure fast fall over to avoid bgp timers once the peering is down. 

[NetworkGuy!]

Thanks both

I have another scenario - 

There is 1 core switches connected to each router (R1 and R2) on either side of buidling

two scenarios (assuming we have active/standby setup with R1 primary and R2 secondary):

a. what will happen if iBGP relationship breaks - I presume traffic will flow through building 2

b. what will happen if the fibre between both core switches break? so R1 and R2 stays up

[chrihussey]

This is where having advertised internet address space unique to each building makes things a lot easier in that any disconnect between the two allows each to operate independently. Although it may not be possible (things are never easy) the assumption is that there are networks in each building using both cores. Again, if the networks were physically unique to each building, any separation of the cores allows each to work independently. That being said:

a. what will happen if iBGP relationship breaks - I presume traffic will flow through building 2

ANS. It all depends on how BGP is set up. If both are advertising the same address space and have the supporting routes then, yes, both would be advertising to the provider. You could potentially have R1 advertise the IBGP networks to R2, so that if connectivity is lost between the two and IBGP fails, R2 would stop advertising to the provider. Conditional advertising is also a possibility. However, this could potentially cause problems with building 2.

b. what will happen if the fibre between both core switches break? so R1 and R2 stays up

ANS: Depending on how the networks are distributed in each building and how address space is advertised to the provider would be the determining factor.

Now, the assumption here is that the fiber could be cut and the internet routers maintain connectivity. Is that possible, or are the cores and routers using the same fiber cable? If it is possible. consider diversely routed fibers between buildings so that connectivity would be maintained in case of a single fiber cut.

There are many ways to accomplish the goal you want to achieve, yet a lot depends on allocation of inside addresses and how you can advertise to the provider. Simplicity works best, but that needs a good foundation.

This response may have been a little vague and not provide you with an answer, but may help in getting to where you want to be. Some questions to ask:

1- Are the inside networks be unique to each building, or could you transition to achieve that goal?

2- Can you advertise unique address space to the provider from each building?

3- What sort of infrastructure needs to be in place to provide redundancy in the event of a fiber cut or switch failure?

4- What will the budget allow (always the big one).

Hope this helps.

[NetworkGuy!]

Hi

Regarding you rquestion about fibre - 2 pair of fibre for switch to switch (iBGP - HSRP) and 2 seperate pair between core switch to core switch 

1- Are the inside networks be unique to each building, or could you transition to achieve that goal? - each side of the user network would be different (user vlan is unique to each building) however the transit vlan would be same across same building- there might be a case where we might have to strech user vlan also amongst buildings

2- Can you advertise unique address space to the provider from each building? you mean the public address? I can try for a /24 but even if i have a /24 i would like to use only /25 (out of /24) for advertisement and the 2nd /25 for DMZ

3- What sort of infrastructure needs to be in place to provide redundancy in the event of a fiber cut or switch failure? I want internet traffic not to be disturbed - ideally if there is a fibre cut between the both bulidngs (unlikely but worse case scneario) then i want internet traffic to flow 

4- What will the budget allow (always the big one). - minimum but can explore options to see if feasible

5. how can i acheive if i had two of public address /24 each?

[chrihussey]

You have two fiber pairs for router and cores, but are they all in the same cable?

1- Transit VLANs, or routed L3 points between buildings wouldn't matter. Stretching a user VLAN across buildings is tricky. But if each building is advertising unique address space to the provider, there would be some functionality.

2- So with a /24, you could advertise a /25 out each building to the ISP and use a /26 (of each /25) for DMZ. One possible solution anyway.

3- For fiber redundant and diversely routed cables. At the switch level, you could stack two switches on each side to provide switch redundancy. For internet access, if done correctly, you could have primary through R1 in normal operation, failover through R2 if R1 fails, and have both buildings operate independently if fiber is cut.

4- If budget is minimal and the likelihood of the fiber being cut between the two buildings is slim, then eliminate that eventuality and let management (or whoever) know. In the end it is all cost.

5 - Whether you have a /24 or two /24s, it is all contingent on how you allocate the space between the buildings and how it is advertised to the ISP.

BTW - I don't mean to over simplify things. Just provide helpful (hopefully) input.

Regards

[NetworkGuy!]

Thanks once again for prompt response

to answer to your question

we have 4 fibres in total out of which i am going to bundle 2 fibres for L2 switch to L2 switch (for iBGP) and the rest two between the core switches in each bulding

1. tranist vlan is streched so basically i have a vlan for all firewalls, routers internally to communicate to each other and run eigrp (for example: i use vlan 100 for management of all devices across both buildings) and generally unique user vlan for each building but sometimes if for example HR want to move to another building then stretch amongst buildings (so i allow complete trunk between both core switches in each building)

2. Yes that i can do, leaving out the dmz for  now, you mean advertise /25 out of each building to ISP? how will failover work if the fibre between the L2 switches break or fibre between core switches break? do you mean active/active scenario?

3- how is this going to work? if all traffic passes through R1 (from both buildings) and then if fibre between L2 switch breaks, yes all traffic passes through R2. but imaging if the fibre between L2 switches stays up but the fibre between core switches break, how will traffic pass through?

sorry to find a good design for this case.