Routing advice

[Vlad Olteanu]

Hello All,

Currently we have 2 sites( as in the diagram B1 and B2) and we are planning to add one more site.(B3).

B1 and B2 are connected with OSPF to each other and they each have one ISP router(R5 and R6) also participating to OSPF and injecting a E1 default route.

The core switches at B1 and B2 are routing those 10.x.x.x subnets towards the routers, statically and they have a default route pointing to the internet firewalls.

The active MPLS link is at B2 and B1 uses it to reach those 10.x.x.x subnets. In case the WAN link at B2 breaks, the WAN link at B1 becomes active and B2 will use it to reach those 10.x.x.x sites.

Now, we are adding a 3rd building with internet connection, but no MPLS. It should use B2 as its break point. It will be connected on 10G fiber to B2 and 100M to B1. The idea is that in case B2 goes down or the links go down, all traffic from B3 should switch to B1

I need advice to find the best way to connect B3 in terms of routing, etc...If any of you has this type of architecture, please feel free to comment. Any idea is welcome.

Attached is the diagram

Thank you for your help!

By 10.x.x.x I mean all those 10.17, 10.18 and 10.20 networks.

[Mark Malone]

Hey This is just off my head without testing it but from what your saying and if I have understood it correctly you don't want B3 to change until B2 is hard down and currently B3 routes by B2 but if B2 fails B1 takes over so how to get B3 to now route by B1 in that instance?

if I have that right you could use on B3 ip sla with pbr verify route reachability and set the next hop ip as B1 but only to occur when B3 cannot connect out through B2 as that's the primary link when there is no issues

This is a Nexus doc but its same concept on IOS and basically same commands you track an upstream interface or route that only B2 has when that fails the ip sla and pbr kicks in verifies the route is no longer reachable from B3 so it redirects traffic to its next preferred ip which in this case would be B1 interface, that should prevent B3 from routing to a device/link which is down

You would use an acl to just match your 10 subnet range so only that's redirected as maybe you still want internet traffic for local users on B3 to flow out the internet link as its closest point for them

Just an option anyway there's probably a few ways to do it

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/6_x/nx-os/IPSLA/configuration/guide/b_Cisco_Nexus_7000_Series_NX-OS_IP_SLAs_Configuration_Guide_rel_6-x/b_Cisco_Nexus_7000_Series_NX-OS_IP_SLAs_Configuration_Guide_rel_6-x_chapter_01000.html

[Vlad Olteanu]

Hi Mark,

You understand right. The only issue is that I am not sure how to connect B3 to all this...in terms or routing protocols. Should I add B3 to OSPF and should I change R5 to do BGP with R1 and R2 and R6 with R3 and R4 respectively and also add the core swicthes to OSPF with the routers R1&2 and R3&4. This way my core switches will do OSPF with the routers and my router will do BGP with R5 and R6. I am looking for the most neat solution, top be honest. As clean as possible.

I am debating if I should connect B3 to the rest on Layer 2 or Layer 3...

[Mark Malone · Accepted solution]

Hey I would just connect B3 as OSPF layer 3 neighbour with the core switches in B1 and 2 but set ospf cost on the links to B1 so its not preferred path and traffic is set to route by primary B2 , and use the pbr that way if for some reason your 10GBs failed traffic reroutes to B1 due to the ospf cost and B1 already knows to route to B2 as its exit in a failure , and then again if your 10GBs are good an only B2 fails pbr will redirect to B1 as its exit and B3 will be aware where to go

I don't think you need to move BGP to R1/2/3/4 once there aware that R5/R6 are the real edge routers with BGP , I would form neighbour relationship  between them in OSPF too but again manipulate the path using cost so these links are only used in event of failures if that's whats required

Either way I would lab this if I were you theres a few things to take into consideration and you don't want to be messing with this live , you should be able to easily put this design into something like gns3 to test and make sure before it goes to production that it works exactly the way you want , to simulate your 10 networks just use loopbacks and advertise them in the lab

You want to make sure no matter what scenario of failure you can still route to R5/6 from any of the cores and with the amount of redundancy you have there you have a few options available , with the lab you can drop links and see for yourself no matter what takes place B1/2/3 work as they should , doesn't matter the bandwidths/speed etc its the traffic path that needs to be right

Anyway as I said there is multiple ways you can probably do this and you want the cleanest , you dont want to over complicate things because when something goes wrong its harder to fix which means more downtime

[Vlad Olteanu]

Thank you very much, Mark! we will definitely test it in the lab.