Highly available data centers and routing traffic.

[t.khan]

Hello,

We have two data centers, each data center is connected to the internet via a T3, and then we have a 10 Mb link between data centers. Location A has two routers, and location B (Backup) has one router. All three routers are in a standby group with the two routers at location A being primary and secondary, and 3rd router being the least preferable. Router 1 at location A has a connection to the internet (t3) and a connection to a switch where router 2 at location A has a connection. Router 2 at location A also has the 10Mb link over to location B router 1. We have an external block of 10.10.10.0/24 (example) and there are static routes from our ISPs (BGP) that sends the whole netblock to location A. If location A's T3 fails, that traffic automatically is routed to location B's T3. The current setup requires that we then take data from Location B and route it over the private 10Mb link back to location A (assuming the 10Mb link is unaffected) and continue processing. Then the next failover would be if the 10Mb link failed and we then kept traffic only at location B. I have setup OSPF and the 'default-information originate' on internal routing to allow for this to occur however it is not as effective as expected. Router A is always the default router, what we are trying to accomplish is two default routes, where one only comes into play when location A's internet connection fails? Are there better ways to do this? Also note, we will have a tunnel between locations for firewall traffic. Sorry for the long post.

Thanks guys.

[tdrais]

Kinda hard without a little more detail but will give a generic solution.

First I assume both your ISP routers have a IBGP session between them and are both in the same OSPF area.

If you are receiving default route from both your ISP connections you can use something like local preference so that both your BGP routers agree which to use. The issue will be that the non preferred BGP location still will have a IBGP default route in his routing table. OSPF does not know the difference when you use default orginate so both routers will advertise the default route into the OSPF area. You can redistribute BGP into OSPF rather than default orginate. Since by default BGP will not redistribute IBGP routes into OSPF you should only get the default route advertised in a single location.

Now your failure of the 10m circuit is much harder. The default route will work and both routers will become the default for their corresponding location. You problem is that if location B send the traffic to the ISP the ISP will return the traffic to location A which will drop it since it has no route to location B. There are a number of ways to solve this but all involve advertising routes to your ISP rather than using static. 2 common ways are to prepend ASPATH on one connection or use conditional advertisement.

[t.khan]

thanks for you quick response. Currently our ISP is using weighted routes, and all traffic goes to location A. Only if location A's T3 interface goes down, will they send traffic to same netblock to location B. So as for the traffic going to both locations if 10Mb I don't see that as a major problem. The problem lies in how to make sure location B sends any traffic over the 10Mb link out location A's T3 and never uses the local T3, as it is only for a disaster scenario. Does this make sense?

[t.khan]

Quick note:

I added the command "redistribute bgp" in the ospf process I had running and removed the default-information originate. Now the router at location A has a default route to serial 0/0 and the router at location B has a default route its own serial 0/0.

[tdrais]

That is a little strange if you are running BGP between routers A and B. It should agree on a best path for the AS.

I assume you are getting the default route via BGP. Try to prepend you own AS on router B as you receive the route. Since ASPATH is very high in the bgp selection it should force it to prefer the route leared by router A.

I would start with router B bgp session shutdown to the ISP and verify that you are getting the default route via IBGP from router A. When router A then redistributes into OSPF you should get the default route via OSPF and the BGP routing table should show a RIB failure on router B since OSPF has a better AD than IBGP.

You can then activate the bgp to the ISP on router B and see what changes. It really should just add the new route to the BGP table but still prefer the route from router A and still have a RIB failure on that route.

[t.khan]

Sorry for any confusion. We are not running BGP internally. Our ISP is running it on our serial interfaces.

----------

router bgp xxxx

no synchronization

bgp log-neighbor-changes

network 65.xxx.xxx.0 mask 255.255.255.0

neighbor 65.xxx.xx.xxx remote-as xxx

no auto-summary

----------

We are not getting anything but a default route to the internet based on serial 0/0.

[tdrais]

The most common solution it to run BGP between the 2 routers. Just put another neighbor statement in each pointing to the loopbacks of the other router.

If you don't want to do this you are going to have to get more creative.

You can add a statement to the BGP configuration on router B to change the administrative distance of all EBGP routes from the default of 20 to something higher than ospf default of 110. DISTANCE 115 200 200 would work.

This would cause router A to redistribute the BGP route into OSPF. When router B receives this route it would override the BGP route since the OSPF route now has a better AD.

When router B no longer sees the ospf route from router A it will now use its BGP route. It will then redistribute it into OSPF. I am pretty sure the router is smart enough to not override its own BGP when it redistributes into OSPF but this is always a concern when distribution from a higher AD into a lower AD.

I would still recommend you put a BGP session between the routers. It is simpler and gives you options in the future if you would ever need to do something like conditional advertisement to your ISP. You could then control which connection they use rather than them deciding.

[t.khan]

Will do. I am wondering how they will know about the loopback id's of each router if I am not advertising those routes to begin with? Should I advertise the loopback addresses through OSPF?

[alexmoya]

How do you have the default-information originate being inject into OSPF type 1 or type 2

[t.khan]

Just the statement of 'default-information originate' with no other parameters. So I assume whatever it defaults to.

Does my issue make sense?

[tdrais]

You don't need the loopbacks it is just common to use them. You could use the ethernet interface on the router. Since that network is in OSPF you should be able to get BGP between them. loopbacks are used because in core routers you many times have multiple paths between them.

To the above post with the default-information originate. After thinking about this you may be able to use default information originate but this will only affect traffic outside of router A and router B. You can set the OSPF cost different on the 2 routers. I went with the other solution because I read this as you wanted router B to send any traffic it received to router A. If no traffic would be required to pass through router B to get to router A then it does not matter if router B itself has its default router to the ISP.