Okay, reposting possible solution, but it does involve a fair bit of config -
To avoid any type of routing loop or blackholing traffic you need each site to check the availability of it's own internet connection and only have a default-route pointing to it's local internet connection if it is up. Note that this default-route is redistributed into EIGRP.
So my original thought was to check for internet availability using IP SLA. If the connection fails remove the default-route and then use the default-routes from the other sites.
The problem with this is that, for example, if the default-route from site A is lost and so site A uses site B the IP SLA will still work because it can go out via the site B connection so the IP SLA ping works and so it thinks the link is back up and reinstalls the route at site A.
The only thing i can think of at the moment is if you picked an IP on the internet for the IP SLA check that is specific to each site ie. the provider next-hop address at each site. So -
site A pings site A provider next-hop address, site A firewall blocks pings to provider B & C next-hop
site B pings site B provider next-hop address, site B firewall blocks pings to provider A & C next-hop
site C pings site C provider next-hop address, site C firewall blocks pings to provider A & B next-hop
that way if site A internet is down, IP SLA in site A keeps trying to ping the provider next-hop address. But it can't get a response through B or C because you have blocked it on their firewalls. So the only way the IP SLA will work at site A is when site A's provider next-hop responds.
So the default-route on each site is redistributed into EIGRP. Each sites actually sees 3 default-routes but should use it's own default-route for normal operations.
If the local internet connection goes down then the alternative default-route(s) are used.
Notes -
1) You would need to ensure within each site all L3 devices prefer the local default-route. You may need to tweak the metrics of the default-routes received from the other sites to ensure this.
2) You will only load-balance between the other 2 sites if the cost to each site is the same ie. the default-routes received from the 2 other sites must have the same metric.
3) it is unlikely that the default-routes received will be seen as feasible successors. What this means is that when the local defaul-route is lost EIGRP may well have to query for a new default-route. It's best to try and limit the scope of EIGRP queries but in this case you will have to use the WAN. If you have an unreliable WAN this could have an impact.
4) Using both sites if the local site fails may raise issues with the firewalls. Certainly you cannot do per-packet load-balancing as that would really mess them up. But even with per-destination load-balancing it something to be aware of.
The key to the whole setup is the checking of the local internet connection. That is why simply using floating statics in each site will not work because a floating static does not know whether the link is up or down. You could use IP SLA on the floating statics but that would involve even more config than you are looking at.
With the above, and with the Notes in mind, you should be able to lose 1 or 2 internet connections and there should be no routing loops. But you can probably see that it involves a fair bit of configuration per site. Not only do you need to configure IP SLA you need to modify your firewalls to limit the IP SLA to each site.
It's a bit messy but you have to have a way of checking whether the default-route in each site is actually available.
If anyone else has a better idea or am i missing something obvious then please jump in.
Thoughts ?
Edit - oh yes, and it would obviously need testing !!
Jon
