Internet redundancy using different sites?

[mwilson]

Hello all,  

Here is the scenario I am working on.   

I have multiple physical sites, each with redundant ISPs running BGP.  2 seperate ASs.  The sites all have Layer 2 adjacency.  I am running ospf for all internal routing.  I would like to eliminate 1 ISP at each site and have the other site be it's backup ISP.  

What is the proper / best practice to accomplish this?

I have tested a scenario where I have border routers peered with ISP.  I then peer the border router with each sites firewall with iBGP.  If I get a default route From ISP it is redistributed to the FW.  I then run OSPF on the LAN connection of the FW peering with each sites core router.  I redistribute the BGP default route only to the LAN via OSPF.  

If there is a failure at the FW level, border router, or ISP level the default route disappears and the other sites route appears.  Seems to be working as expected in LAB.  

Is this the correct method?  I have concerns with the FW running BGP and OSPF.  Are there security concerns doing this?  

Or is there a totally different approach to take?  Thanks in advance!

[e.ciollaro]

Hi
I think that, if properly configured it should work (probably some sort of metric manipulation is required to force firewalls to choose the right path) but in my experience routing protocol between routers and firewalls could bring to problems because:

• firewalls are not router and sometime neighbourship and advertisement don't work properly; in your scenario the default network is the only routes so it's really simple but ...  who will know...

• in many company firewalls and networks are managed by different team so setup and troubleshoting could be harder then expected

So, usually I prefer to just use FHRP.

I have border routers peered with ISP.  I then peer the border router with each sites firewall with iBGP.  If I get a default route From ISP it is redistributed to the FW

Don't forget that you have to properly configure eBGP to make one link primary and the other backup (usually as-prepend is the way)

Bye

enrico

PS: rate if useful

[mwilson]

Enrico,

  I'm not very familiar with FHRP so forgive me.   But I need to protect more than just the first,  or next hop.    The next hop needs to know about the fw status, border router, and ISP.   

Can FHRP do this?  Or just provide redundancy for the gateway?

Thank you.

[Julio E. Moisa]

Hi

Do you have a diagram of the planning?

Thank you in advance. 

[mwilson]

I attached the diagram.

[Julio E. Moisa]

Thank you, my understanding is, if the ISP 1 (left) is down the site A should go through site B and then go to the ISP 2 (right), is that correct?

[mwilson]

That is the goal Julio.  And vice versa.  Thank you again.

[Julio E. Moisa]

Ok perfect, before to study the case and provide the proper config, I would like to know first:

- How the site A and B are getting internet access, the iBGP firewall or router is advertising a default route to each site router or do you have any default static route on each router site A and B?

[mwilson]

Currently both Sites Core routers have a static route to  their sites Firewall.  So if the internet is down they just dead end.  My plan is to have the Default Route advertised by each ISP then "Handed down" to the next device. 

[Julio E. Moisa]

Ok perfect, It is already giving me an idea of your topology. Ok please let me provide you a solution. There are many ways other more complex than others.

[Julio E. Moisa]

Hi 

Below you will find a scheme that I have used before and works perfectly, it is using EEM scripts + IP SLA + Track. 

Router 2

Step 1) Create a static route to reach a public IP through the ISP1, do not advertise it into OSPF, so you can add a tag, example:

Google DNS

ip route 8.8.8.8 255.255.255.255 10.12.0.1  name SLA-TEST tag 777

Step 2) Create a SLA to be making ping periodically to the IP address above from a source who is able to go via Internet (any IP with internet access)

ip sla 10
icmp-echo 8.8.8.8 source-interface or source-ip <ip with internet access, it could be a loopback)
frequency 5   <--- each 5 seconds it will pinging the IP 8.8.8.8

Step 3) Enable the IP SLA

ip sla schedule 10 life forever start-time now

Step 4) Create a track 

track 10 ip sla 10 reachability
delay down 30  <--- it will generate the message %TRACK-6-STATE: 10 ip sla 10 reachability Up -> Down after 30 seconds without response. 

Step 5) Apply the track to the default route

ip route 0.0.0.0 0.0.0.0 10.12.0.1 track 10

Step 6) Create a EEM script to enable an automatic failover once the following message is showed on the router. 

event manager applet FAILOVER-INTERNET
event tag prim syslog occurs 1 pattern "%TRACK-6-STATE: 10 ip sla 10 reachability Up -> Down"
trigger
correlate event prim
action 1.0 cli command "enable"
action 2.0 cli command "configure terminal"
action 3.0 cli command "no ip route 0.0.0.0 0.0.0.0 10.12.0.1 track 10"
action 4.0 cli command "ip route 0.0.0.0 0.0.0.0 10.23.0.3 "
action 5.0 cli command "exit"
action 6.0 cli command "write memory"

So once the EEM is trigerred, it will remove the default route and create a new one pointing to the Router 3, the traffic will be moved to Router 3 and then to the firewall to the edge router and finally to the ISP 2.

The same can be implemented on the Router 3. 

Hope it is useful

:-)