3 ISPs on one router, failover, site to site VPN

[femi.agboade]

Hello,

I have an existing setup as follows:

HO ASA is directly connected to ISP1 running peer-to-peer VPN with 5 remote sites also having ASAs as endpoints terminating the VPN tunnels. Note that the ISPs at the different remote locations are all different so the peer-to-peer VPN is actually run over the internet. 

Because client experiences downtime from ISP1 at HO they went ahead to get 2 new ISPs making 3. A router with additional interface card has been purchased and client wants to configure all 3 ISPs on the HO router in a manner to provide failover from one ISP to the other when there is an issue with the former. VPN traffic between the HO and the sites are what passes through these physical and logical connections.

Task now is to configure the new HO Router with the 3 ISPs such that when ISP1 fails, ISP2 picks up routing traffic. HO ASA will remain in the picture and will maintain its primary function of managing all VPN related traffic between the HO LAN and remote site LAN. Remote site ASAs need to be configured in such a way that they can track when ISP1 at HO is down and accept or initiate VPN traffic from/to the HO via ISP2. I have attached a sketch of what the network topology must look like after the setup is complete.

I have spent some time trying to introduce BGP but the client does not have its own range of public IPs and AS number, so that has been put on ice.

I have also considered DMVPN, but this also is a problem because all the end point devices at the remote sites are all ASA devices.

So i am now stuck and really lost of what next to do. Would appreciate any advice and probably sample configs that can help.

Regards,

Femi

[jc84_]

The IP address of the HO ASA - do you own the public IP address (ie. Provider Independent Address)?

[femi.agboade]

Hi El,

No we don't own the public IP on the HO ASA. All are ISP provided and therefore dependent on the ISPs.

Regards,

Femi

[femi.agboade]

Hello,

So i have the following config for the HO Router, a little something i think for the HO ASA, but not a clue on what to do on the remote ASA to allow it monitor all three HO ISPs and know when to route traffic through a working ISP, any ideas please?

:

HO ROUTER

service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname HO_Router
!
boot-start-marker
boot-end-marker
!
no aaa new-model
ip subnet-zero
ip cef
!
no ip domain lookup
no ip dhcp use vrf connected
!
!
!
no ipv6 cef
!
!
multilink bundle-name authenticated
!
ip audit po max-events 100
!
!
!
crypto isakmp policy 1
 encr aes 256
 authentication pre-share
!
! 
!
crypto ipsec transform-set AES256SHA esp-aes 256 esp-sha-hmac 
 mode transport
!
!
crypto keyring ISP1  
  pre-shared-key address 0.0.0.0 0.0.0.0 key tesTkey1 
crypto keyring ISP2  
  pre-shared-key address 0.0.0.0 0.0.0.0 key tesTkey2 
crypto keyring ISP3  
  pre-shared-key address 0.0.0.0 0.0.0.0 key tesTkey3 
!
!
!
!
crypto isakmp profile ISP1
   keyring ISP1
   match identity address 0.0.0.0 
!
!  
crypto isakmp profile ISP2
   keyring ISP2
   match identity address 0.0.0.0 
!
!
!  
crypto isakmp profile ISP3
   keyring ISP3
   match identity address 0.0.0.0 
!
!
!
!
crypto ipsec profile IpsecProf1
 set transform-set AES256SHA 
 set isakmp-profile ISP1
!
!
crypto ipsec profile IpsecProf2
 set transform-set AES256SHA 
 set isakmp-profile ISP2
!
!
crypto ipsec profile IpsecProf3
 set transform-set AES256SHA 
 set isakmp-profile ISP3
!
!
!
!
!
interface Tunnel1
 bandwidth 2000
 ip address 172.16.10.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication nhrpauth
 ip nhrp map multicast dynamic
 ip nhrp network-id 53
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 ip ospf 1 area 0
 delay 100
 tunnel source GigabitEthernet0/1
 tunnel mode gre multipoint
 tunnel key 1
 tunnel protection ipsec profile IpsecProf1
!
interface Tunnel2
 bandwidth 2000
 ip address 172.16.20.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication nhrpaut2
 ip nhrp map multicast dynamic
 ip nhrp network-id 54
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 ip ospf 1 area 0
 delay 100
 tunnel source GigabitEthernet0/2
 tunnel mode gre multipoint
 tunnel key 2
 tunnel protection ipsec profile IpsecProf2
!
interface Tunnel3
 bandwidth 2000
 ip address 172.16.30.1 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication nhrpaut3
 ip nhrp map multicast dynamic
 ip nhrp network-id 55
 ip tcp adjust-mss 1360
 ip ospf network point-to-multipoint
 ip ospf 1 area 0
 delay 100
 tunnel source FastEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 3
 tunnel protection ipsec profile IpsecProf3
!
!
!
!
interface GigabitEthernet0/0
 desc To ASA
 ip address 10.0.1.1 255.255.255.0
 ip nat inside
 ip ospf 1 area 0
ip ospf authentication message-digest
ip ospf message-digest-key 1 md5 cisco123
 duplex auto
 speed auto
!
!
interface GigabitEthernet0/1
 desc ISP1
 ip address a.a.a.2 255.255.255.254
  ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
!
!
!
interface GigabitEthernet0/2
 desc ISP2
 ip address b.b.b.2 255.255.255.254
  ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
!
!
!
interface FastEthernet0/0/0
 desc ISP3
 ip address c.c.c.2 255.255.255.254
  ip nat outside
 ip virtual-reassembly
 duplex auto
 speed auto
!
!
router ospf 1
 default-information originate always

!
ip nat inside source route-map EXIT_ISP1 interface GigabitEthernet0/1 overload
ip nat inside source route-map EXIT_ISP2 interface GigabitEthernet0/2 overload
ip nat inside source route-map EXIT_ISP3 interface FastEthernet0/0/0 overload
ip forward-protocol nd
ip classless
!
!
ip sla responder
!
!
ip sla 1
 icmp-echo a.a.a.1
ip sla schedule 1 life forever start-time now
ip sla 2
 icmp-echo b.b.b.1
ip sla schedule 2 life forever start-time now
ip sla 3
 icmp-echo c.c.c.1
ip sla schedule 3 life forever start-time now
!
!
track 10 ip sla 1 reachability
 delay down 1 up 1
!
track 20 ip sla 2 reachability
 delay down 1 up 1
!
track 30 ip sla 3 reachability
 delay down 1 up 1
!
!
!
!
no ip http server
no ip http secure-server
ip route 0.0.0.0 0.0.0.0 a.a.a.1 track 10
ip route 0.0.0.0 0.0.0.0 b.b.b.1 track 20
ip route 0.0.0.0 0.0.0.0 c.c.c.1 track 30
ip route 172.16.10.0 255.255.255.0 a.a.a.1
ip route 172.16.20.0 255.255.255.0 b.b.b.1
ip route 172.16.30.0 255.255.255.0 c.c.c.1
!
!
!
!
access-list 110 permit ip 10.0.1.0 0.0.0.255 any
!
route-map EXIT_ISP1 permit 10
 match ip address 110
 match interface GigabitEthernet0/1
!
route-map EXIT_ISP2 permit 10
 match ip address 110
 match interface GigabitEthernet0/2
!
route-map EXIT_ISP3 permit 10
 match ip address 110
 match interface FastEthernet0/0/0
!
!
!
!
!
control-pane
!
!
line con 0
line aux 0
line vty 0 4
 login    
!
end

**************************************************************************************

HO ASA

interface GigabitEthernet0/0
 ip address 10.0.1.2 255.255.255.0
 nameif outside
 security-level 0

 ospf authentication message-digest
 ospf message-digest-key 1 md5 cisco123
 speed auto
 duplex auto
!
!
interface GigabitEthernet0/1
 ip address 192.168.0.1 255.255.255.0
 nameif inside
 security-level 100

 ospf authentication message-digest
 ospf message-digest-key 1 md5 cisco123
 speed auto
 duplex auto
!
!
router ospf 1
 network 10.0.1.0 255.255.255.0 area 0 
 network 192.168.0.1 255.255.255.0 area 0 
!
ip route 0.0.0.0 0.0.0.0 10.0.1.1
!
ip forward-protocol nd
access-group OUTSIDE-IN in interface outside
!
!

[milan.kulik]

Hi. what about running some routing protocol (EIGRP, OSPF, e.g) through the tunnels and advertise the HO prefixes with the best metric through Tunnel1 to the remote sites, middle metric through Tunnel2 and worst metric through Tunnel3? I'm not an ASA expert but I believe it should be able to run dynamic routing protocols? Best regards, Milan

[femi.agboade]

Hello Milan,

Thanks for your response, I however didn't quite catch how your explanation would work. Do you mind being slightly more explicit, maybe with diagrams etc?

Regards, Femi

[milan.kulik]

Hi Femi,

the idea was following:

3 Tunnels in parallel from the HO to each remote site. And the HO subnets advertised through each tunnel via some dynamic routing protocols with different metrics.

The best metric used for Tunnel1, middle metric for Tunnel2 and worst metric for Tunnel3.

So as long as the Tunnel1 would be Up, the remote site woud use it to reach teh HO subnets.

If Tunnel 1 would fail, the remote site would use Tunnel 2, and Tunnel 3 if both Tunnel1 and Tunnel2 would be down.

You would need to configure the same for the opposite direction:

Each site to advertise its subnets with best metrics through Tunnel1, etc.

I know this would be pretty complicated.

But might work possibly?

Best regards,

Milan

[femi.agboade]

Hello Milan,

Thanks again. I never knew "complicated" to be anywhere near pretty, and this definitely isn't looking pretty at all.

If i understand you correctly, for all 5 remote sites, i may have up to 15 tunnels configured on the HO Router, 3 tunnels for each site? I honestly don't even know where to start with this configuration. I also don't know much about the metrics you mentioned.

Would be nice if I can get sample configs to guide me.

Regards,

Femi

[David_Che]

Hi,

It seems to me you have almost all the configuration to accomplish what you wish.

Let me show my solution to this scenario:

ON HO:

1: setup 3 DMVPN tunnels

2: setup ospf neighbor over 3 DMVPN tunnels with remote sites

3: decrease ospf hello time to less than 10 seconds to convergence as fast as possible

ON each Remote sites:

1: setup 3 P2P protected GRE tunnels

2: setup ospf neighbor over 3 protected GRE tunnels with remote sites

3: decrease ospf hello time to less than 10 seconds to convergence as fast as possible

So you can see the solution is same as yours. I think it can works.

Good luck,

David

[milan.kulik]

Hi,

plus you need to play a little with the OSPF costs to assure the tunnel through ISP1 is used and only "...when ISP1 fails, ISP2 picks up routing traffic" (as requested in the original post).

Which also means you should use the same tunnel both directions.

So you should advertise the HO subnets with best OSPF cost via ISP1 tunnel, middle cost via ISP2 tunnel and worst cost via ISP3 tunnel.

And similar way the remote site prefixes should be advertised with best OSPF cost via ISP1 tunnel, middle cost via ISP2 tunnel and worst cost via ISP3 tunnel.

Best regards,

Milan

[femi.agboade]

Hello Milan,

So again, thanks for your valuable input.

What I really need right now is a validation of my HO Router and HO ASA configs, and possibly a sample config for the Remote ASA. Can you help with this please? ASA isn't a strong point for me so i am lacking a bit here.

Regards,

Femi

[milan.kulik]

Hi Femi,

unfortunately, I'm also not an ASA expert.

I'm not sure if DMVPN feature is supprted on it.

I can only say: Generally, you need to establish the tunnels somehow - 3 tunnels (each terminated on one ISP interface on the HO site) from each site.

And then to run some dynamic routing protocol through them.

So maybe I'd start with the current solution - I suppose you've got one tunnel from each site running already?

So why not to take one site as a pilot? And add second tunnel there?

When the second tunnel is Up, I'd try to run a dynamic routing protocol through the two tunnels to the HO. Then  to play with the routing costs to make the site to prefer one tunnel.

Then to add the third tunnel to the site.

And finally repeat the steps on other sites.

Does it make a sense?

I'm sorry I can't help you more but I've got no ASA available at the moment.

Best regards,

Milan

[femi.agboade]

Hello Milan,

Its ok, you've been quite helpful enough already. I'll put your suggestions together along with that from David and see if I can use this to come up with a solution. I'll let you know how it goes.

Thanks again.

Regards,

Femi

[femi.agboade]

Hi David,

Thanks for your comments.

I'm not sure however that you realize that at the remote sites, I only have the ASAs, no routers. As a matter of fact, there already exists the site to site VPN between the remote sites and the HO, both HO and the remote sites all use ASAs. Its the introduction of managing the 3 ISPs at the HO that brought about the HO Router.

So my concern really is how to configure the ASAs at the remote sites to work with the new HO Router and the 3 ISPs on the HO Router.

So according to your comment, is my HO Router config displayed in previous post correct?

I am not very good with ASAs, wouldn't mind if you can help with sample config on what is meant to be added as per your recommendation above on the 3 P2P protected GRE tunnels to be configured at each remote site, and all the associated OSPF config. I am not sure how to configure OSPF routing and metrics on the ASAs.

Also, is my config on the HO ASA as shown above also correct considering my peculiar scenario where the HO ASA only manages all site to site traffic routing?

Thanks as always.

Femi