Setup multiple ISP for failover of VPN traffic - BGP or Static NAT?

[femi.agboade]

Hello,

I have the task of setting up the following:

Router, probably running BGP, terminating 3 ISPs, configured to provide failover. These 3 ISPs are the outside interfaces and they have all provided their separate range of public IP addresses. The 3 ISPs only provide internet access for VPN tunnels to run over them and nothing more. So no need to worry about internet bound traffic or the other way round.

The inside interface of the router is connecting to an ASA that is configured to run site to site VPN configuration between this site and 5 other remote locations. Different ISPs provide the internet access in those remote locations. Each remote location has an ASA where their VPN tunnel runs through.

So what I need to achieve is configure the router and ASA in such a way that the router provides a failover from ISP1 to ISP2 whenever ISP1 fails. This failover would be transparent to end users but will be automatic such that the VPN tunnel will then be established over the ISP that has been failed over to.

I'm thinking BGP on the router with multihoming configuration but then i get stuck when I have no AS number and no internet routable block of IPs to use between the router's inside interface and the outside interface of the ASA.

I have an image attached.

Any ideas on how best and easily to go about this? BGP or any other solution?

Regards,

Femi

[Joseph Nelson]

Hi Femi,

Sounds like you need DMVPN, no? However, DMVPN aside, looks like you only need to do the following:

• On your router, create 3 IP SLA probes, one tracking each ISPs next hop
• Create 3 default routes, tie them to the IP SLA probes, set AD corresponding to your order of preference
• On your ASA, create default route to router

Your VPN is using the router AND the ISPs as transport. If the ASAs at the remote site have public IP addresses, then from the perspective of the local site's VPN, it doesn't matter which ISP ( path) is taken to reach its remote peer. You want to avoid load-balancing between ISPs however, as VPNs are sensitive to latency/packet loss.

HTH

Joe

Rate if helpful.

[femi.agboade]

Hello Joe,

Thanks for your feedback.

Your suggestion does sound interesting and I would be willing to give it a try immediately.

How do you create SLA probes to track the ISPs' respective next hops? Would you be so kind as to give me a sample config capturing your suggestion that I can start with please?

For the VPN on the local (HO) side, yes it doesn't really matter which ISP is taken to any of the remote peer. However, I believe the remote peer must be configured in such a way to allow the setup of a VPN tunnel to the HO site, regardless of which of the 3 ISPs its coming through. So provision, I believe must be made for all 3 ISPs in the peer to peer VPN config, right? This for me is where my second concern lies...

Thanks again and hope to hear back from you soon.

Regards,

Femi

[Joseph Nelson]

Hi Femi,

Based on your requirements, a DMVPN setup definitely seems most appropriate. You should look into it as I believe ASA can do DMVPN.

Now aassuming all you can do is site to site VPNs, then yes you need to configure multiple tunnels at the remote site. The ASA then would need an IP address on each of the IP blocks assigned from the ISPs at the HO for terminating the VPNs.

If you can get your own direct IP allocation then your ASA needs only one IP from your allocation for VPN termination. In this scenario, you could run BGP with the ISP but this may not be strictly necessary (you could get them to just use a stactic route for your allocation with the next hop being your side of the /30 with them). An ASN is required however if you use BGP.

If you go with a DMVPN solution, then your HO ASA need only have an IP from each of the ISPs blocks. Your remote sites then just get configured with 3 next hop servers.

[femi.agboade]

Hello Joe, Thanks again for the feedback. I will read up on DMVPN, would appreciate if you can give me some links or sample config similar to my scenario. Note that I may be unable to configure the DMVPN directly onthe ASA because the router has been made available with the required number of interfaces to accommodate the 3 ISPs while the ASA has just 2 interfaces. Also, client doesn't want me touching or making major config changes to the ASA so I'll focus more on the router. Don't know much about DMVPN yet, but I hope I can get some references from you. Client does not have its own block of public IPs or ASN which is why BGP for me would be an issue and hence reason for exploring other options of getting this done. So we will be making use of range of public IPs provided from each ISP. By site to site VPN config, I assume this is different from DMVPN? Kindly explain a little more about this option of config please? Regards, Femi

[Joseph Nelson]

Femi,

Your question is really design focused, so I start to address it at the design level. I cannot provide any configuration examples until I know specific environment and have your specific configurations. Please check reference appropriate for your version of ASA/ IOS. Please see the attached solution diagram. This solution does not address tunnel security or routing between sites-- I only attempt to address the VPN aspect given your description and your requirements.

Basically, at the HO site:

• On the HO ASA, you crate VLAN interfaces and convert the port facing the router into a trunk port ( see reference)
• The router provides transport by bridging the ISP connections with the ASA. Facing the ISP, you use the "dot1q encap X native" interface command. On the port facing the ASA, you configure sub-interfaces and use the "dot1q encap X" interface command on the sub interface. This allows the router to bridge traffic between the ISP and ASA. 

As far as the HO ASA configuration, you need to do the following:

• In addition to the above, you need to Configure Dynamic IPSec crypto map. Your remote sites probably do not have static IP addresses, this allows their IP address to change while still letting the VPN tunnel come up

On the Remote Site ASAs the configuration is reduced to specifying a single Crypto-Map with multiple peers (see diagram):

• See the "set peer" crypto map reference. Your remote site crypto map needs to set the connection type to "originate-only" in order to use multiple peers with the "set peer" command. You will specify three peers, one for each of the HO ASA VLAN interfaces
• You can establish an order or preference between peers in the crypto map using the "set peer x.x.x.x default" crypto map command. See reference. In this way, the VPN will fall down the list of configured peers, if the HO ASA fails to reply on one interface ( i.e. ISP is down), it will try another interface (hopefully ISP is up)

Now, about DMVPN

You can only use DMVPN with Routers as GRE is not supported on ASA (IIRC). The ASA version of DMVPN is Dynamic IPSec tunnels. From a design perspective, if you used DMVPN on router vs. Dynamic IPSec on ASA, you are just moving the tunnel functionality from one device to the other. In your case, you've already paid for an ASA, might as well use it--DMVPN may require a VPN license on the router which may come as a cost. 

Here is an excellent introduction to DMVPN. And and overview as well. 

HTH.

Rate if helpful

[Joseph Nelson]

Hi Femi,

Also wanted to point out a few more things regarding my solution:

• Asymmetric Routing: There will be some Asymmetric routing due to the ASAs use of default routing. Suppose Remote1 has failed its VPN tunnel over to 2.2.2.2 after HO ASA fails to respond on 1.1.1.1 interface (see diagram). Suppose also that HO Router has figured out that it shouldn't use ISP1 as a next hop ( this is one of the routing issues I have not discussed). In this case, HO router will choose from ISP2 or ISP3--suppose ISP3 is chosen as default route, then from Remote1, the VPN is asymmetric ( i.e. Remote->HO takes ISP2 path, but HO->Remote takes ISP3 path). The same applies to all remote sites. This is not strictly a bad thing, but for troubleshooting/maintenance purposes you should be aware of this behavior
• Bridging ISP to ASA: Strictly speaking, you do not have to use the "encap dot1q x" config on the HO Router interface facing the ISP ( you don't even need to use sub-interfaces there). If you don't bridge the ISP device with the ASA as I described, then the sub-interfaces on the HO router facing the ASA need to be addressed with the IP block given by the ISP. For example, if ISP assigns a 1.1.1.1/30 and a 100.100.100.0/29 block--HO router uses the /30 and the gi0/0.10 interface uses the /29 address space
• Traffic Engineering: Though you only have 5 sites, you need to think about the volume of traffic you're getting from each site as that is the volume of traffic you will see on the VPN from that site. Given that the HO Router<->HO ASA link is only 1Gbps, you can see how the link could be oversubscribed.
• Hair-pining VPN traffic: If you need site-to-site connectivity between your remote sites, you can hair-pin the traffic at the HO ASA. To do this, each of your ASA Vlans will be at the same security level. Then you need to enter the "same-security-trafic inter-inteface" configuration ( see reference)

Please let me know your feedback.

[femi.agboade]

Hello Joe,

Thanks for this new insight. However, I did struggle to understand the concept of introducing the VLANs in this config, and I still am struggling to figure this out.

I however will attempt to shed some more light on the existing config and setup. Please refer to the attached image which pretty much shows a logical connection between the remote sites and the HO.

In summary, each remote site has an ASA, and one single ISP connection which is provided by a different ISP from the others. Each remote site has an ISP provided public IP address configured on the outside interface of the ASA and also used in the setup of the point-to-point tunnel between the Remote site and the HO.

As you already know, the HO also has an ASA directly connected to the router. The router has a physical connection to all 3 ISPs. The ASA has a point-to-point static VPN config to all the remote sites.

See a piece below from the HO ASA:

crypto map Outside_map 1 match address Outside_cryptomap_1
crypto map Outside_map 1 set pfs
crypto map Outside_map 1 set peer a.a.a.a
crypto map Outside_map 1 set transform-set ESP-3DES-SHA
crypto map Outside_map 1 set security-association lifetime seconds 28800
crypto map Outside_map 1 set security-association lifetime kilobytes 4608000
crypto map Outside_map 2 match address Outside_cryptomap
crypto map Outside_map 2 set pfs
crypto map Outside_map 2 set peer b.b.b.b
crypto map Outside_map 2 set transform-set ESP-3DES-SHA
crypto map Outside_map 2 set security-association lifetime seconds 28800
crypto map Outside_map 2 set security-association lifetime kilobytes 4608000
crypto map Outside_map 3 match address Outside_cryptomap_2
crypto map Outside_map 3 set pfs
crypto map Outside_map 3 set peer c.c.c.c
crypto map Outside_map 3 set transform-set ESP-3DES-SHA
crypto map Outside_map 3 set security-association lifetime seconds 28800
crypto map Outside_map 3 set security-association lifetime kilobytes 4608000
crypto map Outside_map 4 match address Outside_cryptomap_4
crypto map Outside_map 4 set pfs
crypto map Outside_map 4 set peer d.d.d.d
crypto map Outside_map 4 set transform-set ESP-3DES-SHA
crypto map Outside_map 4 set security-association lifetime seconds 28800
crypto map Outside_map 4 set security-association lifetime kilobytes 4608000
crypto map Outside_map 10 match address Outside_cryptomap_10
crypto map Outside_map 10 set pfs
crypto map Outside_map 10 set peer e.e.e.e
crypto map Outside_map 10 set transform-set ESP-3DES-SHA
crypto map Outside_map 10 set security-association lifetime seconds 28800
crypto map Outside_map 10 set security-association lifetime kilobytes 4608000
crypto map Outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map Outside_map interface Outside
crypto isakmp enable Outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

tunnel-group a.a.a.a type ipsec-l2l
tunnel-group a.a.a.a ipsec-attributes
pre-shared-key *
tunnel-group b.b.b.b type ipsec-l2l
tunnel-group b.b.b.b ipsec-attributes
pre-shared-key *
tunnel-group c.c.c.c type ipsec-l2l
tunnel-group c.c.c.c ipsec-attributes
pre-shared-key *
tunnel-group d.d.d.d type ipsec-l2l
tunnel-group d.d.d.d ipsec-attributes
pre-shared-key *
tunnel-group TEST type remote-access
tunnel-group TEST general-attributes
address-pool TEST-Pool
default-group-policy TEST
tunnel-group TEST ipsec-attributes
pre-shared-key *
tunnel-group e.e.e.e type ipsec-l2l
tunnel-group e.e.e.e ipsec-attributes
pre-shared-key *
! 

So with all your explanations so far, I'm not sure how to go about this issue.

What i initially had in mind was static routes in each location between that location and the HO. I was also exploring how to use this along with some sort of IP SLA with tracking. What do you think?

Regards,

Femi

[femi.agboade]

Hi Joe,

For the Asymmetric Routing issue, is it not possible to configure the HO Router to select the ISP to use in an order of preference such as ISP1 being preferred, ISP2 second preferred, ISP3 third preferred? Even though, thinking about it again, this may not resolve the issue entirely…

Bridging ISP to ASA: This for me was a confusion and I was lost at some point. Do you mind sharing sample config of what needs to be done on the HO ASA and the HO Router in light of this new development of not bothering to bridge ISP to ASA?
Going by your example, each ISP assigns only 1 range of IP block to use on the out interface of the HO Router. In our case we will have 3 blocks configured on 3 different out interfaces of the router. Does this imply using this sub interface option on the interface between the HO Router and ASA would not fly as we have to use a range of IPs provided by the ISP or can we use private IPs?
Could you help with a sample config, on both ASA and Router sides, to achieve this as I wasn’t clear with what i read up in the reference you provided and how it applies to my scenario?

Traffic Engineering: Yes you are right about this and we will manage this quite well

Hair-pining VPN traffic: The client already has site-to-site configured between remote sites. I however do not know how this option will apply to the new config because I still do not understand how the VLANs come into play here.

Here is what i came up with to be applied at the remote site ASAs
crypto map HO_ISP 10 ipsec-isakmp
crypto map HO_ISP 10 set connection-type originate-only
crypto map HO_ISP 10 set peer 1.1.1.1 default
crypto map HO_ISP 10 set peer 2.2.2.2 3.3.3.3
crypto map HO_ISP 10 set security-association idle time 120 default

I believe this will set preference in this order: 1.1.1.1, 2.2.2.2, 3.3.3.3?

As always, thanks for your valuable contributions.

Regards,

Femi