RV345 Tunnel Failover Non-functional with Network Service Detection

[jzahrte-2]

We have small sites using RV345's with a Dual ISP setup connecting back to a datacenter hub over site-to-site IPSec tunnels. The intention is for when one WAN link is unable to ping 8.8.8.8 via Network Service Detection, it fails over to the secondary WAN link.

This failover appears to succeed, however for the IPSec tunnel to the datacenter, it clings onto the old connection and never forms the failover IPSec tunnel unless we manually go in and disable the original WAN port.

I was hoping that the failover would truly kick the original WAN port out of the equation. I do not want WAN1 sending IPSec keepalives while it should have failed over in it's entirety to WAN2 when the network service detection started failing. 

Is there any way to work around this? Potential bug?

Thanks in advance for any help,

Josh

[nagrajk1969]

Hi

when the network-service-detection fails to reach 8.8.8.8 (using pings) on wan1, the multi-wan service automatically makes the wan1 interface offline, meaning that NO traffic can be passed/routed thru the wan1 interface now. This also means that since wan1 is now made offline/in-active, the VPN tunnel configured on wan1 also does not process any traffic and the tunnel goes down

Therefore when the wan1 interface is offline, the wan-fail-over or Load-Balance mode switches/routes the traffic thru wan2 interface using wan2-ipaddr as the src-addr when sent on internet (as the internal network gets nated)

Now this multi-wan failover/loadbalance on RV340 IS NOT APPLICABLE TO VPN-TUNNELING. Meaning that the multi-wan service (which is the feature that controls the fail-over and load-balance of traffic on RV340) is NOT CONTROLLING THE VPN-SERVICES (s2s, c2s, l2tp-wIPsec, pptp, ssl-vpn)

So to provide for vpn tunnel backup for Site-2-Site tunnels on RV340, you can configure the tunnel-backup(or Tunnel Failover) to wan2 interface. The primary vpn tunnel will be on wan1 to say 100.100.100.x, and when its NOT possible to route over wan1 (due to it being set to offline by the multiwan service due to network-service-detection feature not able to reach 8.8.8.8 on wan1), the S2S tunnel simply starts RE-ESTABLISHING THE IPSEC TUNNEL ON WAN2....and it will continue to be on wan2 till the remote-peer is again reachable via wan1...in which case it simply rolls-back the s2s ipsec tunnel again automatically to wan1..

For the s2s tunnel primary-tunnel/failover-to-backup/roll-back-to-primary config, in the existing s2s tunnel config, ensure that

a) in the advanced config page, you enable keepalive AND DPD too

b) In the tunnel-failover tab, select wan2 and mention the ipaddress of the remote ipsec peer

i guess that should solve your problem

[MSC90233]

Dear nagrajk1969,

I've tested your purposal but it does not work Was your purposal tesed?

Maybe you can check my detailed issue because VPN Failover DOES NOT WORK: VPN backup settings do not re-connect the tunnel

What we have:
- 1 x RV345, 2 x RV160 & 1 x RV160W
- in some days the RV160W will be replace by RV340W

Settings RV345:
- WAN1 reachable by rv345_wan1.dyndns.org
- WAN2 reachable by rv345_wan2.dyndns.org
- as example: VPN tunnel to one RV160 works fine (settings okay)
- Site2Site - Advanced Settings: "NetBIOS Broadcast" and "Keep-Alive" (10sec) activated, "DPD" enable with 10 sec "Delay", 30 sec "Detection timeout" and "Delay Action" with "Clear"
- Site2Site - Failover Settings: activated, "Remote Backup IP Address" with "rv160_1.dyndns.org" and "Local Interface" with "WAN2"

Settings RV160 (one as example):
- WAN reachable by rv160_1.dyndns.org
- VPN tunnel to RV345 (rv345_wan1.dyndns.org) works fine (settings okay)
- Site2Site - Advanced Settings: "NetBIOS Broadcast" and "Keep-Alive" (10sec) activated, "DPD" enable with 10 sec "Delay", 30 sec "Detection timeout" and "Delay Action" with "Clear"
- Site2Site - Failover Settings: activated, "Remote Backup IP Address" with "rv345_wan2.dyndns.org"

Problem:
- works fine: If RV345 WAN1 is offline WAN2 takes over
- VPN Tunnel does not reconnect
- if WAN1 is online again VPN reconnects fine

very confusing addional test result:
- scenario: WAN1 is offline, WAN2 is online and the Tunnel does re-connect
--> if I un-check e.g. "NetBIOS Broadcast" and save the VPN backup tunnel connects and works
--> if I reconnect WAN1 and simulate a WAN1 disconnection again it WILL NOT RE-CONNECT AGAIN
--> if I check e.g. "NetBIOS Broadcast" AGAIN and save the VPN backup tunnel connects again and works

I've contacted CISCO Support and up to now we don't have a running WAN AND VPN failover configuration

Has anyone a tested and working configuration example for me?
Can you help us please?

There are some other questions by my side:
- I had to enable "Tunnel backup" on both sides - right?
- "Delay Action" had to be "clear" - right?

future questions:
- If we'll have the RV345 and RV340W in the future - how we should set up VPN Failover on each router?
- What is in case RV345 WAN1 AND RV340W WAN1 are offline at the same time and both are running on their WAN2 - what should be the "Remote Backup IP Address"?
--> Last week the ISP had a large outtage - both routers were offline.

Normally RV345 would reconnect VPN with his WAN2 (WAN1 offline) to WAN1 (and his dyndns address) of RV340.

If we use 2 x dual-wan routers with 1 Site2Site connection it would be nice to know how to configure VPN failover correctly, for both scenarios: a) just one router has WAN1 down AND worst case b) both wan1 are offline.
Or should be both backup settings with WAN2 remote adress?

kind regards

[nagrajk1969]

Hi

 I suggest that you should take this one step at a time...and first finalize the vpn topology, and which gateway plays which role in the topology (as a hub-gw or a spoke-gw) 

At the outset of your issue description, you mentioned that you will be deploying a Hub-and-Spoke VPN topology with a Hub-Gw at the Data-Center, and all other Spoke-Gws will be RV345s (and/or RV340s, RV160s, RV160W, RV340Ws)

1. So kindly please clarifiy as to who (and what router) is the HUB-GW in your deployment?

2. In a Hub-Spoke topology the vpn-backup tunnel will almost always be configured on the spoke-gws and most of these spoke/branch gateways/routers will be with dual-wan interfaces...so the primary vpn tunnel will be active on wan1, and when wan1 is down, the backup vpn tunnel will become active on wan2 interface. In both cases the s2s vpn tunnel will be to the Hub-Gw at Data-Center

3. And in the Hub-and-Spoke topology vpn tunnels, its generally configured to have spoke-to-spoke communication via hub-gw

4. And in hub-spoke vpn topology, the HUB-GW is configured as a Passive Gateway, and all the vpn tunnels from all spoke-gws (some of them maybe having dyanmic wan connections such as pppoe/dhcp) will be always initiated from spoke-gws only...

5. So your vpn s2s tunnel deployment should be something like in attached schematic

Yes i have a working deployment with primary/backup vpn tunnels on wan1/wan2...and i know it will work. So kindly please clarify which is the hubgw and which are the spoke-gws in your deployment...

6. And as for your additional queries raised by you:

>>>>I had to enable "Tunnel backup" on both sides - right?

NO NEVER. Only one peer will be enabled with tunnel-backup

- becos even if have a s2s tunnel between just 2 peers (instead of say in a hub-spoke topology between spoke to hubgw), if you have enabled backup tunnel on one ipsec peer1gw, the other ipsec peer2gw has to be configured as a passive-gw and on peer2gw, peer1gw is configured as a dynamic-gateway becos peer2gw does not know from which ipaddess the peer1gw will use to establish tunnel with it...so it will always wait for the vpn tunnel to be initiated from peer1gw

- so therefore similarly in Hub-spoke topoogy, hubgw will never know from which ipaddress the spoke-gws (RV34x, RV16x) will connect from...hence the hubgw is configured as  a passive-gw and all vpn tunnels are always have to be initiated from spoke-gws ONLY

>>>"Delay Action" had to be "clear" - right?

Yes absolutely.

[MSC90233]

I'm so glad to read your response

Please be informed: I'm not an expert in networking - But hopefully we get this solved

to your questions:

According to your PPT it is nearly perfect: the RV345 is our Hub Gateway or in my words: he is the master and located in our headquarter. In his local network are clients, different servers (Fileserver, Email Server, ...). To be save he has 2 internet connection to be almost always reachable for the 3 sites: Site 1 (RV160), Site 2 (RV160W --> will be replaced by RV340W) and Site 3 (RV160). All three sites have only 1 S2S tunnel to the RV345 and all sites should communicate to the headquarter only.

By the way:

The headquarter and the 3 Sites are using the same Cable ISP and last week he had a huge outtage. Afterwards my boss recongnized that not only the headquarter should have a internet failover option. Site 2 has 2 internet lines so they told me to buy a dual wan router... And maybe all sites will get a 2nd line - I don't know right now.

Maybe our network with 4 location sounds huge but this not really correct. Site 1 has 5 people, Site 2 has 2 people, Site 3 has 2 people and headquarter 10 people. So it's not big - just parted in 4 locations...

The headquarter has 3 S2S tunnels and with my words: he is the VPN master and main point. These 3 VPN tunnel are running fine and the sites are connected correctly.

I've contacted cisco support and they told me we can create 3 tickets (for each S2S) or we make just 1 ticket. Well - in my opinion I "just" need the solution for 1 S2S failover and I could adapt this to the other 2 S2S by myself.

Well within this ticket I sent configuration files of RV345 and Site 1 (RV160), a lot of screenshots of their settings and we've wrote a few times. Last purposal of the support agent was: please activate for this S2S tunnel the following things: in RV345 failover settings the dyndns address of RV160 and interface WAN2. In RV160 I should activate tunnelback up too, dyndns address of RV345 WAN2 and the interface WAN.

This does not work like mentioned...

You can laugh but up to now I don't know which one of the routers is the Hub/Master/Initiator and who is the spoke/slave :o( I'm checking the settings and the manual but I can not find this.

I'm so unhappy with the manuals or FAQ's: it's not really easy to find the solution or correction configurations. The descriptions to vpn failover and it's settings are pretty sparse. I know there are many and different setup's possibilities in VPN and at he firms - up really high complexity. Well - I thought and up to now I think we have a very similar and expedient plan and topology.

kind regards Marco

[nagrajk1969]

hi

thanks for the details of your deployment.

Ok, i can definitely and i will surely post a complete set of detailed configs & steps required to be applied on each of the routers in your setup...its not so complicated at all, once you get to understand the flow and the logic of why it is to be applied as such

But i do need clarification from you on below points, before i note down the configs that will be optimum and correct for your setup

1. Do you also require the hosts between site1/site2/site3 to communicate with each other via the ipsec tunnel to mainGw/HubGw????

2. Are there multiple subnets used in the vpn tunnel between "SiteN- MainGw"...??

3 Is the Ipsec tunnel using IKEv2 or IKEv1?

I will post the complete set of config details for what i consider to be best/correct/optimum solution for your vpn deployment. Then you can decide whther to apply it or not

thanks & regards

[MSC90233]

Wow - thank you so much

responses:

1) each site should only communicate to the headquarter / RV345

2) well - maybe I don't understand correctly, here my response:

RV345: 192.168.50.0

Site 1 (RV160): 192.168.2.0

Site 2 (RV160W): 192.168.3.0

Site 3 (RV160): 192.168.8.0

3) I've configured v2 for all 3 S2S tunnels on all devices.

Just to be sure: each S2S tunnel of each Site works fine with RV345. Only VPN failover does not work

thanks

[nagrajk1969]

Hi

Please find included here the required s2s vpn tunnel configs in your network deployment.

1. These are working configs and i have checked these in my network setup. Ofcourse in my case i have used different dns-names (and not the dyndns.org domain) and i have RV260s in my setup instead of RV160. 

2. In my case i have also used a RV345 as Main-Hubgw with wan1 and wan2 interfaces active

3. The attached configs are to be configured as given on your setup at main-hubgw and on each of site1, site2 and site3 routers

4. As i had mentioned earlier, these attached configs the optimum and correct configs that should be applied in your deployment.

one important note: (and the reason why it did  not work when you tried to configure the vpn primary/backup with network-service detection)

1. Do not try to enable/configure any primary/backup s2s tunnel (for any of the site1/site2/site3) on the Main-Hub-RV345. Its not correct and bound to fail

2. The vpn backup tunneling should always be configured on Site1/Site2/Site3 routers and due to keepalive being enabled, always either the primary or the backup tunnel will always be up

3. As mentioned in the attached config file in case a RV34X is used in site1/site2/site3 (instead of RV160x), i suggest that you should configure the network-service-detection as mentioned for wan1 (assuming that wan1 is the primary interface configured in multi-wan gui page).

4. On Main-Hub-Gw RV345, you may configure remote-host 8.8.8.8 for the network-service-detection.

It has worked for me, and i truly hope it will work in your setup too

thanks

[MSC90233]

Dear nagrajk1969

Thank you for your patience and your support so far, I'm really exited.

Your configuration method works fine, but this is not even close to what you in read in the manual and what Cisco technical sends me as a solution

After I got some Rest was able to test ist and deal with it, I understood your mentioned principle, was able to adapt ist and already bring it into the live enviroment or switch it. I then created another WAN1 outage and it did exactly what it was supposed to - fantastic

There is one more & new issue:

During the testing a new question came up, which I unfortunately could not solve myself. Then came the RV340W, which I have set up as Site 4 at the moment. In my test, I can currently only connect one active internet line to this. I have se the "same" settings on the hub as for Site 1/2/3. I set the S2S tunnel on the RV340W according to the scheme of the 3 other PeerGW's. In the failover tab I have stored the rv345wan2 dyndns address and set the interface to wan1.

The wan1 tunnel is connected and when i disconnect WAN1 connection of the RV345 - as the other 3 S2S's - the WAN2 tunnel is active. However, if WAN1 is active again, all S2S tunnels go back to their WAN1 tunnel - except that of the RV340W: it remains connected to WAN2.

Do you have any idea?

It is quite conceivable that perhaps more dual wan will be purchased, but these are initially only connected to just one internet line. I just want to be prepared and I just noticved that while testing.

Would be so nice If you can help me one more time

A question form me by the way: you are a network specialist right?

kind regards

[nagrajk1969]

Hi

>>>It is quite conceivable that perhaps more dual wan will be purchased, but these are initially only connected to just one internet line.
>>>>I just want to be prepared and I just noticved that while testing.

On the SiteX/SpokeGw/BranchRouters, there is no mandatory rule that there has to be 2 wan-internet links.

- You can use only 1 internet link on RV34X, and still get to have vpn-tunnel backup to the Main-HubGw router.
- But in such cases of 1 internet-link on siteX routers, the expectation/assumption is that the internet-link does not go down...then you dont have any backup at all on the siteX/branch router

- For example RV160/RV260 have only 1 wan interface (if you ignore the support for 4G-USB dongle support, which would be quite expensive link to use anyways for most times). Here you cant do anything about the 1 internet link only.

- So presently if you have a RV340W in site4 and there is only 1 internet link (which we will assume that it never goes down), then its still ok, you can and you should be and you will be able to configure and use the vpn backup functionality becos in your case the main-HubGw has 2 wan links...so you can connect to wan2-link of HubGw if your are unable to reach to wan1-hubgw-link from the RV340W-site4 router

>>>Would be so nice If you can help me one more time

1. So ok you must have already applied this, but to confirm again:

a) you now added a new RV340W router in site4, and you have to configure the s2s tunnel as in attached doc - OnRV34X_Site4Gw_S2S-TunnelConfig_toHubGw.txt

b) And on the Main-HUBGw-RV345, your s2s tunnel config for Site4 would be as in attached doc - OnRV345MainHubGw_S2S-TunnelConfigs-to-Site4.txt

2. Only on Site4 RV340W,

a) can you provide the config-details of your wan-interfaces (from the WAN page of GUI) - Is WAN1 configured with a static-IPaddr or PPPoE-WAN or DHCP?

b) In the WAN/MULTI-WAN page, is wan1 the primary interface (with 1) and wan2 the secondary (with 2)? Or is it wan2 the primary (with weight 1) and wan1 the secondary (with 2)?

c) In the Multi-WAN page, if you click on WAN1 interface settings for "Networ Service detection", is it enabled now? If yes, what is the configs applied presently?

d) Can you do just once, a permanent save of existing config and REBOOT once? and then check the vpn tunnel behavior (when the wan1-link of HUBGW becomes unreachable from this RV340W-site4?

e) If after the reboot, there is still an issue, can you just check by disabling the "network service detection" for wan1 interface and do a apply/save and reboot once more?

Note: iam suggesting "reboot" to be done only 1 time (or maybe 2nd time in point-e) as a last resort...just to see if it works after the router has been re-initialized after a reboot...its not to be done everytime there is an issue...this is only for debugging effort.

>>>>A question form me by the way: you are a network specialist right?
- Yes Iam, since last 24+ years. But i will be and should be humble enough to say that i am not an expert in every aspect of networking.
- I just know some of the things only. I still am and will always will be a beginner and have lots to learn and understand.
- So i keep reading and learning from every person (beginner or expert).
- Hence my policy is to share everything of what "i think" i know and keep learning from the knowledge sharing feedback and experience, and if i don't know any aspects of networking then i simply say i don't know and i dont have knowledge in that area...presently.

- Life is simpler and happy when you speak the truth everytime becos Nobody is an expert or perfect. 

Sorry if i seem to be lecturing and too philosiphical for my own good...

So coming to reality..lets hope that your issues get solved one by one...best regards

[MSC90233]

Dear nagrajk1969

This the point to laugh, be happy and shake your head at same time: it works...

I really treid it 3 times, deleted the configuration for the Site4 tunnel on both sides, set it up again, and again the problem that it remainded with WAN2 and did not go back to wan1. Thank you for your answer, I checked the settings again afterwards, didn't change anything and now it works. I think I'm going crazy

That with Site4 was really just for testing. I will soon replace the RV160W with the RV340W and then it will run with 2 lines. I will then test your configuration for the spoke with 2 active WAN interfaces - but I am very confident. Now I've practicakky confirmed that your configuration also works for a dual WAN router with only one active line.

I will post my next experiences

I am so grateful and impressed by your support - thank you so much

[MSC90233]

Dear nagrajk1969

The RV160W has now been replaced by the RV340W and the last step has also been successfully implemented. The mix of RV345 as a hub and the various spokes (RV160's, RV340W) works flawlessly: the failure safety - especially in the article on VPN - works 100% and now exactly what was planned happens.

It is a shame that it could not be solved with the manual, the FAQs and the CISCO support. I am incredibly happy and relieved that you helped me exactly to the point and I would like to thank you again 1000 times for your patience, your nerves, your knowledge and your explanations. Once understood, it works as it should.

Thanks!