DMVPN IPsec BGP tunnels with backup

[Matthewmunhall]

I have a site that currently has 2 DMVPN tunnels with BGP.  Tunnel 1 routes to our primary DC and Tunnels 2 routes to our backup DC.  We are going to be adding a backup circuit to the site and will need to configure a tunnel3 that goes to the primary DC and tunnel4 that goes to the backup DC.  In testing I am having a hard time getting tunnels 3 to come up when I bring down tunnel 1.  The config for tunnel 1 is:
bandwidth 
backup interface Tunnel3
ip address (site code BGP)
no ip redirects
ip mtu 1350
ip nhrp authentication odomvpn
ip nhrp map (DC BGP route with ip address)
ip nhrp map multicast (DC ip address)
ip nhrp network-id 1
ip nhrp holdtime 450
ip nhrp nhs (DC BGP Route)
ip tcp adjust-mss 1360
delay 1000
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 100
tunnel path-mtu-discovery
tunnel vrf VPN-OUT
tunnel protection ipsec profile DMVPN shared
ip virtual-reassembly

Tunnel 3 config:
description DMVPN CLIENT Site Code 63
bandwidth 
ip flow monitor Intermapper input
ip flow monitor Intermapper output
ip address (site code BGP different that tunnel1)
no ip redirects
ip mtu 1350
ip nhrp authentication odomvpn
ip nhrp map (DC BGP route with ip address)
ip nhrp map multicast (PRI DC IP)
ip nhrp network-id 1
ip nhrp holdtime 450
ip nhrp nhs (DC BGP ROUTE)
ip tcp adjust-mss 1360
delay 1000
qos pre-classify
tunnel source GigabitEthernet0/0/2
tunnel mode gre multipoint
tunnel key 100
tunnel path-mtu-discovery
tunnel vrf VPN-OUT
tunnel protection ipsec profile DMVPN shared
ip virtual-reassembly

What is missing or 

[Larry Sullivan]

Is the backup circuit installed already and you're testing it now?

Are these tunnels all on the same router?

You're not testing by sharing the same public IP on VPN 1 and 3 are you?

[Matthewmunhall]

Yes, the backup circuit is installed, and I have tried failing over a few times with no success.  Yes, tunnels 1-4 are all on the same router.  For your last question if I understand correctly, I have made the IP addresses of each tunnel the same and also having a different one on tunnel 3.  An example would be tunnel1 and IP would be192.168.10.1 255.255.255.0 or Tunnel3 would be 192.168.20.1 255.255.255.0. 

[Larry Sullivan]

What do you mean IP address of each tunnel is the same?  The actual tunnel IPs can't be the same if they all connect to same Hub, and they need to be on the same subnet as the hub tunnel, also you will run into issues if the public IP address is the same for more than one tunnel if terminating on the same Hub.  So if the remote tunnel IPs are all different than each other, are on the same subnet as the hub, and all have a different "public" IP/egress interface, then you should be good there.

[Matthewmunhall]

I think I misunderstood what you were asking.  The public IPs address are different being the VRF VPN-OUT.  its 0.0.0.0 0.0.0.0 (public IP).  Each tunnel has its own IP address on the same subnet as the Hub router which is why tunnel 1 and 2 work. For tunnel 3 and 4 I just changed the 3rd octet.  Hope this clarifies.

[Larry Sullivan]

Makes a little more sense.  I'm assuming your DMVPN subnet is larger than a /24, because if it's a /24 or smaller, changing the third octet would put those spokes outside the DMVPN subnet.  Also, for future reference, if your tunnel IPs and source interface IPs are private IPs, it won't hurt to post them here and helps us clarify the situation better.  Just don't post any public IPs.  As Cristian Matei said, your MTU and ip tcp adjust-mss are a little funky.  Standard is ip mtu 1400 and ip tcp adjust-mss 1360, but tunnels 1 and 2 work with your funky configs, so likely not the issue.  At this point if it were me I would start running crypto debugs to see if I can get an indication of what the router is saying.  Also it's good to confirm end to end connectivity with pings, make sure you can ping DC public IP sourcing tunnel 3s source interface/IP.   While you're sourcing  GigabitEthernet0/0/2 for tunnel 3, the routing table could be taking g0/0/0 depending on how you have your routes set up.

[Matthewmunhall]

Here is the config the private IP's to make this a little easier.

The config for tunnel 1 is:
bandwidth
backup interface Tunnel3
ip address 10.1.12.110 255.255.254.0
no ip redirects
ip mtu 1350
ip nhrp authentication odomvpn
ip nhrp map 10.1.12.1 (Public IP)
ip nhrp map multicast (Public IP)
ip nhrp network-id 1
ip nhrp holdtime 450
ip nhrp nhs 10.1.12.1
ip tcp adjust-mss 1360
delay 1000
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 100
tunnel path-mtu-discovery
tunnel vrf VPN-OUT
tunnel protection ipsec profile DMVPN shared
ip virtual-reassembly

Tunnel 3 config:
description DMVPN CLIENT Site Code 63
bandwidth
ip address 10.1.12.110 255.255.254.0
no ip redirects
ip mtu 1350
ip nhrp authentication odomvpn
ip nhrp map 10.1.12.1 (Public IP)
ip nhrp map multicast (Public IP)
ip nhrp network-id 1
ip nhrp holdtime 450
ip nhrp nhs 10.1.12.1
ip tcp adjust-mss 1360
delay 1000
qos pre-classify
tunnel source GigabitEthernet0/0/2
tunnel mode gre multipoint
tunnel key 100
tunnel path-mtu-discovery
tunnel vrf VPN-OUT
tunnel protection ipsec profile DMVPN shared
ip virtual-reassembly

[Larry Sullivan]

They have the same tunnel IP.  Seems like you're trying to do some sort of active/passive.  You should give your tunnel 3 a unique IP and it's own VRF and do active/active and use a dynamic routing protocol like BGP to do the failover.

[Matthewmunhall]

Yes, I am looking to do an active passive option.  I can update tunnel3 ip address to be a 10.1.13.110 and also add if-state nhrp for the tunnel.  With doing that do separate vrf VPN-OUT and get that active passive that I am going for?

[Larry Sullivan]

EDIT: Ignore the below.  Looks like I'm still not sure why you're trying to do what you're trying to do and why you're doing it the way you're trying to do it.  Good luck.  Hope you get it sorted.  

To be clear, when I say active/passive I mean you have to do a manual failover where tunnel 3 directly replaces tunnel 1, hence why you had the same tunnel IP, this is making things harder on yourself.  By active/active I mean both tunnels are up, and dynamic routing protocol is up, but is only using tunnel 1, and if tunnel 1 goes down, tunnel 2/3/or 4 will automatically take over (in whatever order you prefer).  That being said, if you give tunnel 3 it's own tunnel IP, and it's own VRF (not the same VRF as tunnel 1), it should work.  Did you ever ping DC public IP sourcing tunnel 3 WAN IP?  What were the results?

[Cristian Matei]

Hi,

   Outside of your problems, worth mentioning by looking at your DMVPN config: having MTU lower than TCP adjust-MSS will lead to unneeded fragmentation, use MSS values to be MTU value - 40; assuming you're using routers of last 2-3 generation, MTU on the tunnel with IPSec is self-adjusted based on encryption algorithm and is best to leave it self-adjustable (test it by removing MTU value, clear IPsec / IKE SA and look at "show Interface tunnel 1" and "show tunnel endpoints" to se the adjusted MTU); when using IPsec with DMVPN, using NHRP authentication is pointless and it should be removed; when using  multipoint GRE / DMVPN, tunnel key is required only when sourcing two multipoint GRE tunnels off same interface so that it acts as a discriminator, otherwise it's useless and adds extra overhead; I see you're using multipoint GRE tunnels on the spokes, so you'r running a Phase 2 DMVPN, highly recommended to move to Phase 3 ("ip nhrp redirect" on hub and "ip nhrp shortcut" on spokes, however for proper functionality Phase 3 requires routing to be changed as opposed to Phase2, on spokes all routes needs a next-hop of the hub); you can replace the two commands of "ip nhrp map multicast" and " ip nhrp map" with single command of "ip nhrp nhs HUB_VPN_IP nbma HUB_NBMA_IP multicast". 

    Now, to your point: remove the "shared" keyword from "tunnel protection ipsec profile DMVPN" as this feature does not apply to your use-case, since Tunnel1 and Tunnel3 are not sourced from same physical interface; I assume that both Gi0/0/0 and Gi0/0/2 are members of VRF VPN-OUT and that spoke to hub NBMA routing is functional across both ISP's; although using "backup" functionality is pretty creative, however in your case it will not work as for Tunnel3 to come UP, Tunnel1 needs to go down and this happens only if you manually shut it down or Gi0/0/0 goes down or if you enable "if-state nhrp" and spoke looses NHRP communication with the hub based on hub configured NHRP timers. Perform these changes and provide output of "show dmvpn detail". 

Best,

Cristian.

[Matthewmunhall]

Each public IP has its own VRF VPN-OUT.  Can I keep everything on 1 profile and just add the if-state nhrp statement to get tunnel 3 to come up if tunnel1 goes down or is this a reconfig of the profiles?

[Cristian Matei]

Hi,

    Long story short, yes; keep sam FVRF for all tunnels, no need to add extra VRF's. However, assuming there's a WAN failure for transport network that Tunnel 1 uses, how fast do you want NHRP to detect that, so tunnel goes down because you've configured if-state NHRP? Whatever the answer is to the question, this is the NHRP holdtime value you need to configure on the hub, yes, on the hub. Note that for NHRP hold time configured on hub to be pushed down to spokes, all spoked need to re-register to the hub (so flap the spoke tunnel).

    I guess you're looking to use the backup interface option, to avoid the routing complexity which shows up if all 4 tunnels are UP at the same time, right?

    Once everything is deployed, test that tunnel failover happens as expected, within the time period you're looking for (easiest way of doing this is to configure a temporary static NBMA to VPN mapping on the hub, for tunnel1 NBMA and VPN of the spoke you're performing testing on, and this static entry needs to be incorrect, like use a wrong NBMA address).

Best,

Cristian.

[MHM Cisco World]

Share the ike config, I need to check vrf Config 

Also since each tunnel have it different tunnel source you not need "shared" keyword in tunnel protection. <<- ""shared"" is need if tunnel 3/4 shared same tunnel source with tunnel 1/2

Lastly if you want to config one tunnel as primary and other as backup use weight in bgp to prefer one path, I. E. Both tunnels is UP and let bgp prefer path

MHM

[MHM Cisco World]