Automatically clear DMVPN GRE when peer is down

[Travis-Fleming]

Hey guys! We have a site with two internet connections. We utilize DMVPN GRE tunnels back to our DMVPN concentrator at our HQ fro the remote site. When the primary internet goes down at the remote site, we have the DMVPN auto NAT out the secondary internet connection. However our DMVPN does not come back up. Believe that is because at our HQ the DMVPN session is still up to the old public IP address of the peer.

Is there a way to have our HQ do an sla that would automatically clear the tunnel of any down peer address dynamically? Below is an example of the tunnel interfaces of the remote peer and HQ concentrator for the DMVPN tunnels.We map the peer to our two public IP's for our two redundent DMVPN routers at HQ (1.2.3.4 and 4.3.2.1)

PEER:

interface Tunnel0
bandwidth 15000
ip address 172.17.110.24 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication #XXXX
ip nhrp map 172.17.110.1 1.2.3.4
ip nhrp map multicast 1.2.3.4
ip nhrp map 172.17.110.2 4.3.2.1
ip nhrp map multicast 4.3.2.1
ip nhrp network-id 1
ip nhrp nhs 172.17.110.1
ip nhrp nhs 172.17.110.2
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/1.1
tunnel mode gre multipoint
tunnel protection ipsec profile ipsec-prof

CONCENTRATOR:

interface Tunnel0
bandwidth 100000
ip address 172.17.110.1 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 100
no ip split-horizon eigrp 100
ip flow monitor NetFlowMonitor1 input
ip flow monitor NetFlowMonitor1 output
ip nhrp authentication #XXXX
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel protection ipsec profile ipsec-prof

[Georg Pauwen]

Hello,

you could use an IP SLA on the peer that tracks the primary interface, when the connection to the primary ISP is down, the router will telnet to the Concentrator and send a message to the log. If the Concentrator sees the message in the log, it will run an EEM script that clears the DMVPN peer.

It would look like this:

PEER

track 1 ip sla 1 reachability
!
ip sla 1
icmp-echo 8.8.8.8 source-ip 1.2.3.2
timeout 500
threshold 500
frequency 1
!
ip sla schedule 1 life forever start-time now
!
event manager applet TEST
event track 1 state down
action 1.0 cli command "enable"
action 2.0 cli command "telnet 1.2.3.4" pattern "Username:"
action 3.0 cli command "Remote" pattern "Password:"
action 4.0 cli command "Alert" pattern "#"
action 5.0 cli command "send log DMVPN DOWN"

CONCENTRATOR

username Remote privilege 15 password 0 Alert
!
line vty 0 4
login local
!
event manager applet CLEAR_DMVPN_PEER
event syslog pattern "DMVPN DOWN"
action 1.0 cli command "enable"
action 2.0 cli command "clear dmvpn session peer 1.2.3.1

[Travis-Fleming]

That looks like we are on the right track! Two small hickups. We have about 20 remote peers using DMVPN, so we would need a way for the concentrator to know which peer to clear. Perhaps the log the peer generates could put something into the log with it's IP, then the EMM on the concentrator can extract that IP from the log to clear?

Also, we utilize a clearpass radius server with ssh. Would the commands to enable login via ssh be the same as telnet like you have it?

Also, another layer of complexity, we have a Cisco FTD landing both internet connections one hop from the peer router. The default route on the router points to the FTD LAN address (ip route 0.0.0.0 0.0.0.0 172.28.8.1). To the Router, the default route never changes. We could setup the sla to perhaps ping the default gateway of the primary internet connection? Then if it's down to write to the log?

[Georg Pauwen]

Should be doable, I'll see what I can come up with...

[Travis-Fleming]

In my testing the convergence from one ISP to another was a few pings on the router. Maybe there will also be a way to add a few second delay to the EMM to allow that? That way the peer router will be able to successfully ssh the concentrater. 

[Travis-Fleming]

Here is my rough draft, but the SSH part isn't working on my router. It could be because I"m testing with a 2911? I can't ssh from the router itself in the cli so I"ll have to troubleshoot there...But there would then need to be an event manager applet for each of our DMVPN peers with more then one internet connection. At this point it will be two, but could be more in the future. Thoughts Georg oh wise one?

Remote Peer:

event manager applet PeerDMVPNDown
event track 1 state down
action 1.0 wait 15
action 2.0 cli command "enable"
action 3.0 cli command "ssh -l sa.at.scmn.eem 1.2.3.4" pattern "Password:"
action 4.0 cli command "Password" pattern "#"
action 5.0 cli command "send log Peer 1.2.3.4 DMVPN Down"

Concentrator:

event manager applet Clear_DMVPN_PEER_1.2.3.4
event systlog pattern "Peer 1.2.3.4 DMVPN Down"
action 1.0 cli command "enable"
action 2.0 cli command "clear dmvpn session peer 1.2.3.4"

[Georg Pauwen]

Hello,

I am currently testing, the syntax for ssh seems to be a bit different indeed. 2911 should not matter though...will let you know...

[Georg Pauwen]

Hello

I have done some more testing. I have used Cisco routers as SSH clients and servers, but the commands used should be similar. When you execute the 'ssh -l sa.at.scmn.eem 1.2.3.4' command, do you get to the 'Password" prompt at all ?

Either way, I have changed the EEM scripts so they work with any IP address. On the peer, the message you send is 'DMVPN Peer down 1.2.3.1', for each peer, (obviously) use a different IP address. The concentrator will run the EEM script every time the 'DMVPN Peer down' message is sent to the log, extract the IP address, and clear the session that is associated with that IP address.

Peer

event manager applet PRIMARY_ISP_DOWN
event track 1 state down
action 1.0 cli command "enable"
action 2.0 cli command "ssh -v 2 -c aes256-cbc -m hmac-sha1-160 -l admin 1.2.3.4" pattern "Password:"
action 3.0 cli command "cisco" pattern "#"
action 4.0 cli command "send log DMVPN Peer down 1.2.3.1"

Concentrator

username admin privilege 15 password 0 cisco

!

line vty 0 4

transport input shh

login local

!
event manager applet CLEAR_DMVPN_PEER
event syslog pattern "DMVPN Peer down"
action 1.0 regexp "([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)" $_syslog_msg ignore ipaddress
action 2.0 syslog msg "DMVPN IP address is $ipaddress"
action 3.0 cli command "enable"
action 4.0 cli command "clear dmvpn session peer $ipaddress"

action 5.0 cli command "clear log" pattern "confirm"
action 6.0 cli command "y"
action 7.0 cli command "end"

[Travis-Fleming]

This looks great! One question, if you don't do the "clear log" pattern "confirm", are you thinking it would just keep clearing the dmvpn? Also, it was a firewall rule on my side not allowing that router to SSH on port 22. That is allowed and working now.