DMVPN Spoke Stops passing traffic after inactivity

[Timothy Patrick]

I  am trying out a solution using an ISR 829 with a cellular connection. This device will be a spoke in a DMVPN setup. It successfully connects with the hub and I  am able to ping the Hubs tunnel address as well as ping between the sites behind the tunnel addresses.

The ISR829 is using a Verizon connection and sits behind a NAT address of 100.108.7.202/32

I  have tried both transport mode and tunnel mode in my IPSEC configuration and both have worked with one showing the N attribute (Transport) and no N attribute with Tunneled.

(Transport)

# Ent Peer NBMA Addr Peer Tunnel Add     State  UpDn Tm     Attrb
----- --------------- --------------- ----- --------    -----
      1   xxx.xxx.xxx.xxx      172.16.124.5       UP    00:20:07     DN 

DMVPN_POC_HUB#ping 172.16.124.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.124.5, timeout is 2 seconds:
!!!!!

(Tunneled)
# Ent Peer NBMA Addr Peer Tunnel Add      State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
     1           100.108.7.202        172.16.124.5      UP     00:00:46 D

DMVPN_POC_HUB#ping 172.16.124.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.124.5, timeout is 2 seconds:
!!!!!

After a certain amount of time(as early as a couple of minutes) traffic stops flowing even though the tunnel still shows up and connected.

# Ent   Peer NBMA Addr      Peer Tunnel   Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
      1 xxx.xxx.xxx.xxx.      172.16.124.5          UP     00:03:04 DN

DMVPN_POC_HUB#ping 172.16.124.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.124.5, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

#HUB

interface Tunnel0
description -> DMVPN HUB Tunnel
ip address 172.16.124.1 255.255.255.0
no ip redirects
ip nhrp authentication dmvpnpoc
ip nhrp network-id 10
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile IPSEC_PROFILE
end

#SPOKE

interface Tunnel1

description -> Spoke Tunnel

ip address 172.16.124.5 255.255.255.0

no ip redirects

ip mtu 1440

ip nhrp authentication dmvpnpoc

ip nhrp map multicast xxx.xxx.xxx.xxx

ip nhrp map 172.16.124.1 xxx.xxx.xxx

ip nhrp network-id 10

ip nhrp nhs 172.16.124.1

tunnel source Cellular0/0

tunnel mode gre multipoint

tunnel key 1

tunnel protection ipsec profile IPSEC_PROFILE

I  have tried issue the Debug Crypto IPSEC and ISAKMP but nothing useful has come from those logs. 

Any help on next steps in troubleshooting would be appreciated.

[MHM Cisco World]

how many Spokes behind NAT (same NAT router )???

[Timothy Patrick]

Just one

The spoke is the ISR 829 Ithe NAT is happening at the cellular level.  

CIsco ISR829(Spoke) <-----------> (HUB) Cisco ISR4331

[paul driver]

Hello

sh ip protocols
sh ip route
sh ip nhrp

try the following:
interface Tunnel x
ip mtu 1400
ip tcp adjust-mss 1360

[Timothy Patrick]

IR800#show ip protocols

*** IP Routing is NSF aware ***

Routing Protocol is "application"

  Sending updates every 0 seconds

  Invalid after 0 seconds, hold down 0, flushed after 0

  Outgoing update filter list for all interfaces is not set

  Incoming update filter list for all interfaces is not set

  Maximum path: 32

  Routing for Networks:

  Routing Information Sources:

    Gateway         Distance      Last Update

  Distance: (default is 4)

Routing Protocol is "nhrp"

  Maximum path: 32

  Routing Information Sources:

    Gateway         Distance      Last Update

  Distance: (default is 250)

IR800#show ip route

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

       E1 - OSPF external type 1, E2 - OSPF external type 2

       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

       ia - IS-IS inter area, * - candidate default, U - per-user static route

       o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP

       a - application route

       + - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

S*    0.0.0.0/0 is directly connected, Cellular0/0

      10.0.0.0/32 is subnetted, 1 subnets

C        10.1.1.1 is directly connected, wlan-ap0

      100.0.0.0/32 is subnetted, 1 subnets

C        100.93.34.92 is directly connected, Cellular0/0

      172.16.0.0/16 is variably subnetted, 2 subnets, 2 masks

C        172.16.124.0/24 is directly connected, Tunnel1

L        172.16.124.5/32 is directly connected, Tunnel1

      172.30.0.0/32 is subnetted, 1 subnets

S        172.30.36.210 is directly connected, Tunnel1

      192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks

C        192.168.1.0/24 is directly connected, Vlan1

L        192.168.1.1/32 is directly connected, Vlan1

      192.168.2.0/32 is subnetted, 1 subnets

C        192.168.2.1 is directly connected, Loopback100

IR800# show ip nhr

IR800# show ip nhrp

172.16.124.1/32 via 172.16.124.1

   Tunnel1 created 00:13:43, never expire

   Type: static, Flags: used

   NBMA address: xxx.xxx.xxx.xxx

[Georg Pauwen]

Hello,

--> DMVPN Spoke Stops passing traffic after inactivity

I have read through the post, but I am not really clear on what the problem actually is. Do you want the Cellular to be up all the time ?

If so, try and configure NTP (which permanently tries to synchronize and hence sends out interesting traffic). Also, you could set the 'dialer idle-timeout 0' if your router uses a dialer interface...

[Timothy Patrick]

The cellular interface does not go down. I  maintain connectivity to the internet at all times. My problem is when I  establish a tunnel to my hub I  am able to pass traffic across that tunnel to the other side with no issues and the other side can reach the spoke. After a few minutes something happens and I  am no longer able to ping or reach anything across the tunnel from either side

[paul driver]

Hello
Do you lose connectivity without IPSEC if not then it has to relate with the encryption.
Can you post the output this debug please

debug dmvpn all all

[MHM Cisco World]

First thanks for sharing and help us to detect the issue, 
let start troubleshooting 
IR800(Spoke)-NAT-Hub 

IR800#show ip nhrp traffic
Tunnel1: Max-send limit:10000Pkts/10Sec, Usage:0%
   Sent: Total 197
         1 Resolution Request  0 Resolution Reply  196 Registration Request 
         0 Registration Reply  0 Purge Request  0 Purge Reply 
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress 
   Rcvd: Total 55
         0 Resolution Request  1 Resolution Reply  0 Registration Request 
         54 Registration Reply  0 Purge Request  0 Purge Reply 
         0 Error Indication  0 Traffic Indication  0 Redirect Suppress 

IKEv1 SA: local 100.93.34.92/4500 remote 216.185.188.10/4500 Active

Capabilities:N connid:1207 lifetime:00:29:44

Session ID: 0

IKEv1 SA: local 100.93.34.92/4500 remote 216.185.188.10/4500 Inactive

Capabilities:N connid:1206 lifetime:0

Crypto Session Status: UP-ACTIVE

fvrf: (none), Phase1_id: 216.185.188.10

IPSEC FLOW: permit 47 host 100.93.34.92 host xxx.xxx.xxx.xxx

Active SAs: 2, origin: crypto map

Inbound: #pkts dec'ed 8 drop 0 life (KB/Sec) 4169134/3585

Outbound: #pkts enc'ed 8 drop 0 life (KB/Sec) 4169134/3585

so according to above 
Spoke send registration request and get reply, 
and in IPSec there is no drop.
so every think from my view is OK.

then suddenly the traffic drop (no ping)

the issue come from 
The Cellular change it IP !!
Yes if the Cellular change it IP then the source address of packet is change 
how we can know that 
show ip nhrp <<- in Hub
there are two IP 
A-NBMA Address 
B- claimed Address  <<- claimed address is pre-NAT address (before NAT to NBMA) 
if the cellular IP and claimed Address is not match then this issue !!
why ?
the Spoke can send new ISAKMP packet to Hub, here Hub refuse the new ISAKMP new SA because it already have one and it lifetime not end. 
here the packet start to drop 
how we know that Spoke ask new ISAKMP 
show crypto ISAKMP sa 
see if there is any delete and new ISAKMP entry add to table 

How we can solve this issue 
as mention before 
1- non-Unique command 
2- crypto isakmp keepalive <small interval> periodic 
3- if-state nhrp <<- spoke send periodic NHRP message to hub

hope this solve your issue 

and one more think 
add command 
crypto isakmp NAT keepalive <<- this also can add to protect tunnel from timeout of PAT