cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2284
Views
20
Helpful
22
Replies

DMVPN Spoke Stops passing traffic after inactivity

Timothy Patrick
Level 1
Level 1

I  am trying out a solution using an ISR 829 with a cellular connection. This device will be a spoke in a DMVPN setup. It successfully connects with the hub and I  am able to ping the Hubs tunnel address as well as ping between the sites behind the tunnel addresses.

The ISR829 is using a Verizon connection and sits behind a NAT address of 100.108.7.202/32

I  have tried both transport mode and tunnel mode in my IPSEC configuration and both have worked with one showing the N attribute (Transport) and no N attribute with Tunneled.

(Transport)

# Ent Peer NBMA Addr Peer Tunnel Add     State  UpDn Tm     Attrb
----- --------------- --------------- ----- --------    -----
      1   xxx.xxx.xxx.xxx      172.16.124.5       UP    00:20:07     DN 

DMVPN_POC_HUB#ping 172.16.124.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.124.5, timeout is 2 seconds:
!!!!!

(Tunneled)
# Ent Peer NBMA Addr Peer Tunnel Add      State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
     1           100.108.7.202        172.16.124.5      UP     00:00:46 D

DMVPN_POC_HUB#ping 172.16.124.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.124.5, timeout is 2 seconds:
!!!!!

After a certain amount of time(as early as a couple of minutes) traffic stops flowing even though the tunnel still shows up and connected.

# Ent   Peer NBMA Addr      Peer Tunnel   Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
      1 xxx.xxx.xxx.xxx.      172.16.124.5          UP     00:03:04 DN

DMVPN_POC_HUB#ping 172.16.124.5
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 172.16.124.5, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

#HUB

interface Tunnel0
description -> DMVPN HUB Tunnel
ip address 172.16.124.1 255.255.255.0
no ip redirects
ip nhrp authentication dmvpnpoc
ip nhrp network-id 10
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile IPSEC_PROFILE
end

#SPOKE

interface Tunnel1

description -> Spoke Tunnel

ip address 172.16.124.5 255.255.255.0

no ip redirects

ip mtu 1440

ip nhrp authentication dmvpnpoc

ip nhrp map multicast xxx.xxx.xxx.xxx

ip nhrp map 172.16.124.1 xxx.xxx.xxx

ip nhrp network-id 10

ip nhrp nhs 172.16.124.1

tunnel source Cellular0/0

tunnel mode gre multipoint

tunnel key 1

tunnel protection ipsec profile IPSEC_PROFILE

 

I  have tried issue the Debug Crypto IPSEC and ISAKMP but nothing useful has come from those logs. 

Any help on next steps in troubleshooting would be appreciated.

22 Replies 22

in Spoke do you check the cellular is it UP or not ?
in Spoke can you share show ip route ?

Cellular0/0                100.108.7.202   YES IPCP   up                    up      

Cellular1/0                unassigned      YES NVRAM  up                    up      

Cellular0/1                unassigned      YES NVRAM  up                    up      

Tunnel1                    172.16.124.5    YES manual up                    up    

I  am able to ping the Hubs external IP address from the spoke and I  am able to get to the internet as well

Timothy Patrick
Level 1
Level 1

Here are some additional debugs before the tunnel stops passing traffic:

DMVPN_POC_HUB#
*Nov 27 23:56:08.368: NHRP: Receive Registration Request via Tunnel0 vrf global(0x0), packet size: 108
*Nov 27 23:56:08.369: NHRP: Tunnels gave us pak src addr: 100.108.7.202
*Nov 27 23:56:08.369: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.369: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.369: NHRP: Peer capability:0
*Nov 27 23:56:08.369: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.369: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.370: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.370: NHRP: Peer capability:0
*Nov 27 23:56:08.370: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.370: NHRP: nhrp_subblock_check_for_map() - Map Already Exists
*Nov 27 23:56:08.370: NHRP: Sending Registration Reply if_in Tunnel0 vrf global(0x0) dst 172.16.124.5 nbma 100.108.7.202 code: no error(0)
*Nov 27 23:56:08.370: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 172.16.124.5
*Nov 27 23:56:08.370: NHRP: Send Registration Reply via Tunnel0 vrf global(0x0), packet size: 128
*Nov 27 23:56:08.371: src: 172.16.124.1, dst: 172.16.124.5
*Nov 27 23:56:08.371: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 100.108.7.202
*Nov 27 23:56:08.371: NHRP: 156 bytes out Tunnel0
*Nov 27 23:56:08.828: NHRP: Receive Registration Request via Tunnel0 vrf global(0x0), packet size: 108
*Nov 27 23:56:08.829: NHRP: Tunnels gave us pak src addr: 100.108.7.202
*Nov 27 23:56:08.829: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.829: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.829: NHRP: Peer capability:0
*Nov 27 23:56:08.829: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.829: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.829: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.830: NHRP: Peer capability:0
*Nov 27 23:56:08.830: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:08.830: NHRP: nhrp_subblock_check_for_map() - Map Already Exists
*Nov 27 23:56:08.830: NHRP: Sending Registration Reply if_in Tunnel0 vrf global(0x0) dst 172.16.124.5 nbma 100.108.7.202 code: no error(0)
*Nov 27 23:56:08.830: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 172.16.124.5
*Nov 27 23:56:08.830: NHRP: Send Registration Reply via Tunnel0 vrf global(0x0), packet size: 128
*Nov 27 23:56:08.831: src: 172.16.124.1, dst: 172.16.124.5
*Nov 27 23:56:08.831: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 100.108.7.202
*Nov 27 23:56:08.831: NHRP: 156 bytes out Tunnel0
*Nov 27 23:56:10.807: NHRP: Receive Registration Request via Tunnel0 vrf global(0x0), packet size: 108
*Nov 27 23:56:10.807: NHRP: Tunnels gave us pak src addr: 100.108.7.202
*Nov 27 23:56:10.807: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:10.807: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:10.807: NHRP: Peer capability:0
*Nov 27 23:56:10.808: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:10.808: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:10.808: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:10.808: NHRP: Peer capability:0
*Nov 27 23:56:10.808: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:10.808: NHRP: nhrp_subblock_check_for_map() - Map Already Exists
*Nov 27 23:56:10.808: NHRP: Sending Registration Reply if_in Tunnel0 vrf global(0x0) dst 172.16.124.5 nbma 100.108.7.202 code: no error(0)
*Nov 27 23:56:10.809: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 172.16.124.5
*Nov 27 23:56:10.809: NHRP: Send Registration Reply via Tunnel0 vrf global(0x0), packet size: 128
*Nov 27 23:56:10.809: src: 172.16.124.1, dst: 172.16.124.5
*Nov 27 23:56:10.809: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 100.108.7.202
*Nov 27 23:56:10.809: NHRP: 156 bytes out Tunnel0
*Nov 27 23:56:14.589: NHRP: Receive Registration Request via Tunnel0 vrf global(0x0), packet size: 108
*Nov 27 23:56:14.589: NHRP: Tunnels gave us pak src addr: 100.108.7.202
*Nov 27 23:56:14.590: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:14.590: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:14.590: NHRP: Peer capability:0
*Nov 27 23:56:14.590: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:14.590: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:14.590: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 27 23:56:14.590: NHRP: Peer capability:0
*Nov 27 23:56:14.591: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)

https://blog.ipspace.net/2010/09/dmvpn-non-unique-nhrp-registrations.html

 ip nhrp registration non-unique <<- this what you need only I think 

after run command clear nhrp in hub to take effect 

I  tried that on my spoke but with no change. My external NAT address from the carrier and my cellular ip address do not appear to be changing.

*Nov 28 00:22:10.254: NHRP: Receive Registration Request via Tunnel0 vrf global(0x0), packet size: 108
*Nov 28 00:22:10.254: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Nov 28 00:22:10.254: shtl: 4(NSAP), sstl: 0(NSAP)
*Nov 28 00:22:10.254: pktsz: 108 extoff: 52
*Nov 28 00:22:10.254: (M) flags: "nat ", reqid: 546
*Nov 28 00:22:10.254: src NBMA: 100.108.7.202
*Nov 28 00:22:10.254: src protocol: 172.16.124.5, dst protocol: 172.16.124.1
*Nov 28 00:22:10.255: (C-1) code: no error(0)
*Nov 28 00:22:10.255: prefix: 32, mtu: 17912, hd_time: 600
*Nov 28 00:22:10.255: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Nov 28 00:22:10.255: NHRP: Tunnels gave us pak src addr: 100.108.7.202
*Nov 28 00:22:10.255: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 28 00:22:10.255: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 28 00:22:10.255: NHRP: Peer capability:0
*Nov 28 00:22:10.255: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 28 00:22:10.256: NHRP: Adding Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 28 00:22:10.256: NHRP: NHRP subblock already exists for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 28 00:22:10.256: NHRP: Peer capability:0
*Nov 28 00:22:10.256: NHRP: Cache already has a subblock node attached for Tunnel Endpoints (VPN: 172.16.124.5, NBMA: 100.108.7.202)
*Nov 28 00:22:10.256: NHRP: nhrp_subblock_check_for_map() - Map Already Exists
*Nov 28 00:22:10.256: NHRP: Sending Registration Reply if_in Tunnel0 vrf global(0x0) dst 172.16.124.5 nbma 100.108.7.202 code: no error(0)
*Nov 28 00:22:10.256: NHRP: Attempting to send packet through interface Tunnel0 via DEST dst 172.16.124.5
*Nov 28 00:22:10.257: NHRP: Send Registration Reply via Tunnel0 vrf global(0x0), packet size: 128
*Nov 28 00:22:10.257: src: 172.16.124.1, dst: 172.16.124.5
*Nov 28 00:22:10.257: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Nov 28 00:22:10.257: shtl: 4(NSAP), sstl: 0(NSAP)
*Nov 28 00:22:10.257: pktsz: 128 extoff: 52
*Nov 28 00:22:10.257: (M) flags: "nat ", reqid: 546
*Nov 28 00:22:10.257: src NBMA: 100.108.7.202
*Nov 28 00:22:10.257: src protocol: 172.16.124.5, dst protocol: 172.16.124.1
*Nov 28 00:22:10.257: (C-1) code: no error(0)
*Nov 28 00:22:10.257: prefix: 32, mtu: 17912, hd_time: 600
*Nov 28 00:22:10.258: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Nov 28 00:22:10.258: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 100.108.7.202

It tries to send it but nothing goes through. They only way I  can get to send traffic again is to shut/ no shut the tunnel interface on the spoke

with non-unique and transport IPSec Mode, can you share 
show crypto iskamp sa ?

I  ran the clear ip NHRP on the hub, bounced both tunnel interfaces and it came but stopped sending traffic shortly after. I  changed IPSEC mode to transport. Also noted that the "ip nhrp registration no-unique " did not show up in my tunnel interface config even though I  added it. 

DMVPN_POC_HUB#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
xxx.xx.xxx.xxx  70.184.116.40 QM_IDLE 15191 ACTIVE

Spoke:

IR800#show crypto isakmp sa

IPv4 Crypto ISAKMP SA

dst             src             state          conn-id status

 xxx.xxx.xxx.xxx 100.108.7.202   QM_IDLE           1128 ACTIVE dmvpn-tun0

I will run lab, need some info. 
are you use IKEv1 or IKEv2 ?
are Spoke behind NAT or Hub behind NAT or both ?

I run lab and same issue as you, 
but I detect issue from where, 
do 
show dmvpn detail 
there is only outbound in Spoke and in hub there is no Inbound and Outbound. 

the issue is NAT router not NATing udp 4500 port 
it can be bug if you use ISR as NAT router. 

So my hub router has a public IP and does not NAT. It is an ISR 4331
My Spoke Router is an ISR829 with a cellular connection as the outbound connection. There is no other connection on this router providing outbound services. The ip that is given to the cellular interface comes from Verizon and is100.108.7.202. This is an IP address that is given and is is not a public reachable address. Verizon uses this to NST to a public address.

can I see 
show dmvpn detail 
show ip nhrp traffic 
in spoke ?

I solve the issue I face in my lab, waiting your share of output

show dmvpn detail 
show ip nhrp traffic 

Sending 5, 100-byte ICMP Echos to 172.16.124.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 50/55/60 ms

IR800#show dmvpn detail
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable, I2 - Temporary
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel1 is up/up, Addr. is 172.16.124.5, VRF ""
Tunnel Src./Dest. addr: 100.93.34.92/MGRE, Tunnel VRF ""
Protocol/Transport: "multi-GRE/IP", Protect "shiva"
Interface State Control: Disabled
nhrp event-publisher : Disabled
IPv4 Registration Timer: 65535 seconds

IPv4 NHS:
172.16.124.1 RE priority = 0 cluster = 0
Type:Spoke, Total NBMA Peers (v4/v6 1

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
1 xxx.xxx.xxx.xxx 172.16.124.1 UP 00:00:14 S 172.16.124.1/32
 

Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel1
Session: [0x141A9B28]
Session ID: 0
IKEv1 SA: local 100.93.34.92/4500 remote 216.185.188.10/4500 Active
Capabilities:N connid:1207 lifetime:00:29:44
Session ID: 0
IKEv1 SA: local 100.93.34.92/4500 remote 216.185.188.10/4500 Inactive
Capabilities:N connid:1206 lifetime:0
Crypto Session Status: UP-ACTIVE
fvrf: (none), Phase1_id: 216.185.188.10
IPSEC FLOW: permit 47 host 100.93.34.92 host xxx.xxx.xxx.xxx
Active SAs: 2, origin: crypto map
Inbound: #pkts dec'ed 8 drop 0 life (KB/Sec) 4169134/3585
Outbound: #pkts enc'ed 8 drop 0 life (KB/Sec) 4169134/3585
Outbound SPI : 0xCA3CE5F1, transform : esp-aes esp-sha-hmac
Socket State: Open

Pending DMVPN Sessions:
 
IR800#show ip nhrp traffic
Tunnel1: Max-send limit:10000Pkts/10Sec, Usage:0%
Sent: Total 197
1 Resolution Request 0 Resolution Reply 196 Registration Request
0 Registration Reply 0 Purge Request 0 Purge Reply
0 Error Indication 0 Traffic Indication 0 Redirect Suppress
Rcvd: Total 55
0 Resolution Request 1 Resolution Reply 0 Registration Request
54 Registration Reply 0 Purge Request 0 Purge Reply
0 Error Indication 0 Traffic Indication 0 Redirect Suppress
IR800#ping 172.16.124.1
Sending 5, 100-byte ICMP Echos to 172.16.124.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 48/55/60 ms
IR800#show ip nhrp traffic
Tunnel1: Max-send limit:10000Pkts/10Sec, Usage:0%
Sent: Total 197
1 Resolution Request 0 Resolution Reply 196 Registration Request
0 Registration Reply 0 Purge Request 0 Purge Reply
0 Error Indication 0 Traffic Indication 0 Redirect Suppress
Rcvd: Total 55
0 Resolution Request 1 Resolution Reply 0 Registration Request
54 Registration Reply 0 Purge Request 0 Purge Reply
0 Error Indication 0 Traffic Indication 0 Redirect Suppress
IR800#

After a few minutes 


IR800#ping 172.16.124.1    

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 172.16.124.1, timeout is 2 seconds:

.....

Success rate is 0 percent (0/5)

IR800#show dmvpn detail    

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete

        N - NATed, L - Local, X - No Socket

        T1 - Route Installed, T2 - Nexthop-override

        C - CTS Capable, I2 - Temporary

        # Ent --> Number of NHRP entries with same NBMA peer

        NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting

        UpDn Time --> Up or Down Time for a Tunnel

==========================================================================

 

Interface Tunnel1 is up/up, Addr. is 172.16.124.5, VRF ""

   Tunnel Src./Dest. addr: 100.93.34.92/MGRE, Tunnel VRF ""

   Protocol/Transport: "multi-GRE/IP", Protect "shiva"

   Interface State Control: Disabled

   nhrp event-publisher : Disabled

IPv4 Registration Timer: 65535 seconds

 

IPv4 NHS:

172.16.124.1  RE priority = 0 cluster = 0

Type:Spoke, Total NBMA Peers (v4/v6): 1

 

# Ent  Peer NBMA Addr Peer Tunnel Add State  UpDn Tm Attrb    Target Network

----- --------------- --------------- ----- -------- ----- -----------------

    xxx.xxx.xxx.xxx     172.16.124.1    UP 00:06:34     S    172.16.124.1/32

          

 

Crypto Session Details:

--------------------------------------------------------------------------------

 

Interface: Tunnel1

Session: [0x141A9B28]

  Session ID: 0 

  IKEv1 SA: local 100.93.34.92/4500 remote 216.185.188.10/4500 Active

          Capabilities:N connid:1207 lifetime:00:23:24

  Crypto Session Status: UP-ACTIVE     

  fvrf: (none), Phase1_id:  xxx.xxx.xxx.xxx

  IPSEC FLOW: permit 47 host 100.93.34.92 host. xxx.xxx.xxx.xxx

        Active SAs: 2, origin: crypto map

        Inbound:  #pkts dec'ed 18 drop 0 life (KB/Sec) 4169132/3204

        Outbound: #pkts enc'ed 23 drop 0 life (KB/Sec) 4169132/3204

   Outbound SPI : 0xCA3CE5F1, transform : esp-aes esp-sha-hmac

    Socket State: Open

 

Pending DMVPN Sessions:

 

IR800#show ip nhrp traffic

Tunnel1: Max-send limit:10000Pkts/10Sec, Usage:0%

   Sent: Total 197

         1 Resolution Request  0 Resolution Reply  196 Registration Request 

         0 Registration Reply  0 Purge Request  0 Purge Reply 

         0 Error Indication  0 Traffic Indication  0 Redirect Suppress 

   Rcvd: Total 55

         0 Resolution Request  1 Resolution Reply  0 Registration Request 

         54 Registration Reply  0 Purge Request  0 Purge Reply 

         0 Error Indication  0 Traffic Indication  0 Redirect Suppress 

IR800#    

Review Cisco Networking for a $25 gift card