CSR1000v Issues with Dmvpn

[gservicios]

Hello all,

we have an issue in a DMVPN Phase 2 topology, one spoke is having a strange problem:

The spoke is not an hub but is having a lot of dynamic tunnels with many spokes because there are a lot of services that remote spokes need to access there, it is able to create a spole-to-spoke communication with another spoke, after some time we see the dynamic tunnel active but there is no connectivity betweem the two spokes and routing seems down even if I can see the routes... it is very strange because we are using the same configuration with other spokes, I will paste some "shows" to better explain the issue:

CHQRTRDV01#sh ip route 10.8.158.3
Routing entry for 10.8.158.0/24
Known via "ospf 101", distance 110, metric 4100
Tag 1002, type extern 1
Redistributing via ospf 1
Advertised by ospf 1 metric 4100 metric-type 1 subnets route-map DomDmvpnFiltering_RM
Last update from 10.2.250.79 on Tunnel1, 1d15h ago
Routing Descriptor Blocks:
* 10.2.250.79, from 10.2.250.79, 1d15h ago, via Tunnel1
Route metric is 4100, traffic share count is 1
Route tag 1002
CHQRTRDV01#

CHQRTRDV01#sh dm | s 10.2.250.79
1 82.163.3.122 10.2.250.79 UP 22:25:08 D <----- here we have the dynamic tunnel up for more than one hour and it seems  stucked

There is no crypto sa information at this moment:

CMEX-GBR-CHQRTRDV01#sh crypto isakmp sa | s 82.163.3.122

CHQRTRDV01#ping 10.8.158.3 source gigabitEthernet 2 <---- ping fails and there is no connectivity
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.8.158.3, timeout is 2 seconds:
Packet sent with a source address of 10.9.111.2
.....
Success rate is 0 percent (0/5)

However when I clear manually the dmvpn session the connectivity recovers:

CHQRTRDV01#clear dmvpn session peer 82.163.3.122

CHQRTRDV01#ping 10.8.158.3 source gigabitEthernet 2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.8.158.3, timeout is 2 seconds:
Packet sent with a source address of 10.9.111.2
.!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 34/35/37 ms
CMEX-GBR-CHQRTRDV01#sh dm | s 10.2.250.79
1 82.163.3.122 10.2.250.79 UP 00:00:06 D
CMEX-GBR-CHQRTRDV01#sh crypto isakmp sa | s 82.163.3.122
62.30.184.178 82.163.3.122 QM_IDLE 50417 ACTIVE
CMEX-GBR-CHQRTRDV01#

we are using a CSR1000v with IOS xd Fuji 16.09.1 

If you want to check I attach the DMVPN configuration for the two spokes and the HUB

HUB:

Current configuration : 645 bytes
!
interface Tunnel1
description WAN - HOUCRTDV00
bandwidth 100000
ip flow monitor FlowMonitor1 input
ip flow monitor FlowMonitor1 output
ip address 10.2.250.3 255.255.254.0
no ip redirects
ip mtu 1400
ip nhrp authentication DMVPN101
ip nhrp network-id 101
ip nhrp holdtime 60
ip tcp adjust-mss 1360
ip policy route-map ClearRedirectedMark_RM
ip ospf network broadcast
ip ospf priority 100
ip ospf cost 1
load-interval 30
cdp enable
if-state nhrp
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 101
tunnel vrf INET
tunnel protection ipsec profile DMVPN_IPSEC shared
end

SPOKE 1 (that one affected by this issue)

Building configuration...

Current configuration : 650 bytes
!
interface Tunnel1
description WAN  CHQCRTDV00
bandwidth 300000
ip address 10.2.251.180 255.255.254.0
no ip redirects
ip mtu 1400
ip nhrp authentication DMVPN101
ip nhrp map multicast 80.69.23.227
ip nhrp map 10.2.250.3 80.69.23.227
ip nhrp network-id 101
ip nhrp holdtime 60
ip nhrp nhs 10.2.250.3
ip tcp adjust-mss 1360
ip policy route-map ClearRedirectedMark_RM
ip ospf network broadcast
ip ospf priority 0
ip ospf cost 1000
load-interval 30
cdp enable
if-state nhrp
tunnel source GigabitEthernet1
tunnel mode gre multipoint
tunnel key 101
tunnel vrf INET
tunnel protection ipsec profile DMVPN_IPSEC shared
end

REMOTE SPOKE:

Current configuration : 682 bytes
!
interface Tunnel1
description WAN - LCCCRTDV00
bandwidth 1024
ip address 10.2.250.79 255.255.254.0
no ip redirects
ip mtu 1400
ip flow ingress
ip flow egress
ip nhrp authentication DMVPN101
ip nhrp map multicast 80.69.23.227
ip nhrp map 10.2.250.3 80.69.23.227
ip nhrp network-id 101
ip nhrp holdtime 60
ip nhrp nhs 10.2.250.3
ip tcp adjust-mss 1360
ip policy route-map Dscp14ToAf11_RM
ip ospf network broadcast
ip ospf priority 0
ip ospf cost 1000
load-interval 30
if-state nhrp
qos pre-classify
cdp enable
tunnel source Dialer1
tunnel mode gre multipoint
tunnel key 101
tunnel vrf INET
tunnel protection ipsec profile DMVPN_IPSEC shared
end

Has anybody had the same issue before or someone that can help us with troubleshooting?

Regards

[paul driver]

Hello
When the NHRP tunnel fails, you have no crypto session correct which would indeed negate the transfer of traffic.
On the spoke can you enable and post the output of the following:

debug dmvpn all all 
sh crypto isakmp sa detail
sh icrypto ipsec sa detail
sh crypto session 
sh crypto sockets

[gservicios]

Hi Paul,

first of all thanks for your support and thanks to everybody who has posted,

since the output of those command is quite long, I will attach the logs.
One thing I have noticed from the command "show crypto sockets" is that the spokes the spoke in question cant reach are shown in "Socket state: Closed"

Example:

Shd Peers (local/remote): 62.30.184.178/82.109.243.50
Local Ident (addr/mask/port/prot): (62.30.184.178/255.255.255.255/0/47)
Remote Ident (addr/mask/port/prot): (82.109.243.50/255.255.255.255/0/47)
Flags: shared
IPSec Profile: "DMVPN_IPSEC"
Socket State: Closed

Client: "TUNNEL SEC" (Client State: Active)

When I log show dmvpn I can see the tunnel up but is not working

CHQRTRDV01#sh dm | s 82.163.2.236
1 82.163.2.236 10.2.250.183 UP 06:37:21 D

CHQRTRDV01#ping 10.2.250.183 source gigabitEthernet 2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.250.183, timeout is 2 seconds:
Packet sent with a source address of 10.9.111.2
.....

Success rate is 0 percent (0/5)

If I clear dmvpn session it starts to work

CHQRTRDV01#clear dmvpn SESSion peer 82.163.2.236
CHQRTRDV01#ping 10.2.250.183 source gigabitEthernet 2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.2.250.183, timeout is 2 seconds:
Packet sent with a source address of 10.9.111.2
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 32/32/33 ms

CHQRTRDV01#sh dm | s 82.163.2.236
1 82.163.2.236 10.2.250.183 UP 00:00:03 D

CHQRTRDV01#sh crypto sockets | s 82.163.2.236
Shd Peers (local/remote): 62.30.184.178/82.163.2.236
Local Ident (addr/mask/port/prot): (62.30.184.178/255.255.255.255/0/47)
Remote Ident (addr/mask/port/prot): (82.163.2.236/255.255.255.255/0/47)
Flags: shared
IPSec Profile: "DMVPN_IPSEC"
Socket State: Open
Client: "TUNNEL SEC" (Client State: Active)

Do you have any idea?

I will investigate the Socket State: Closed and why the router cant reset this state to a new one

regards

[MHM Cisco World]

I run lab it it was easy it work as your config 

CHQRTRDV01#ping 10.2.250.183 source gigabitEthernet 2 <<<<<-this wrong source of ping, it must be tunnel IP. 

and for more run traceroute not ping to see if the first traceroute show hub and second one not show hub ip.

[Georg Pauwen]

Hello,

as far as I recall, on the hub, you need to configure 'ip nhrp map multicast dynamic', in order for the spokes to send OSPF 'hellos' to the hub...