Solved: DMVPN - Tunnel issues after HUB restart

Elkem ITS · ‎02-18-2016

Hey all,

I have some strange problems with DMVPN that I recently found out after I booted my DMVPN hub.

It seems that the tunnels will not re establish automatically after the hub restart.

After the HUB restarted, the tunnel came up with IKE / NHRP issues, and did not go UP.

#Sh dmvpn

Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:2,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 Public-IP 10.10.1.1 IKE 16:12:34 S

After waiting for an hour, I had to manually log into the Spoke router and shut/no shut the tunnel interface to get the tunnel UP.

Any ideas of what is wrong or how I can fix this?

I have experienced a bit with holdtime and registration-timeout without any results.

HUB - CISCO2951 - Version 15.4(3)M

interface Tunnel1
bandwidth 100000
ip address 10.10.1.1 255.255.255.0
no ip redirects
ip mtu 1400
ip hello-interval eigrp 1 30
ip hold-time eigrp 1 90
no ip next-hop-self eigrp 1
no ip split-horizon eigrp 1
ip nhrp authentication PASSWORD
ip nhrp map multicast dynamic
ip nhrp network-id 2
ip nhrp holdtime 200
ip summary-address eigrp 1 0.0.0.0 0.0.0.0
ip tcp adjust-mss 1360
ip policy route-map VPN-DEFAULT-GW
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key PASSWORD
tunnel protection ipsec profile DMVPN

Spoke: Cisco 881 - Version 15.4(3)M2

interface Tunnel1
bandwidth 100000
ip address 10.10.1.5 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication PASSWORD
ip nhrp map multicast "Public-IP"
ip nhrp map 10.10.1.1 "Public-IP"
ip nhrp network-id 2
ip nhrp holdtime 60
ip nhrp nhs 10.10.1.1
ip nhrp registration timeout 10
ip tcp adjust-mss 1360
tunnel source FastEthernet4
tunnel mode gre multipoint
tunnel key PASSWORD
tunnel vrf Internet
tunnel protection ipsec profile DMVPN shared

Best regards,

Daniel

Philip D'Ath · ‎02-19-2016

I just spotted the hub is missing the nhs server command.

On the Hub add:

interface Tunnel1
  ip nhrp nhs 10.10.1.1

I also think you should have the same "ip nhrp holdtime" on the hub and spoke. Could you make them the same please. I would recommend 300s.

View solution in original post

Philip D'Ath · ‎02-18-2016

I would upgrade at least the hub it not the spoke as well to 15.4.3M4 before doing too much more. This may well be an issues that is already resolved.

Elkem ITS · ‎02-18-2016

Hey,

I did upgrade both hub and spoke to: Version 15.4(3)M4

Sadly, that did not do any difference :/

And a bit strange, after the upgrade the hub router comes up with none of my tunnels:

hub#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

hub#

And I cannot ping it open from my spoke.

Debuging crypto isakmp says alot of vs my spokes:

Feb 18 11:14:13.063: ISAKMP: ignoring request to send delete notify (no ISAKMP sa) src HUBIP dst PublicIP for SPI 0
Feb 18 11:14:18.871: ISAKMP: ignoring request to send delete notify (no ISAKMP sa) src HUBIP dst PublicIP for SPI 0x90851AE0

Philip D'Ath · ‎02-18-2016

Looking at the ISAKMP debug - have you got ISAKMP keepalives enabled? It doe snot look like the existing SPI has been cleared out.

Below is an example to enable keepalives. This needs to be done on the hub and spoke.

crypto isakmp keepalive 10 3

Philip D'Ath · ‎02-18-2016

You hub tunnel should have a "ip nhrp map ..." command, like the spoke. Try adding to the hub:

interface Tunnel1
  ip nhrp map 10.10.1.1 "Public-IP"

You said you had dual DMVPN tunnels. Is the other tunnel using a unique NHRP network ID (aka is not using 2 like this tunnel)?

Do any of your spokes have dynamic IP addresses? If so you should add this on the spoke (only on spokes with dynamic IP addresses):

interface Tunnel1
  ip nhrp registration no-unique

Elkem ITS · ‎02-19-2016

Hello Philip,

Thanx alot for the help, I do appreciate it.

-----------

First off, I tried to change the

crypto isakmp keepalive 10 3

Also, for fun I tried, just to let it live a bit longer:

crypto isakmp keepalive 10 10

Sadly, that did not help. I booted my hub and nothing happend for the next 30 minutes. Tunnels did not get re-activated.

At the moment its looking like this:

hub02#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

-----------

Funny enough, I did boot my HUB yesterday before I went home, and it seems it took about 12 hours for the tunnels to re establish:

hub02 uptime is 22 hours, 7 minutes

1 PUBLIC-IP 10.10.1.5 UP 08:55:08 D

1 PUBLIC-IP 10.10.1.6 UP 08:08:07 D

1 PUBLIC-IP 10.10.1.7 UP 02:33:32 DN

1 PUBLIC-IP 10.10.1.8 UP 07:52:19 D

1 PUBLIC-IP 10.10.1.9 UP 08:20:04 D

1 PUBLIC-IP 10.10.1.10 UP 08:31:15 D

1 PUBLIC-IP 10.10.1.11 UP 08:56:21 D

1 PUBLIC-IP 10.10.1.12 UP 08:57:47 D

-----------

Yes, the other DMVPN hub is using NHRP Network ID 1.

-----------

We have both static IP's and dynamic IP addresses on our spokes. And the dynamic have that ip nhrp registration no-unique command.

-----------

I did try to ping from tunnel interface on hub02 to one of the spokes, and this is what i saw:

hub02#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel1, IPv4 NHRP Details
Type:Unknown, NHRP Peers:1,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
2 UNKNOWN 10.10.1.5 NHRP never IX
0 UNKNOWN 10.10.1.6 NHRP never IX

Nothing important in the debug crypto isakmp, only:

ISAKMP: ignoring request to send delete notify (no ISAKMP sa) src HUBIP dst PublicIP for SPI 0

-------------

Other good ideas that I might test?

Elkem ITS · ‎02-19-2016

Update!

Took 2 hours before the tunnel re established:

hub02#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel1, IPv4 NHRP Details
Type:Hub, NHRP Peers:1,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 PUBLIC-IP 10.10.1.5 UP 02:11:27 D

glo-vpn02#

Philip D'Ath · ‎02-19-2016

I just spotted the hub is missing the nhs server command.

On the Hub add:

interface Tunnel1
  ip nhrp nhs 10.10.1.1

I also think you should have the same "ip nhrp holdtime" on the hub and spoke. Could you make them the same please. I would recommend 300s.

Elkem ITS · ‎02-22-2016

Adding "ip nhrp nhs 10.10.1.1" on the hub, seems to have helped alot. I also have added the holdtime (same on hub and spoke).

2 minutes re-establish:

hub02#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel1, IPv4 NHRP Details
Type:Hub, NHRP Peers:2,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 PUBLIC-IP 10.10.1.5 UP 00:02:37 D
1 PUBLIC-IP 10.10.1.6 UP 00:02:45 D

Waiting for the other tunnels to show up now.

Elkem ITS · ‎02-22-2016

Update:

The other tunnels came also.

Nice, this has been solved with the following things done:

Hub:

crypto isakmp keepalive 10 10

Interface Tunnel1

ip nhrp registration timeout 10

ip nhrp holdtime 300

ip nhrp nhs 10.10.1.1

Spoke:

crypto isakmp keepalive 10 10

Interface Tunnel1

ip nhrp registration timeout 10

ip nhrp holdtime 300

Thank you Philip D'Ath

Philip D'Ath · ‎02-22-2016

Your welcome.

ps. I think your "ip nhrp registration timeout 10" setting is very aggressive. I would personally remove this line completely from both hub and spoke.