Small DMVPN Issue

Ricky S · ‎04-25-2013

Hey everyone, I have a weird problem where two spokes stop talking to each other until I go in and issue a "Clear ip nhrp" command.

For example:

Today, I couldn't ping from a router in Toronto (10.10.200.11) to a router in Thunderbay (10.10.200.38)

Below is a show DMVPN from both routers. Here I have put 1.1.1.1 as the IP of the main hub router that each of these spokes are associated with.

Notice RTRTOR001 knows about RTRTBAY001 (10.10.200.38) via the HUB whereas RTRTBAY001 currently has a dynamic entry for RTRTOR001. This is what is causing for the communication to fail. I just don't know why this would happen? This happens quite a lot and causes me to issue clear IP NHRP command frantically on both ends trying to resolve the issue which also resets all other peers.

I have checked and crypto and ipsec settings match on both ends. Any ideas?

RTRTOR001#sh dmvpn

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete

N - NATed, L - Local, X - No Socket

# Ent --> Number of NHRP entries with same NBMA peer

NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting

UpDn Time --> Up or Down Time for a Tunnel

==========================================================================

Interface: Tunnel0, IPv4 NHRP Details

Type:Spoke, NHRP Peers:55,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb

----- --------------- --------------- ----- -------- -----

3 1.1.1.1 10.10.200.1 UP 00:50:35 S

10.10.200.38 UP 00:01:29 D

10.10.200.170 UP 00:02:05 D

1 x.x.x.x 10.10.200.2 UP 1d17h D

1 x.x.x.x 10.10.200.3 UP 1d19h D

**************************************************************************************

RTRTBAY001#sh dmvpn

Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete

N - NATed, L - Local, X - No Socket

# Ent --> Number of NHRP entries with same NBMA peer

NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting

UpDn Time --> Up or Down Time for a Tunnel

==========================================================================

Interface: Tunnel0, IPv4 NHRP Details

Type:Spoke, NHRP Peers:9,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb

----- --------------- --------------- ----- -------- -----

1 1.1.1.1 10.10.200.1 UP 03:21:31 S

1 x.x.x.x 10.10.200.11 UP 00:00:01 D

1 x.x.x.x 10.10.200.23 UP 00:01:46 D

skarthic · ‎04-28-2013

Hi Ricky,

I see that your spoke(200.11) points to the hub's physical ip for NHRP mapping of 200.38 and this in turn is breaking the tunnel.

- Are you running dmvpn phase 2 or phase 3?

- Is this a single hub,single dmvpn cloud setup ?

- How is this issue getting triggered? Does it happen every time when you want to pass traffic between these sites?

- Is this issue happening only on RTRTOR001? If so, can you check the dmvpn configuration on this router?

- If not, let me know as to how many sites you are facing this problem?

Thanks,

Karthic

Ricky S · ‎04-28-2013

Hi Karthic, thanks for your your response. I am desparate to find the cure.

You are absolutely correct and I believe that is the reason why communication breaks. I am just not sure as to why this happens when one side obtains a correct NHRP mapping to other side's public IP however the other side points to the hub's physical IP to connect to the original side.

- I believe we are running DMVPN Phase 2 ( I have posted my tunnel config from both Hub and one of the spokes)

- We have a Dual hub Dual DMVPN cloud configuration where we have a primary and a secondary hub router in the data center each terminating Tunnel0 (10.10.200.1) and Tunnel1 (10.10.201.1) for all spokes.

- This issue gets triggered at random. Sometimes if a spoke gets rebooted due to power failure etc., as it comes back online, it communicates with some spokes but not with others until I go in an issue clear ip nhrp command on both sides.

- This issue happens at random spokes and not just RTRTOR001

HUB CONFIG

crypto isakmp policy 10

encr 3des

hash md5

authentication pre-share

group 5

crypto isakmp key xxxx address 0.0.0.0 0.0.0.0

crypto isakmp keepalive 30 5 periodic

!

crypto ipsec security-association lifetime seconds 86400

crypto ipsec security-association idle-time 86400

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set TS1 esp-3des esp-md5-hmac

mode transport

!

crypto ipsec profile main_profile

set transform-set TS1

interface Tunnel0

bandwidth 102400

ip address 10.10.200.1 255.255.255.0

no ip redirects

ip mtu 1420

ip hello-interval eigrp 1 30

ip hold-time eigrp 1 90

no ip next-hop-self eigrp 1

ip flow egress

ip nhrp authentication xxxx

ip nhrp map multicast dynamic

ip nhrp network-id 200

ip nhrp holdtime 300

ip tcp adjust-mss 1380

no ip split-horizon eigrp 1

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile main_profile

end

SPOKE TUNNEL CONFIG

crypto isakmp policy 10

encr 3des

hash md5

authentication pre-share

group 5

crypto isakmp key xxxx address 0.0.0.0 0.0.0.0

crypto isakmp keepalive 120 10 periodic

!

crypto ipsec security-association lifetime seconds 86400

crypto ipsec security-association idle-time 86400

!

crypto ipsec transform-set TS1 esp-3des esp-md5-hmac

mode transport

!

crypto ipsec profile main_profile

set transform-set TS1

!

interface Tunnel0

bandwidth 100000

ip address 10.10.200.11 255.255.255.0

no ip redirects

no ip unreachables

no ip proxy-arp

ip mtu 1492

ip hello-interval eigrp 1 30

ip hold-time eigrp 1 90

no ip next-hop-self eigrp 1

ip flow egress

ip nhrp authentication xxxx

ip nhrp map multicast dynamic

ip nhrp map multicast 1.1.1.1

ip nhrp map 10.10.200.1 1.1.1.1

ip nhrp network-id 200

ip nhrp holdtime 300

ip nhrp nhs 10.10.200.1

ip nhrp registration no-unique

ip tcp adjust-mss 1380

load-interval 30

delay 5

no snmp trap link-status

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile main_profile shared

Thanks

Ricky S · ‎04-29-2013

Another thing I notice when two spokes stop communicating with one another, is that when I issue the show crypto ipsec sa command, I notice SAs for all destinations are there over Tunnel0 (primary dmvpn tunnel) however for the spoke that is unreachable, SA flips over to Tunnel1 (secondary). See below: 222.222.222.222 is the IP address of the Secondary DMVPN hub connected over Tunnel1. All SAs to spokes should always establish over Tunnel0 however in my case SA to RTRTOR001 (216.216.216.216) is established over Tunnel1. When I issue the clear ip nhrp command, the latter clears out and re-establishes under Tunnel0 and the communication starts to flow.

Only SA utilizing Tunnel1 should be between the spoke and secondary DMVPN hub. Rest should all be under Tunnel0.

RTRSND001#sh crypto ipsec sa

interface: Tunnel1

Crypto map tag: MAINIPSECPROF1-head-1, local addr 666.666.666.666

protected vrf: (none)

local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)

remote ident (addr/mask/prot/port): (222.222.222.222/255.255.255.255/47/0)

current_peer 222.222.222.222 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 22450, #pkts encrypt: 22450, #pkts digest: 22450

#pkts decaps: 23383, #pkts decrypt: 23383, #pkts verify: 23383

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 3, #recv errors 5

local crypto endpt.: 666.666.666.666, remote crypto endpt.: 222.222.222.222

path mtu 1500, ip mtu 1500, ip mtu idb (none)

current outbound spi: 0x10CE7D4E(281967950)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0xF286B037(4068913207)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2283, flow_id: NETGX:283, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1

sa timing: remaining key lifetime (k/sec): (4587185/27595)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x10CE7D4E(281967950)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2284, flow_id: NETGX:284, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1

sa timing: remaining key lifetime (k/sec): (4587202/27595)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

protected vrf: (none)

local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)

remote ident (addr/mask/prot/port): (216.216.216.216/255.255.255.255/47/0)

current_peer 216.216.216.216 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0

#pkts decaps: 5623, #pkts decrypt: 5623, #pkts verify: 5623

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 2

local crypto endpt.: 666.666.666.666, remote crypto endpt.: 216.216.216.216

path mtu 1500, ip mtu 1500, ip mtu idb (none)

current outbound spi: 0xB1103D10(2970631440)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0xDBE4BAB4(3689200308)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2177, flow_id: NETGX:177, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1

sa timing: remaining key lifetime (k/sec): (4519361/86369)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0xB1103D10(2970631440)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2178, flow_id: NETGX:178, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1

sa timing: remaining key lifetime (k/sec): (4519364/86369)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

interface: Tunnel0

Crypto map tag: MAINIPSECPROF0-head-1, local addr 666.666.666.666

protected vrf: (none)

local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)

remote ident (addr/mask/prot/port): (444.444.444.444/255.255.255.255/47/0)

current_peer 444.444.444.444 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 123065, #pkts encrypt: 123065, #pkts digest: 123065

#pkts decaps: 122609, #pkts decrypt: 122609, #pkts verify: 122609

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 144

local crypto endpt.: 666.666.666.666, remote crypto endpt.: 444.444.444.444

path mtu 1500, ip mtu 1500, ip mtu idb (none)

current outbound spi: 0x4326F7E7(1126627303)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0xE6676207(3865534983)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2883, flow_id: NETGX:883, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1

sa timing: remaining key lifetime (k/sec): (4512528/4071)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x4326F7E7(1126627303)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2884, flow_id: NETGX:884, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1

sa timing: remaining key lifetime (k/sec): (4512539/4071)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

protected vrf: (none)

local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)

remote ident (addr/mask/prot/port): (127.127.127.127/255.255.255.255/47/0)

current_peer 127.127.127.127 port 500

PERMIT, flags={origin_is_acl,}

#pkts encaps: 101568, #pkts encrypt: 101568, #pkts digest: 101568

#pkts decaps: 101049, #pkts decrypt: 101049, #pkts verify: 101049

#pkts compressed: 0, #pkts decompressed: 0

#pkts not compressed: 0, #pkts compr. failed: 0

#pkts not decompressed: 0, #pkts decompress failed: 0

#send errors 0, #recv errors 89

local crypto endpt.: 666.666.666.666, remote crypto endpt.: 127.127.127.127

path mtu 1500, ip mtu 1500, ip mtu idb (none)

current outbound spi: 0x7D1C9153(2099024211)

PFS (Y/N): N, DH group: none

inbound esp sas:

spi: 0x9F0D0B84(2668432260)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2083, flow_id: NETGX:83, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1

sa timing: remaining key lifetime (k/sec): (4562931/2346)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

inbound ah sas:

inbound pcp sas:

outbound esp sas:

spi: 0x7D1C9153(2099024211)

transform: esp-3des esp-md5-hmac ,

in use settings ={Transport, }

conn id: 2084, flow_id: NETGX:84, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1

sa timing: remaining key lifetime (k/sec): (4562930/2346)

IV size: 8 bytes

replay detection support: Y

Status: ACTIVE

outbound ah sas:

outbound pcp sas:

***********Output Trunkated for remaining SAs**********

Ricky S · ‎04-29-2013

I did some more digging into this issue. I setup a lab DMVPN environment with 1 hub and 2 spoke routers. Everything was setup and worked as expected. Both spokes were able to setup SAs between each other to transfer data as expected. I didn't change the default SA timeout of 24 hours.

I then pulled the power on one spoke and rebooted it. The other spoke still had an active SA however the spoke that got rebooted did not. Now both spokes were unreachable from each other until I issued clear ip nhrp command on the spoke that did not get rebooted.

This is starting to seem like an issue with DMVPN where, if one spoke abruptly reboots etc., other spokes do not drop their SAs until someone manually issues the clear ip nhrp command.

skarthic · ‎05-01-2013

Hi Ricky,

Since clearing NHRP mappings on the spoke fixes the issue, I checked the NHRP configuration on the spoke.

ip nhrp map multicast dynamic <<<<<

This command will only be required on the hub for it to send multicasts on dynamically learned spoeks. So please try removing this on the spoke and check again.

If you are able to replicate this behavior in lab using 1hub and 2 spokes can you attach the configs?

Thanks,

Karthic

Ricky S · ‎05-02-2013

Hi, I think I have resolved my issue (atleast it has not occured for last 4 days). Turns out I just had to configure keepalives and also invalid-spi-recovery feature for ISAKMP

crypto isakmp keepalive 30 5

crypto isakmp invalid-spi-recovery

As per the below link, the issue I was having is one of the most common problems with IPSEC as the SAs can become out of sync.

http://www.cisco.com/image/gif/paws/115801/115801-ipsec-spi-errors-technologies_tech_note-00.pdf

Here is my LAB config that I tested this on after adding the above mentioned features and it seems to be working. I tested by sending a spoke-to-spoke ping and then rebooting the remote end. Once the remote end reboots, it sends an invalid SPI message to my router which then drops the old SA and re-establishes a new one.

***HUB***

version 15.2

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname HUB

!

boot-start-marker

boot-end-marker

!

no aaa new-model

!

no ipv6 cef

ip auth-proxy max-login-attempts 5

ip admission max-login-attempts 5

!

ip cef

!

multilink bundle-name authenticated

!

redundancy

!

crypto isakmp policy 10

encr aes 256

hash sha256

authentication pre-share

group 5

crypto isakmp key cisco123 address 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 30 5

!

crypto ipsec transform-set TSET esp-aes esp-sha256-hmac

mode transport

!

crypto ipsec profile main_profile

set transform-set TSET

!

interface Loopback0

ip address 192.168.1.1 255.255.255.255

!

interface Tunnel0

ip address 10.10.200.1 255.255.255.0

no ip redirects

no ip next-hop-self eigrp 1

no ip split-horizon eigrp 1

ip nhrp authentication cisco

ip nhrp map multicast dynamic

ip nhrp network-id 200

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile main_profile

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

ip address 1.1.1.1 255.255.255.0

duplex auto

speed auto

!

interface GigabitEthernet0/1

no ip address

shutdown

duplex auto

speed auto

!

router eigrp 1

network 10.10.200.0 0.0.0.255

network 192.168.1.1 0.0.0.0

passive-interface default

no passive-interface Tunnel0

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

control-plane

!

line con 0

line aux 0

line 2

no activation-character

no exec

transport preferred none

transport input all

transport output pad telnet rlogin lapb-ta mop udptn v120 ssh

stopbits 1

line vty 0 4

login

transport input all

!

scheduler allocate 20000 1000

!

end

***SPOKE A***

version 15.1

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname SPOKEA

!

boot-start-marker

boot-end-marker

!

no aaa new-model

!

no ipv6 cef

ip source-route

ip cef

!

multilink bundle-name authenticated

!

crypto pki token default removal timeout 0

!

redundancy

!

crypto isakmp policy 10

encr aes 256

hash sha256

authentication pre-share

group 5

crypto isakmp key cisco123 address 0.0.0.0 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 30 5

!

crypto ipsec transform-set TSET esp-aes esp-sha256-hmac

mode transport

!

crypto ipsec profile main_profile

set transform-set TSET

!

interface Loopback0

ip address 192.168.1.2 255.255.255.255

!

interface Tunnel0

ip address 10.10.200.2 255.255.255.0

no ip redirects

ip nhrp authentication cisco

ip nhrp map 10.10.200.1 1.1.1.1

ip nhrp map multicast 1.1.1.1

ip nhrp network-id 200

ip nhrp nhs 10.10.200.1

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile main_profile

!

interface Embedded-Service-Engine0/0

no ip address

shutdown

!

interface GigabitEthernet0/0

ip address 1.1.1.2 255.255.255.0

duplex auto

speed auto

!

interface GigabitEthernet0/1

no ip address

shutdown

duplex auto

speed auto

!

router eigrp 1

network 10.10.200.0 0.0.0.255

network 192.168.1.2 0.0.0.0

passive-interface default

no passive-interface Tunnel0

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

control-plane

!

line con 0

line aux 0

line 2

no activation-character

no exec

transport preferred none

transport input all

transport output pad telnet rlogin lapb-ta mop udptn v120 ssh

stopbits 1

line vty 0 4

login

transport input all

!

scheduler allocate 20000 1000

end

***SPOKE B***

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

!

hostname SPOKEB

!

boot-start-marker

boot-end-marker

!

no aaa new-model

!

service-module wlan-ap 0 bootimage autonomous

!

no ipv6 cef

ip source-route

ip cef

!

multilink bundle-name authenticated

!

crypto pki token default removal timeout 0

!

license udi pid CISCO1941W-A/K9 sn FTX152302Q0

hw-module ism 0

!

redundancy

!

crypto isakmp policy 10

encr aes 256

hash sha256

authentication pre-share

group 5

crypto isakmp key cisco123 address 0.0.0.0 0.0.0.0

crypto isakmp invalid-spi-recovery

crypto isakmp keepalive 30 5

!

crypto ipsec transform-set TSET esp-aes esp-sha256-hmac

mode transport

!

crypto ipsec profile main_profile

set transform-set TSET

!

interface Loopback0

ip address 192.168.1.3 255.255.255.255

!

interface Tunnel0

ip address 10.10.200.3 255.255.255.0

no ip redirects

ip nhrp authentication cisco

ip nhrp map 10.10.200.1 1.1.1.1

ip nhrp map multicast 1.1.1.1

ip nhrp network-id 200

ip nhrp nhs 10.10.200.1

tunnel source GigabitEthernet0/0

tunnel mode gre multipoint

tunnel key 200

tunnel protection ipsec profile main_profile

!

interface Wlan-GigabitEthernet0/0

description Internal switch interface connecting to the embedded AP

!

interface GigabitEthernet0/0

ip address 1.1.1.3 255.255.255.0

duplex auto

speed auto

!

interface wlan-ap0

description Service module interface to manage the embedded AP

no ip address

arp timeout 0

no mop enabled

no mop sysid

!

interface GigabitEthernet0/1

no ip address

shutdown

duplex auto

speed auto

!

interface Vlan1

no ip address

!

router eigrp 1

network 10.10.200.0 0.0.0.255

network 192.168.1.3 0.0.0.0

passive-interface default

no passive-interface Tunnel0

!

ip forward-protocol nd

!

no ip http server

no ip http secure-server

!

control-plane

!

line con 0

line aux 0

line 67

no activation-character

no exec

transport preferred none

transport input all

transport output pad telnet rlogin lapb-ta mop udptn v120 ssh

line vty 0 4

login

transport input all

!

scheduler allocate 20000 1000

end