04-25-2013 10:02 AM - edited 03-04-2019 07:43 PM
Hey everyone, I have a weird problem where two spokes stop talking to each other until I go in and issue a "Clear ip nhrp" command.
For example:
Today, I couldn't ping from a router in Toronto (10.10.200.11) to a router in Thunderbay (10.10.200.38)
Below is a show DMVPN from both routers. Here I have put 1.1.1.1 as the IP of the main hub router that each of these spokes are associated with.
Notice RTRTOR001 knows about RTRTBAY001 (10.10.200.38) via the HUB whereas RTRTBAY001 currently has a dynamic entry for RTRTOR001. This is what is causing for the communication to fail. I just don't know why this would happen? This happens quite a lot and causes me to issue clear IP NHRP command frantically on both ends trying to resolve the issue which also resets all other peers.
I have checked and crypto and ipsec settings match on both ends. Any ideas?
RTRTOR001#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:55,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
3 1.1.1.1 10.10.200.1 UP 00:50:35 S
10.10.200.38 UP 00:01:29 D
10.10.200.170 UP 00:02:05 D
1 x.x.x.x 10.10.200.2 UP 1d17h D
1 x.x.x.x 10.10.200.3 UP 1d19h D
**************************************************************************************
RTRTBAY001#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================
Interface: Tunnel0, IPv4 NHRP Details
Type:Spoke, NHRP Peers:9,
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 1.1.1.1 10.10.200.1 UP 03:21:31 S
1 x.x.x.x 10.10.200.11 UP 00:00:01 D
1 x.x.x.x 10.10.200.23 UP 00:01:46 D
04-28-2013 12:06 PM
Hi Ricky,
I see that your spoke(200.11) points to the hub's physical ip for NHRP mapping of 200.38 and this in turn is breaking the tunnel.
- Are you running dmvpn phase 2 or phase 3?
- Is this a single hub,single dmvpn cloud setup ?
- How is this issue getting triggered? Does it happen every time when you want to pass traffic between these sites?
- Is this issue happening only on RTRTOR001? If so, can you check the dmvpn configuration on this router?
- If not, let me know as to how many sites you are facing this problem?
Thanks,
Karthic
04-28-2013 09:26 PM
Hi Karthic, thanks for your your response. I am desparate to find the cure.
You are absolutely correct and I believe that is the reason why communication breaks. I am just not sure as to why this happens when one side obtains a correct NHRP mapping to other side's public IP however the other side points to the hub's physical IP to connect to the original side.
- I believe we are running DMVPN Phase 2 ( I have posted my tunnel config from both Hub and one of the spokes)
- We have a Dual hub Dual DMVPN cloud configuration where we have a primary and a secondary hub router in the data center each terminating Tunnel0 (10.10.200.1) and Tunnel1 (10.10.201.1) for all spokes.
- This issue gets triggered at random. Sometimes if a spoke gets rebooted due to power failure etc., as it comes back online, it communicates with some spokes but not with others until I go in an issue clear ip nhrp command on both sides.
- This issue happens at random spokes and not just RTRTOR001
HUB CONFIG
crypto isakmp policy 10
encr 3des
hash md5
authentication pre-share
group 5
crypto isakmp key xxxx address 0.0.0.0 0.0.0.0
crypto isakmp keepalive 30 5 periodic
!
crypto ipsec security-association lifetime seconds 86400
crypto ipsec security-association idle-time 86400
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set TS1 esp-3des esp-md5-hmac
mode transport
!
crypto ipsec profile main_profile
set transform-set TS1
interface Tunnel0
bandwidth 102400
ip address 10.10.200.1 255.255.255.0
no ip redirects
ip mtu 1420
ip hello-interval eigrp 1 30
ip hold-time eigrp 1 90
no ip next-hop-self eigrp 1
ip flow egress
ip nhrp authentication xxxx
ip nhrp map multicast dynamic
ip nhrp network-id 200
ip nhrp holdtime 300
ip tcp adjust-mss 1380
no ip split-horizon eigrp 1
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 200
tunnel protection ipsec profile main_profile
end
SPOKE TUNNEL CONFIG
crypto isakmp policy 10
encr 3des
hash md5
authentication pre-share
group 5
crypto isakmp key xxxx address 0.0.0.0 0.0.0.0
crypto isakmp keepalive 120 10 periodic
!
crypto ipsec security-association lifetime seconds 86400
crypto ipsec security-association idle-time 86400
!
crypto ipsec transform-set TS1 esp-3des esp-md5-hmac
mode transport
!
crypto ipsec profile main_profile
set transform-set TS1
!
interface Tunnel0
bandwidth 100000
ip address 10.10.200.11 255.255.255.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1492
ip hello-interval eigrp 1 30
ip hold-time eigrp 1 90
no ip next-hop-self eigrp 1
ip flow egress
ip nhrp authentication xxxx
ip nhrp map multicast dynamic
ip nhrp map multicast 1.1.1.1
ip nhrp map 10.10.200.1 1.1.1.1
ip nhrp network-id 200
ip nhrp holdtime 300
ip nhrp nhs 10.10.200.1
ip nhrp registration no-unique
ip tcp adjust-mss 1380
load-interval 30
delay 5
no snmp trap link-status
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 200
tunnel protection ipsec profile main_profile shared
Thanks
04-29-2013 06:30 AM
Another thing I notice when two spokes stop communicating with one another, is that when I issue the show crypto ipsec sa command, I notice SAs for all destinations are there over Tunnel0 (primary dmvpn tunnel) however for the spoke that is unreachable, SA flips over to Tunnel1 (secondary). See below: 222.222.222.222 is the IP address of the Secondary DMVPN hub connected over Tunnel1. All SAs to spokes should always establish over Tunnel0 however in my case SA to RTRTOR001 (216.216.216.216) is established over Tunnel1. When I issue the clear ip nhrp command, the latter clears out and re-establishes under Tunnel0 and the communication starts to flow.
Only SA utilizing Tunnel1 should be between the spoke and secondary DMVPN hub. Rest should all be under Tunnel0.
RTRSND001#sh crypto ipsec sa
interface: Tunnel1
Crypto map tag: MAINIPSECPROF1-head-1, local addr 666.666.666.666
protected vrf: (none)
local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (222.222.222.222/255.255.255.255/47/0)
current_peer 222.222.222.222 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 22450, #pkts encrypt: 22450, #pkts digest: 22450
#pkts decaps: 23383, #pkts decrypt: 23383, #pkts verify: 23383
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 3, #recv errors 5
local crypto endpt.: 666.666.666.666, remote crypto endpt.: 222.222.222.222
path mtu 1500, ip mtu 1500, ip mtu idb (none)
current outbound spi: 0x10CE7D4E(281967950)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0xF286B037(4068913207)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2283, flow_id: NETGX:283, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1
sa timing: remaining key lifetime (k/sec): (4587185/27595)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x10CE7D4E(281967950)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2284, flow_id: NETGX:284, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1
sa timing: remaining key lifetime (k/sec): (4587202/27595)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
protected vrf: (none)
local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (216.216.216.216/255.255.255.255/47/0)
current_peer 216.216.216.216 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 5623, #pkts decrypt: 5623, #pkts verify: 5623
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 2
local crypto endpt.: 666.666.666.666, remote crypto endpt.: 216.216.216.216
path mtu 1500, ip mtu 1500, ip mtu idb (none)
current outbound spi: 0xB1103D10(2970631440)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0xDBE4BAB4(3689200308)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2177, flow_id: NETGX:177, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1
sa timing: remaining key lifetime (k/sec): (4519361/86369)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0xB1103D10(2970631440)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2178, flow_id: NETGX:178, sibling_flags 80000006, crypto map: MAINIPSECPROF1-head-1
sa timing: remaining key lifetime (k/sec): (4519364/86369)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
interface: Tunnel0
Crypto map tag: MAINIPSECPROF0-head-1, local addr 666.666.666.666
protected vrf: (none)
local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (444.444.444.444/255.255.255.255/47/0)
current_peer 444.444.444.444 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 123065, #pkts encrypt: 123065, #pkts digest: 123065
#pkts decaps: 122609, #pkts decrypt: 122609, #pkts verify: 122609
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 144
local crypto endpt.: 666.666.666.666, remote crypto endpt.: 444.444.444.444
path mtu 1500, ip mtu 1500, ip mtu idb (none)
current outbound spi: 0x4326F7E7(1126627303)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0xE6676207(3865534983)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2883, flow_id: NETGX:883, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1
sa timing: remaining key lifetime (k/sec): (4512528/4071)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x4326F7E7(1126627303)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2884, flow_id: NETGX:884, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1
sa timing: remaining key lifetime (k/sec): (4512539/4071)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
protected vrf: (none)
local ident (addr/mask/prot/port): (666.666.666.666/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (127.127.127.127/255.255.255.255/47/0)
current_peer 127.127.127.127 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 101568, #pkts encrypt: 101568, #pkts digest: 101568
#pkts decaps: 101049, #pkts decrypt: 101049, #pkts verify: 101049
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 0, #recv errors 89
local crypto endpt.: 666.666.666.666, remote crypto endpt.: 127.127.127.127
path mtu 1500, ip mtu 1500, ip mtu idb (none)
current outbound spi: 0x7D1C9153(2099024211)
PFS (Y/N): N, DH group: none
inbound esp sas:
spi: 0x9F0D0B84(2668432260)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2083, flow_id: NETGX:83, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1
sa timing: remaining key lifetime (k/sec): (4562931/2346)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
inbound ah sas:
inbound pcp sas:
outbound esp sas:
spi: 0x7D1C9153(2099024211)
transform: esp-3des esp-md5-hmac ,
in use settings ={Transport, }
conn id: 2084, flow_id: NETGX:84, sibling_flags 80000006, crypto map: MAINIPSECPROF0-head-1
sa timing: remaining key lifetime (k/sec): (4562930/2346)
IV size: 8 bytes
replay detection support: Y
Status: ACTIVE
outbound ah sas:
outbound pcp sas:
outbound pcp sas:
***********Output Trunkated for remaining SAs**********
04-29-2013 09:59 AM
I did some more digging into this issue. I setup a lab DMVPN environment with 1 hub and 2 spoke routers. Everything was setup and worked as expected. Both spokes were able to setup SAs between each other to transfer data as expected. I didn't change the default SA timeout of 24 hours.
I then pulled the power on one spoke and rebooted it. The other spoke still had an active SA however the spoke that got rebooted did not. Now both spokes were unreachable from each other until I issued clear ip nhrp command on the spoke that did not get rebooted.
This is starting to seem like an issue with DMVPN where, if one spoke abruptly reboots etc., other spokes do not drop their SAs until someone manually issues the clear ip nhrp command.
05-01-2013 12:06 PM
Hi Ricky,
Since clearing NHRP mappings on the spoke fixes the issue, I checked the NHRP configuration on the spoke.
ip nhrp map multicast dynamic <<<<<
This command will only be required on the hub for it to send multicasts on dynamically learned spoeks. So please try removing this on the spoke and check again.
If you are able to replicate this behavior in lab using 1hub and 2 spokes can you attach the configs?
Thanks,
Karthic
05-02-2013 07:58 AM
Hi, I think I have resolved my issue (atleast it has not occured for last 4 days). Turns out I just had to configure keepalives and also invalid-spi-recovery feature for ISAKMP
crypto isakmp keepalive 30 5
crypto isakmp invalid-spi-recovery
As per the below link, the issue I was having is one of the most common problems with IPSEC as the SAs can become out of sync.
http://www.cisco.com/image/gif/paws/115801/115801-ipsec-spi-errors-technologies_tech_note-00.pdf
Here is my LAB config that I tested this on after adding the above mentioned features and it seems to be working. I tested by sending a spoke-to-spoke ping and then rebooting the remote end. Once the remote end reboots, it sends an invalid SPI message to my router which then drops the old SA and re-establishes a new one.
***HUB***
version 15.2
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname HUB
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
no ipv6 cef
ip auth-proxy max-login-attempts 5
ip admission max-login-attempts 5
!
!
!
!
!
ip cef
!
multilink bundle-name authenticated
!
!
!
!
!
!
redundancy
!
!
!
!
!
!
!
crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
group 5
crypto isakmp key cisco123 address 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 30 5
!
crypto ipsec transform-set TSET esp-aes esp-sha256-hmac
mode transport
!
!
crypto ipsec profile main_profile
set transform-set TSET
!
!
!
!
!
!
interface Loopback0
ip address 192.168.1.1 255.255.255.255
!
interface Tunnel0
ip address 10.10.200.1 255.255.255.0
no ip redirects
no ip next-hop-self eigrp 1
no ip split-horizon eigrp 1
ip nhrp authentication cisco
ip nhrp map multicast dynamic
ip nhrp network-id 200
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 200
tunnel protection ipsec profile main_profile
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
ip address 1.1.1.1 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
!
router eigrp 1
network 10.10.200.0 0.0.0.255
network 192.168.1.1 0.0.0.0
passive-interface default
no passive-interface Tunnel0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
!
!
!
!
control-plane
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login
transport input all
!
scheduler allocate 20000 1000
!
end
***SPOKE A***
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname SPOKEA
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
!
no ipv6 cef
ip source-route
ip cef
!
!
!
!
!
!
multilink bundle-name authenticated
!
crypto pki token default removal timeout 0
!
!
!
!
redundancy
!
!
!
!
!
!
crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
group 5
crypto isakmp key cisco123 address 0.0.0.0 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 30 5
!
crypto ipsec transform-set TSET esp-aes esp-sha256-hmac
mode transport
!
crypto ipsec profile main_profile
set transform-set TSET
!
!
!
!
!
!
interface Loopback0
ip address 192.168.1.2 255.255.255.255
!
interface Tunnel0
ip address 10.10.200.2 255.255.255.0
no ip redirects
ip nhrp authentication cisco
ip nhrp map 10.10.200.1 1.1.1.1
ip nhrp map multicast 1.1.1.1
ip nhrp network-id 200
ip nhrp nhs 10.10.200.1
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 200
tunnel protection ipsec profile main_profile
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
ip address 1.1.1.2 255.255.255.0
duplex auto
speed auto
!
interface GigabitEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
!
router eigrp 1
network 10.10.200.0 0.0.0.255
network 192.168.1.2 0.0.0.0
passive-interface default
no passive-interface Tunnel0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
!
!
!
!
!
!
!
control-plane
!
!
!
line con 0
line aux 0
line 2
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
stopbits 1
line vty 0 4
login
transport input all
!
scheduler allocate 20000 1000
end
***SPOKE B***
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname SPOKEB
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
!
service-module wlan-ap 0 bootimage autonomous
!
no ipv6 cef
ip source-route
ip cef
!
!
!
!
!
!
multilink bundle-name authenticated
!
crypto pki token default removal timeout 0
!
!
license udi pid CISCO1941W-A/K9 sn FTX152302Q0
hw-module ism 0
!
!
!
!
redundancy
!
!
!
!
!
!
crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
group 5
crypto isakmp key cisco123 address 0.0.0.0 0.0.0.0
crypto isakmp invalid-spi-recovery
crypto isakmp keepalive 30 5
!
!
crypto ipsec transform-set TSET esp-aes esp-sha256-hmac
mode transport
!
crypto ipsec profile main_profile
set transform-set TSET
!
!
!
!
!
!
interface Loopback0
ip address 192.168.1.3 255.255.255.255
!
interface Tunnel0
ip address 10.10.200.3 255.255.255.0
no ip redirects
ip nhrp authentication cisco
ip nhrp map 10.10.200.1 1.1.1.1
ip nhrp map multicast 1.1.1.1
ip nhrp network-id 200
ip nhrp nhs 10.10.200.1
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 200
tunnel protection ipsec profile main_profile
!
interface Wlan-GigabitEthernet0/0
description Internal switch interface connecting to the embedded AP
!
interface GigabitEthernet0/0
ip address 1.1.1.3 255.255.255.0
duplex auto
speed auto
!
interface wlan-ap0
description Service module interface to manage the embedded AP
no ip address
arp timeout 0
no mop enabled
no mop sysid
!
interface GigabitEthernet0/1
no ip address
shutdown
duplex auto
speed auto
!
interface Vlan1
no ip address
!
!
router eigrp 1
network 10.10.200.0 0.0.0.255
network 192.168.1.3 0.0.0.0
passive-interface default
no passive-interface Tunnel0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
!
!
!
!
!
!
!
control-plane
!
!
line con 0
line aux 0
line 67
no activation-character
no exec
transport preferred none
transport input all
transport output pad telnet rlogin lapb-ta mop udptn v120 ssh
line vty 0 4
login
transport input all
!
scheduler allocate 20000 1000
end
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide