DMVPN Tunnel down

[Chandhuru sekaran marimuthu]

Hello All,

We have configured two Tunnels in single ISP link for dual connectivity.

Data center1 router <-------------> Edge Router(Tunnel1)

Data center2 router <-------------> Edge Router(Tunnel2)

Above is the setup of DMVPN Tunnel.

Edge Router#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel2, IPv4 NHRP Details
Type:Spoke, NHRP Peers:2,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 Public IP DC1 10.XX.XX.1 UP 13:21:24 S
1 Public IP DC2 10.XX.XX.2 NHRP 07:37:04 S

neighbor 10.XX.XX.1 remote-as 13567
neighbor 10.XX.XX.1 description DC1 Router
neighbor 10.XX.XX.1 password 7 password
neighbor 10.XX.XX.1 update-source Tunnel2
neighbor 10.XX.XX.1 timers 180 540
neighbor 10.XX.XX.1 send-community both
neighbor 10.XX.XX.1 soft-reconfiguration inbound
neighbor 10.XX.XX.1 route-map BGP_INBOUND_3GDMVPN in
neighbor 10.XX.XX.1 route-map BGP_OUTBOUND_3GDMVPN out
neighbor 10.XX.XX.2 remote-as 13567
neighbor 10.XX.XX.2 description DC2 Router
neighbor 10.XX.XX.2 password 7 password
neighbor 10.XX.XX.2 update-source Tunnel2
neighbor 10.XX.XX.2 timers 180 540
neighbor 10.XX.XX.2 send-community both
neighbor 10.XX.XX.2 soft-reconfiguration inbound
neighbor 10.XX.XX.2 route-map BGP_INBOUND_3GDMVPN in
neighbor 10.XX.XX.2 route-map BGP_OUTBOUND_3GDMVPN out

DC2 Router#sh dmv 

0 UNKNOWN 10.XX.XX.104 NHRP never IX

Could you please let us know what would be the reason? and i am sure, it is not issue with ISP since one tunnel is up.

Thanks in advance.

Regards,

Chandhuru

[Georg Pauwen]

Hello,

post the full configs of both ends of the DMVPN...

[Chandhuru sekaran marimuthu]

Hello Georg,

Thanks for prompt response.

I Have attached edge router config. Could you please confirm any specific running config from DC router so that i can pull and give it for you. 

Regards,

Chandhuru

[Georg Pauwen]

Hello,

from the hub, the parts that are needed to compare if there is a mismatch are all the crypto parts as well as the tunnel config:

crypto keyring kcen-keyring-3g vrf 3g
pre-shared-key address 0.0.0.0 0.0.0.0 key XXXXXXXXXXXXXXX
!
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp keepalive 60
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set gre_set esp-3des esp-sha-hmac
mode transport
crypto ipsec df-bit clear
!
crypto ipsec profile gre_prof
set transform-set gre_set

!
interface Tunnel2
description 3G-DMVPN Tunnel Interface
ip address 10.XX.XX.104 255.255.254.0
ip access-group ALLOWED_TRAFFIC_3G in
no ip redirects
ip mtu 1400
ip flow ingress
ip flow egress
ip nhrp authentication kcen
ip nhrp map multicast dynamic
ip nhrp map multicast Public IP DC2
ip nhrp map 10.XX.XX.2 Public IP DC2
ip nhrp map multicast Public IP DC1
ip nhrp map 10.XX.XX.1 Public IP DC1
ip nhrp network-id 1
ip nhrp holdtime 600
ip nhrp nhs 10.XX.XX.1
ip nhrp nhs 10.XX.XX.2
ip tcp adjust-mss 1360
load-interval 30
if-state nhrp
qos pre-classify
tunnel source GigabitEthernet0/2/0
tunnel mode gre multipoint
tunnel vrf 3g
tunnel protection ipsec profile gre_prof

[Chandhuru sekaran marimuthu]

Hello Georg,

Please find the config below,

DC2 Router#sh run | sec crypto
crypto keyring keyring-vpn-AWS_NorthVirginia_2_2 vrf internet
local-address GigabitEthernet0/0/0
pre-shared-key address XXXXXXXX key ccccccccccc
crypto keyring keyring-vpn-AWS_NorthVirginia_2_1 vrf internet
local-address GigabitEthernet0/0/0
pre-shared-key address XXXXXXXX key cccccccc
crypto keyring keyring-AWS_NorthVirginia_1_2
local-address Public IP DC2
pre-shared-key address XXXXXXXXX key xccccccccccc
crypto keyring kcen-keyring vrf internet
pre-shared-key address 0.0.0.0 0.0.0.0 key XXXXXXXXXXXXXXX
crypto isakmp policy 10
encr 3des
authentication pre-share
group 2
crypto isakmp policy 20
encr aes
authentication pre-share
group 2
lifetime 28800
crypto isakmp policy 200
encr aes
authentication pre-share
group 2
lifetime 28800
crypto isakmp keepalive 10 10
crypto isakmp profile isakmp-vpn-AWS_NorthVirginia_2_1
keyring keyring-vpn-AWS_NorthVirginia_2_1
match identity address XXXXXXX 255.255.255.255
local-address GigabitEthernet0/0/0
crypto isakmp profile isakmp-vpn-AWS_NorthVirginia_2_2
keyring keyring-vpn-AWS_NorthVirginia_2_2
match identity address XXXXXXXX 255.255.255.255
local-address Public IP DC2
crypto ipsec security-association replay window-size 128
crypto ipsec transform-set gre_set esp-3des esp-sha-hmac
mode transport
crypto ipsec transform-set ipsec-prop-vpn-AWS_NorthVirginia_2_1 esp-aes esp-sha-hmac
mode tunnel
crypto ipsec transform-set ipsec-prop-vpn-AWS_NorthVirginia_2_2 esp-aes esp-sha-hmac
mode tunnel
crypto ipsec df-bit clear
crypto ipsec nat-transparency spi-matching
crypto ipsec profile gre_prof
set transform-set gre_set
crypto ipsec profile ipsec-vpn-AWS_NorthVirginia_2_1
set transform-set ipsec-prop-vpn-AWS_NorthVirginia_2_1
set pfs group2
crypto ipsec profile ipsec-vpn-AWS_NorthVirginia_2_2
set transform-set ipsec-prop-vpn-AWS_NorthVirginia_2_2
set pfs group2
crypto ipsec df-bit clear
DC2 Router# sh int desc
Interface Status Protocol Description
Gi0/0/0 up up [CKT] *
Gi0/0/1 admin down down
Gi0/0/2 up up **LAN
Gi0/0/3 up up **LAN
Gi0 admin down down
Lo1 up up
Tu1 up up - DMVPN Tunnel Interface
Tu11 up up AmazonWebServices (AWS)
Tu22 up up
DC2 Router#sh run int tu1
Building configuration...

Current configuration : 4225 bytes
!
interface Tunnel1
description - DMVPN Tunnel Interface
ip address 10.XX.XX.2 255.255.254.0
no ip redirects
no ip unreachables
no ip proxy-arp
ip mtu 1400
ip flow monitor XXXX_Monitor input
ip flow monitor YYYY_Monitor output
ip nhrp authentication kcen
ip nhrp map multicast dynamic
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_1000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_1000K
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_5000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_5000K
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_10000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_10000K
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_20000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_20000K
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_50000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_50000K
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_100000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_100000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_10000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_10000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_50000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_50000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_100000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_100000K
ip nhrp map group VPN_SHAPE1_0-0-40-5-50-5_384K service-policy output VPN_SHAPE1_0-0-40-5-50-5_384K
ip nhrp map group VPN_SHAPE1_0-0-40-5-50-5_1544K service-policy output VPN_SHAPE1_0-0-40-5-50-5_1544K
ip nhrp map group VPN_SHAPE1_0-0-40-5-50-5_768K service-policy output VPN_SHAPE1_0-0-40-5-50-5_768K
ip nhrp map group VPN_SHAPE1_0-0-40-5-50-5_2000K service-policy output VPN_SHAPE1_0-0-40-5-50-5_2000K
ip nhrp map group VPN_SHAPE1_0-0-40-5-50-5_3000K service-policy output VPN_SHAPE1_0-0-40-5-50-5_3000K
ip nhrp map group VPN_SHAPE1_0-0-40-5-50-5_20000K service-policy output VPN_SHAPE1_0-0-40-5-50-5_20000K
ip nhrp map group VPN_SHAPE1_0-0-40-10-40-10_344K service-policy output VPN_SHAPE1_0-0-40-10-40-10_344K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_2000k service-policy output VPN_SHAPE_15-20-30-5-40-5_2000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_3000K service-policy output VPN_SHAPE_15-20-30-5-40-5_3000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_4000K service-policy output VPN_SHAPE_15-20-30-5-40-5_4000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_6000K service-policy output VPN_SHAPE_15-20-30-5-40-5_6000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_8000K service-policy output VPN_SHAPE_15-20-30-5-40-5_8000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_10000K service-policy output VPN_SHAPE_15-20-30-5-40-5_10000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_15000K service-policy output VPN_SHAPE_15-20-30-5-40-5_15000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_20000K service-policy output VPN_SHAPE_15-20-30-5-40-5_20000K
ip nhrp map group VPN_SHAPE1_15-20-30-5-40-5_25000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_25000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_30000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_30000K
ip nhrp map group VPN_SHAPE_15-20-30-5-40-5_36000K service-policy output VPN_SHAPE1_15-20-30-5-40-5_36000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_4000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_4000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_18000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_18000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_20000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_20000K
ip nhrp map group VPN_SHAPE1_15-60-15-5-15-5_30000K service-policy output VPN_SHAPE1_15-60-15-5-15-5_30000K
ip nhrp map group VPN_SHAPE1_0-0-50-0-25-25_6000K service-policy output VPN_SHAPE1_0-0-50-0-25-25_6000K
ip nhrp map group VPN_SHAPE1_30-20-20-15-40-5_10000K service-policy output VPN_SHAPE1_30-20-20-15-40-5_10000K
ip nhrp map group VPN_SHAPE1_30-20-20-15-40-5_6000K service-policy output VPN_SHAPE1_30-20-20-15-40-5_6000K
ip nhrp network-id 1
ip nhrp holdtime 600
ip tcp adjust-mss 1360
load-interval 30
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel vrf internet
tunnel protection ipsec profile gre_prof
crypto ipsec df-bit clear
bgp-policy destination ip-qos-map
end

Regards,

Chandhuru

[Georg Pauwen]

Hello,

thanks for the configs. I think the problem is that a tunnel can only be in one VRF. Since your DC2 router tunnel is in a different VRF, it won't work. So your best option is to create two tunnels on each spoke, one for each VRF.

[Chandhuru sekaran marimuthu]

Thanks for your reply Georg!

I guess you did misunderstood. only one spoke and two hubs.

DC1 ----------> Spoke1

DC2 ----------> Spoke1

One more thing. We have shut and no shut the tunnel at Spoke end has fixed this issue. Could you please clarify what will be the issue?

Regards,

Chandhuru

[Georg Pauwen]

Hello,

do you have a log that gives some sort of indication what went on ? I don't want to speculate.

Either way, glad it is resolved.

[Chandhuru sekaran marimuthu]

Hello Georg,

There is no specific logs related to it. Just the log stating that "Neighbor 10.XX.XX.2 is Up"

We are seeing this issue only happening between DC2 Hub to the Spoke.

I mean not only this Spoke few other spoke also same problem. Some of the cases it become Up by itself few cases need to shut and no shut Tunnel at Spoke end to regain connectivity. 

In this scenario, what would be the probable cause. any hold on timer issue or something else.

Any guess? that would be helpful for me to troubleshoot further.

Regards,

Chandhuru

[Georg Pauwen]

Hello,

when the tunnel is down, is the BGP neighbor down, too ?

[Chandhuru sekaran marimuthu]

Yes Georg.

When DMVPN Tunnel going down. BGP neighbor also going down.

[Chandhuru sekaran marimuthu]

Any update?

[Georg Pauwen]

Hello,

since you have configured your BGP neighbors with the tunnel addresses, both depend on each other. However, the tunnel (most likely) only goes down when there is no connectivity between the two public IP addresses, that is, when the connection to your ISP goes down. Is there any indication of that in your logs ?

[Chandhuru sekaran marimuthu]

Hello ,

Yes ISP dropped and Tunnel 1 back online but only Tunnel2 come up. That is
the question here. I mean Tunnel 2 at Spoke end not back online.

[Georg Pauwen]

Hello,

can you try and configure tunnel keepalives ?

interface Tunnel2

keepalive 10 5

You would need to configure that on the hub tunnels as well...