dmvpn tunnel down frequently ( up after shutdown no shutdown)

saif · ‎12-12-2020

Dears

kindly , we have problem in dmvpn tunnel which not work frequently , it work only after shutdown , no shutdown tunnel

below is dmvpn configuration

spoke (ISR4321/K9)

-----------------------
Cisco IOS XE Software, Version 16.09.02
Cisco IOS Software [Fuji], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.9.2, RELEASE SOFTWARE (fc4)
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
GPL code under the terms of GPL Version 2.0. For more details, see the

--------------------------------------

interface Tunnel1
ip address 192.168.100.14 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication PaSS_tun
ip nhrp map multicast 172.29.100.254
ip nhrp map 192.168.100.254 172.29.100.254
ip nhrp map multicast 172.29.100.253
ip nhrp map 192.168.100.253 172.29.100.253
ip nhrp network-id 1
ip nhrp holdtime 10
ip nhrp nhs 192.168.100.254
ip nhrp nhs 192.168.100.253
ip tcp adjust-mss 1360
nhrp group SHAPE-8M
bfd template sample
tunnel source Loopback0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile PaSS-TS
end

-----------------------------------------------------------
crypto isakmp policy 10
encr aes
hash md5
authentication pre-share
group 2
crypto isakmp key xxxxx address 172.29.100.254
crypto isakmp key xxxxx address 172.29.100.253
crypto ipsec transform-set PaSS-TS esp-aes esp-sha256-hmac
mode transport
crypto ipsec profile PaSS-TS
set transform-set PaSS-TS

hub router (ASR1001-X )
-------------------

ASR-MOI-HQ1#sh version | in Version
Cisco IOS XE Software, Version 16.09.02
Cisco IOS Software [Fuji], ASR1000 Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 16.9.2, RELEASE SOFTWARE (fc4)
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
GPL code under the terms of GPL Version 2.0. For more details, see the

do sh run | sec ip route
ip route 172.29.100.253 255.255.255.255 172.29.14.5
ip route 172.29.100.254 255.255.255.255 172.29.14.6
==================================================================
ip route 172.29.100.14 255.255.255.255 172.29.14.1

interface Tunnel254
description Hub-Main-Passports
ip address 192.168.100.254 255.255.255.0
no ip redirects
ip mtu 1400
no ip split-horizon eigrp 100
ip nhrp authentication PaSS_tun
ip nhrp network-id 1
ip nhrp holdtime 50
ip nhrp bfd notify transport never
ip nhrp bfd notify services never
ip tcp adjust-mss 1360
delay 100
nhrp map group SHAPE-8M service-policy output SHAPE-8M
nhrp map group SHAPE-4M service-policy output SHAPE-4M
bfd template sample
tunnel source Loopback0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile PASSPORTS-PROF
==================================================================

crypto isakmp policy 10
encr aes
hash md5
authentication pre-share
group 2
crypto isakmp key xxxxx address 0.0.0.0
crypto ipsec transform-set PaSS-TS esp-aes esp-sha256-hmac
mode transport
crypto ipsec profile PASSPORTS-PROF
set transform-set PaSS-TS

marce1000 · ‎12-13-2020

- Check the logs on both platforms when this happens.

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

saif · ‎12-14-2020

dear sir

kindly find below log at spoke site

Syslog logging: enabled (0 messages dropped, 2 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 21713418 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 21713418 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level informational, 355 message lines logged
Logging Source-Interface: VRF Name:

Log Buffer (4096 bytes):
:09:40.312: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Dec 13 12:09:40.312: shtl: 4(NSAP), sstl: 0(NSAP)
*Dec 13 12:09:40.312: pktsz: 125 extoff: 52
*Dec 13 12:09:40.312: (M) flags: "nat ", reqid: 60994
*Dec 13 12:09:40.312: src NBMA: 172.29.100.14
*Dec 13 12:09:40.312: src protocol: 192.168.100.14, dst protocol: 192.168.100.254
*Dec 13 12:09:40.312: (C-1) code: no error(0)
*Dec 13 12:09:40.312: prefix: 32, mtu: 9972, hd_time: 10
*Dec 13 12:09:40.313: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Dec 13 12:09:40.313: Responder Address Extension(3):
*Dec 13 12:09:40.313: Forward Transit NHS Record Extension(4):
*Dec 13 12:09:40.313: Reverse Transit NHS Record Extension(5):
*Dec 13 12:09:40.313: Authentication Extension(7):
*Dec 13 12:09:40.313: type:Cleartext(1), data:PaSS_tun
*Dec 13 12:09:40.313: NAT address Extension(9):
*Dec 13 12:09:40.313: (C-1) code: no error(0)
*Dec 13 12:09:40.313: prefix: 32, mtu: 9972, hd_time: 0
*Dec 13 12:09:40.313: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Dec 13 12:09:40.313: client NBMA: 172.29.100.254
*Dec 13 12:09:40.313: client protocol: 192.168.100.254
*Dec 13 12:09:40.313: Vendor Private Extension(8):
*Dec 13 12:09:40.313: vendor_id:00000C, total len :13
*Dec 13 12:09:40.313: subtype:1, len:8, data: 53 48 41 50 45 2D 38 4D
*Dec 13 12:09:40.314: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.29.100.254
*Dec 13 12:09:40.314: NHRP: 153 bytes out Tunnel1
*Dec 13 12:09:40.314: NHRP: Resetting retransmit due to hold-timer for 192.168.100.254
*Dec 13 12:09:40.314: NHRP: No SNMP node found to add requestID
*Dec 13 12:09:40.314: IPSEC-IFC MGRE/Tu1(172.29.100.14/172.29.100.253): connection lookup returned 80007F3E8B0656C0
*Dec 13 12:09:40.314: NHRP: NHRP Group Name is configured
*Dec 13 12:09:40.315: NHRP: Cisco Vendor Private Ext flag is set
*Dec 13 12:09:40.315: NHRP: added Cisco vendor extn, total len 7
*Dec 13 12:09:40.315: NHRP: Coyping VPE client type : GROUP and Length :10

*Dec 13 12:09:40.315: NHRP: Total len of the VPE : 13
*Dec 13 12:09:40.315: NHRP: Attempting to send packet through interface Tunnel1 via DEST dst 192.168.100.253
*Dec 13 12:09:40.315: NHRP: Send Registration Request via Tunnel1 vrf global(0x0), packet size: 125
*Dec 13 12:09:40.315: src: 192.168.100.14, dst: 192.168.100.253
*Dec 13 12:09:40.316: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Dec 13 12:09:40.316: shtl: 4(NSAP), sstl: 0(NSAP)
*Dec 13 12:09:40.316: pktsz: 125 extoff: 52
*Dec 13 12:09:40.316: (M) flags: "nat ", reqid: 60995
*Dec 13 12:09:40.316: src NBMA: 172.29.100.14
*Dec 13 12:09:40.316: src protocol: 192.168.100.14, dst protocol: 192.168.100.253
*Dec 13 12:09:40.316: (C-1) code: no error(0)
*Dec 13 12:09:40.316: prefix: 32, mtu: 9972, hd_time: 10
*Dec 13 12:09:40.316: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Dec 13 12:09:40.316: Responder Address Extension(3):
*Dec 13 12:09:40.316: Forward Transit NHS Record Extension(4):
*Dec 13 12:09:40.316: Reverse Transit NHS Record Extension(5):
*Dec 13 12:09:40.316: Authentication Extension(7):
*Dec 13 12:09:40.316: type:Cleartext(1), data:PaSS_tun
*Dec 13 12:09:40.316: NAT address Extension(9):
*Dec 13 12:09:40.316: (C-1) code: no error(0)
*Dec 13 12:09:40.316: prefix: 32, mtu: 9972, hd_time: 0
*Dec 13 12:09:40.317: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Dec 13 12:09:40.317: client NBMA: 172.29.100.253
*Dec 13 12:09:40.317: client protocol: 192.168.100.253
*Dec 13 12:09:40.317: Vendor Private Extension(8):
*Dec 13 12:09:40.317: vendor_id:00000C, total len :13
*Dec 13 12:09:40.317: subtype:1, len:8, data: 53 48 41 50 45 2D 38 4D
*Dec 13 12:09:40.317: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.29.100.253
*Dec 13 12:09:40.318: NHRP: 153 bytes out Tunnel1
*Dec 13 12:09:40.318: NHRP: Resetting retransmit due to hold-timer for 192.168.100.253

balaji.bandi · ‎12-13-2020

Is this a new setup or an upgrade, or is this a recurring problem? how long this DMVPN Setup up and running?

how many Hub and spoke config is this setup.

A couple of things need to check when the tunnel goes down.

1. Do you have reachability to HUB

2. is there any Link issue gone down and come up - keep monitor

3. post-show logging both the side.

4. show dmvpn understand the issue.

For now easy fix is -running EEM script make tunnel up based on the syslog message :

event manager applet DMVPN_TUNNEL_RESET
event syslog pattern "XXXXXXXXXXXXXXXXXXXXXX" <---- this need to capture when the tunnel go down and add to EEM so tunnel will be unshut automatically.
action 1.0 cli command "enable"
action 2.0 cli command "configure term"
action 3.0 cli command "interface Tunnel1"
action 4.0 cli command "shut"
action 4.0 cli command "no shut"

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

saif · ‎12-14-2020

event manager applet DMVPN_TUNNEL_RESET
event syslog pattern "%DUAL-5-NBRCHANGE: EIGRP-IPv4 100: Neighbor 192.168.100.254 (Tunnel1) is down"
action 1.0 cli command "enable"
action 2.0 cli command "configure term"
action 3.0 cli command "interface Tunnel1"
action 4.0 cli command "shut"
action 4.0 cli command "no shut"

Dear sir

kindly ,first of all thanks for your reply , find in below answer on your question

1.script above not work

2. Do you have reachability to HUB

yes , at same time tunnel ip 192.168.100.254 not reachable

3. is there any Link issue gone down and come up - keep monitor

no i ping between loopbacks ( hub-spoke) during tunnel down ping is stable

4.# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 172.29.100.253 192.168.100.253 IKE 01:28:14 S
1 172.29.100.254 192.168.100.254 UP 00:19:15 S

5. eem script not work as above

log as below

BGD-MSL-R-NEW#sh log
Syslog logging: enabled (0 messages dropped, 2 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: level debugging, 21713418 messages logged, xml disabled,
filtering disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 21713418 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level informational, 355 message lines logged
Logging Source-Interface: VRF Name:

Log Buffer (4096 bytes):
:09:40.312: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Dec 13 12:09:40.312: shtl: 4(NSAP), sstl: 0(NSAP)
*Dec 13 12:09:40.312: pktsz: 125 extoff: 52
*Dec 13 12:09:40.312: (M) flags: "nat ", reqid: 60994
*Dec 13 12:09:40.312: src NBMA: 172.29.100.14
*Dec 13 12:09:40.312: src protocol: 192.168.100.14, dst protocol: 192.168.100.254
*Dec 13 12:09:40.312: (C-1) code: no error(0)
*Dec 13 12:09:40.312: prefix: 32, mtu: 9972, hd_time: 10
*Dec 13 12:09:40.313: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Dec 13 12:09:40.313: Responder Address Extension(3):
*Dec 13 12:09:40.313: Forward Transit NHS Record Extension(4):
*Dec 13 12:09:40.313: Reverse Transit NHS Record Extension(5):
*Dec 13 12:09:40.313: Authentication Extension(7):
*Dec 13 12:09:40.313: type:Cleartext(1), data:PaSS_tun
*Dec 13 12:09:40.313: NAT address Extension(9):
*Dec 13 12:09:40.313: (C-1) code: no error(0)
*Dec 13 12:09:40.313: prefix: 32, mtu: 9972, hd_time: 0
*Dec 13 12:09:40.313: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Dec 13 12:09:40.313: client NBMA: 172.29.100.254
*Dec 13 12:09:40.313: client protocol: 192.168.100.254
*Dec 13 12:09:40.313: Vendor Private Extension(8):
*Dec 13 12:09:40.313: vendor_id:00000C, total len :13
*Dec 13 12:09:40.313: subtype:1, len:8, data: 53 48 41 50 45 2D 38 4D
*Dec 13 12:09:40.314: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.29.100.254
*Dec 13 12:09:40.314: NHRP: 153 bytes out Tunnel1
*Dec 13 12:09:40.314: NHRP: Resetting retransmit due to hold-timer for 192.168.100.254
*Dec 13 12:09:40.314: NHRP: No SNMP node found to add requestID
*Dec 13 12:09:40.314: IPSEC-IFC MGRE/Tu1(172.29.100.14/172.29.100.253): connection lookup returned 80007F3E8B0656C0
*Dec 13 12:09:40.314: NHRP: NHRP Group Name is configured
*Dec 13 12:09:40.315: NHRP: Cisco Vendor Private Ext flag is set
*Dec 13 12:09:40.315: NHRP: added Cisco vendor extn, total len 7
*Dec 13 12:09:40.315: NHRP: Coyping VPE client type : GROUP and Length :10

*Dec 13 12:09:40.315: NHRP: Total len of the VPE : 13
*Dec 13 12:09:40.315: NHRP: Attempting to send packet through interface Tunnel1 via DEST dst 192.168.100.253
*Dec 13 12:09:40.315: NHRP: Send Registration Request via Tunnel1 vrf global(0x0), packet size: 125
*Dec 13 12:09:40.315: src: 192.168.100.14, dst: 192.168.100.253
*Dec 13 12:09:40.316: (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
*Dec 13 12:09:40.316: shtl: 4(NSAP), sstl: 0(NSAP)
*Dec 13 12:09:40.316: pktsz: 125 extoff: 52
*Dec 13 12:09:40.316: (M) flags: "nat ", reqid: 60995
*Dec 13 12:09:40.316: src NBMA: 172.29.100.14
*Dec 13 12:09:40.316: src protocol: 192.168.100.14, dst protocol: 192.168.100.253
*Dec 13 12:09:40.316: (C-1) code: no error(0)
*Dec 13 12:09:40.316: prefix: 32, mtu: 9972, hd_time: 10
*Dec 13 12:09:40.316: addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
*Dec 13 12:09:40.316: Responder Address Extension(3):
*Dec 13 12:09:40.316: Forward Transit NHS Record Extension(4):
*Dec 13 12:09:40.316: Reverse Transit NHS Record Extension(5):
*Dec 13 12:09:40.316: Authentication Extension(7):
*Dec 13 12:09:40.316: type:Cleartext(1), data:PaSS_tun
*Dec 13 12:09:40.316: NAT address Extension(9):
*Dec 13 12:09:40.316: (C-1) code: no error(0)
*Dec 13 12:09:40.316: prefix: 32, mtu: 9972, hd_time: 0
*Dec 13 12:09:40.317: addr_len: 4(NSAP), subaddr_len: 0(NSAP), proto_len: 4, pref: 255
*Dec 13 12:09:40.317: client NBMA: 172.29.100.253
*Dec 13 12:09:40.317: client protocol: 192.168.100.253
*Dec 13 12:09:40.317: Vendor Private Extension(8):
*Dec 13 12:09:40.317: vendor_id:00000C, total len :13
*Dec 13 12:09:40.317: subtype:1, len:8, data: 53 48 41 50 45 2D 38 4D
*Dec 13 12:09:40.317: NHRP: Encapsulation succeeded. Sending NHRP Control Packet NBMA Address: 172.29.100.253
*Dec 13 12:09:40.318: NHRP: 153 bytes out Tunnel1
*Dec 13 12:09:40.318: NHRP: Resetting retransmit due to hold-timer for 192.168.100.253

saif · ‎12-14-2020

Dear sir

kindly about date of problem , this related with new project connect dual hub & 40 spokes