Re: EIGRP over DMVPN stops working after tunnel interface rename

mcegielka · ‎03-06-2023

Hello all,

I have a problem with DMVPN network configured over ISP provided WAN network. It consists of 35 4331 routers as spokes and 881G router as a hub. EIGRP is configured over DMVPN. Maybe it's overkill, but I wanted to gain some DMVPN experience.

Recently I wanted to make configuration uniform among spokes and decided to renumber tunnel interfaces on spokes from Tunnel 1 to Tunnel 0. I've prepared following config snippet and uploaded it by SCP to the spoke router:

interface Tunnel1
shutdown
exit
no interface Tunnel1
interface Tunnel0
!parameters copied from Tunnel 1
bandwidth 2048
ip address 192.168.51.21 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication NHRPpass
ip nhrp map 192.168.51.29 192.168.29.3
ip nhrp map multicast 192.168.29.3
ip nhrp network-id 2
ip nhrp holdtime 450
ip nhrp nhs 192.168.51.29
ip tcp adjust-mss 1360
delay 1000
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile prm_dmvpn
end

Next, I've used "copy flash:/merge.conf running-config" to merge this snippet into running configuration. Unfortunately, although DMVPN sessions are reestablished, EIGRP stops working. I think I've traced it down to multicast traffic not flowing through tunnel; I can ping tunnel ip addresses in both directions, but pinging 224.0.0.10 from spoke does not work:

#ping 224.0.0.10
Type escape sequence to abort.
Sending 1, 100-byte ICMP Echos to 224.0.0.10, timeout is 2 seconds:
.

After setting up debugging:

access-list 100 permit ip any host 224.0.0.10

debug ip packet 100 detail

Mar 6 13:15:16.274: IP: s=192.168.20.2 (local), d=224.0.0.10 (GigabitEthernet0/0/0), len 100, sending broad/multicast
Mar 6 13:15:16.275: ICMP type=8, code=0
Mar 6 13:15:16.275: IP: s=192.168.240.254 (local), d=224.0.0.10 (GigabitEthernet0/0/1), len 100, sending broad/multicast
Mar 6 13:15:16.275: ICMP type=8, code=0
Mar 6 13:15:16.275: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 100, sending broad/multicast
Mar 6 13:15:16.275: ICMP type=8, code=0
Mar 6 13:15:16.275: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 100, output feature
Mar 6 13:15:16.276: ICMP type=8, code=0, feature skipped, TCP Adjust MSS(58), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar 6 13:15:16.276: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 100, encapsulation failed
Mar 6 13:15:16.276: ICMP type=8, code=0
Mar 6 13:15:16.276: IP: s=10.0.27.97 (local), d=224.0.0.10 (Vlan1), len 100, sending broad/multicast
Mar 6 13:15:16.276: ICMP type=8, code=0
Mar 6 13:15:16.790: IP: s=192.168.51.29 (Tunnel0), d=224.0.0.10 (nil), len 60, enqueue feature, proto=88, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE.
Mar 6 13:15:20.436: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 60, sending broad/multicast, proto=88
Mar 6 13:15:20.436: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 60, output feature, proto=88, feature skipped, TCP Adjust MSS(58), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar 6 13:15:20.436: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 60, encapsulation failed, proto=88
Mar 6 13:15:21.625: IP: s=192.168.51.29 (Tunnel0), d=224.0.0.10 (nil), len 60, enqueue feature, proto=88, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar 6 13:15:25.150: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 60, sending broad/multicast, proto=88
Mar 6 13:15:25.150: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 60, output feature, proto=88, feature skipped, TCP Adjust MSS(58), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar 6 13:15:25.150: IP: s=192.168.51.20 (local), d=224.0.0.10 (Tunnel0), len 60, encapsulation failed, proto=88

Connecting without IPSec, removing and adding tunnel 0 interface again corrects this problem, but this connection has to be made by an inconvenient proxy device.
Does anyone have any idea, what causes this problem and what can be done to prevent it?

Hub software is IOS 15.4(3)M8, spoke IOS XE 16.9.5

Relevant configuration:

192.168.20.2 192.168.29.1
[Spoke]--------------------[ISP R1]...........[ISP R2]--------------------[Hub]
192.168.20.1 192.168.20.3

Hub:

crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
group 19
crypto isakmp key IPSecKey12345 address 0.0.0.0
crypto isakmp keepalive 60 periodic

crypto ipsec transform-set dmvpn_set esp-aes 256 esp-sha256-hmac
mode transport

crypto ipsec profile prm_dmvpn
set transform-set dmvpn_set
set pfs group19

interface Tunnel0
bandwidth 2048
ip address 192.168.51.29 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 1
no ip split-horizon eigrp 1
ip nhrp authentication NHRPpass
ip nhrp map multicast dynamic
ip nhrp network-id 2
ip nhrp holdtime 450
ip tcp adjust-mss 1360
delay 1000
tunnel source Vlan44
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile prm_dmvpn

router eigrp 1
network 10.10.254.0 0.0.0.15
network 192.168.51.0
network 192.168.52.0
redistribute static metric 2048 1000 255 1 1400 route-map ROZGLASZAJ_LAN_10
passive-interface Vlan45

Spoke (after changing Tunnel 1 to Tunnel 0):

crypto isakmp policy 10
encr aes 256
hash sha256
authentication pre-share
group 19
crypto isakmp key IPSecKey12345 address 0.0.0.0
crypto isakmp keepalive 60 periodic

crypto ipsec transform-set dmvpn_set esp-aes 256 esp-sha256-hmac
mode transport

crypto ipsec profile prm_dmvpn
set transform-set dmvpn_set
set pfs group19

interface Tunnel0
bandwidth 2048
ip address 192.168.51.20 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication NHRPpass
ip nhrp map 192.168.51.29 192.168.29.3
ip nhrp map multicast 192.168.29.3
ip nhrp network-id 2
ip nhrp holdtime 450
ip nhrp nhs 192.168.51.29
ip tcp adjust-mss 1360
delay 1000
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile prm_dmvpn

router eigrp 1
network 10.0.24.0 0.0.7.255
network 192.168.51.0
network 192.168.52.0
passive-interface Vlan1

Thanks in advance,

Mariusz Cegiełka

MHM Cisco World · ‎03-06-2023

points must do here to eliminate this issue
1- IPsec run isakmp keepalive
2- DMVPN tunnel
need
ip nhrp registration non-unique
if-state nhrp

that it do and check again

mcegielka · ‎03-06-2023

Hi MHM Cisco World,
thank you for your reply.

1. Command "crypto isakmp keepalive 60 periodic" was already in configuration, did you meant it?

2. I've added these commands to spoke's tunnel 1 configuration, then merged my configuration snippet, effective configuration of tunnel 0 is like this (it's another spoke, so IP addresses are different):

interface Tunnel0
bandwidth 2048
ip address 192.168.51.22 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication NHRPpass
ip nhrp map 192.168.51.29 192.168.29.3
ip nhrp map multicast 192.168.29.3
ip nhrp network-id 2
ip nhrp holdtime 450
ip nhrp nhs 192.168.51.29
ip tcp adjust-mss 1360
delay 1000
if-state nhrp
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile prm_dmvpn
end

Unfortunately, interface stayed in "Tunnel0 is up, line protocol is down" state, dmvpn debug showed no activity except "No peer struct to get peer description" every several minutes and "show dmvpn" showed single static entry in state "IKE". There were no IKE or IPSEC SAs. Shutting down end re-enabling Tunnel 0 caused previous behavior: tunnel worked with unicast, but no multicast traffic. This time, however, I found error "Multicast replication process not ready" in debug:

Mar  6 23:55:00.448: NHRP: Attempting to send packet through interface Tunnel0 via DEST  dst 192.168.51.29
Mar  6 23:55:00.448: NHRP: Send Registration Request via Tunnel0 vrf global(0x0), packet size: 108
Mar  6 23:55:00.448:       src: 192.168.51.22, dst: 192.168.51.29
Mar  6 23:55:00.448: NHRP: Encapsulation succeeded.  Sending NHRP Control Packet  NBMA Address: 192.168.29.3
Mar  6 23:55:00.449: NHRP: 136 bytes out Tunnel0
Mar  6 23:55:00.458: NHRP: Receive Registration Reply via Tunnel0 vrf global(0x0), packet size: 128
Mar  6 23:55:00.458: NHRP-DETAIL: netid_in = 0, to_us = 1
Mar  6 23:55:01.555: IP: s=192.168.51.22 (local), d=224.0.0.10 (Tunnel0), len 60, sending broad/multicast, proto=88
Mar  6 23:55:01.555: IP: s=192.168.51.22 (local), d=224.0.0.10 (Tunnel0), len 60, output feature, proto=88, feature skipped, TCP Adjust MSS(58), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar  6 23:55:01.555: NHRP: Error: Multicast replication process not ready
Mar  6 23:55:01.555: IP: s=192.168.51.22 (local), d=224.0.0.10 (Tunnel0), len 60, encapsulation failed, proto=88
Mar  6 23:55:01.678: IP: s=192.168.51.29 (Tunnel0), d=224.0.0.10 (nil), len 60, enqueue feature, proto=88, TCP Adjust MSS(5), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar  6 23:55:01.679: %DUAL-5-NBRCHANGE: EIGRP-IPv4 1: Neighbor 192.168.51.29 (Tunnel0) is up: new adjacency
Mar  6 23:55:01.680: IP: s=192.168.51.22 (local), d=224.0.0.10 (Tunnel0), len 70, sending broad/multicast, proto=88
Mar  6 23:55:01.681: IP: s=192.168.51.22 (local), d=224.0.0.10 (Tunnel0), len 70, output feature, proto=88, feature skipped, TCP Adjust MSS(58), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Mar  6 23:55:01.681: NHRP: Error: Multicast replication process not ready
Mar  6 23:55:01.681: IP: s=192.168.51.22 (local), d=224.0.0.10 (Tunnel0), len 70, encapsulation failed, proto=88

After removing and recreating Tunnel 0 everything worked properly.

Maybe this is some kind of bug and removing and creating interface in the same configuration snippet just won't work?

Thanks in advance,
Mariusz Cegiełka

MHM Cisco World · ‎03-06-2023

if you only renumber the tunnel in same Spoke that will NEVER work for two reason
1- IP overlapping between two tunnel
if point one solve by add new subnet for new number tunnel
2- then you need to add shared to Ipsec profile since both tunnel now share same source interface
3-tunnel key must change, the tunnel key is used by hub to separate between traffic come from same spoke.

so in one Spoke try this Seq to solve all three point above
in SCP config first
no interface tunnel 1
interface tunnel 0
.....omitted

mcegielka · ‎03-07-2023

Hi MHM Cisco World,

I'm already doing that, its literally my last config file SCPed to router:

interface Tunnel1
 shutdown
exit
no interface Tunnel1
interface Tunnel0
 bandwidth 2048
 ip address 192.168.51.22 255.255.255.0
 no ip redirects
 ip mtu 1400
 ip nhrp authentication Ivs0fREg
 ip nhrp map 192.168.51.29 192.168.29.3
 ip nhrp map multicast 192.168.29.3
 ip nhrp network-id 2
 ip nhrp holdtime 450
 ip nhrp nhs 192.168.51.29
 ip tcp adjust-mss 1360
 delay 1000
 if-state nhrp
 tunnel source GigabitEthernet0/0/0
 tunnel mode gre multipoint
 tunnel key 1
 tunnel protection ipsec profile prm_dmvpn
end

Maybe adding Tunnel 0 is too fast after removing Tunnel 1 and some process on router does not catch up?

Thanks in advance,
Mariusz Cegiełka

MHM Cisco World · ‎03-07-2023

Ivs0fREg <<- this not same NHRP auth password appear in tunnel 1

mcegielka · ‎03-07-2023

Hi MHM Cisco World,

I knowed I would forget to replace my password at some point... Fortunately this network is not accessible from Internet. It should be "ip nhrp authentication NHRPpass" as in rest of the configuration.

Thanks in advance,
Mariusz Cegiełka

MHM Cisco World · ‎03-07-2023

Yes but this Password must same Hub use ?
are Hub use NHRPpass or this new one ?

mcegielka · ‎03-07-2023

Hi MHM Cisco World,

yes, NHRP passwords are the same on all spokes and a hub.

Regards,
Mariusz Cegiełka

MHM Cisco World · ‎03-07-2023

Yes it same but which one ?
if renumber tunnel with new password then you need to change the Hub password so.

mcegielka · ‎03-07-2023

Hi MHM Cisco World,

I do not change NHRP password at all, let's say its NHRPpass everywhere.

Regards,
Mariusz Cegiełka

mcegielka · ‎03-07-2023

Hi all,

I've solved this problem by creating two configuration snippets: one removing Tunnel 1 interface, and another adding Tunnel 0 interface with same configuration. My Python script copies these files to each spoke router, schedules in 1 minute cron job to copy second file to running-config and then merges first file into running-config. This removes Tunnel 1 and after about 30 seconds cron job recreates Tunnel 0 interface.

Regards,
Mariusz Cegiełka

MHM Cisco World · ‎03-07-2023

glad your issue solve.
good job friend
have a nice day