DMVPN Config Issue or Bug?

corycandia · ‎11-08-2020

Experts,

We either have a config issue, or there's a bug in IOS 15.4, maybe you can help.

Issue: DMVPN isn't passing traffic, EIGRP can't form neighbor relationship.

Possible cause: Does IPSEC SA look weird? Is spoke misconfigured or is there a bug when DMVPN goes over a dialer/PPPoE interface?

Note: The branch/spoke config was built and tested using regular ethernet internet (on cellular hotspot). It worked perfect. The branch internet is PPPoE over fiber, so the config was adjusted for the PPPoE once it arrived there. The DMVPN is not working correctly after adjusting for dialer interface (tunnel source dialer, changed NAT to dialer overload, etc.).

Local spoke subnets = 172.20.32.0/255.255.248.0

Remote (hub) subnets = 172.20.16.0/255.255.248.0

Here's the spoke's relevant pieces:

crypto ikev2 proposal IKEV2-PROPOSAL_CISCO
encryption aes-cbc-256
integrity sha512
group 24
!
crypto ikev2 profile IKEV2-PROFILE_DMVPN
match identity remote any
authentication remote pre-share key ####
authentication local pre-share key ####
!
crypto ipsec transform-set TRANSFORMSET-IPSEC_DMVPN esp-aes 256 esp-sha256-hmac
mode transport
!
crypto ipsec profile IPSEC-PROFILE_DMVPN
set transform-set TRANSFORMSET-IPSEC_DMVPN
set ikev2-profile IKEV2-PROFILE_DMVPN
!
interface Tunnel0
description DMVPN
bandwidth 100000
bandwidth qos-reference 100000
ip address 172.20.254.2 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 1
no ip split-horizon eigrp 1
ip pim sparse-mode
ip nat inside
ip nhrp authentication CNDMNTCS
ip nhrp map multicast dynamic !Added
ip nhrp network-id 1
ip nhrp holdtime 300
ip virtual-reassembly in
zone-member security LAN
ip summary-address eigrp 1 172.20.16.0 255.255.248.0
ip tcp adjust-mss 1360
delay 100
nhrp group DMVPN-QOS_100MBPS
nhrp map group DMVPN-QOS_100MBPS service-policy output POLICYMAP_DMVPN-100MBPS-LINE-RATE
nhrp map group DMVPN-QOS_60MBPS service-policy output POLICYMAP_DMVPN-60MBPS-LINE-RATE
qos pre-classify
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile IPSEC-PROFILE_DMVPN
end

!
interface GigabitEthernet0
ip address dhcp
no ip proxy-arp
ip virtual-reassembly in
duplex auto
speed auto
no cdp enable
!
interface GigabitEthernet0.7
description TELEKOM SUB-INTERFACE
encapsulation dot1Q 7
ip virtual-reassembly in
pppoe enable group global
pppoe-client dial-pool-number 1
no cdp enable
!
interface Vlan320
description Default gateway for LAN
ip address 172.20.32.254 255.255.255.0
no ip proxy-arp
ip nat inside
ip virtual-reassembly in
zone-member security LAN
!
interface Dialer1
description PPPoE configuration
mtu 1492
ip address negotiated
ip nat outside
ip virtual-reassembly in
zone-member security INTERNET
encapsulation ppp
ip tcp adjust-mss 1452
dialer pool 1
dialer-group 1
ppp authentication chap pap callin
ppp chap hostname #####@t-online.de
ppp chap password #####
ppp pap sent-username #####@t-online.de password #####
ppp ipcp route default
!
router eigrp 1
network 172.20.32.0 0.0.0.255
network 172.20.39.254 0.0.0.0
network 172.20.42.0 0.0.0.255
network 172.20.254.0 0.0.0.255
passive-interface default
no passive-interface Tunnel0
!
ip nat inside source list ACL_NAT interface Dialer1 overload
ip route 0.0.0.0 0.0.0.0 62.155.246.30 5
ip route 0.0.0.0 0.0.0.0 Dialer1 10
!
ip access-list extended ACL_NAT
deny ip 172.20.0.0 0.0.255.255 172.20.0.0 0.0.255.255
deny ip 172.20.42.0 0.0.0.255 any
permit ip 172.20.32.0 0.0.0.255 any
permit ip host 172.20.39.254 any

Here's the Hub Config:

rypto ikev2 proposal IKEV2-PROPOSAL_CISCO 
encryption aes-cbc-256
integrity sha512
group 24
!
crypto ikev2 profile IKEV2-PROFILE_DYNAMIC
description ** Allows dynamic tunnels **
match identity remote any
identity local address 199.27.251.53
authentication remote pre-share key ###
authentication local pre-share key ###
!
crypto ipsec transform-set TRANSFORMSET-IPSEC_DMVPN esp-aes 256 esp-sha256-hmac 
mode transport
!
crypto ipsec profile IPSEC-PROFILE_DMVPN
set transform-set TRANSFORMSET-IPSEC_DMVPN 
set ikev2-profile IKEV2-PROFILE_DYNAMIC
!
interface Tunnel0
description DMVPN
bandwidth 100000
bandwidth qos-reference 100000
ip address 172.20.254.2 255.255.255.0
no ip redirects
ip mtu 1400
no ip next-hop-self eigrp 1
no ip split-horizon eigrp 1
ip pim sparse-mode
ip nhrp authentication CNDMNTCS
ip nhrp network-id 1
ip nhrp holdtime 300
zone-member security LAN
ip summary-address eigrp 1 172.20.16.0 255.255.248.0
ip tcp adjust-mss 1360
delay 100
nhrp group DMVPN-QOS_100MBPS
nhrp map group DMVPN-QOS_100MBPS service-policy output POLICYMAP_DMVPN-100MBPS-LINE-RATE
nhrp map group DMVPN-QOS_60MBPS service-policy output POLICYMAP_DMVPN-60MBPS-LINE-RATE
qos pre-classify
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key 1
tunnel protection ipsec profile IPSEC-PROFILE_DMVPN
!
interface GigabitEthernet0/1
description ** METRONET ISP @ 100MBPS **
mac-address XXX.9b1d.9954
bandwidth 100000
ip address XXX.27.251.53 255.255.255.192
ip nat outside
ip virtual-reassembly in
zone-member security INTERNET
ip traffic-export apply TRAFFIC_EXPORT
duplex auto
speed auto
service-policy input POLICYMAP_MARK-WAN-INGRESS
service-policy output POLICYMAP_ISP-SUB-LINE-RATE
!
router eigrp 1
network 172.20.17.64 0.0.0.15
network 172.20.17.254 0.0.0.0
network 172.20.19.246 0.0.0.0
network 172.20.19.248 0.0.0.7
network 172.20.254.0 0.0.0.255
redistribute static
redistribute ospf 1 metric 100 1 255 255 1500
passive-interface default
no passive-interface Tunnel0
no passive-interface GigabitEthernet0/0
!
ip nat inside source list ACL_NAT interface GigabitEthernet0/1 overload
!
ip access-list extended ACL_NAT
deny ip 172.20.0.0 0.0.255.255 172.20.0.0 0.0.255.255
permit ip 172.20.30.0 0.0.0.255 any
permit ip host 172.20.16.3 any
permit ip host 172.20.16.2 any
permit ip host 172.20.16.11 any
permit ip host 172.20.19.253 any
permit ip host 172.20.19.32 any
permit ip host 172.20.19.38 any
permit ip host 172.20.19.39 any
permit ip host 172.20.19.249 any
permit ip 172.20.42.0 0.0.0.255 any
permit ip 172.20.16.128 0.0.0.127 any
permit ip host 172.20.19.254 any
permit ip 172.20.19.240 0.0.0.7 any
permit ip host 172.20.19.246 any
permit ip 172.20.31.0 0.0.0.255 any
permit ip host 172.20.17.254 any

Here's the output of the show crypto IPSEC SA's that shows incorrect protection:

interface: Tunnel0
Crypto map tag: Tunnel0-head-0, local addr XX.199.198.71

protected vrf: (none)
local ident (addr/mask/prot/port): (79.199.198.71/255.255.255.255/47/0)  ! should be (0.0.0.0/0.0.0.0/0/0) right?
remote ident (addr/mask/prot/port): (XXX.27.251.53/255.255.255.255/47/0) ! should be (0.0.0.0/0.0.0.0/0/0) right?
current_peer XXX.27.251.53 port 500
PERMIT, flags={origin_is_acl,ipsec_sa_request_sent}
#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 1646, #recv errors 0

local crypto endpt.: XX.199.198.71, remote crypto endpt.: XX.27.251.53
plaintext mtu 1500, path mtu 1500, ip mtu 1500, ip mtu idb (none)
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none

inbound esp sas:

inbound ah sas:

inbound pcp sas:

outbound esp sas:

outbound ah sas:

outbound pcp sas:

The other VTI VPNs show all 0's and let routing statements direct traffic as normal, I think DMVPN is supposed to be the same?

Lastly, show DMVPN detail:

gateway#show dmvpn detail
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override
C - CTS Capable
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface Tunnel0 is up/up, Addr. is 172.20.254.3, VRF ""
Tunnel Src./Dest. addr: XX.199.198.71/MGRE, Tunnel VRF ""
Protocol/Transport: "multi-GRE/IP", Protect "IPSEC-PROFILE_DMVPN"
Interface State Control: Disabled
nhrp event-publisher : Disabled

IPv4 NHS:
172.20.254.2 E priority = 0 cluster = 0
Type:Spoke, Total NBMA Peers (v4/v6): 2

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
----- --------------- --------------- ----- -------- ----- -----------------
5 UNKNOWN 172.20.16.1 NHRP never IX 172.20.16.1/32
UNKNOWN 172.20.16.7 NHRP never IX 172.20.16.7/32
UNKNOWN 172.20.16.8 NHRP never IX 172.20.16.8/32
UNKNOWN 172.20.16.13 NHRP never IX 172.20.16.13/32
UNKNOWN 172.20.16.201 NHRP never IX 172.20.16.201/32
1 XXX.27.251.53 172.20.254.2 NHRP 01:26:48 S 172.20.254.2/32


Crypto Session Details:
--------------------------------------------------------------------------------

Interface: Tunnel0
Session: [0x119CF024]
Crypto Session Status: DOWN
fvrf: (none), IPSEC FLOW: permit 47 host XX.199.198.71 host XXX.27.251.53
Active SAs: 0, origin: crypto map
Inbound: #pkts dec'ed 0 drop 0 life (KB/Sec) 0/0
Outbound: #pkts enc'ed 0 drop 1653 life (KB/Sec) 0/0
Outbound SPI : 0x 0, transform :
Socket State: Closed

Notes: I can ping each end of the tunnel from the routers (172.20.254.2 and 172.20.254.3), but that's it, no other subnets. EIGRP is not sharing routes. The hub sees the branch as a neighbor, but the spoke has no neighbors.

Rob Ingram · ‎11-08-2020

Hi @corycandia

protected vrf: (none)
local ident (addr/mask/prot/port): (79.199.198.71/255.255.255.255/47/0)  ! should be (0.0.0.0/0.0.0.0/0/0) right?
remote ident (addr/mask/prot/port): (XXX.27.251.53/255.255.255.255/47/0) ! should be (0.0.0.0/0.0.0.0/0/0) right?

These local/remote ident are correct, DMVPN uses GRE (protocol 47). I'd expect the ident to be 0.0.0.0/0.0.0.0 if you were using a VTI, which you aren't.

You have no inbound/outbound ESP SAs established, so obviously an issue there. Can you run some ikev2/ipsec debugs and upload the output please?

I assume you've established IKEv2 SA correctly, can you provide the output of "show crypto ikev2 sa detail"

As you are using ZBFW, can you provide the configuration for sefl and internet zones for review please?

If you took of the ZBFW temporarily for testing, does the VPN establish correctly?

corycandia · ‎11-08-2020

Rob,

Brace yourself, I attached the ZBF pieces.

corycandia · ‎11-08-2020

Rob,

Attached is debug for IKEv2. It looks like it establishes and then gets deleted, reason unknown.

I have seen this before, but I can't remember why. I reviewed the IKEv2, Transform, IPSEC policy, and they all look like they match.

Rob Ingram · ‎11-08-2020

The tunnel source on the Hub's tunnel interface is loopback 0, but the IP address of the hub's Gig0/1 is xxx.27.251.53 - which is the IP address defined as the NBMA on the spoke. Double check the tunnel source on the hub is correct.

corycandia · ‎11-08-2020

I adjusted it back to gi 0/1 (outside).

I'll edit the previous posted config, and send new debugs.

MHM Cisco World · ‎11-08-2020

Tunnel ip in hub and spoke must be in same subnet

ip summary not required for phase 2 dmvpn

corycandia · ‎11-08-2020

They should be, 172.20.254.2 hub, 172.20.254.3 /24

MHM Cisco World · ‎11-08-2020

Yes

corycandia · ‎11-08-2020

Here's updated show command output:

Looks like crypto pieces are working, but not EIGRP:

*Nov 9 00:50:57.440: EIGRP: Sending HELLO on Tu0 - paklen 20
*Nov 9 00:50:57.440: AS 1, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
*Nov 9 00:50:57.568: EIGRP: Received UPDATE on Tu0 - paklen 0 nbr 172.20.254.2
*Nov 9 00:50:57.568: AS 1, Flags 0x1:(INIT), Seq 26/0 interfaceQ 0/0
*Nov 9 00:50:57.568: EIGRP: Neighbor(172.20.254.2) not yet found
gateway#
*Nov 9 00:51:02.068: EIGRP: Received UPDATE on Tu0 - paklen 0 nbr 172.20.254.2
*Nov 9 00:51:02.068: AS 1, Flags 0x1:(INIT), Seq 26/0 interfaceQ 0/0
*Nov 9 00:51:02.068: EIGRP: Neighbor(172.20.254.2) not yet found
*Nov 9 00:51:02.256: EIGRP: Sending HELLO on Tu0 - paklen 20
*Nov 9 00:51:02.256: AS 1, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
gateway#
*Nov 9 00:51:06.711: EIGRP: Sending HELLO on Tu0 - paklen 20
*Nov 9 00:51:06.711: AS 1, Flags 0x0:(NULL), Seq 0/0 interfaceQ 0/0 iidbQ un/rely 0/0
*Nov 9 00:51:07.063: EIGRP: Received UPDATE on Tu0 - paklen 0 nbr 172.20.254.2
*Nov 9 00:51:07.063: AS 1, Flags 0x1:(INIT), Seq 26/0 interfaceQ 0/0
*Nov 9 00:51:07.067: EIGRP: Neighbor(172.20.254.2) not yet found
gateway#

I double checked the eigrp network statements, they look right. tunnel interface IPs are in the same network. They can actually ping each other (172.20.254.2 <--> 172.20.254.3), but no neighbor. I've been reseaching the 'not yet found', but haven't gotten anywhere yet.

MHM Cisco World · ‎11-08-2020

.....

corycandia · ‎11-08-2020

Latest Update:

I had to add neighbor statements on both routers, then the adjacency was good.

I think this means there is a multicast problem, since EIGRP should have done this on it's own without manual/static programming. I removed the pim dense statements, and that didn't make any difference.

Reading the other posts about this issue didn't really shed light on why it's screwed up, just a way to cheat and force it to work.

Any ideas?

MHM Cisco World · ‎11-09-2020

Do you add

ip nhrp map multicast dynamic

to hub?

corycandia · ‎11-09-2020

I was missing this for some reason, went back through the config guide.

Frustrating part, I tested removing the forced neighbor statement on the hub and the tunnel dropped. I need to check what ROb said and see if firewall is blocking EIGRP, but I didn't see any dropped packets using term mon. I saw an article talking about 15.x ZBF breaking DMVPN EIGRP, I'll find that again and test.

MHM Cisco World · ‎11-10-2020

Hi
do you check after add this command ?