Solved: DMVPN Issues

CarsonDavis6833 · ‎02-16-2021

Hey,

I am currently configuring a Hub (Router A) and Spoke(Rotuer B) topology using Cisco 1921's. I have already configured the neccessary ISKAMP/IPSEC policies, gre tunnel interfaces, nhrp policies however I am having issues forming peer relationships with one another. I can confirm that I am able to ping the spoke's WAN interface from the Hub WAN interface, however I can not ping the spoke's configured tunnel interface or see any ospf learned routes through the tunnel. I have listed by configurations for both nodes below as well as debugs that were ran on both nodes. Both of these routers are connected to internet to their dedicated internet circuits using default static routes. Thanks!

Running Configurations

Router A (HUB):

Current configuration : 2186 bytes
!
! Last configuration change at 01:35:18 UTC Wed Feb 17 2021
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname DATACENTER-HUB
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
ethernet lmi ce
bsd-client server url https://cloudsso.cisco.com/as/token.oauth2
!
!
!

DATACENTER-HUB# show run
Building configuration...

Current configuration : 2186 bytes
!
! Last configuration change at 01:35:18 UTC Wed Feb 17 2021
!
version 15.5
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname DATACENTER-HUB
!
boot-start-marker
boot-end-marker
!
!
!
no aaa new-model
ethernet lmi ce
bsd-client server url https://cloudsso.cisco.com/as/token.oauth2
!
!
!
!
!
!
!
!
!
!
ip dhcp pool WAN
network 99.147.220.8 255.255.255.252
default-router 99.147.220.10
!
!
!
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
cts logging verbose
!
!
license udi pid CISCO1921/K9 sn FJC1927E1QH
!
!
!
redundancy
!
!
!
!
!
!
!
crypto isakmp policy 10
encr aes
authentication pre-share
group 2
crypto isakmp key cisco123 address 0.0.0.0
!
!
crypto ipsec transform-set MYSET esp-aes esp-sha-hmac
mode tunnel
!
crypto ipsec profile MYPROFILE
set transform-set MYSET
!
!
!
!
!
!
!
interface Tunnel1
ip address 172.16.0.1 255.255.255.248
tunnel source GigabitEthernet0/1
tunnel destination 72.186.132.98
!
interface Tunnel2
ip address 172.16.1.1 255.255.255.248
tunnel source GigabitEthernet0/1
tunnel destination 201.202.55.41
!
interface Tunnel10
bandwidth 1000
ip address 10.0.0.1 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 5
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key 6
tunnel protection ipsec profile MYPROFILE
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
ip address 99.147.220.10 255.255.255.252
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address dhcp
duplex auto
speed auto
!
router ospf 1
auto-cost reference-bandwidth 1000
network 10.0.100.0 0.0.1.255 area 0
network 172.16.0.0 0.0.0.7 area 0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 99.72.114.2
!
!
!
!

Router B (SPOKE):

! Last configuration change at 02:28:38 UTC Wed Feb 17 2021 by cdavis
version 15.1
service timestamps debug datetime msec
service timestamps log datetime msec
no service password-encryption
!
hostname NVRT01-SPOKE
!
boot-start-marker
boot-end-marker
!
!
enable secret 4 tufkM91tBB9Py9beFGqfA3hqeUl/fF80mj5ufl2/r0U
!
no aaa new-model
!
!
no ipv6 cef
ip source-route
ip cef
!
!
!
!
!
ip name-server 68.94.157.1
!
multilink bundle-name authenticated
!
crypto pki token default removal timeout 0
!
!
license udi pid CISCO1921/K9 sn FGL164325TY
!
!
username cdavis secret 4 tufkM91tBB9Py9beFGqfA3hqeUl/fF80mj5ufl2/r0U
!
redundancy
!
!
!
!
!
!
crypto isakmp policy 10
encr aes
authentication pre-share
group 2
crypto isakmp key 6 cisco1234 address 0.0.0.0 0.0.0.0
!
!
crypto ipsec transform-set SPOKE-SET esp-aes esp-sha-hmac
crypto ipsec transform-set MYSET esp-aes esp-sha-hmac
!
crypto ipsec profile MYPROFILE
set transform-set MYSET
!
crypto ipsec profile SPOKE-PROFILE
set transform-set SPOKE-SET
!
!
!
!
!
!
interface Tunnel1
ip address 172.16.0.2 255.255.255.252
tunnel source GigabitEthernet0/0
tunnel destination 99.72.114.1
!
interface Tunnel10
bandwidth 1000
ip address 10.0.0.2 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication cisco123
ip nhrp map 10.0.0.1 99.72.114.1
ip nhrp map multicast 99.72.114.1
ip nhrp network-id 5
ip nhrp nhs 10.0.0.1
tunnel source GigabitEthernet0/0
tunnel mode gre multipoint
tunnel key 6
tunnel protection ipsec profile MYPROFILE
!
interface Embedded-Service-Engine0/0
no ip address
shutdown
!
interface GigabitEthernet0/0
ip address dhcp
ip nat outside
ip virtual-reassembly in
duplex auto
speed auto
!
interface GigabitEthernet0/1
ip address 10.0.100.1 255.255.255.0
ip nat inside
ip virtual-reassembly in
glbp 1 ip 10.0.100.7
duplex auto
speed auto
!
router ospf 1
network 23.117.195.32 0.0.0.7 area 0
network 99.23.11.0 0.0.0.7 area 0
!
ip forward-protocol nd
!
no ip http server
no ip http secure-server
!
ip nat inside source list 1 interface GigabitEthernet0/0 overload
ip nat inside source list 11 interface GigabitEthernet0/0 overload
ip nat inside source static 10.0.100.10 23.117.195.36
ip route 0.0.0.0 0.0.0.0 23.117.195.38
ip route 0.0.0.0 0.0.0.0 72.186.132.97
ip route 10.0.101.0 255.255.255.0 10.0.100.2
ip route 56.23.16.0 255.255.255.248 99.23.11.3
ip route 192.168.2.0 255.255.255.0 10.0.100.2
!
access-list 1 permit any
access-list 1 permit 10.0.100.0 0.0.0.255
access-list 11 permit 0.0.0.0
access-list 101 permit tcp any host 23.117.195.34 neq 443
!
!
!
!
!
!

Router A (Hub) Debug:

*Feb 17 02:20:16.391: %CRYPTO-4-IKMP_BAD_MESSAGE: IKE message from 72.186.132.98 failed its sanity check or is malformed

*Feb 17 02:19:46.927: NHRP: nhrp_ifcache: Avl Root:0
*Feb 17 02:19:46.927: NHRP-MPLS: Could not find AVL node for tableid: 0 vrf:
*Feb 17 02:19:46.927: NHRP: NHRP could not map 10.0.0.2 to NBMA, cache entry not found
*Feb 17 02:19:46.927: NHRP: MACADDR: if_in null netid-in 0 if_out Tunnel10 netid-out 5
*Feb 17 02:19:46.927: NHRP: Checking for delayed event NULL/10.0.0.2 on list (Tunnel10).
*Feb 17 02:19:46.927: NHRP-MPLS: tableid: 0 vrf:
*Feb 17 02:19:46.927: NHRP: No delayed event node found.
*Feb 17 02:19:46.927: NHRP: nhrp_ifcache: Avl Root:0
*Feb 17 02:19:46.927: NHRP-MPLS: Could not find AVL node for tableid: 0 vrf:
*Feb 17 02:19:46.927: NHRP: No cache for forwarding(0)
*Feb 17 02:19:46.927: NHRP: NHRP AVL root is NULL
*Feb 17 02:19:46.927: NHRP-MPLS: Inserted AVL node tableid: 0 vrf:
*Feb 17 02:19:46.927: NHRP-MPLS: Success: initialized cache radix tree head tableid: 0 vrf:
*Feb 17 02:19:46.927: NHRP: swidb Tunnel10, nhrp_cache_update
*Feb 17 02.:19:46.927: NHRP-MPLS: tableid: 0 vrf:
*Feb 17 02:19:46.927: NHRP: nhrp_ifcache: Avl Root:30DF10D8
*Feb 17 02:19:46.927: NHRP: Enqueued NHRP Resolution Request for destination: 10.0.0.2
*Feb 17 02:19:46.927: NHRP-MPLS: tableid: 0 vrf:
*Feb 17 02:19:46.939: NHRP: Checking for delayed event NULL/10.0.0.2 on list (Tunnel10).
*Feb 17 02:19:46.939: NHRP-MPLS: tableid: 0 vrf:
*Feb 17 02:19:46.939: NHRP: No delayed event node found.
*Feb 17 02:19:46.939: NHRP: nhrp_ifcache: Avl Root:30DF10D8
*Feb 17 02:19:46.939: NHRP: nhrp_ifcache: Avl Root:30DF10D8
*Feb 17 02:19:46.939: NHRP: NHRP attribute flag is false
*Feb 17 02:19:46.939: NHRP: NHRP Group map is not configure

Router B (Spoke) Debug :

Feb 17 03:03:25.531: NHRP: NHRP successfully resolved 10.0.0.1 to NBMA 99.72.114.1
Feb 17 03:03:25.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:03:25.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check.
Feb 17 03:03:27.531: NHRP: NHRP successfully resolved 10.0.0.1 to NBMA 99.72.114.1.
Feb 17 03:03:29.531: NHRP: NHRP successfully resolved 10.0.0.1 to NBMA 99.72.114.1.
Feb 17 03:03:31.531: NHRP: NHRP successfully resolved 10.0.0.1 to NBMA 99.72.114.1.
Feb 17 03:03:33.531: NHRP: NHRP successfully resolved 10.0.0.1 to NBMA 99.72.114.1.
Success rate is 0 percent (0/5)
NVRT01-SPOKE#
Feb 17 03:03:35.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:03:35.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:03:45.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:03:45.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
NVRT01-SPOKE#
Feb 17 03:03:55.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:03:55.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
NVRT01-SPOKE#
Feb 17 03:04:05.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:04:05.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:04:11.371: NHRP: Setting retrans delay to 64 for nhs dst 10.0.0.1
Feb 17 03:04:11.371: NHRP: Attempting to send packet via DEST 10.0.0.1
Feb 17 03:04:11.371: NHRP: NHRP successfully resolved 10.0.0.1 to NBMA 99.72.114.1
Feb 17 03:04:11.371: NHRP: Encapsulation succeeded. Tunnel IP addr 99.72.114.1
Feb 17 03:04:11.371: NHRP: Send Registration Request via Tunnel10 vrf 0, packet size: 108
Feb 17 03:04:11.371: src: 10.0.0.2, dst: 10.0.0.1
Feb 17 03:04:11.371: NHRP: 136 bytes out Tunnel10
Feb 17 03:04:15.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:04:15.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:04:25.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check
Feb 17 03:04:25.799: NHRP: Failed to retrieve NHRP IDB in IF ctrl check

Rob Ingram · ‎02-17-2021

Is that all the debug from the spoke, there should be a lot more than that? Try debug crypto isakmp 128

If there is no debug on the hub, that would seem to indicate that no traffic was received. Is there a firewall in front of the hub router?

The Hub interface is configured for DHCP, does it receive the same IP address everytime?

OSPF adjacency will only form once you have an IPSec SA established. Resolve that issue and then confirm OSPF is working with the suggestion.

View solution in original post

Rob Ingram · ‎02-17-2021

@CarsonDavis6833

You've provide the debug information for NHRP, but has the IPSec SAs been formed?

Provide the output of "show crypto isakmp sa" and "show crypto ipsec sa" - if they haven't been formed, then turn on debugs for isakmp and ipsec and provide the output for review.

The spoke has 2 tunnel interfaces, shutdown tunnel1. The spoke also has 2 default routes, remove the one the incorrect route.

CarsonDavis6833 · ‎02-17-2021

I have removed secondary static as well as shut down second static gre tunnel.

Requested Show Command(s):

Spoke:

NVRT01-SPOKE#show crypto ipsec sa

interface: Tunnel10
Crypto map tag: Tunnel10-head-0, local addr 72.186.132.98

protected vrf: (none)
local ident (addr/mask/prot/port): (72.186.132.98/255.255.255.255/47/0)
remote ident (addr/mask/prot/port): (99.72.114.1/255.255.255.255/47/0)
current_peer 99.72.114.1 port 500
PERMIT, flags={origin_is_acl,}
#pkts encaps: 60, #pkts encrypt: 60, #pkts digest: 60
#pkts decaps: 0, #pkts decrypt: 0, #pkts verify: 0
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts compr. failed: 0
#pkts not decompressed: 0, #pkts decompress failed: 0
#send errors 50, #recv errors 0

local crypto endpt.: 72.186.132.98, remote crypto endpt.: 99.72.114.1
path mtu 1500, ip mtu 1500, ip mtu idb GigabitEthernet0/0
current outbound spi: 0x0(0)
PFS (Y/N): N, DH group: none

inbound esp sas:

inbound ah sas:

NVRT01-SPOKE#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status
99.72.114.1 72.186.132.98 MM_NO_STATE 0 ACTIVE (deleted)

IPv6 Crypto ISAKMP SA

NVRT01-SPOKE#

Hub: (SA's have not yet formed)

DATACENTER-HUB#show crypto isak
DATACENTER-HUB#show crypto isakmp sa
IPv4 Crypto ISAKMP SA
dst src state conn-id status

IPv6 Crypto ISAKMP SA

DATACENTER-HUB#

Requested Debug:

Spoke

Feb 17 15:38:01.676: IPSEC(key_engine): request timer fired: count = 1,
(identity) local= 72.186.132.98:0, remote= 99.72.114.1:0,
local_proxy= 72.186.132.98/255.255.255.255/47/0 (type=1),
remote_proxy= 99.72.114.1/255.255.255.255/47/0 (type=1)
Feb 17 15:38:01.676: IPSEC(sa_request): ,
(key eng. msg.) OUTBOUND local= 72.186.132.98:500, remote= 99.72.114.1:500,
local_proxy= 72.186.132.98/255.255.255.255/47/0 (type=1),
remote_proxy= 99.72.114.1/255.255.255.255/47/0 (type=1),
ip
NVRT01-SPOKE#show ip protocol= ESP, transform= esp-aes esp-sha-hmac (Tunnel),
lifedur= 3600s and 4608000kb,
spi= 0x0(0), conn_id= 0, keysize= 128, flags= 0x0

Hub

No debugging messages were propagated.

Rob Ingram · ‎02-17-2021

Is that all the debug from the spoke, there should be a lot more than that? Try debug crypto isakmp 128

If there is no debug on the hub, that would seem to indicate that no traffic was received. Is there a firewall in front of the hub router?

The Hub interface is configured for DHCP, does it receive the same IP address everytime?

OSPF adjacency will only form once you have an IPSec SA established. Resolve that issue and then confirm OSPF is working with the suggestion.

CarsonDavis6833 · ‎02-17-2021

There is no firewall sitting in front of the two nodes, they are directly connected to their dedicated emulated ISP handoff circuit.

However, after changing the WAN interface on my Hub to a static and readjusting the Spoke configurations to match the new config , I am now able to ping the Hub tunnel interface. It totally went over my head that the wan interface was configured for dhcp. Couldve sworn i set it on static. Thanks!!

MHM Cisco World · ‎02-17-2021

Ospf of spoke not include the network of the tunnel ip.

Second when you ping use source "which will be tunnel ip"

CarsonDavis6833 · ‎02-17-2021

Ospf of spoke has been adjusted. All pings source from 10.0.0.2 (Spoke) source 10.0.0.1(Hub) fail.