Re: Tunnel VRF DMVPN

martinbuff · ‎01-18-2022

I have a head office with Internet over MPLS (MPLS.35). I then have a multi homed branch office that has Line of Site to the head office(MPLS.50) and also BT fibre on a separate VRF (BT.99)
I'm trying to run DMVPN to go over the LoS and the BT link. So if the LoS drops we can use BT as a backup path. I have configured two tunnels with different network keys.

DMVPN Tunnel 0 over the LoS establishes fine. and also home users tunnels come up without issue.
DMVPN Tunnel 2 over the BT fibre fails between the two main sites, but home users can connect to each main siteVPN and AnyConnect without issue.

I'm completely at a loss. I've tried to include the config I think is relevant:

Head Office Router
-----------------------

interface Port-channel1
description to MPLS
ip address MPLS.35 255.255.255.248
ip nat outside
zone-member security Internet
no negotiation auto

interface Tunnel0
description COMPANY-VPN
bandwidth 40000
ip address 10.255.250.1 255.255.255.0
no ip redirects
ip mtu 1400
ip hello-interval eigrp 10 1
ip hold-time eigrp 10 3
no ip split-horizon eigrp 10
ip nhrp authentication M4
ip nhrp network-id 100000
ip nhrp holdtime 360
ip nhrp nhs 10.255.250.1
zone-member security Trusted
ip tcp adjust-mss 1360
cdp enable
tunnel source Port-channel1
tunnel mode gre multipoint
tunnel key 100000
tunnel protection ipsec profile protect-gre shared

interface Tunnel2
description DMVPNwithBT
bandwidth 20000
ip address 10.255.2.1 255.255.255.0
no ip redirects
ip mtu 1400
ip hello-interval eigrp 10 1
ip hold-time eigrp 10 3
no ip split-horizon eigrp 10
ip nhrp authentication New
ip nhrp network-id 100001
ip nhrp holdtime 360
ip nhrp nhs 10.255.2.6 nbma BT.99 multicast
zone-member security Trusted
ip tcp adjust-mss 1360
tunnel source Port-channel1
tunnel mode gre multipoint
tunnel key 100001
tunnel protection ipsec profile protect-gre shared
end

crypto ipsec profile protect-gre
set security-association lifetime seconds 86400
set transform-set COMPANY-Transform


Multi Home Branch Office
-------------------------

interface GigabitEthernet0/0/0
description Link to BCB-LoS-Rt via LoS
ip address 172.16.193.4 255.255.255.248 secondary
!used to manage the LoS equipment)
ip address MPLS.50 255.255.255.248
no ip redirects
no ip unreachables
no ip proxy-arp
ip nat outside
ip verify unicast source reachable-via rx allow-default 100
zone-member security Zone-Outside
negotiation auto
no cdp enable
crypto map SDM_CMAP_1
ip virtual-reassembly
end

interface GigabitEthernet0/0/1
description WAN:BT Fibre Broadband (areacode259888)
ip vrf forwarding BT
ip address dhcp client-id GigabitEthernet0/0/2
ip nat outside
zone-member security Zone-BTOutside
negotiation auto
no cdp enable
ip virtual-reassembly
end

interface Tunnel0
description COMPANY-VPN
bandwidth 40000
ip address 10.255.250.6 255.255.255.0
no ip redirects
ip mtu 1400
ip nbar protocol-discovery
ip hello-interval eigrp 10 1
ip hold-time eigrp 10 3
no ip split-horizon eigrp 10
ip nat inside
ip nhrp authentication M4
ip nhrp map 10.255.250.1 MPLS.35
ip nhrp map multicast MPLS.35
ip nhrp network-id 100000
ip nhrp holdtime 360
ip nhrp nhs 10.255.250.1
zone-member security Zone-TunnelToCOMPANY
ip tcp adjust-mss 1360
ip ospf network broadcast
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 100000
tunnel protection ipsec profile protect-gre
ip virtual-reassembly
end

interface Tunnel2
description DMVPNwithBT
bandwidth 20000
ip address 10.255.2.6 255.255.255.0
no ip redirects
ip mtu 1400
ip hello-interval eigrp 10 1
ip hold-time eigrp 10 3
no ip split-horizon eigrp 10
ip nhrp authentication New
ip nhrp network-id 100001
ip nhrp holdtime 360
zone-member security Zone-TunnelToCOMPANY
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/0/1
tunnel mode gre multipoint
tunnel key 100001
tunnel vrf BT
tunnel protection ipsec profile protect-gre2
end

crypto ipsec profile protect-gre
set security-association lifetime seconds 86400
set transform-set COMPANY-Transform
crypto ipsec profile protect-gre2
set security-association lifetime seconds 86400
set transform-set COMPANY-Transform


HeadOffice#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override, B - BGP
C - CTS Capable, I2 - Temporary
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel0, IPv4 NHRP Details
Type:Hub, NHRP Peers:3,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 MPLS.50 10.255.250.6 UP 1w0d D
1 HomeUser 10.255.250.14 UP 5d22h DN
1 HomeUserB 10.255.250.16 UP 00:29:25 DN

Interface: Tunnel2, IPv4 NHRP Details
Type:Hub/Spoke, NHRP Peers:2,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 BT.99 10.255.2.6 IKE 3d22h S
1 HomeUserB 10.255.2.16 UP 00:29:25 DN

MultiHome#sh dmvpn
Legend: Attrb --> S - Static, D - Dynamic, I - Incomplete
N - NATed, L - Local, X - No Socket
T1 - Route Installed, T2 - Nexthop-override, B - BGP
C - CTS Capable, I2 - Temporary
# Ent --> Number of NHRP entries with same NBMA peer
NHS Status: E --> Expecting Replies, R --> Responding, W --> Waiting
UpDn Time --> Up or Down Time for a Tunnel
==========================================================================

Interface: Tunnel0, IPv4 NHRP Details
Type:Hub/Spoke, NHRP Peers:3,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 MPLS.35 10.255.250.1 UP 1w0d S
1 HomeUSerA 10.255.250.14 UP 6d15h DN
1 HOmeUSerB 10.255.250.16 UP 00:30:49 DN

Interface: Tunnel2, IPv4 NHRP Details
Type:Hub, NHRP Peers:1,

# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb
----- --------------- --------------- ----- -------- -----
1 HomeUserB 10.255.2.16 UP 00:30:48 DN

martinbuff · ‎01-18-2022

sh log | s 15230
Jan 18 15:09:40.797: ISAKMP: (15230):processing vendor id payload
Jan 18 15:09:40.798: ISAKMP: (15230):vendor ID is Unity
Jan 18 15:09:40.798: ISAKMP: (15230):processing vendor id payload
Jan 18 15:09:40.798: ISAKMP: (15230):vendor ID is DPD
Jan 18 15:09:40.798: ISAKMP: (15230):processing vendor id payload
Jan 18 15:09:40.798: ISAKMP: (15230):speaking to another IOS box!
Jan 18 15:09:40.798: ISAKMP: (15230):received payload type 20
Jan 18 15:09:40.798: ISAKMP: (15230):His hash no match - this node outside NAT
Jan 18 15:09:40.798: ISAKMP: (15230):received payload type 20
Jan 18 15:09:40.798: ISAKMP: (15230):No NAT Found for self or peer
Jan 18 15:09:40.798: ISAKMP: (15230):Input = IKE_MESG_INTERNAL, IKE_PROCESS_MAIN_MODE
Jan 18 15:09:40.798: ISAKMP: (15230):Old State = IKE_I_MM4  New State = IKE_I_MM4
Jan 18 15:09:40.799: ISAKMP: (15230):Send initial contact
Jan 18 15:09:40.799: ISAKMP: (15230):processing CERT_REQ payload. message ID = 0
Jan 18 15:09:40.799: ISAKMP: (15230):peer wants a CT_X509_SIGNATURE cert
Jan 18 15:09:40.799: ISAKMP: (15230):peer wants cert issued by cn=sh-net-TH-SVR-DC1-CA,dc=sh-net,dc=local,ou=sh-net.local,o=Company Group Ltd,l=XXX,st=XXX,c=UK
Jan 18 15:09:40.799: ISAKMP: (15230):processing CERT_REQ payload. message ID = 0
Jan 18 15:09:40.799: ISAKMP: (15230):peer wants a CT_X509_SIGNATURE cert
Jan 18 15:09:40.799: ISAKMP: (15230):peer wants cert issued by cn=Company-HEDGE-CA,dc=Company,dc=local
Jan 18 15:09:40.799: ISAKMP: (15230):processing CERT_REQ payload. message ID = 0
Jan 18 15:09:40.799: ISAKMP: (15230):peer wants a CT_X509_SIGNATURE cert
Jan 18 15:09:40.800: ISAKMP: (15230):peer wants cert issued by cn=Cisco Licensing Root CA,o=Cisco
Jan 18 15:09:40.800: ISAKMP: (15230):IKE->PKI Get self CertificateChain state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:09:40.800: ISAKMP: (15230):PKI->IKE Got self CertificateChain state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:09:40.800: ISAKMP: (15230):IKE->PKI Get SubjectName state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:09:40.802: ISAKMP: (15230):PKI->IKE Got SubjectName state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:09:40.802: ISAKMP-ERROR: (15230):My ID configured as IPv4 Addr, but Addr not in Cert!
Jan 18 15:09:40.802: ISAKMP-ERROR: (15230):Using FQDN as My ID
Jan 18 15:09:40.803: ISAKMP: (15230):SA is doing
Jan 18 15:09:40.803: ISAKMP: (15230):RSA signature authentication using id type ID_FQDN
Jan 18 15:09:40.804: ISAKMP: (15230):ID payload
Jan 18 15:09:40.804: ISAKMP: (15230):   FQDN name    : HeadOffice-Rt1.Company.local
Jan 18 15:09:40.804: ISAKMP: (15230):   protocol     : 17
Jan 18 15:09:40.804: ISAKMP: (15230):Total payload length: 29
Jan 18 15:09:40.804: ISAKMP: (15230):IKE->PKI Get CertificateChain to be sent to peer state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:09:40.806: ISAKMP: (15230):PKI->IKE Got CertificateChain to be sent to peer state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:09:40.808: ISAKMP: (15230):constructing CERT payload for cn=HeadOffice-Rt1.sh-net.com,ou=sh-net.local,st=Herefordshire O=Company,c=GB
Jan 18 15:09:40.808: ISAKMP: (15230):using the shTrustPoint2023 trustpoint's keypair to sign
Jan 18 15:09:40.847: ISAKMP-PAK: (15230):sending packet to BT.99 my_port 500 peer_port 500 (I) MM_KEY_EXCH
Jan 18 15:09:40.847: ISAKMP: (15230):Sending an IKE IPv4 Packet.
Jan 18 15:09:40.847: ISAKMP: (15230):Input = IKE_MESG_INTERNAL, IKE_PROCESS_COMPLETE
Jan 18 15:09:40.847: ISAKMP: (15230):Old State = IKE_I_MM4  New State = IKE_I_MM5
Jan 18 15:09:50.775: ISAKMP-PAK: (15230):received packet from BT.99 dport 500 sport 500 Global (I) MM_KEY_EXCH
Jan 18 15:09:50.775: ISAKMP: (15230):phase 1 packet is a duplicate of a previous packet.
Jan 18 15:09:50.775: ISAKMP: (15230):retransmitting due to retransmit phase 1
Jan 18 15:09:51.275: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH...
Jan 18 15:09:51.275: ISAKMP: (15230):: incrementing error counter on sa, attempt 1 of 5: retransmit phase 1
Jan 18 15:09:51.275: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH
Jan 18 15:09:51.275: ISAKMP-PAK: (15230):sending packet to BT.99 my_port 500 peer_port 500 (I) MM_KEY_EXCH
Jan 18 15:09:51.275: ISAKMP: (15230):Sending an IKE IPv4 Packet.
Jan 18 15:10:00.774: ISAKMP-PAK: (15230):received packet from BT.99 dport 500 sport 500 Global (I) MM_KEY_EXCH
Jan 18 15:10:00.774: ISAKMP: (15230):phase 1 packet is a duplicate of a previous packet.
Jan 18 15:10:00.774: ISAKMP: (15230):retransmitting due to retransmit phase 1
Jan 18 15:10:01.274: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH...
Jan 18 15:10:01.274: ISAKMP: (15230):: incrementing error counter on sa, attempt 2 of 5: retransmit phase 1
Jan 18 15:10:01.274: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH
Jan 18 15:10:01.274: ISAKMP-PAK: (15230):sending packet to BT.99 my_port 500 peer_port 500 (I) MM_KEY_EXCH
Jan 18 15:10:01.274: ISAKMP: (15230):Sending an IKE IPv4 Packet.
Jan 18 15:10:10.773: ISAKMP-PAK: (15230):received packet from BT.99 dport 500 sport 500 Global (I) MM_KEY_EXCH
Jan 18 15:10:10.773: ISAKMP: (15230):phase 1 packet is a duplicate of a previous packet.
Jan 18 15:10:10.773: ISAKMP: (15230):retransmitting due to retransmit phase 1
Jan 18 15:10:10.816: ISAKMP: (15230):set new node 0 to QM_IDLE
Jan 18 15:10:10.816: ISAKMP-ERROR: (15230):SA is still budding. Attached new ipsec request to it. (local MPLS.35, remote BT.99)
Jan 18 15:10:11.273: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH...
Jan 18 15:10:11.273: ISAKMP: (15230):: incrementing error counter on sa, attempt 3 of 5: retransmit phase 1
Jan 18 15:10:11.273: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH
Jan 18 15:10:11.273: ISAKMP-PAK: (15230):sending packet to BT.99 my_port 500 peer_port 500 (I) MM_KEY_EXCH
Jan 18 15:10:11.273: ISAKMP: (15230):Sending an IKE IPv4 Packet.
Jan 18 15:10:20.774: ISAKMP-PAK: (15230):received packet from BT.99 dport 500 sport 500 Global (I) MM_KEY_EXCH
Jan 18 15:10:20.774: ISAKMP: (15230):phase 1 packet is a duplicate of a previous packet.
Jan 18 15:10:20.774: ISAKMP: (15230):retransmitting due to retransmit phase 1
Jan 18 15:10:21.274: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH...
Jan 18 15:10:21.274: ISAKMP: (15230):: incrementing error counter on sa, attempt 4 of 5: retransmit phase 1
Jan 18 15:10:21.274: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH
Jan 18 15:10:21.274: ISAKMP-PAK: (15230):sending packet to BT.99 my_port 500 peer_port 500 (I) MM_KEY_EXCH
Jan 18 15:10:21.274: ISAKMP: (15230):Sending an IKE IPv4 Packet.
Jan 18 15:10:30.774: ISAKMP-PAK: (15230):received packet from BT.99 dport 500 sport 500 Global (I) MM_KEY_EXCH
Jan 18 15:10:30.774: ISAKMP: (15230):phase 1 packet is a duplicate of a previous packet.
Jan 18 15:10:30.774: ISAKMP: (15230):retransmitting due to retransmit phase 1
Jan 18 15:10:31.275: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH...
Jan 18 15:10:31.275: ISAKMP: (15230):: incrementing error counter on sa, attempt 5 of 5: retransmit phase 1
Jan 18 15:10:31.275: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH
Jan 18 15:10:31.275: ISAKMP-PAK: (15230):sending packet to BT.99 my_port 500 peer_port 500 (I) MM_KEY_EXCH
Jan 18 15:10:31.275: ISAKMP: (15230):Sending an IKE IPv4 Packet.
Jan 18 15:10:40.816: ISAKMP: (15230):set new node 0 to QM_IDLE
Jan 18 15:10:40.816: ISAKMP-ERROR: (15230):SA is still budding. Attached new ipsec request to it. (local MPLS.35, remote BT.99)
Jan 18 15:10:41.275: ISAKMP: (15230):retransmitting phase 1 MM_KEY_EXCH...
Jan 18 15:10:41.275: ISAKMP: (15230):peer does not do paranoid keepalives.
Jan 18 15:10:41.275: ISAKMP-ERROR: (15230):deleting SA reason "Death by retransmission P1" state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:10:41.276: ISAKMP (15230): IPSec has no more SA's with this peer.  Won't keepalive phase 1.
Jan 18 15:10:41.276: ISAKMP-ERROR: (15230):deleting SA reason "Death by retransmission P1" state (I) MM_KEY_EXCH (peer BT.99)
Jan 18 15:10:41.278: ISAKMP: (15230):deleting node 2523801782 error FALSE reason "IKE deleted"
Jan 18 15:10:41.278: ISAKMP: (15230):deleting node 3642562032 error FALSE reason "IKE deleted"
Jan 18 15:10:41.278: ISAKMP: (15230):deleting node 4100597187 error FALSE reason "IKE deleted"
Jan 18 15:10:41.278: ISAKMP: (15230):IKE->PKI End PKI Session state (I) MM_NO_STATE (peer BT.99)
Jan 18 15:10:41.279: ISAKMP: (15230):PKI->IKE Ended PKI Session state (I) MM_NO_STATE (peer BT.99)
Jan 18 15:10:41.279: ISAKMP: (15230):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL
Jan 18 15:10:41.279: ISAKMP: (15230):Old State = IKE_I_MM5  New State = IKE_DEST_SA

MHM Cisco World · ‎01-18-2022

Hi friend,
you have DMVPN VRF aware, this is nice design
Tunnel 0 is up because it is in global VRF and the destination of tunnel is reachable via global.
Tunnel 2 is down because it is in different VRF "BT" and the destination of tunnel is unreachable via this VRF...
so you need static route from VRF BT to global to make Branch VRF BT reach the destination of tunnel.

martinbuff · ‎01-18-2022

Thanks for the reply. Unfortunately the BT VRF does have a default GW so MPLS.35, which is public, is routable.

#sh ip route vrf BT

Routing Table: BT
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2, m - OMP
       n - NAT, Ni - NAT inside, No - NAT outside, Nd - NAT DIA
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       H - NHRP, G - NHRP registered, g - NHRP registration summary
       o - ODR, P - periodic downloaded static route, l - LISP
       a - application route
       + - replicated route, % - next hop override, p - overrides from PfR
       & - replicated local route overrides by connected

Gateway of last resort is BT.97 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via BT.97
      10.0.0.0/16 is subnetted, 1 subnets
S        10.0.0.0 [1/0] via 10.0.1.1
      81.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        BT.96/29 is directly connected, GigabitEthernet0/0/1
L        BT.99/32 is directly connected, GigabitEthernet0/0/1
      172.31.0.0/32 is subnetted, 1 subnets
S        172.31.255.254 [254/0] via BT.97, GigabitEthernet0/0/1
      192.168.6.0/24 is variably subnetted, 2 subnets, 2 masks
C        192.168.6.0/24 is directly connected, GigabitEthernet0/0/2.5
L        192.168.6.1/32 is directly connected, GigabitEthernet0/0/2.5

MHM Cisco World · ‎01-18-2022

OK for mpls.35 you need to config NHRP map in tunnel2.

martinbuff · ‎01-18-2022

Tunnel 2 on the main Head office achieves that doesn't it. It's just coming from the other direction

ip nhrp nhs 10.255.2.6 nbma BT.99 multicast

MHM Cisco World · ‎01-18-2022

the Tunnel config in both Hub and Spoke,
you config the NHS map which is not need since it it Hub not Spoke.
Hub you need only the map for multicast "dynamic"
for spoke you need two map one for NHS and other for multicast. "after the config the NHS ip address".
I check this missing from the Tunnel2 of spoke