MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche · ‎12-05-2020

# The problem:

I have configured MPLS over FlexVPN following the configuration snipet of Cisco live (page 39)

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKSEC-3036.pdf

The spoke-to-spoke traffic shortcut is not working so all the traffic goes via the hub.

When debugging NHRP I see the error: Could not find AVL node for vrf

Does anyone know what this error mean and how to fix it?

I am running: Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.7(3)M3, RELEASE SOFTWARE (fc2)

# Diagram

R1 (spoke LAN = 10.1.10.1/24 - vrf BLUE)======== R3(hub)============R2(spoke LAN = 10.1.20.1/24 - vrf BLUE)

# Troubleshooting steps

R2 sends ICMP traffic to R1 LAN. R2 receives a redirect from the hub and sends back a Resolution Request.

r2#ping vrf BLUE 10.1.10.1 source gi0/1 repeat 3


r2#sl
Log Buffer (8192 bytes):

 NHRP: Receive Traffic Indication via Tunnel0 vrf global(0x0), packet size: 84
  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
      shtl: 4(NSAP), sstl: 0(NSAP)
      pktsz: 84 extoff: 68
  (M) traffic code: redirect(0)
      src NBMA: 198.51.100.7
      src protocol: 10.1.30.0, dst protocol: 10.1.20.1
      Contents of nhrp traffic indication packet:
         45 00 00 64 00 67 00 00 FE 01 8A 2E 0A 01 14 01
         0A 01 0A 01 08 00 64 11 00 19 00
 NHRP-DETAIL: netid_in = 1, to_us = 0
 NHRP-DETAIL: Multipath IP route lookup for 10.1.20.1 in vrf BLUE(0x1) yielded GigabitEthernet0/1, pfx:10.1.20.0/24 (netid_in:1 if_in:Tunnel0)
 NHRP: nhrp_rtlookup yielded GigabitEthernet0/1
 NHRP-DETAIL: netid_out 0, netid_in 1
 NHRP: Parsing NHRP Traffic Indication

 NHRP: Enqueued NHRP Resolution Request for destination: 10.1.10.1
 NHRP: Checking for delayed event NULL/10.1.10.1 on list (Tunnel0 vrf: BLUE(0x1))
 NHRP: No delayed event node found.

R3 (hub) receive the resolution request but is unable to respond.

The error seen is : 'NHRP: Could not find AVL node for vrf:BLUE(0x1)'

NHRP: Receive Resolution Request via Virtual-Access1 vrf global(0x0), packet size: 79
 (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
     shtl: 4(NSAP), sstl: 0(NSAP)
     pktsz: 79 extoff: 52
 (M) flags: "router auth src-stable nat ", reqid: 20
     src NBMA: 198.51.100.3
     src protocol: 10.1.30.2, dst protocol: 10.1.10.1
 (C-1) code: no error(0)
       prefix: 32, mtu: 17874, hd_time: 600
       addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
NHRP-DETAIL: netid_in = 1, to_us = 0
NHRP: Could not find AVL node for vrf:BLUE(0x1)
NHRP-DETAIL: Multipath IP route lookup for 10.1.10.1 in vrf BLUE(0x1) yielded Null0, pfx:10.1.10.0/24 (netid_in:1 if_in:Virtual-Access1)
NHRP: Route lookup for destination 10.1.10.1 in vrf BLUE(0x1) yielded interface Null0, prefixlen 24
NHRP: Could not find AVL node for vrf:BLUE(0x1)

Yet the hub does have a route for the prefix 10.1.10.0/24

r3#sh ip route vrf BLUE | b ^G
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S        10.1.0.0/16 is directly connected, Null0
B        10.1.10.0/24 [200/0] via 10.1.30.1, 00:37:56
B        10.1.20.0/24 [200/0] via 10.1.30.2, 00:37:10
C        10.1.30.30/32 is directly connected, Loopback10

r3#sh ip cef vrf BLUE 10.1.10.1
10.1.10.0/24
  nexthop 10.1.30.1 Virtual-Access2 label 16-(local:18)

r2 never gets a reply so the shortcut does not work

r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:00:04, expire 00:03:00
   Type: incomplete, Flags: negative
   Cache hits: 2


r2#traceroute vrf BLUE 10.1.10.1 source gi0/1
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 61 msec 57 msec 31 msec
  2 10.1.10.1 87 msec 102 msec 124 msec

# Configuration Snipet on Hub

vrf definition BLUE
 rd 1:1
 !
 address-family ipv4
  route-target export 1:1
  route-target import 1:1
 exit-address-family
!
vrf definition RED
 rd 1:2
 !
 address-family ipv4
  route-target export 1:2
  route-target import 1:2
 exit-address-family
!
interface Loopback1
 ip address 10.1.30.0 255.255.255.255
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel source GigabitEthernet0/2
 tunnel protection ipsec profile default
!
router bgp 1
 bgp log-neighbor-changes
 bgp listen range 10.1.30.0/24 peer-group Flex
 neighbor Flex peer-group
 neighbor Flex remote-as 1
 neighbor Flex update-source Loopback1
 neighbor Flex timers 5 15
 !
 address-family vpnv4
  neighbor Flex activate
  neighbor Flex send-community extended
 exit-address-family
 !
 address-family ipv4 vrf BLUE
  network 10.1.0.0 mask 255.255.0.0
  network 10.1.30.30 mask 255.255.255.255
 exit-address-family
 !
 address-family ipv4 vrf RED
  network 10.1.0.0 mask 255.255.0.0
 exit-address-family

Georg Pauwen · ‎12-05-2020

Hello,

post the full configs of the hub and the two spokes (sh run). IOSv means you are doing this in VIRL or GNS3 ?

Tyche · ‎12-06-2020

Hello,

I am using GNS3.

Please see full config and diagram attached.

Thanks

Georg Pauwen · ‎12-06-2020

Hello,

post the configuration of the Internet router as well, so we can lab this.

Tyche · ‎12-06-2020

Yes, sure.

I have attached both the INTERNET1 config and the NET_SRV config (this router plays the role CA and NTP server)

Georg Pauwen · ‎12-07-2020

Hello,

on r3, remove the 'tunnel source' from the virtual template:

interface Virtual-Template1 type tunnel
ip unnumbered Loopback1
ip nhrp network-id 1
ip nhrp redirect
mpls nhrp
--> no tunnel source GigabitEthernet0/2
tunnel protection ipsec profile default

Tyche · ‎12-07-2020

Hello,

I removed the tunnel source on Gi0/2

This did not solve the problem.

Did you manage to get it working on your side?

# Modification on R3


r3(config)#interface Virtual-Template1 type tunnel
r3(config-if)#no tunnel source GigabitEthernet0/2
r3(config-if)#end


r3#sh run int Virtual-Template1
Building configuration...

Current configuration : 164 bytes
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel protection ipsec profile default


# Verification 

r3#clear crypto ikev2 sa


r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 90 msec 52 msec 16 msec
  2 10.1.10.1 58 msec 65 msec 78 msec

r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 27 msec 51 msec 38 msec
  2 10.1.10.1 94 msec 124 msec 37 msec

r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:00:25, expire 00:02:39
   Type: incomplete, Flags: negative
   Cache hits: 2

Same issue on R3 during redirection

093386: Dec  7 05:28:31.873: NHRP: Receive Resolution Request via Virtual-Access2 vrf global(0x0), packet size: 79
093387: Dec  7 05:28:31.876:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
093388: Dec  7 05:28:31.876:      shtl: 4(NSAP), sstl: 0(NSAP)
093389: Dec  7 05:28:31.877:      pktsz: 79 extoff: 52
093390: Dec  7 05:28:31.878:  (M) flags: "router auth src-stable nat ", reqid: 22
093391: Dec  7 05:28:31.879:      src NBMA: 198.51.100.1
093392: Dec  7 05:28:31.880:      src protocol: 10.1.30.1, dst protocol: 10.1.20.1
093393: Dec  7 05:28:31.884:  (C-1) code: no error(0)
093394: Dec  7 05:28:31.885:        prefix: 32, mtu: 17874, hd_time: 600
093395: Dec  7 05:28:31.886:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
093396: Dec  7 05:28:31.887: NHRP-DETAIL: netid_in = 1, to_us = 0
093397: Dec  7 05:28:31.888: NHRP: Could not find AVL node for vrf:BLUE(0x1)

Georg Pauwen · ‎12-07-2020

Hello,

actually, not really. I copied the exact configs you have, but r3 says that NHRP is not enabled. I get no NHRP debug output at all on r3.

Tyche · ‎12-07-2020

Thanks,

What IOS version are you using?

Did you use GNS3 as well?

MHM Cisco World · ‎12-06-2020

...

Tyche · ‎12-06-2020

Hello

Thanks for the configurations but you are using DMVPN.

In my topology I want to use MPLS over FLEX VPN.

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_ike2vpn/configuration/15-mt/sec-flex-vpn-15-mt-book/sec-cfg-mpls-flex.html

MHM Cisco World · ‎12-06-2020

I check config

only hub have virtual interface

spoke must have tunnel interface

and again check tunnel and virtual source must be in global.

Tyche · ‎12-06-2020

Flex VPN is the new way of building VPNs. It support all the use cases of other VPN technologies (including DMVPN) and more.

In this scenario I am using MPLS on top of FLEX VPN to build a multi tenant topology.

(Please see my original post for Cisco pdf).

A spoke uses Tunnel 0 to create a secured control plane connection with the hub, then receives routes for various VRFs via MBGP ... (You can have sources in many VRFs, in this case I have physical links in the BLUE and RED VRFs).

Later on in the process (at the data plane) a NHRP redirect is sent by the hub and the spoke creates (using its virtual template) a direct encrypted point-to-point link to the other spoke.

In my lab, the redirect I sent by the hub but the resolution fails ...

Georg Pauwen · ‎12-06-2020

Post the config of the Internet router as well. I want to lab this up.

Tyche · ‎12-06-2020

I have just sent.

Thank you !