topic Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf in Routing and SD-WAN

MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Sat, 05 Dec 2020 22:09:56 GMT

# The problem:

I have configured MPLS over FlexVPN following the configuration snipet of Cisco live (page 39)

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKSEC-3036.pdf

The spoke-to-spoke traffic shortcut is not working so all the traffic goes via the hub.

When debugging NHRP I see the error: Could not find AVL node for vrf

Does anyone know what this error mean and how to fix it?

I am running: Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.7(3)M3, RELEASE SOFTWARE (fc2)

# Diagram

R1 (spoke LAN = 10.1.10.1/24 - vrf BLUE)======== R3(hub)============R2(spoke LAN = 10.1.20.1/24 - vrf BLUE)

# Troubleshooting steps

R2 sends ICMP traffic to R1 LAN. R2 receives a redirect from the hub and sends back a Resolution Request.

r2#ping vrf BLUE 10.1.10.1 source gi0/1 repeat 3


r2#sl
Log Buffer (8192 bytes):

 NHRP: Receive Traffic Indication via Tunnel0 vrf global(0x0), packet size: 84
  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
      shtl: 4(NSAP), sstl: 0(NSAP)
      pktsz: 84 extoff: 68
  (M) traffic code: redirect(0)
      src NBMA: 198.51.100.7
      src protocol: 10.1.30.0, dst protocol: 10.1.20.1
      Contents of nhrp traffic indication packet:
         45 00 00 64 00 67 00 00 FE 01 8A 2E 0A 01 14 01
         0A 01 0A 01 08 00 64 11 00 19 00
 NHRP-DETAIL: netid_in = 1, to_us = 0
 NHRP-DETAIL: Multipath IP route lookup for 10.1.20.1 in vrf BLUE(0x1) yielded GigabitEthernet0/1, pfx:10.1.20.0/24 (netid_in:1 if_in:Tunnel0)
 NHRP: nhrp_rtlookup yielded GigabitEthernet0/1
 NHRP-DETAIL: netid_out 0, netid_in 1
 NHRP: Parsing NHRP Traffic Indication

 NHRP: Enqueued NHRP Resolution Request for destination: 10.1.10.1
 NHRP: Checking for delayed event NULL/10.1.10.1 on list (Tunnel0 vrf: BLUE(0x1))
 NHRP: No delayed event node found.

R3 (hub) receive the resolution request but is unable to respond.

The error seen is : 'NHRP: Could not find AVL node for vrf:BLUE(0x1)'

NHRP: Receive Resolution Request via Virtual-Access1 vrf global(0x0), packet size: 79
 (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
     shtl: 4(NSAP), sstl: 0(NSAP)
     pktsz: 79 extoff: 52
 (M) flags: "router auth src-stable nat ", reqid: 20
     src NBMA: 198.51.100.3
     src protocol: 10.1.30.2, dst protocol: 10.1.10.1
 (C-1) code: no error(0)
       prefix: 32, mtu: 17874, hd_time: 600
       addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
NHRP-DETAIL: netid_in = 1, to_us = 0
NHRP: Could not find AVL node for vrf:BLUE(0x1)
NHRP-DETAIL: Multipath IP route lookup for 10.1.10.1 in vrf BLUE(0x1) yielded Null0, pfx:10.1.10.0/24 (netid_in:1 if_in:Virtual-Access1)
NHRP: Route lookup for destination 10.1.10.1 in vrf BLUE(0x1) yielded interface Null0, prefixlen 24
NHRP: Could not find AVL node for vrf:BLUE(0x1)

Yet the hub does have a route for the prefix 10.1.10.0/24

r3#sh ip route vrf BLUE | b ^G
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S        10.1.0.0/16 is directly connected, Null0
B        10.1.10.0/24 [200/0] via 10.1.30.1, 00:37:56
B        10.1.20.0/24 [200/0] via 10.1.30.2, 00:37:10
C        10.1.30.30/32 is directly connected, Loopback10

r3#sh ip cef vrf BLUE 10.1.10.1
10.1.10.0/24
  nexthop 10.1.30.1 Virtual-Access2 label 16-(local:18)

r2 never gets a reply so the shortcut does not work

r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:00:04, expire 00:03:00
   Type: incomplete, Flags: negative
   Cache hits: 2


r2#traceroute vrf BLUE 10.1.10.1 source gi0/1
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 61 msec 57 msec 31 msec
  2 10.1.10.1 87 msec 102 msec 124 msec

# Configuration Snipet on Hub

vrf definition BLUE
 rd 1:1
 !
 address-family ipv4
  route-target export 1:1
  route-target import 1:1
 exit-address-family
!
vrf definition RED
 rd 1:2
 !
 address-family ipv4
  route-target export 1:2
  route-target import 1:2
 exit-address-family
!
interface Loopback1
 ip address 10.1.30.0 255.255.255.255
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel source GigabitEthernet0/2
 tunnel protection ipsec profile default
!
router bgp 1
 bgp log-neighbor-changes
 bgp listen range 10.1.30.0/24 peer-group Flex
 neighbor Flex peer-group
 neighbor Flex remote-as 1
 neighbor Flex update-source Loopback1
 neighbor Flex timers 5 15
 !
 address-family vpnv4
  neighbor Flex activate
  neighbor Flex send-community extended
 exit-address-family
 !
 address-family ipv4 vrf BLUE
  network 10.1.0.0 mask 255.255.0.0
  network 10.1.30.30 mask 255.255.255.255
 exit-address-family
 !
 address-family ipv4 vrf RED
  network 10.1.0.0 mask 255.255.0.0
 exit-address-family

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Georg Pauwen — Sun, 06 Dec 2020 07:40:57 GMT

Hello,

post the full configs of the hub and the two spokes (sh run). IOSv means you are doing this in VIRL or GNS3 ?

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

MHM Cisco World — Sun, 06 Dec 2020 20:15:01 GMT

...

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Sun, 06 Dec 2020 19:31:13 GMT

Hello,

I am using GNS3.

Please see full config and diagram attached.

Thanks

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Sun, 06 Dec 2020 19:42:55 GMT

Hello

Thanks for the configurations but you are using DMVPN.

In my topology I want to use MPLS over FLEX VPN.

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_ike2vpn/configuration/15-mt/sec-flex-vpn-15-mt-book/sec-cfg-mpls-flex.html

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

MHM Cisco World — Sun, 06 Dec 2020 20:18:44 GMT

I check config

only hub have virtual interface

spoke must have tunnel interface

and again check tunnel and virtual source must be in global.

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Georg Pauwen — Sun, 06 Dec 2020 20:51:44 GMT

Hello,

post the configuration of the Internet router as well, so we can lab this.

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Sun, 06 Dec 2020 21:00:11 GMT

Flex VPN is the new way of building VPNs. It support all the use cases of other VPN technologies (including DMVPN) and more.

In this scenario I am using MPLS on top of FLEX VPN to build a multi tenant topology.

(Please see my original post for Cisco pdf).

A spoke uses Tunnel 0 to create a secured control plane connection with the hub, then receives routes for various VRFs via MBGP ... (You can have sources in many VRFs, in this case I have physical links in the BLUE and RED VRFs).

Later on in the process (at the data plane) a NHRP redirect is sent by the hub and the spoke creates (using its virtual template) a direct encrypted point-to-point link to the other spoke.

In my lab, the redirect I sent by the hub but the resolution fails ...

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Georg Pauwen — Sun, 06 Dec 2020 21:13:53 GMT

Post the config of the Internet router as well. I want to lab this up.

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Sun, 06 Dec 2020 21:14:45 GMT

Yes, sure.

I have attached both the INTERNET1 config and the NET_SRV config (this router plays the role CA and NTP server)

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Sun, 06 Dec 2020 21:16:00 GMT

I have just sent.

Thank you !

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

MHM Cisco World — Tue, 08 Dec 2020 23:01:12 GMT

....

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Georg Pauwen — Sun, 06 Dec 2020 22:59:52 GMT

Hello,

thanks for the files, I got it running in GNS3. I'll investigate and get back with you. I am in the GMT +1 timezone, so bear with me...

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Georg Pauwen — Mon, 07 Dec 2020 08:42:42 GMT

Hello,

on r3, remove the 'tunnel source' from the virtual template:

interface Virtual-Template1 type tunnel
ip unnumbered Loopback1
ip nhrp network-id 1
ip nhrp redirect
mpls nhrp
--> no tunnel source GigabitEthernet0/2
tunnel protection ipsec profile default

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Mon, 07 Dec 2020 09:30:58 GMT

Hello,

I removed the tunnel source on Gi0/2

This did not solve the problem.

Did you manage to get it working on your side?

# Modification on R3


r3(config)#interface Virtual-Template1 type tunnel
r3(config-if)#no tunnel source GigabitEthernet0/2
r3(config-if)#end


r3#sh run int Virtual-Template1
Building configuration...

Current configuration : 164 bytes
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel protection ipsec profile default


# Verification 

r3#clear crypto ikev2 sa


r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 90 msec 52 msec 16 msec
  2 10.1.10.1 58 msec 65 msec 78 msec

r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 27 msec 51 msec 38 msec
  2 10.1.10.1 94 msec 124 msec 37 msec

r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:00:25, expire 00:02:39
   Type: incomplete, Flags: negative
   Cache hits: 2

Same issue on R3 during redirection

093386: Dec  7 05:28:31.873: NHRP: Receive Resolution Request via Virtual-Access2 vrf global(0x0), packet size: 79
093387: Dec  7 05:28:31.876:  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
093388: Dec  7 05:28:31.876:      shtl: 4(NSAP), sstl: 0(NSAP)
093389: Dec  7 05:28:31.877:      pktsz: 79 extoff: 52
093390: Dec  7 05:28:31.878:  (M) flags: "router auth src-stable nat ", reqid: 22
093391: Dec  7 05:28:31.879:      src NBMA: 198.51.100.1
093392: Dec  7 05:28:31.880:      src protocol: 10.1.30.1, dst protocol: 10.1.20.1
093393: Dec  7 05:28:31.884:  (C-1) code: no error(0)
093394: Dec  7 05:28:31.885:        prefix: 32, mtu: 17874, hd_time: 600
093395: Dec  7 05:28:31.886:        addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
093396: Dec  7 05:28:31.887: NHRP-DETAIL: netid_in = 1, to_us = 0
093397: Dec  7 05:28:31.888: NHRP: Could not find AVL node for vrf:BLUE(0x1)

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Georg Pauwen — Mon, 07 Dec 2020 10:09:06 GMT

Hello,

actually, not really. I copied the exact configs you have, but r3 says that NHRP is not enabled. I get no NHRP debug output at all on r3.

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Mon, 07 Dec 2020 10:27:51 GMT

Hello,

Thank you for your observations. I have modified.

r1#sh run all | sec profile default
crypto ikev2 profile default
 description
 match identity remote fqdn domain lab.net
 identity local fqdn r1.lab.net
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint Trusted-CA
 lifetime 86400
 lifetime certificate
 aaa authentication eap
 aaa authentication anyconnect-eap
 aaa authorization group cert list default default local
 virtual-template 1
 config-exchange set send
 config-exchange set accept
 config-exchange request
 no shutdown



r3#sh run int Virtual-Template1
Building configuration...

Current configuration : 164 bytes
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel protection ipsec profile default

Unfortunately the problem remains:

r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 75 msec 55 msec 39 msec
  2 10.1.10.1 88 msec 55 msec 43 msec
  
r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 85 msec 44 msec 37 msec
  2 10.1.10.1 55 msec 45 msec 43 msec
  
r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:01:24, expire 00:01:40
   Type: incomplete, Flags: negative
   Cache hits: 2

I looked a bit further at the error message. R3 is saying it does not have route to 10.1.20.1 in vrf BLUE

110421: Dec  7 06:08:14.515: NHRP: Receive Resolution Request via Virtual-Access2 vrf global(0x0), packet size: 79
110422: Dec  7 06:08:14.517: NHRP-DETAIL: netid_in = 1, to_us = 0
110423: Dec  7 06:08:14.518: NHRP: Could not find AVL node for vrf:BLUE(0x1)
110424: Dec  7 06:08:14.520: NHRP-DETAIL: Multipath IP route lookup for 10.1.20.1 in vrf BLUE(0x1) yielded Null0, pfx:10.1.20.0/24 (netid_in:1 if_in:Virtual-Access2)
110425: Dec  7 06:08:14.521: NHRP: Route lookup for destination 10.1.20.1 in vrf BLUE(0x1) yielded interface Null0, prefixlen 24
110426: Dec  7 06:08:14.522: NHRP: Could not find AVL node for vrf:BLUE(0x1)
110427: Dec  7 06:08:14.523: NHRP-DETAIL: First hop route lookup for 10.1.20.1 yielded 10.1.30.2, Virtual-Access1
110428: Dec  7 06:08:14.524: NHRP: Route lookup for 10.1.20.1 in BLUE(0x1) yielded nexthop 10.1.30.2 interface Virtual-Access1
110429: Dec  7 06:08:14.525: NHRP: Could not find AVL node for vrf:BLUE(0x1)
110430: Dec  7 06:08:14.526: NHRP: Cache lookup for nexthop 10.1.30.2 on Virtual-Access1 returned nbma Null

From the perspective of the RIB, this is incorrect, the route exists:

r3#sh ip route vrf BLUE 10.1.20.1

Routing Table: BLUE
Routing entry for 10.1.20.0/24
  Known via "bgp 1", distance 200, metric 0, type internal
  Last update from 10.1.30.2 00:21:05 ago
  Routing Descriptor Blocks:
  * 10.1.30.2 (default), from 10.1.30.2, 00:21:05 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0
      MPLS label: 16
      MPLS Flags: MPLS Required

However from the perspective of the LFIB the route 10.1.20.0/24 does not exist:

r3#sh mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         No Label   10.1.0.0/16[V]   3556          aggregate/BLUE
17         No Label   10.1.0.0/16[V]   0             aggregate/RED
18         16         10.1.10.0/24[V]  0             Vi2        point2point
19         17         10.1.11.0/24[V]  0             Vi2        point2point
20         Pop Label  10.1.30.30/32[V] 14468         aggregate/BLUE
r3#

I suspect NHRP is looking in the LFIB thus the error. If we can get r3 to install the 10.1.20.0/24 in the LFIB we might be able to resolve the issue.

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Mon, 07 Dec 2020 10:30:32 GMT

Regarding R3 loopback 1, what issue do you see with using 10.1.30.0/32?

This is a host address, there is no notion of subnet ID or broadcast ...

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Mon, 07 Dec 2020 11:09:25 GMT

Just in case I changed loopback1 on r3 to 10.1.30.100/32

r3#sh bgp vpnv4 un all summary
BGP router identifier 10.1.30.100, local AS number 1
BGP table version is 12, main routing table version 12
7 network entries using 1092 bytes of memory
7 path entries using 588 bytes of memory
4/4 BGP path/bestpath attribute entries using 672 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2400 total bytes of memory
BGP activity 21/14 prefixes, 21/14 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
*10.1.30.1      4            1      57      58       12    0    0 00:04:19        2
*10.1.30.2      4            1      58      57       12    0    0 00:04:22        2
* Dynamically created based on a listen range command
Dynamically created neighbors: 2, Subnet ranges: 1

r1#sh bgp vpnv4 un all summary
BGP router identifier 10.1.30.1, local AS number 1
BGP table version is 18, main routing table version 18
5 network entries using 780 bytes of memory
5 path entries using 420 bytes of memory
4/4 BGP path/bestpath attribute entries using 672 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1920 total bytes of memory
BGP activity 8/3 prefixes, 8/3 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.30.100 4 1 74 73 18 0 0 00:05:38 3


r2#sh bgp vpnv4 un all summary
BGP router identifier 10.1.30.2, local AS number 1
BGP table version is 18, main routing table version 18
5 network entries using 780 bytes of memory
5 path entries using 420 bytes of memory
4/4 BGP path/bestpath attribute entries using 672 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1920 total bytes of memory
BGP activity 8/3 prefixes, 8/3 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.30.100 4 1 76 77 18 0 0 00:06:00 3

I then rebooted r3. I am not installing any prefix learned by MBGP the LFIB.

Before I was getting at least entries for r1. This looks like a bug ...

r3#sh ip route vrf BLUE | b ^G
Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S 10.1.0.0/16 is directly connected, Null0
B 10.1.10.0/24 [200/0] via 10.1.30.1, 00:09:38
B 10.1.20.0/24 [200/0] via 10.1.30.2, 00:09:41
C 10.1.30.30/32 is directly connected, Loopback10


r3#sh mpls forwarding-table vrf BLUE
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
19         No Label   10.1.0.0/16[V]   0             aggregate/BLUE
20         Pop Label  10.1.30.30/32[V] 0             aggregate/BLUE

Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf

Tyche — Mon, 07 Dec 2020 11:12:59 GMT

Thanks,

What IOS version are you using?

Did you use GNS3 as well?