Re: MPLS over FLEX VPN shortcut does not work - NHRP error: Could not find AVL node for vrf - Page 2

Tyche · ‎12-05-2020

# The problem:

I have configured MPLS over FlexVPN following the configuration snipet of Cisco live (page 39)

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2016/pdf/BRKSEC-3036.pdf

The spoke-to-spoke traffic shortcut is not working so all the traffic goes via the hub.

When debugging NHRP I see the error: Could not find AVL node for vrf

Does anyone know what this error mean and how to fix it?

I am running: Cisco IOS Software, IOSv Software (VIOS-ADVENTERPRISEK9-M), Version 15.7(3)M3, RELEASE SOFTWARE (fc2)

# Diagram

R1 (spoke LAN = 10.1.10.1/24 - vrf BLUE)======== R3(hub)============R2(spoke LAN = 10.1.20.1/24 - vrf BLUE)

# Troubleshooting steps

R2 sends ICMP traffic to R1 LAN. R2 receives a redirect from the hub and sends back a Resolution Request.

r2#ping vrf BLUE 10.1.10.1 source gi0/1 repeat 3


r2#sl
Log Buffer (8192 bytes):

 NHRP: Receive Traffic Indication via Tunnel0 vrf global(0x0), packet size: 84
  (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
      shtl: 4(NSAP), sstl: 0(NSAP)
      pktsz: 84 extoff: 68
  (M) traffic code: redirect(0)
      src NBMA: 198.51.100.7
      src protocol: 10.1.30.0, dst protocol: 10.1.20.1
      Contents of nhrp traffic indication packet:
         45 00 00 64 00 67 00 00 FE 01 8A 2E 0A 01 14 01
         0A 01 0A 01 08 00 64 11 00 19 00
 NHRP-DETAIL: netid_in = 1, to_us = 0
 NHRP-DETAIL: Multipath IP route lookup for 10.1.20.1 in vrf BLUE(0x1) yielded GigabitEthernet0/1, pfx:10.1.20.0/24 (netid_in:1 if_in:Tunnel0)
 NHRP: nhrp_rtlookup yielded GigabitEthernet0/1
 NHRP-DETAIL: netid_out 0, netid_in 1
 NHRP: Parsing NHRP Traffic Indication

 NHRP: Enqueued NHRP Resolution Request for destination: 10.1.10.1
 NHRP: Checking for delayed event NULL/10.1.10.1 on list (Tunnel0 vrf: BLUE(0x1))
 NHRP: No delayed event node found.

R3 (hub) receive the resolution request but is unable to respond.

The error seen is : 'NHRP: Could not find AVL node for vrf:BLUE(0x1)'

NHRP: Receive Resolution Request via Virtual-Access1 vrf global(0x0), packet size: 79
 (F) afn: AF_IP(1), type: IP(800), hop: 255, ver: 1
     shtl: 4(NSAP), sstl: 0(NSAP)
     pktsz: 79 extoff: 52
 (M) flags: "router auth src-stable nat ", reqid: 20
     src NBMA: 198.51.100.3
     src protocol: 10.1.30.2, dst protocol: 10.1.10.1
 (C-1) code: no error(0)
       prefix: 32, mtu: 17874, hd_time: 600
       addr_len: 0(NSAP), subaddr_len: 0(NSAP), proto_len: 0, pref: 255
NHRP-DETAIL: netid_in = 1, to_us = 0
NHRP: Could not find AVL node for vrf:BLUE(0x1)
NHRP-DETAIL: Multipath IP route lookup for 10.1.10.1 in vrf BLUE(0x1) yielded Null0, pfx:10.1.10.0/24 (netid_in:1 if_in:Virtual-Access1)
NHRP: Route lookup for destination 10.1.10.1 in vrf BLUE(0x1) yielded interface Null0, prefixlen 24
NHRP: Could not find AVL node for vrf:BLUE(0x1)

Yet the hub does have a route for the prefix 10.1.10.0/24

r3#sh ip route vrf BLUE | b ^G
Gateway of last resort is not set

      10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S        10.1.0.0/16 is directly connected, Null0
B        10.1.10.0/24 [200/0] via 10.1.30.1, 00:37:56
B        10.1.20.0/24 [200/0] via 10.1.30.2, 00:37:10
C        10.1.30.30/32 is directly connected, Loopback10

r3#sh ip cef vrf BLUE 10.1.10.1
10.1.10.0/24
  nexthop 10.1.30.1 Virtual-Access2 label 16-(local:18)

r2 never gets a reply so the shortcut does not work

r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:00:04, expire 00:03:00
   Type: incomplete, Flags: negative
   Cache hits: 2


r2#traceroute vrf BLUE 10.1.10.1 source gi0/1
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 61 msec 57 msec 31 msec
  2 10.1.10.1 87 msec 102 msec 124 msec

# Configuration Snipet on Hub

vrf definition BLUE
 rd 1:1
 !
 address-family ipv4
  route-target export 1:1
  route-target import 1:1
 exit-address-family
!
vrf definition RED
 rd 1:2
 !
 address-family ipv4
  route-target export 1:2
  route-target import 1:2
 exit-address-family
!
interface Loopback1
 ip address 10.1.30.0 255.255.255.255
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel source GigabitEthernet0/2
 tunnel protection ipsec profile default
!
router bgp 1
 bgp log-neighbor-changes
 bgp listen range 10.1.30.0/24 peer-group Flex
 neighbor Flex peer-group
 neighbor Flex remote-as 1
 neighbor Flex update-source Loopback1
 neighbor Flex timers 5 15
 !
 address-family vpnv4
  neighbor Flex activate
  neighbor Flex send-community extended
 exit-address-family
 !
 address-family ipv4 vrf BLUE
  network 10.1.0.0 mask 255.255.0.0
  network 10.1.30.30 mask 255.255.255.255
 exit-address-family
 !
 address-family ipv4 vrf RED
  network 10.1.0.0 mask 255.255.0.0
 exit-address-family

Georg Pauwen · ‎12-06-2020

Hello,

thanks for the files, I got it running in GNS3. I'll investigate and get back with you. I am in the GMT +1 timezone, so bear with me...

MHM Cisco World · ‎12-06-2020

....

Tyche · ‎12-07-2020

Hello,

Thank you for your observations. I have modified.

r1#sh run all | sec profile default
crypto ikev2 profile default
 description
 match identity remote fqdn domain lab.net
 identity local fqdn r1.lab.net
 authentication remote rsa-sig
 authentication local rsa-sig
 pki trustpoint Trusted-CA
 lifetime 86400
 lifetime certificate
 aaa authentication eap
 aaa authentication anyconnect-eap
 aaa authorization group cert list default default local
 virtual-template 1
 config-exchange set send
 config-exchange set accept
 config-exchange request
 no shutdown



r3#sh run int Virtual-Template1
Building configuration...

Current configuration : 164 bytes
!
interface Virtual-Template1 type tunnel
 ip unnumbered Loopback1
 ip nhrp network-id 1
 ip nhrp redirect
 mpls nhrp
 tunnel protection ipsec profile default

Unfortunately the problem remains:

r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 75 msec 55 msec 39 msec
  2 10.1.10.1 88 msec 55 msec 43 msec
  
r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.1.30.30 85 msec 44 msec 37 msec
  2 10.1.10.1 55 msec 45 msec 43 msec
  
r2#sh ip nhrp
10.1.10.1/32 (BLUE)
   Tunnel0 created 00:01:24, expire 00:01:40
   Type: incomplete, Flags: negative
   Cache hits: 2

I looked a bit further at the error message. R3 is saying it does not have route to 10.1.20.1 in vrf BLUE

110421: Dec  7 06:08:14.515: NHRP: Receive Resolution Request via Virtual-Access2 vrf global(0x0), packet size: 79
110422: Dec  7 06:08:14.517: NHRP-DETAIL: netid_in = 1, to_us = 0
110423: Dec  7 06:08:14.518: NHRP: Could not find AVL node for vrf:BLUE(0x1)
110424: Dec  7 06:08:14.520: NHRP-DETAIL: Multipath IP route lookup for 10.1.20.1 in vrf BLUE(0x1) yielded Null0, pfx:10.1.20.0/24 (netid_in:1 if_in:Virtual-Access2)
110425: Dec  7 06:08:14.521: NHRP: Route lookup for destination 10.1.20.1 in vrf BLUE(0x1) yielded interface Null0, prefixlen 24
110426: Dec  7 06:08:14.522: NHRP: Could not find AVL node for vrf:BLUE(0x1)
110427: Dec  7 06:08:14.523: NHRP-DETAIL: First hop route lookup for 10.1.20.1 yielded 10.1.30.2, Virtual-Access1
110428: Dec  7 06:08:14.524: NHRP: Route lookup for 10.1.20.1 in BLUE(0x1) yielded nexthop 10.1.30.2 interface Virtual-Access1
110429: Dec  7 06:08:14.525: NHRP: Could not find AVL node for vrf:BLUE(0x1)
110430: Dec  7 06:08:14.526: NHRP: Cache lookup for nexthop 10.1.30.2 on Virtual-Access1 returned nbma Null

From the perspective of the RIB, this is incorrect, the route exists:

r3#sh ip route vrf BLUE 10.1.20.1

Routing Table: BLUE
Routing entry for 10.1.20.0/24
  Known via "bgp 1", distance 200, metric 0, type internal
  Last update from 10.1.30.2 00:21:05 ago
  Routing Descriptor Blocks:
  * 10.1.30.2 (default), from 10.1.30.2, 00:21:05 ago
      Route metric is 0, traffic share count is 1
      AS Hops 0
      MPLS label: 16
      MPLS Flags: MPLS Required

However from the perspective of the LFIB the route 10.1.20.0/24 does not exist:

r3#sh mpls forwarding-table
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
16         No Label   10.1.0.0/16[V]   3556          aggregate/BLUE
17         No Label   10.1.0.0/16[V]   0             aggregate/RED
18         16         10.1.10.0/24[V]  0             Vi2        point2point
19         17         10.1.11.0/24[V]  0             Vi2        point2point
20         Pop Label  10.1.30.30/32[V] 14468         aggregate/BLUE
r3#

I suspect NHRP is looking in the LFIB thus the error. If we can get r3 to install the 10.1.20.0/24 in the LFIB we might be able to resolve the issue.

Tyche · ‎12-07-2020

Regarding R3 loopback 1, what issue do you see with using 10.1.30.0/32?

This is a host address, there is no notion of subnet ID or broadcast ...

Tyche · ‎12-07-2020

Just in case I changed loopback1 on r3 to 10.1.30.100/32

r3#sh bgp vpnv4 un all summary
BGP router identifier 10.1.30.100, local AS number 1
BGP table version is 12, main routing table version 12
7 network entries using 1092 bytes of memory
7 path entries using 588 bytes of memory
4/4 BGP path/bestpath attribute entries using 672 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2400 total bytes of memory
BGP activity 21/14 prefixes, 21/14 paths, scan interval 60 secs

Neighbor        V           AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
*10.1.30.1      4            1      57      58       12    0    0 00:04:19        2
*10.1.30.2      4            1      58      57       12    0    0 00:04:22        2
* Dynamically created based on a listen range command
Dynamically created neighbors: 2, Subnet ranges: 1

r1#sh bgp vpnv4 un all summary
BGP router identifier 10.1.30.1, local AS number 1
BGP table version is 18, main routing table version 18
5 network entries using 780 bytes of memory
5 path entries using 420 bytes of memory
4/4 BGP path/bestpath attribute entries using 672 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1920 total bytes of memory
BGP activity 8/3 prefixes, 8/3 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.30.100 4 1 74 73 18 0 0 00:05:38 3


r2#sh bgp vpnv4 un all summary
BGP router identifier 10.1.30.2, local AS number 1
BGP table version is 18, main routing table version 18
5 network entries using 780 bytes of memory
5 path entries using 420 bytes of memory
4/4 BGP path/bestpath attribute entries using 672 bytes of memory
2 BGP extended community entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 1920 total bytes of memory
BGP activity 8/3 prefixes, 8/3 paths, scan interval 60 secs

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.1.30.100 4 1 76 77 18 0 0 00:06:00 3

I then rebooted r3. I am not installing any prefix learned by MBGP the LFIB.

Before I was getting at least entries for r1. This looks like a bug ...

r3#sh ip route vrf BLUE | b ^G
Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S 10.1.0.0/16 is directly connected, Null0
B 10.1.10.0/24 [200/0] via 10.1.30.1, 00:09:38
B 10.1.20.0/24 [200/0] via 10.1.30.2, 00:09:41
C 10.1.30.30/32 is directly connected, Loopback10


r3#sh mpls forwarding-table vrf BLUE
Local      Outgoing   Prefix           Bytes Label   Outgoing   Next Hop
Label      Label      or Tunnel Id     Switched      interface
19         No Label   10.1.0.0/16[V]   0             aggregate/BLUE
20         Pop Label  10.1.30.30/32[V] 0             aggregate/BLUE

MHM Cisco World · ‎12-07-2020

...

Tyche · ‎12-08-2020

Thank you MHM,

Modifying the neighborship to peer groups on the spokes did not make a difference.

I am in the process of reproducing the set-up on a totally different platerform (CSR1000V), in case this is a bug.

Either way I will post my result so everyone can benefit from the research.

Kind Regards

MHM Cisco World · ‎12-08-2020

can I see the show ip route for each one after modify?

Tyche · ‎12-10-2020

------------------------------------ MODIFICATIONS YOU REQUESTED ------------------

r1#sh run | sec er bgp
router bgp 1
bgp log-neighbor-changes
neighbor Flex peer-group
neighbor 10.1.30.100 remote-as 1
neighbor 10.1.30.100 peer-group Flex
!
address-family vpnv4
neighbor Flex send-community both
neighbor 10.1.30.100 activate
exit-address-family
!
address-family ipv4 vrf BLUE
redistribute connected
exit-address-family
!
address-family ipv4 vrf RED
redistribute connected
exit-address-family

r2#sh run | sec er bgp
router bgp 1
bgp log-neighbor-changes
neighbor Flex peer-group
neighbor 10.1.30.100 remote-as 1
neighbor 10.1.30.100 peer-group Flex
!
address-family vpnv4
neighbor Flex send-community both
neighbor 10.1.30.100 activate
exit-address-family
!
address-family ipv4 vrf BLUE
redistribute connected
exit-address-family
!
address-family ipv4 vrf RED
redistribute connected
exit-address-family

r3#sh run | sec er bgp
router bgp 1
bgp router-id interface Loopback1
bgp log-neighbor-changes
bgp listen range 10.1.30.0/24 peer-group Flex
neighbor Flex peer-group
neighbor Flex remote-as 1
neighbor Flex timers 5 15
!
address-family vpnv4
neighbor Flex activate
neighbor Flex send-community extended
exit-address-family
!
address-family ipv4 vrf BLUE
network 10.1.0.0 mask 255.255.0.0
exit-address-family
!
address-family ipv4 vrf RED
network 10.1.0.0 mask 255.255.0.0
exit-address-family

----------------------- ROUTING TABLE OK ------------------------------------

r3# clear ip bgp *

r3#sh ip route vrf BLUE

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
S 10.1.0.0/16 is directly connected, Null0
B 10.1.10.0/24 [200/0] via 10.1.30.12, 00:00:13
B 10.1.20.0/24 [200/0] via 10.1.30.2, 00:00:14
C 10.1.30.30/32 is directly connected, Loopback10

r3#sh ip route vrf RED

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
S 10.1.0.0/16 is directly connected, Null0
B 10.1.11.0/24 [200/0] via 10.1.30.12, 00:00:19
B 10.1.21.0/24 [200/0] via 10.1.30.2, 00:00:20

------------------------------- MPLS LFIB not populated correctly -----------------------

r3#sh mpls forwarding-table vrf BLUE
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
16 No Label 10.1.0.0/16[V] 0 aggregate/BLUE
r3#
r3#sh mpls forwarding-table vrf RED
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
17 No Label 10.1.0.0/16[V] 0 aggregate/RED

---------------- -----------------SHORTCUT NOT WORKING ----------------------- -----------------------

r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
1 10.1.30.30 53 msec 22 msec 23 msec
2 10.1.10.1 68 msec 99 msec 42 msec

r2#traceroute vrf BLUE 10.1.10.1 source gi0/1 numeric
Type escape sequence to abort.
Tracing the route to 10.1.10.1
VRF info: (vrf in name/id, vrf out name/id)
1 10.1.30.30 45 msec 21 msec 18 msec
2 10.1.10.1 56 msec 37 msec 41 msec

MHM Cisco World · ‎12-10-2020

10.1.30.100<- this is the BGP peer which must be the unnumbered loopback of virtual-template of hub.

Tyche · ‎12-10-2020

I believe I have set-up the hub correctly.

r3#sh run int lo 1

interface Loopback1
ip address 10.1.30.100 255.255.255.255

r3#sh run int virtual-template 1
Building configuration...

Current configuration : 185 bytes
!
interface Virtual-Template1 type tunnel
ip unnumbered Loopback1
ip nhrp network-id 1
ip nhrp redirect
mpls nhrp
tunnel protection ipsec profile default

MHM Cisco World · ‎12-10-2020

whenever you have time send to me we can do together.

Tyche · ‎12-10-2020

Thank you for your support. I will get back to you tomorrow and we can do a Webex if you like.

( I am in the process of reproducing the lab on a different platform)

Tyche · ‎12-15-2020

I implemented the lab on physical hardware: Cisco 2901 running IOS: c2900-universalk9-mz.SPA.157-3.M7.bin

I also tried on CSR1000V running IOS XE, Version 16.12.4a

In both cases the MBGP routes were received however the the labels were not installed in the MPLS forwarding table.

The problem does not seem related to the IOS or the platform (virtual or physical).

MHM Cisco World · ‎12-17-2020

Hi friend
I finish the Lab yesterday and it successfully work I will send to you detail later today.