EVPN Configuration Centralized VRF Route-Leaking - Shared Internet with Custom VRF

[f00z]

This documentation seems wrong, and doesn't work. Am I missing something or does the documentation need to be fixed.

Talking about :

Configuration Centralized VRF Route-Leaking - Shared Internet with Custom VRF

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/vxlan/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x_chapter_011111.html

This has ip route 0.0.0.0/0 Null0 in the tenant vrf and advertises it to the tenant vrf.

Obviously this causes the default route to send it to the tenant vrf on the border and gets dropped.  The imported 0.0.0.0/0 route from the shared internet vrf won't override the 0.0.0.0/0 static route, so the documentation doesn't make sense to me.

It seems to be missing a command to re-advertise previously imported routes, so it would import the default route from internet vrf and then use the 'advertise-as-vpn' command like in IOS-XR to re-advertise the imported route (with route map of course). 

This command(advertise-as-vpn) doesn't seem to exist on nxos, and the way this documentation has the config, it doesn't work. 

Either I'm missing a single command somewhere in my test lab , or the documentation is broken.

Has anyone else tried this?

[f00z]

Seriously no replies? I was hoping for at least a 'yeah the documentation looks wrong and doesn't work' 

[Christopher Hart]

Hello!

I believe this particular section of the document is incorrect - I will work internally to verify this and get it corrected.

The majority of the configuration example is correct - however, the ip route 0.0.0.0/0 Null0 static route under each tenant VRF (Red, Blue, etc.) should instead be ip route 0.0.0.0/0 10.9.9.1 vrf Shared. This will put a default route in each tenant VRF that has a next-hop of 10.9.9.1 in the Shared VRF (which would be the external next-hop of your Internet-facing gateway).

As mentioned, I will check internally to verify whether this is correct and have the document corrected. Note that since the holiday season is around the corner, it might take some time for an updated copy to be published.

Thank you!

-Christopher

[f00z]

I tried that as well, and it sort-of worked. I need it to be dynamic though as in bgp default route is injected from '10.9.9.1' and imported into red/blue and then red/blue redistribute it (i.e. advertise-vpn/allow-vpn; i know these commands exist in some vrf context but I couldn't get it to work right with EVPN, although it may have been a bug, I will reinvestigate).   

The static route worked but if 10.9.9.1 disappears and reappears the route doesn't come back properly, plus it's not a very elegant solution. 

Thanks for looking into it at least! I hope to see a better solution proposed, and hopefully more features added to nxos, like being able to force re-advertisement of imported routes to evpn if necessary and also centralized gateway on evpn.

[Authur_Hong]

Hello!

I'm also confused about that documenttion

The section "Some pointers follow:" tells the right way to deal with this occasion

1）The default-route is made exported from the Shared Internet VRF and re-advertisement within VRF Blue and VRF Red on the Border Node.

leaking default route to VRF Blue and VRF Red can attract traffic towards Border Node with Shared Internet VRF,and go to the external router.

And I think the ip route 0.0.0.0/0 Null0 under the VRF Blue and VRF Red is needless,cus you are already have the default route leaking from the shared internet VRF.

2)Ensure the default-route in VRF Blue and VRF Red is not leaked to the Shared Internet VRF

that means the import map RM_DENY_IMPORT should under the VRF shared.

3)

aggregate-address 10.10.0.0/16
aggregate-address 10.20.0.0/16

just creat the aggreat for VRF Blue and VRF Red

So,in this way,you have the route towards the external router on the control plane,vice versa

VRF Blue host--->default-router leaking from VRF shared--->VRF Blue border--->VRF shared---> ip route 0.0.0.0/0 10.9.9.1--->external router

external router --->static route about the VRF Blue and Red--->VRF shared---aggregated address（10.10.0.0/16 and 10.20.0.0/16）--->VRF Blue or VRF Red host

[TCed]

Did anyone get this to work?

I've tried this but I have a hard time getting everything to work.

I've pretty much followed the instructions. I've set up the SHARED VRF and two CUSTOM VRFs
The SHARED VRF only have a static route for external connectivity (no dynamic routing right now)
External GW = 192.168.10.1
Border Leaf = 192.168.10.6

The CUSTOM VRFs are also set up on non-border leafs.

I've tried both with and without "ip route 0.0.0.0/0 Null0" in the CUSTOM VRFs.

Result:

1. With "ip route 0.0.0.0/0 Null0" in the CUSTOM VRFs (as per the instructions):
   
   The default route set in the CUSTOM VRFs overides the leaked default route from the SHARED VRF.
   
   Border leaf:
   
   0.0.0.0/0, ubest/mbest: 1/0
   *via Null0, [1/0], 00:00:23, static
   ...(vrf-local route)
   ...(vrf-local route)
   192.168.10.0/29, ubest/mbest: 1/0, attached
   *via 192.168.10.6%SHARED_VRF, Vlan20, [20/0], 2d16h, bgp-65201, external, tag 65201
   
   On a non-border leaf I get a default route pointing to the border-leaf:
   
   Non Border-leaf:
   
   0.0.0.0/0, ubest/mbest: 1/0
   *via 10.10.10.11%default, [200/0], 00:02:18, bgp-65201, internal, tag 65201, segid: 10555 tunnelid: 0xa159670 encap: VXLAN
   ...(vrf-local route)
   ...(vrf-local route)
   
   (10.10.10.11 = TEP-adress on border leaf)
   
   This results in no connectivity to the outside.
2. With "ip route 0.0.0.0/0 Null0" removed from the CUSTOM VRFs:
   
   Now things looks a bit better.
   
   Border-leaf:
   
   0.0.0.0/0, ubest/mbest: 1/0
   *via 192.168.10.1%SHARED_VRF, [20/0], 00:00:07, bgp-65201, external, tag 65201
   ...(vrf-local route)
   ...(vrf-local route)
   192.168.10.0/29, ubest/mbest: 1/0, attached
   *via 192.168.10.6%SHARED_VRF, Vlan20, [20/0], 2d16h, bgp-65201, external, tag 65201
   
   Non Border-leaf:
   
   (No default route, only vrf-local routes)
   
   
   Now I have external connectivity from endpoints in the CUSTOM VRFs, but only if the endpoints are located on the border leaf. If endpoint are located on other leafs, they get no external connectivity.
   
   What am I missing... Why isn't the default route leaked from the SHARED VRF distrubuted to other leafs?
   What do I have to do to get this to work?

[Ifeanyi]

The document is certainly not correct. The default route from the vrf share is not propagated to other tenants connected to other vteps. Only the hosts connected to the border switch have access to the internet.

[Ifeanyi]

This document should be corrected and reposted.

[mpetcu]

Please check this document, maybe it will help!

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/221709-configure-and-verify-vxlan-vrf-leaking-o.html