EVPN Configuration Centralized VRF Route-Leaking - Shared Internet with Custom VRF

[f00z]

This documentation seems wrong, and doesn't work. Am I missing something or does the documentation need to be fixed.

Talking about :

Configuration Centralized VRF Route-Leaking - Shared Internet with Custom VRF

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/vxlan/configuration/guide/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x/b-cisco-nexus-9000-series-nx-os-vxlan-configuration-guide-93x_chapter_011111.html

This has ip route 0.0.0.0/0 Null0 in the tenant vrf and advertises it to the tenant vrf.

Obviously this causes the default route to send it to the tenant vrf on the border and gets dropped.  The imported 0.0.0.0/0 route from the shared internet vrf won't override the 0.0.0.0/0 static route, so the documentation doesn't make sense to me.

It seems to be missing a command to re-advertise previously imported routes, so it would import the default route from internet vrf and then use the 'advertise-as-vpn' command like in IOS-XR to re-advertise the imported route (with route map of course). 

This command(advertise-as-vpn) doesn't seem to exist on nxos, and the way this documentation has the config, it doesn't work. 

Either I'm missing a single command somewhere in my test lab , or the documentation is broken.

Has anyone else tried this?

[f00z]

Seriously no replies? I was hoping for at least a 'yeah the documentation looks wrong and doesn't work' 

[Christopher Hart]

Hello!

I believe this particular section of the document is incorrect - I will work internally to verify this and get it corrected.

The majority of the configuration example is correct - however, the ip route 0.0.0.0/0 Null0 static route under each tenant VRF (Red, Blue, etc.) should instead be ip route 0.0.0.0/0 10.9.9.1 vrf Shared. This will put a default route in each tenant VRF that has a next-hop of 10.9.9.1 in the Shared VRF (which would be the external next-hop of your Internet-facing gateway).

As mentioned, I will check internally to verify whether this is correct and have the document corrected. Note that since the holiday season is around the corner, it might take some time for an updated copy to be published.

Thank you!

-Christopher

[f00z]

I tried that as well, and it sort-of worked. I need it to be dynamic though as in bgp default route is injected from '10.9.9.1' and imported into red/blue and then red/blue redistribute it (i.e. advertise-vpn/allow-vpn; i know these commands exist in some vrf context but I couldn't get it to work right with EVPN, although it may have been a bug, I will reinvestigate).   

The static route worked but if 10.9.9.1 disappears and reappears the route doesn't come back properly, plus it's not a very elegant solution. 

Thanks for looking into it at least! I hope to see a better solution proposed, and hopefully more features added to nxos, like being able to force re-advertisement of imported routes to evpn if necessary and also centralized gateway on evpn.

[Authur_Hong]

Hello!

I'm also confused about that documenttion

The section "Some pointers follow:" tells the right way to deal with this occasion

1）The default-route is made exported from the Shared Internet VRF and re-advertisement within VRF Blue and VRF Red on the Border Node.

leaking default route to VRF Blue and VRF Red can attract traffic towards Border Node with Shared Internet VRF,and go to the external router.

And I think the ip route 0.0.0.0/0 Null0 under the VRF Blue and VRF Red is needless,cus you are already have the default route leaking from the shared internet VRF.

2)Ensure the default-route in VRF Blue and VRF Red is not leaked to the Shared Internet VRF

that means the import map RM_DENY_IMPORT should under the VRF shared.

3)

aggregate-address 10.10.0.0/16
aggregate-address 10.20.0.0/16

just creat the aggreat for VRF Blue and VRF Red

So,in this way,you have the route towards the external router on the control plane,vice versa

VRF Blue host--->default-router leaking from VRF shared--->VRF Blue border--->VRF shared---> ip route 0.0.0.0/0 10.9.9.1--->external router

external router --->static route about the VRF Blue and Red--->VRF shared---aggregated address（10.10.0.0/16 and 10.20.0.0/16）--->VRF Blue or VRF Red host

[TCed]

Did anyone get this to work?

I've tried this but I have a hard time getting everything to work.

I've pretty much followed the instructions. I've set up the SHARED VRF and two CUSTOM VRFs
The SHARED VRF only have a static route for external connectivity (no dynamic routing right now)
External GW = 192.168.10.1
Border Leaf = 192.168.10.6

The CUSTOM VRFs are also set up on non-border leafs.

I've tried both with and without "ip route 0.0.0.0/0 Null0" in the CUSTOM VRFs.

Result:

1. With "ip route 0.0.0.0/0 Null0" in the CUSTOM VRFs (as per the instructions):
   
   The default route set in the CUSTOM VRFs overides the leaked default route from the SHARED VRF.
   
   Border leaf:
   
   0.0.0.0/0, ubest/mbest: 1/0
   *via Null0, [1/0], 00:00:23, static
   ...(vrf-local route)
   ...(vrf-local route)
   192.168.10.0/29, ubest/mbest: 1/0, attached
   *via 192.168.10.6%SHARED_VRF, Vlan20, [20/0], 2d16h, bgp-65201, external, tag 65201
   
   On a non-border leaf I get a default route pointing to the border-leaf:
   
   Non Border-leaf:
   
   0.0.0.0/0, ubest/mbest: 1/0
   *via 10.10.10.11%default, [200/0], 00:02:18, bgp-65201, internal, tag 65201, segid: 10555 tunnelid: 0xa159670 encap: VXLAN
   ...(vrf-local route)
   ...(vrf-local route)
   
   (10.10.10.11 = TEP-adress on border leaf)
   
   This results in no connectivity to the outside.
2. With "ip route 0.0.0.0/0 Null0" removed from the CUSTOM VRFs:
   
   Now things looks a bit better.
   
   Border-leaf:
   
   0.0.0.0/0, ubest/mbest: 1/0
   *via 192.168.10.1%SHARED_VRF, [20/0], 00:00:07, bgp-65201, external, tag 65201
   ...(vrf-local route)
   ...(vrf-local route)
   192.168.10.0/29, ubest/mbest: 1/0, attached
   *via 192.168.10.6%SHARED_VRF, Vlan20, [20/0], 2d16h, bgp-65201, external, tag 65201
   
   Non Border-leaf:
   
   (No default route, only vrf-local routes)
   
   
   Now I have external connectivity from endpoints in the CUSTOM VRFs, but only if the endpoints are located on the border leaf. If endpoint are located on other leafs, they get no external connectivity.
   
   What am I missing... Why isn't the default route leaked from the SHARED VRF distrubuted to other leafs?
   What do I have to do to get this to work?

[Ifeanyi]

The document is certainly not correct. The default route from the vrf share is not propagated to other tenants connected to other vteps. Only the hosts connected to the border switch have access to the internet.

[Ifeanyi]

This document should be corrected and reposted.

[mpetcu]

Please check this document, maybe it will help!

https://www.cisco.com/c/en/us/support/docs/switches/nexus-9000-series-switches/221709-configure-and-verify-vxlan-vrf-leaking-o.html

[cemosoccer1]

Hello,

I ran into the same issue recently. I was wanting to get the Default route into the tenant vrf from another vrf.

To achieve this I firstly created an extended community lists matching both the route target values of each vrf.

ip extcommunity-list standard abc seq 10 permit rt 65111:299900
ip extcommunity-list standard xyz seq 10 permit rt 65111:255500

I then created a prefix list which had a permit for 0.0.0.0/0.

ip prefix-list abc-to-xyz seq 10 permit 0.0.0.0/0

I then created a route-map, the first seq would match the prefix-list and extcommunity from which we are importing from. Then followed by a permit 20 to match its own RT. (without having its own RT value in the route map, no prefixes would get imported, even from its own tables)

route-map abc-to-xyz permit 10
  match ip address prefix-list abc-to-xyz
  match extcommunity abc
route-map abc-to-xyz permit 20
  match extcommunity xyz

Now under the vrf context xyz, I applied the route-target import matching the abc vrf, followed by the import map and the last peace that took me a while to figure out was the command I think you are after "import vrf advertise-vpn"

vrf context xyz
 address-family ipv4 unicast
   route-target import 65111:299900
   route-target import 65111:299900 evpn
   import map abc-to-xyz
   import vrf advertise-vpn

Hope this helps.

[f00z]

This is missing the point entirely.. it was never that the route couldn't be sent to the VRF , it was two 'bugs', plus a third issue.. 

1. The readvertise like XR has wasn't there so it couldn't be done this way (and now this command is in NXOS , not exactly the same but similar)

2. The VRF wouldn't allow it to work how we normally think VRFs work like in MPLS, and this now works which what cisco calls 'Downstream VNI'  which to me isn't a feature and should be how it worked in the first place, more of a bugfix they are calling a feature.

3.  Documentation says routing default to null will send default to VRF but all it sends to the other VRF is a default to NULL, lol.. so obviously this just blackholes all the traffic. The documentation is wrong, the default can't be to null, it has to be to the egress point but it wouldn't work correctly due to the #1 and #2 -- either way the doc is very wrong 

When I initially tested this, neither one of those 'bugfixes' was there.  This basically made it so we used a different design entirely and also used arista and juniper equipment because those worked properly at the time and also have centralized gateway (instead of anycast) functionality which I think cisco still doesn't have in nx-os but I haven't checked newer versions of the nxos code.

Nx-os at least of 9.3.11 or so seems to have fixed most of the issues except the centralized gw, and more issues are fixed in the 10.2 release I believe, but I haven't really tested it just perused release notes.