ACI Inter VRF/Tenant Route Leaking Configuration Example

[gkumark]

Introduction

This document will run you through a step by step configuration example of Inter VRF route leaking in Cisco Application Centric Infrastructure(ACI). This article describes the steps with screen shots of example configuration and some verification on the command line interface where relevant.

This procedure is applicable for 2 different VRF's in same tenant or separate tenant.

Note: This configuration has been tested on version 2.3(1f). Screen shots below may vary slightly with later versions.

Prerequisites

In this document, we will be configuring and describing only about configurations relating to inter VRF leaking. Hence prior knowledge of ACI concepts and configurations like EPG, Bridge Domain, VRF and L3out is essential. 

Components Used

For the purpose of this document, below devices/components have been used.

1. ACI fabric with 3 leafs, 2 spines and 3 APICs running version 2.3(1f)
2. Nexus 7000, configured as a router running OSPF advertising routes to ACI fabric

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any configuration/command.

Network Diagram

Below topology will be used for the purpose of this configuration example. 

IP and other details:

Consumer VM : VM has the IP of 10.10.10.1/24 and has the gateway 10.10.10.254/24 in ACI deployed in VRF consumer-vrf.
Provider VM: VM has the IP of 10.10.20.1/24 and has the gateway 10.10.20.254/24 in ACI deployed in VRF provider-vrf.
L3out: L3out is deployed in ACI leaf 101 and 102 (vPC) connecting to Nexus 7K acting as router here. SVI interface vlan15 (15.15.15.3/24) is used for forming the OSPF adjacency and loopback 200 interface(200.200.200.200/32) will be used for testing the connectivity from Provider EPG to the l3out.

Configuration

Before configuring anything related to inter-VRF route leaking, let's verify below configurations.

1. VRF provider-vrf has been created with Bridge Domain(BD) provider-bd. The endpoint 10.10.20.1 has been learned in EPG provider-epg through leaf 201/1/1

2. VRF consumer-vrf has been created with Bridge Domain(BD) consumer-bd. The endpoint 10.10.10.1 has been learned in EPG consumer-epg through leaf 201/1/17

3. Both above endpoints are able to ping their respective gateways deployed in ACI.

4. The l3out consumer-l3out has been deployed and ACI has learned the routes from the external router.
5. As you see below, consumer-vrf on leaf 201 has the route 10.10.10.0/24 which is local and learned 200.200.200.200/32 through the overlay-1 vrf from leaf 101 and 102 where the l3out is deployed.

6. Also, as you see below the provider-vrf just has its local route 10.10.20.0/24.

Now, let us start with the Inter-VRF configurations

Inter-VRF route leaking for communication between VMs in 2 different tenants/vrf's

Step1: Configure shared subnet under the provider-epg as opposed to configuring under BD.

Step2: Configure shared subnet under consumer-bd

Step3: Create a global contract in tenant provider-tenant

Step4: Export above contract to the consumer-tenant

Make sure to export the right contract and select the right tenant to export the contract to.

NOTE: If you are configuring inter-VRF routing between two VRFs in the same Tenant, there is no need to export/import the contract.

Step5: Apply the above contract as provided contract under provider-epg

Step6: Apply above exported global contract as consumed contract interface under the consumer-vrf

Now the communication between the VM's in provider-epg and consumer-epg should work based on the contract filter. In my case, I am using default filter which is allow-all.

As you can see below, the provider subnet 10.10.20.0/24 has been advertised to consumer-vrf and the consumer subnet 10.10.10.0/24 has been advertised to provider-vrf.

Inter-VRF route leaking for communication from a router connected to ACI through an l3out in consumer-vrf to the VM in provider-epg

In this case, I already have the l3out created in consumer-tenant and we saw earlier that the subnet 200.200.200.200/32 is being advertised from the external router to ACI. Let us make some changes to the l3out now so that we can advertise this route to the provider-vrf.

Note: Above EPG's are configured in leaf 201 and hence both consumer-vrf and provider-vrf has been deployed in leaf 201. However, we only have a l3out configured in leaf 101 and 102 which is in consumer-vrf. So, we would only see consumer-vrf deployed in leaf 101 and 102. Also, consumer-vrf here wouldn't learn the routes yet for the provider-vrf as it need not be programmed in leaf 101 and 102 as yet.

First, let's advertise subnet 200.200.200.200/32 learned from external router to provider-vrf

Step1: Under the l3out EPG in consumer-tenant, tick the check box for "Shared route control subnet" and "Shared security import subnet" along with the default "External subnet for External EPG"

Note:
Shared route control subnet - Advertises the routes to another VRF based on where the contract is applied
Shared security import subnet - Allows the communication between the subnet and another VRF.

In this example, I am configuring only 200.200.200.200/32 subnet to be advertised to ACI. This can be configured to 0.0.0.0/0 to allow all subnets from external router to be advertised to ACI.

Step2: Under the l3out EPG in consumer-tenant, apply the consumed contract interface that was imported from provider-tenant.

Now, the route 200.200.200.200/32 should have been advertised to the provider-vrf in compute leaf 201 and the provider subnet 10.10.20.0/24 should have been advertised to consumer-vrf in border leaf 101 and 102.

We need to complete 1 more step to actually establish the communication between a VM in provider-vrf to an interface in external router belonging to consumer-vrf. We now need to advertise the provider subnet 10.10.20.0/24 out of l3out to the external router so that the external router knows the path for the traffic to provider-vrf. 

Step3: Advertise the provider subnet 10.10.20.0/24 out of l3out to the external router in consumer-vrf.

From the external router, if we check the routes now, we should be able to see ACI advertising the route 10.10.20.0/24. Ping to the VM 10.10.20.1 in provider-vrf should now work fine from the external router in consumer-vrf.

 

 

 

 

 

 

 

 

 

 

 

 

Verify

Check the routes in consumer-vrf and provider-vrf  on compute  leaf 201 to confirm the routes are leaked across the VRF.

[Rick1776]

Awesome job. This is a question that gets asked a lot in the support communities.

[kthned]

Hi @gkumark thanks for a very good description. 

It actually work without creating a shared subnet as you mentioned in "Step1: Configure shared subnet under the provider-epg as opposed to configuring under BD. "

in my scenario, one Tenant and two vrf. VRF consumer and Provider. VRF Consumer has the L3 OUT to the internet. 

I created common/default contract from L3-OUT epg (provider) to the VRF provider. I can see the VRF provider routes in L3 out as well as VRF comsumer and ping test works fine. 

Just wondering why we need step 1 to share subnet under EPG ? I have migration scenario with 1 EPG=1 VLAN = 1 Subnet. 

Could sharing subnet handy with single EPG across many subnets ? 

[mihaitanasescu]

Hi there,

Can I also bump into this thread?
I have a setup that interconnects ACI fabric (Spines) with GOLF routers via BGP-EVPN.
Do you know if it’s possible to do inter-tenant VRF leaking but not inside the ACI fabric, rather inside the GOLF router?
I tried this here in a lab scenario and although I saw the routes being advertised with the proper route-targets via EVPN, transit did not work (an mtr from a VM always showing the GOLF as the last hop where everything stopped and tcpdump on the end linux systems not showing traffic in both directions flowing)

I had:
Tenant1 –> VRF/EPG/BD –> Golf 1 (import 1234:1000 + import 1234:1000 stitching / export same = Tenant 1)
Tenant2 –> VRF/EPG/BD –> Golf 1 (import 1234:2000 + import 1234:2000 stitching / export same = Tenant 2)

In both of these on the Golf I also added:
Tenant1 VRF: import 1234:2000 + stitching (importing Tenant 2)
Tenant 2 VRF: import 1234:1000 + stitching (importing Tenant 1)

Any clue as to what I might be missing out or is this not supposed to work like this?
(I know it is suboptimal and that inter-tenant vrf leaking inside the ACI fabric is the way to go but I am now just curious for my own know-how)

[NDP]

Thanks for this post. 

All good, except  this point 

2. VRF consumer-vrf has been created with Bridge Domain(BD) consumer-bd. The endpoint 10.10.20.1 has been learned in EPG consumer-epg through leaf 201/1/17

I believe you wanted to say that it was 10.10.10.1 in consumer-epg.

We followed same article and it's working :-) 

[sandevsingh]

Great explanation.

[sandevsingh]

Imagine there is a firewall instead of the router in the Consumer-vrf. It is doing a static NAT on the outside interface. For example: destination ip of 15.15.15.10 (on outside) gets translated to 10.10.15.10 (on inside). So basically the firewall will be doing Proxy ARP for the ip: 15.15.15.10.

Question is - Will the route leaking work to the provider-vrf for this proxy ip also ? Will it be able to reach 15.15.15.10? 

[Adham Abdallah]

Hi,

I tried same setup myself and not working , the search to troubleshoot landing me here and it is the same as what i did...

After troubleshooting I found that consumer routes only leaked in the provider tenant....I tried to add another contract and export it and it is working finally!!!!!

so provider tenant provide contract1 and export it to consumer tenant.

the same way consumer tenant provide contract2 and export it to provider tenant.