cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Cisco ACI Inter VRF/Tenant Route Leaking Design – Simplified!

11705
Views
90
Helpful
7
Comments
Jayesh Singh
Cisco Employee

There is a difference between something you know and something you understand. Recently, I came across such kind of a situation, when I realized I perfectly knew how to configure Inter VRF communication in ACI, but the in-depth understanding was missing. The intention of this article is not to share my experience, but the learning.

Please note that Shared Services term which is widely used in most of the documents is nothing but about Inter VRF/Tenant Communication.

I would like to mention, the level of documentation Cisco has done for ACI is commendable. You can find a detailed guide for all the bells and whistle in ACI on the Cisco official site.

Then, Why Am I Writing This?

Well, I thought there is a very critical design guideline around Inter VRF/Tenant route leaking methods in ACI, which I should highlight. So, just trying to do that…

Note: This is an advanced topic in ACI and I assume you have the working knowledge of ACI components like Tenant, EPG, BD, VRF, Contract etc. So, let’s begin…

Need for Inter VRF/Tenant Communication

The first thing we need to understand is in which cases Inter VRF/Tenant communication will be needed:

  1. Shared Services – Meaning, you have an EPG hosting common shared services in one VRF (shared service provider) and there can be single or multiple EPGs in another VRF (shared service consumers) using the shared services.
  2. Ad-Hoc – There can be specific two or more EPGs in separate VRFs and you might need them to communicate with each other as per requirement E.g. Multi-tier application setup with EPGs corresponding to every tier in separate VRFs.

Role of Contracts

As you must be aware of the fact that all the communication in ACI is governed by Contracts, even in case of inter vrf communication, we will need contracts between the EPGs. This is where the duality of the contract comes into the picture. So, there are two roles of contracts in case of inter vrf/tenant communication:

  1. Access control with the help of subjects and filters defined in the contracts
  2. Route Leaking between consumer and provider VRFs (I know many of us looked at contracts as ACLs, it’s more than that)

Contract Scope is an important attribute, it can be either of the two mentioned below:

  1. Scope: Tenant – Provider and consumer EPGs are in different VRFs but the same tenant.
  2. Scope: Globe – Provider and Consumer EPGS are in different VRFs as well as the different tenant.

Few Considerations

  • In the case of inter tenant communication between two user tenants, the contract must be created in provider tenant and has to be exported to the consumer tenant. On the consumer side, the exported contract has to be attached to the consumer EPG as Consumed Contract Interface.
  • Common tenant comes with the superpowers, in case you have one EPG in user tenant and another EPG in common tenant, create a contract in common tenant. That eliminates the overhead of exporting the contract, as contracts created in common tenant can be directly attached to EPGs in user tenant as provided or consumed contract.
  • In the case of inter vrf communication within the same tenant, just the contract has to be provided and consumed into respective EPGs like we do normally and contract export is not required.
  • Since the subnets between two VRFs will be exported to each other, they have to be unique and non-overlapping.

Design Approach

The design approach is selected based on whether there will be a single EPG or multiple EPGs (in Provider VRF) serving as the shared service provider to the EPGs in consumer VRF. Basically, there are two ways of doing the route leaking and policy enforcement between EPGs in separate VRFs.

1. Shared Services with Subnet defined under provider EPG (Usage: when there is only single EPG in provider VRF which will serve as a shared service provider) 

2. BD-BD Shared Services (Usage: when there are multiple EPGs in provider VRF that can serve as the shared service providers)

Let's get into more details of both methods one by one:

1. Shared Services with Subnet defined under provider EPG

This is the preferred method to configure inter vrf/tenant communication as per ACI Best practices guide and works perfectly fine for shared services requirement (until you reach the Limitations section below). There are about tons of documents online that would serve as a step by step configuration guide for this method. I am taking an example of EPGs in separate tenant and VRFs to explain this method.

Procedure:

  • Define the Provider side subnet at EPG level and mark it as Shared between VRFs (define the subnet at EPG level even if you have the same subnet defined at BD level in case it is being used by other EPGs also, as in application-centric deployment).
  • Define the Consumer side subnet at BD or EPG level and mark it as Shared between VRFs.
  • Create a contract at provider tenant and attach to the provider EPG as provided contract.
  • Export that contract from provider tenant to consumer tenant, attach to the consumer EPG as consumed contract interface.

A logical representation of the setup with the provider and consumer EPGs in different Tenants is shown in the diagram below: InterVRF.png

What Happens Under the Hood?

Provider Side:

  • As soon as the contract is attached at both provider and consumer EPGs, consumer side prefix P2 is leaked into the routing table of provider side VRF (VRF1).
  • Along with the routes, VNID rewrite information corresponding to the consumer side VRF (VRF2) and ClassID is also installed in VRF1. Please note that ClassID, which is nothing but PC Tag (EPG ID) is set as 0, meaning there is no consumer EPG information associated with the leaked prefix.
  • Policy is also not programmed in provider VRF. So, on the provider side, it just forwards the packet based on routing information and policy is never applied on this side.

Consumer Side:

  • Provider side prefix P1 defined at EPG level is installed into the routing table of consumer side VRF (VRF2).
  • Along with the prefix, VNID rewrite information corresponding to provider side VRF (VRF1) and ClassID is also installed in VRF2. Please note that ClassID at consumer side is corresponding to EPG1 (provider EPG). This can also be considered as 1:1 static mapping of provider side subnet to the provider side EPG.
  • Any incoming or outgoing traffic belonging to the provider subnet(P1) will be classified as EPG1.
  • The contract is programmed in consumer VRF (VRF2). So, in this method, the policy is applied at the consumer side only and never on the provider side.

Limitations

In a nutshell, this method works like having 1:1 static mapping of provider side subnet to the provider side EPG in consumer VRF.

What if there are two provider EPGs using the same subnet as we normally have in application-centric deployment and both need to provide services to the consumer, if we follow the same method, what would be the ClassID (EPG ID) for provider’s leaked subnet installed at Consumer side?

Well, both providers will keep fighting (not literally) and at a time ClassID of only one provider EPG will be programmed in the consumer VRF, keeping another one inaccessible for the consumer (contract drop condition). 

Interesting isn’t it? That’s when the second method A.K.A Poor brother of the first method comes into rescue… 

2. BD-BD Shared Services

In my opinion, this method has been highly underrated due to its low profile appearance in shared services deployment guides. This method is also treated as the workaround for the limitation we saw in the first method.

In BD-BD shared services method, EPGs in both the VRFs form provider as well as consumer relation with each other. It’s kind of 360-degree relationship making both of them as a shared service provider as well as the shared service consumer. Again, I am taking the example of EPGs in separate tenant and VRFs to explain this method.

 Procedure:

  • Define the subnets at BD level for both the EPGs in both VRF and mark them as Shared between VRFs.
  • There is no need to define subnet at EPG level; route leaking is done between BDs.
  • Create a contract in both tenants, provide and consume on both sides. It’s like configuring contracts as we did in the first method but twice and in both directions.
  • What if you don’t need both EPGs to be the provider as well as the consumer? Well, still you need to form consumer as well as provider relation on both sides.
  • So, in that case, you can make the second contract as a dummy contract, with filter mentioning a port on which there is no service running, or denying any unrequired port. It doesn’t need to enable communication, just required to form relationships. To save TCAM space, don’t check on apply in both directions.

A logical representation of the setup with the provider and consumer EPGs in different Tenants is shown in the diagram below:InterVRF2.png

What Happens Under the Hood?

If you see the diagram, everything is identical on both sides. Routes are leaked between both VRFs along with VNID rewrite info; contracts (policies) are programmed in both VRFs.

One thing to notice is ClassID at both the sides are set as 0. This means, there is no EPG information associated with the leaked subnets. Hence, there is no static mapping of the subnet with EPG ID.

In this case what essentially happens is that, to classify the endpoint a lookup is done in actual endpoint table which contains exact /32 host IPs and EPG information as compared to the first method, where the endpoint is classified based on the provider subnet defined at EPG level.

So even if two EPGs are using the same subnet, there won’t be any impact as there is no static mapping of the subnet to EPG is involved, as the endpoint classification process is more granular.

Once the endpoint is classified, the policy is applied as per the contract between both the EPGs.

Limitations

BD-BD shared services method requires both sides to have a provider and consumer contracts, regardless of the fact whether it is needed for communication or not.

This eats up the TCAM space and should be chosen only when you have more than one EPG in the same VRF to be the shared service provider to the consumer.

Quickbits

Verify VNID rewrite information and ClassID on the leaf switches using following commands:

Leaf-1#vsh

Leaf-1#show ip route vrf <tenant:vrf_name> <imported_IP_prefix> detail

Read the VRF crossing information section to get the details. ClassID will be shown in hex and when converted to decimal, gives us the pc tag of EPG.

Hope this post is helpful for all the ACI Experts out there. Your feedback will be highly appreciated!

Regards,

Jayesh

References:

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices.html

https://ciscolive.cisco.com/on-demand-library/?search=aci&search.event=ciscoliveus2018#/session/1509501653465001PRkT

7 Comments
pavan.goyal
Beginner

Excellent Knowledge sharing!!!!

mmiljkovic
Beginner

Hi, Jayesh,

 

Thanks for this good explanation.

 

I was looking at BD-BD shared services example, but in my example consuming contract is done on consumer VRF side at EPG collection level (vzAny), and as I saw EPG VRF Collection can't be a provider for dummy contract from your example.

 

As I saw, it is enough dummy contract to be provided from one EPG in consumer VRF to have leaked prefixes from provider BD VRF into consumer VRF, but it seems to me like clean design this dummy contract to be provided from all EPGs on consumer VRFs side, but in situation there are to many EPGs there, it could be a problem.

 

From your point of view, is it enough dummy contract to be provided from only one EPG from consumer VRF side?

 

Thanks,

 

Regards,

 

Marko

GMichael6
Beginner

Hello

 

I was wondering what can we do if we have a subnet in VRF 2 that is already leaked in VRF 1, to be Leaked in VRF 3

assuming that all VRFs in the same Tenant

 

to be more precise

 

                                                               Tenant A

 

           

VRF1,Ext EPG 1, Subnet 1 <--leak--->  VRF2,Ext EPG 1, Subnet 2                        

           L3OUt 1                                      VRF2,Ext EPG 2, Subnet 2<---leak--->  VRF3,Ext EPG 3, Subnet 3 

                                                                            L3OUT2                                                 L3OUT3   

 

leak between  ext EPG 1 VRF 1 and Ext EPG 1 VRF 2 works fine with Subnet 1 and subnet 2

As soon as I     leak same subnet 2 in EPG2 VRF 2 with EPG1 in VRF3

 

Subnet 2 starts fllaping every one minute in the routing table of VRF3             

 

I tried many scenarios like using same EPG 1 in VRF 2 to do the leak, but the route kept flapping in VRF 3

 

Thanks,

moredipak2006
Beginner

Excellent Knowledge in Very Simple Way.

Hi Jayesh,

 

Could you explain more in detail what happen in solution 2 bd-to-bd , I mean how are contracts enforced (provider vs customer side).

 

In this case what essentially happens is that, to classify the endpoint a lookup is done in actual endpoint table which contains exact /32 host IPs and EPG information as compared to the first method, where the endpoint is classified based on the provider subnet defined at EPG level.

 

I understand the exported subnet is not flagged with the EPG info and thus the fabric will perform some lookup on the endpoint table to derive the EPG membership. Does it happen on both sides ? What happen if customer (or provider) side is an unenforced VRF for instance ?

nkhawaja1
Beginner

For multiple EPG in the same subnet leaking scenario, could you use "no default gateway SVI" feature on the EPG and assign the IP on the EPG levels?

prasanth_k
Beginner

Brilliant explanation!!!. Thanks Jayesh!!