External EPG subnet under multiple VRFs

[Luke Poskitt]

Hi All,

We have a MultiPod fabric with multiple L3Outs and wish to enforce contracts consistently between EPGs and hosts external to the fabric. 

For example; an administrator on the access layer external to the fabric registers the IP address of their workstation, and (using the API) the IP address of the workstation is added to an External Network under the L3Outs as a /32 External Subnet.  Contracts between the External Network containing the /32 IP address and the EPGs then grant the administrator appropriate access.

Unfortunately, it appears as though it is not possible to add the same /32 IP address to an External Network under multiple L3Outs as doing so generates fault code F0467.

Is there any way at all to consistently enforce contracts between External Networks and EPGs where multiple L3Outs are used?

Cheers,

-Luke

[Jayesh Singh]

Hi @Luke Poskitt ,

The external subnets that you add under external epg with tick mark set to "External Subnets for External EPG" essentially means that you are creating a static mapping of IP subnet (in ur case /32 host ip) and External EPG.

This is required for classifying the traffic to and fro external network and applying the policy as per contract.

Multiple External EPG within an L3 out or multiple L3outs in the same VRF cannot have exact same external subnet defined under multiple external EPGs. That throws a fault and it is similar to the internal EPGs where in you can have an endpoint to be part of just single internal EPG.

When traffic arrives on the leaf switch it classifies source and destination IPs into EPGs and lookup the contracts and takes the decision whether to allow or not. If an endpoint is part of two EPGs how will it classify it to one EPG? Thats the idea basically and hope that helps! Let me know if you have any further query.

Regards,

Jayesh

***Rate all posts that are helpful. Mark it as a solution if that answers your query as it may help other users who might have the same query***

[Luke Poskitt]

Hi Jayesh,

Thanks for the reply explaining the behaviour - per my original question; is there any way in which to consistently enforce contracts for external EPGs when using multiple L3Outs?

Cheers,

-Luke

[Jayesh Singh]

Hi @Luke Poskitt,

My understanding is that for a set of external users you would have single L3out in a VRF.

We would essentially have 2 different L3outs with same external subnets if they want to access EPGs in 2 different VRFs. If you look at it at routing level, how would you route your traffic from your external network to the ACI fabric if your have 2 L3 paths to the ACI.

You can have multiple L3 connections be part of same L3 out and with same external EPG. That is one method to having unified policies between your external hosts and internal EPGs.

One more point here is, the external EPG to external subnet mapping works on lowest prefix match. For e.g. if I have an external EPG1 with 10.0.0.1/32 as external subnets and another external EPG2 with 10.0.0.0/24, you would be allowed to do that and no fault will show up. However, 10.0.0.1 will be classified as external EPG1 only and traffic to and from rest all IPs in 10.0.0.0/24 will be classified as external EPG2.

You can have different contracts between your external EPGs and internal EPGs defining the appropriate access level. But within a VRF, you can't have 2 L3Outs with the external EPGs having the same external subnets. 

You can share some details on what exactly you want to achieve, probably we can figure out best designing method to meet the requirement.

Regards,

Jayesh

[Luke Poskitt]

Hi Jayesh,

What we are trying to achieve by using two different L3Outs within the same VRF is some level of traffic engineering for each pod in the fabric.  Essentially, in our fabric each pod can be thought of as equivalent to a datacenter and with the exception of a couple of "stretched" EPGs, our EPGs are "local" to a pod/datacenter.  Likewise, each L3Out is "local" to a pod in the sense that the physical equipment underlying the L3Out is within the same datacenter.  In a network-centric design, it therefore makes sense to have a BD associated with the "local" L3Out and therefore advertised to the routers directly connected to the pod.

Given that an external IP address (e.g. an administrator's workstation) may require the same level of privileged access to endpoints in EPGs in both pods, and that the BDs for those EPGs would be associated with the "local" L3Out of the pod, then it would be necessary to apply a contract between the external IP address and the EPG regardless of which L3Out the external IP address used to access the EPG.

To me, these requirements do not seem altogether unusual, that is; using multiple L3Outs to allow a network to be selectively advertised via a specific route, and consistently enforcing a contract for an external network.

Cheers,

-Luke

[Jayesh Singh]

Hey Luke,

Thank you for the information.

What comes into my mind that can work for you is having both of your L3 connectivites under same L3 out.

For example, create an L3 out and configure node profile and interface profile corresponding to external networks for both pods under same L3 out. That way you will have two seperate set of external connectivities but same external EPG and all under a single L3 out.

L3 out necessarily doesn't mean that it can contain only one set of connectivities. Instead it is collection of external connectivites. In your case it will be collection of external connectivites in both the pods within a VRF.

I am no genius in multipod setups but as far as L3 out is concerned it will never allow you to have exact same external subnets in multiple external epgs in L3outs under same vrf.

You can try method mentioned above (Same L3out and Multiple External Connections) and see if that works!

Regards,

Jayesh

[Luke Poskitt]

Hi Jayesh,

Thanks for the reply.

We have previously considered placing all connections under a single L3Out as this appears to be the only way in which to consistently enforce policy for an external network, however (as mentioned previously) the reason for using two L3Outs is that our pods are located in two respective datacenters and are geographically separated by ~80km.

Consequently, we wish to have some level of being able to implement "traffic engineering" by associating a BD with the pod-local L3Out, rather than associating it with a single L3Out which would result in less control in how the BD network was advertised to the connected routers.

Cheers,

-Luke

[DanLaden · Accepted solution]

Luke,

Take  a look at this doc.  In particular, search for "External Subnets for an External EPG".

"Even though the external subnet for the external EPG is configured with the L3Out, the ACL is applied at the VRF level. This means that if a prefix is configured for L3Out-1 and traffic with a source address matching that prefix arrives on L3Out-2 the traffic is classified to the external EPG of L3Out-1. The following diagram explains this behavior:"

You probably dont need any EEPGs on your second L3Out if its one your first L3Out.

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices/b_ACI_Best_Practices_chapter_01001.html

Enjoy your day,

Dan Laden

[Luke Poskitt]

Thanks Dan, I can't believe that it was as simple as configuring the external network under a single L3Out only.

I noticed that the section to which you referred in the best practices guide has been slightly revised to clarify this behavior when comparing to an earlier version, but I still feel very silly for overlooking this detail.

Cheers,

-Luke