Subnet on EPG and/or BD for Co-existence of Network-centric and application-centric mode

[clement_cheung]

Hi experts

I have an ACI fabric currently designed based on network-centric mode and needs inter-VRF route leaking.  Therefore there is one-one mapping among subnet, EPG and BD, and the subnet is defined under EPG rather than the BD.  This design is aimed for ease of migration of endpoints from classical network to ACI.

I however foresee that after migration, we may need to 'rewire' some migrated endpoints, without changing IP addresses, to new EPG which acts as Application Component Group so special contracts can be applied for micro-segmentation.

How to do that ? Where should the subnets be placed EPG and/or BD ?

Regards

[Rick1776]

The Bride Domain.

[clement_cheung]

If putting subnet on BD, how to support inter VRF route leaking that is supposed to be achieved by putting subnet under epg?

[Rick1776]

See this link for explanation...

https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/200242-Configuring-Inter-Context-Communication.html

[RedNectar]

Adding Subnets under EPG configuration is mainly used to allow route leaking as you described. However, there is nothing wrong with having the subnet ALSO defined under the Bridge Domain, and indeed, if you have multiple EPGs using the same default gateway, you will probably end up doing that, especially if some of those EPGs do not need to have contracts with EPGs in other VRFs.

[clement_cheung]

Thanks!

yes that’s my case. I’ll need two EPGs having the same default gateway so I intend to put the subnet with the gateway IP under their BD in common. You mean I can also add the exactly same subnet (with the same gateway IP) to the one EPG that needs inter-VRF route leaking? What will happen inside ACI if the same subnet is defined in both an EPG as well as its underlying BD.

[RedNectar]

---

What will happen inside ACI if the same subnet is defined in both an EPG as well as its underlying BD?

Well, as far as the tenant's routing perspective, nothing changes.  The IP addresses really only exists once, but the thing is, by adding the IP address to the EPG, it makes it possible for ACI to ADVERTISE that route to another VRF (ie Route Leaking).

Now the explanation of WHY you have to assign the IP to the EPG rather than just marking the IP in the BD as being Shared between VRFs is a lot more complicated, and to find the answer you'd have to read this Cisco Techzone article (requires internal Cisco Access - if you don't have access, get back to me in a week or so when I have more time to digest it and reproduce without breaking any rules)

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[clement_cheung]

Need your further explanation

[Rick1776]

Please look at the link I provided in my post for further explanation.

[RedNectar · Accepted solution]

Hi Clement,

[Disclaimer. This is my interpretation of how inter-VRF contracts work. I have taken some liberties with the technical details, and if I have made mistakes in my interpretation, I welcome anyone with better knowledge to enlighten us all]

No doubt you read the link that Rick suggested at https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/200242-Configuring-Inter-Context-Communication.html

And you will have noticed that that document explains that you need:

1. Configure the shared subnet for EPG-X under the EPG (as opposed to under the Bridge Domain (BD).  This will be the provider EPG.
2. Configure the shared subnet for EPG-Y under either the EPG or BD.  This will be the consumer EPG.

But to understand WHY the Provider of the contract must define a subnet under the EPG rather than the Bridge Domain, you have to start looking at WHICH VRF the contract is actually implemented in. Is it implemented in the provider or the consumer VRF? When the contract is implemented entirely within the same VRF, this consern doesn't arise.

As you probably know, traffic is subjected to polcy on ingress, and the first packet to flow in a session is always (as far as TCP is concerned) is the TCP SYN packet from the consumer to the provider.

So it makes sense that the contract is implemented in the CONSUMER VRF. 

And you also probably know that when policy is implemented within a VRF, it is applied on a host to host basis. Along the lines of (from the viewpoint of a Leaf Switch processing an arriving packet)

1. Who sent this? (identify the host that sent the packet)
2. Which EPG is this host in? (identify the source EPG)
3. What is the destination host? (look up the MAC/IP address in LST/GST)
4. Which EPG is the destination in? (identify the destination EPG)
5. Apply the policy according to the filters created by the contracts between the two EPGs

But when policy is applied to traffic going to or coming from an external host, such as a host on the internet or some other external host, step 3 above changes slightly and the system reverts to a more tarditional LPM (Longest Prefix Match) lookup, and the policy is applied not so much on the destination host, but on the destination network.  Something like this:

1. Who sent this? (identify the host that sent the packet)
2. Which EPG is this host in? (identify the source EPG)
3. What is the destination host? (identify the destination IP address)
4. Is there a LPM path match for this IP? (Routing lookup)
5. Which L3 EPG is the destination in? (identify the destination EPG)
6. Apply the policy according to the filters created by the contracts between the two EPGs

Of course, when contracts are between VRFs, the situation is similar, and indeed is implemented in much the same way as far as step 5 above, up to the part where there is a LPM lookup. But in the case of contracts betwen EPGs in different VRFs, there is no destination EPG information intrinsically contained in the prefix as there is with an L3 EPG.

So ACI needs a way to ensure that the consumer VRF has knowledge of both the destination network (route) and the policy that needs to be applied.

This is achieved by associating a prefix (the subnet) with an EPG, which is of course implemented by defining a subnet under the EPG.  So the logic in this case is:

1. Who sent this? (identify the host that sent the packet)
2. Which EPG is this host in? (identify the source EPG)
3. What is the destination host? (identify the destination IP address)
4. Is there a LPM path match for this IP? (Routing lookup)
5. Which EPG is the destination in? (And the leaf switch can only know this if the provider has defined the subnet under the EPG)
6. Apply the policy according to the filters created by the contracts between the two EPGs

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[Rick1776]

Fantastic explanation. Thanks for explaining.

[randreetta]

... regarding step 5, wouldn't it be possible for the leaf to send the packet anyway, because it still doesn't know the destination EPG, and let the egress leaf filter the packet ? Then the /32 -> EPG association would be dinamically learnt in case someone answers to this first packet, through what was called 'conversational learning' (which I'm not sure if it still applies and is used inside ACI ...).

[Rick1776 · Accepted solution]

See this link.

https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/200242-Configuring-Inter-Context-Communication.html

[Jason Williams · Accepted solution]

Clement, 

Looks like you've gotten some good feedback for this topic. I would like to add a simple note in regards to minimum contract requirements for inter-VRF route leaking. 

Let's assume there are 2 subnets in 2 different VRFs which need to be able to reach each other. Below are different scenarios to permit route leaking. 

- If both subnets are defined in the EPG, then it doesn't matter which side provides or consumes in the contract relationship. 

- If one subnet is under the BD and one subnet is under the EPG, then the EPG subnet must provide and the BD subnet must consume at minimum. BD subnet side does not have to provide a contract to the EPG subnet.

- If both subnets are defined under the BD, then both EPGs must provide and consume a contract with each other. 

Each scenario assumes both subnets have "Shared between VRFs"enabled and the contract scope is global. 

-JW

[clement_cheung]

Thanks all for the feedback!

Hi Jason

I am especially interested in the scenario in which both subnets are defined under the BD; both EPGs must provide and consume a contract with each other.

May I know how such mutual provider-consumer relationship for both EPGs allows the consumer side leaf switch to learn the destination EPG for policy enforcement ?

Regards

Clement