IP Subnet on EPG vs BD

[RedNectar]

Hi experts.

I have a challenge for you!

When deploying a Network Centric design, I've always assigned IP addresses to Bridge Domains, as per just about every document I've read.

But I want to challenge that.

I'm searching for reasons why I shouldn't assign IP addresses to EPGs when deploying a Network Centric design - and that's my challenge to you - tell me WHY I should assign the default gateway IP address for a subnet to a Bridge Domain rather than an EPG when deploying a Network Centric design!

To me, assigning default gateway IP addresses to EPGs rather than BDs seems like an easier way to explain the transition from a traditional network to an ACI Network Centric Design.  But I want to be sure before I start suggesting this.  It would certainly make it easier to explain how to leak routes between Tenants (where you are forced to assign an IP address to an EPG anyway!!! - why not just assign it to the EPG from teh get-go?)

Of course I've done some simple experimenting, but not comprehensive. For instance, I've not had a chance to test DHCP relay when IP addresses are assigned to EPGs rather than BDs.  But I have tested normal EPG to EPG communication, EPG to external L3 destinations, and Tenant to Tenant communication.  These scenarios seem to work just as well (somewhat better you may argue for Tenant to Tenant communication) when IP addresses are assigned to EPGs rather than BDs.

[micgarc2]

As you mentioned I think the main use case of the subnet under the EPG was for route leaking. If you have a scenario where multiple EPGs (hence multiple sclasses or pcTags) are tied to the same BD you can run into issues with contract enforcement. The best practice is to always define the subnet for the provider of a route leaking scenario at the EPG level. This way the sclass for the EPG is tied to the subnet when the route leaking takes place.  The consumer VRF needs to map a provider subnet to a pcTag. You can run into problem where the return traffic could get misclassified. 

BRKACI-3101 explains this further.

As far as scenarios where route leaking is not taking place and not just using the EPG subnet as the GW rather than the BD, let me think about this one to get back to you.

[RedNectar]

Thanks for your reply.  I've been so busy lately I just haven't had a chance to re-visit this.

I can see that "If you have a scenario where multiple EPGs (hence multiple sclasses or pcTags) are tied to the same BD you can run into issues with contract enforcement." might possibly be an issue, but that's why I specified "Network Centric Design".  And as far as I know, not only is "The best practice is to always define the subnet for the provider of a route leaking scenario at the EPG level.", but it is the ONLY way to make this work.

BRKACI-3101 is the BEST ACI presentation I've ever seen - it is now my ACI go-to doc.  But it doesn't really cover this case.

But thanks again

[Jayesh Singh]

Interesting topic Chris!

Its actually tough to identify the need to have subnet defined at BD level in case of network centric approach. Subnets configured at EPG level just gets the job done.

After a deep dive into the fundamentals only two points related to this comes into my mind,

1. DHCP Relay Function: This might need subnet to be configured at BD level as DHCP relay label is attached to the BD. In DHCP request, under option 82 sub option 5 (link selection sub option) the subnet/ip pool is mentioned from which the ip needs to be provided by dhcp server. This subnet is nothing but primary subnet configured at BD level.

Not sure how dhcp will work in our scenario of having subnet configured at just EPG level. Theoretically, I don't think it will work. But testing it out would be totally worth it.

2. Route advertising in L3 out: If we have static routing between border leafs and external network then it should work fine. However, in case of L3out with dynamic routing protocol we associate L3out with BD for advertising internal subnets to external network. Does this work if we have subnet at EPG level? I haven't tried this, in your case did u use static routing or dymanic routing for external connectivity?

Regards,

Jayesh

[RedNectar]

Hi Jayesh,

Thanks for replying  - I've just been too busy lately to get back to this.  I was hoping to test DHCP relay, but didn't get the chance yet.

But I have tested  the case of L3out with dynamic routing protocol where we associate L3out with BD for advertising internal subnets to external network - it still works just fine. 

I'll add more if I find out more

[Jayesh Singh]

Hi Chris,

Thank you! Good to know that internal route advertising to external network works fine with subnet at EPG level in case of dynamic routing in L3 out.

I got the opportunity to test DHCP relay function with inter-vrf scenario(i.e. user epg and dhcp server epg are in separate vrf) and surprisingly - It works absolutely fine!

Testing scenario is as follows:

DHCP server details -

In Tenant: common

VRF: common

Endpoint Details -

In Tenant: User Tenant

Subnet set at EPG level

VRF: User VRF

BD Config:

- DHCP relay label set at infra level

- Unicast routing enabled

- subnet section left blank

So even DHCP function doesn't have any dependency of subnet to be defined at BD level.

Further to this, if we summarize then most of the functionalities in case of subnet at EPG level(only) works just same as subnet at BD level, below are the basic network requirements for any service:

1. External connectivity with static/dynamic routing - Tested OK

2. Inter EPG/Tenant/VRF communication - Tested OK

3. DHCP relay function - Tested OK

In purely network-centric approach, defining subnet at EPG level(only) and not at BD level doesn't seem to have any limitation with basic functionalities that we have tested so far!

Would be happy to know if this information is helpful and more insights on this topic are welcome. That will help us derive a conclusion.

Regards,

Jayesh