Re: ACI Tenant / VRF / L3Out Choices

Alex Moore · ‎01-22-2021

Hi, I am new to ACI, and currently trying to figure out how best to achieve my goals in terms of the high-level constructs, and could do with some advice/guidance. I have been reading numerous guides, but am still a little unsure whether what I am hoping to do makes sense & if I am going about it the right way.

I would like to create two distinct partitions within the network ("prod" and "test"), and am thinking that the right approach would be to define two tenants to achieve that. Each subnet and VLAN defined in ACI (I am likely to take the BD = VLAN = EPG approach) will either belong to one tenant or the other, however there will need to be a lot of cross-tenant connectivity, and no need for overlapping addresses, so I imagine it makes sense to have a single shared VRF defined in the common tenant (to avoid having to configure route-leaking), but to have individual BDs & EPGs configured in their respective tenants. Does that sound reasonable?

I would also like each tenant to have its own separate L3 Out. Each L3 Out will provide connectivity to the same external networks, however I'd like the "prod" tenant to only use its L3 Out to reach the external networks, and the "test" tenant to only use its L3 Out. Similarly, I'd like the "prod" tenant's networks to only be advertised externally only over its L3 Out, and the "test" tenant's networks to only be advertised externally over its L3 Out. Is that how it will work if I define an L3 Out in each tenant, despite the fact everything is using the same shared VRF?

The alternative I am considering is to have a separate VRF for each tenant. With that approach I am more confident that each L3 Out will only be used by traffic to/from the associated tenant's networks, however I imagine it would become more painful to permit inter-tenant connectivity, as route leaking will be required - which is why I'm thinking the first approach I mentioned might make more sense, assuming it will work the way I'm imagining.

Thanks a lot,
Alex

Sergiu.Daniluk · ‎01-24-2021

Hi @Alex Moore

Different tenants + shared VRF sounds good, and here are a couple of remarks based on your concerns:

1. The external prefixes learned from the two L3Outs must not overlap.

Basically, having a single VRF (a single routing instance) you need to make sure that you do not learn the same external prefix from both L3Outs. Otherwise, you will encounter problems both in routing and policy enforcement.

2. The "Prod" and "Test" subnets can be exported in a controlled manner using the "associated L3Out" under each BD.
3. The "external subnets" configured in the ExtEPG, must not be 0/0 for both tenants. Easy solution is to make one as "0/0" and the second one with "0.0.0.0/1" and "128.0.0.0/1"

4. Communication of EPGs with the L3Outs will be done based on the contracts, so no need to worry about communication with the "wrong" L3Out, as long as conditions #1and #3 are present.

Stay safe,

Sergiu

6askorobogatov · ‎01-24-2021

Some comments about that. design.

1. Overlapping prefixes will create faults in zoning table, In unenforced VRF there will be no issues.

2. Will not work with (3)

3. 0/1 and 128/1 will make zoning table useless, there will be permission for 0/1, 128/1 applicable to all L3outs 'cos zoning table is for VRF not for L3OUT. That is the reason why 0/0 has special tag, but 0/1, 128/1 are regular subnets.

4. Will not work with (3)

In other words, on the same VRF any EPG that has contact with Net-EPG 0/1, 128/1 will have in zoning table entry permitting that EPG to talk to ANY IP on ANY L3Out.

====================================

1. The external prefixes learned from the two L3Outs must not overlap.

Basically, having a single VRF (a single routing instance) you need to make sure that you do not learn the same external prefix from both L3Outs. Otherwise, you will encounter problems both in routing and policy enforcement.

2. The "Prod" and "Test" subnets can be exported in a controlled manner using the "associated L3Out" under each BD.
3. The "external subnets" configured in the ExtEPG, must not be 0/0 for both tenants. Easy solution is to make one as "0/0" and the second one with "0.0.0.0/1" and "128.0.0.0/1"

4. Communication of EPGs with the L3Outs will be done based on the contracts, so no need to worry about communication with the "wrong" L3Out, as long as conditions #1and #3 are present.

Sergiu.Daniluk · ‎01-25-2021

Hi @6askorobogatov

Thanks for the intervention. I stand corrected. There are a couple of things which I presumed to be true when suggesting my solution. So allow me to clear some details up and avoid creating confusion for @Alex Moore or anyone else reading this topic.

1. Internal subnets advertisement: When advertising a BD subnet, routes are redistributed into OSPF LSDB (Link-State Database) via a route map that is automatically created on a border leaf. This route map is shared with EIGRP and other OSPF L3Outs on the same leaf in the same VRF. It implies that subnet advertisement configuration on one L3Out may affect other L3Outs. In other words, if the two L3Outs are configured on the same border leaf, and you are using OSPF in both L3Outs or OSPF for one L3out and EIGRP for the second L3Out, then BD subnets will be advertised to both L3Outs even if you are associating only one L3Out.

Hence, awareness of this implementation is required when there are multiple L3Outs on the same leaf in the same VRF.

Solution1: use different border leaf switches for each L3Out

Solution2: use BGP as routing protocol

2. External Subnets for External EPG: if using 0/0 in one ExtEPG/L3Out and 0/1 + 128.0.0.0/1 on the second ExtEPG/L3Out, will allow traffic from the internal EPG to both ExtEPG because of this:

Leaf301# vsh_lc -c "show system internal aclqos prefix "

Vrf Vni Addr           Mask     Scope Class  Shared Remote

======= ============== ======== ===== ====== ====== ======
3047424 0::/0 0::/0 3047424 15     FALSE FALSE
3047424 0.0.0.0        ffffffff 3047424 15     FALSE FALSE
3047424 0.0.0.0        7fffffff 3047424 32772  FALSE FALSE
3047424 128.0.0.0      7fffffff 3047424 32772  FALSE FALSE

What you are seeing here, is how the ExtEPG subnets are allocating sclass ids. Since we are not very specific, an external prefix can match to wrong ExtEPG thus policy enforcement will not work as intended.

Solution: Configure 0/0 Ext Subnet for Ext EPG only on the ExtEPG where the default route should be. For the second ExtEPG, configure the exact subnets (or supranets) accessible through that ExtEPG.

Leaf301# vsh_lc -c "show system internal aclqos prefix "

Vrf Vni Addr           Mask     Scope Class  Shared Remote

======= ============== ======== ===== ====== ====== ======
3047424 0::/0 0::/0 3047424 15     FALSE FALSE
3047424 0.0.0.0        ffffffff 3047424 15     FALSE FALSE    -> "Default L3Out" for PROD
3047424 198.18.0.0     1ffff    3047424 32772  FALSE FALSE    -> external subnets for Testing ENV

This will work as long as the 198.18.0.0/15 subnet is not behind PROD L3Out.

As a conclusion and a general rule, as long as the two L3Out are configured on distinct border leaf switches (lets call them Test BL and Prod BL) AND there are no endpoints Test are connected on the Prod BL, and vice-versa, the solution should work in the same VRF as well.

Hope this is much clear now

Stay safe,

Sergiu

6askorobogatov · ‎01-25-2021

Hi Sergiu,

Very interesting post, thank you. However, there are still no solution for 2 L3OUTs that are default gateways for the 3 sets of EPGs.

Regardless of static route or dynamic protocol, one of the set of EPGs (Test or Prod in this. example) will not have route to 0/0.

Otherwise the will be conflict or in routing table or/and in zoning table. It still will be the same one routing table and zoning table for VRF.

Sergiu.Daniluk · ‎01-25-2021

Totally agree! It's impossible to have 2x L3Outs in the same VRF, with both having a default route, while keeping in mind the described requirements. If this is needed, the solution of having dedicated VRFs is the only way to go ahead.

Stay safe,

Sergiu

6askorobogatov · ‎01-24-2021

In traditional network, will you put you dev and prod on the same VRF ? Probably not. You probably will prefer to have some layer of L3 control between environments. Also, in my experience, design with network objects in common tenant will need full understanding of the compete implementation. If you start to populate "Common" before you know what you fabric will be at the end of implementation, there will be very hard way back.

Alex Moore · ‎01-24-2021

Thanks both for the advice. It sounds like I should stick with separate VRFs, as the way I was hoping to use the L3Outs would not have met those constraints (I effectively wanted each tenant to use its own L3Out as a "default route"). Glad I asked in here rather than spending time trying to test it out myself!

As I'm beginning to learn more about ACI, I am definitely a little concerned about the fact that many things appear to be difficult to change later on (at least without disruption / significant effort). So I'm very keen to make good decisions at the start. But that also means it's taking me a lot longer than I had anticipated to begin the implementation in earnest, as I'm trying to build as complete a picture as I can before properly starting!

6askorobogatov · ‎01-24-2021

It will be very basic design, You will have 2 tenants Test and Prod with a single VRF each. L3Out on each VRF will be the default gateway.

L3 device outside of the fabric should take care of routing between Test and Prod. Obviously you can use OSPF or BGP on L3Outs.