EPG Scalability

Alex Moore · ‎01-29-2021

I am working on re-implementing an existing network in ACI - it's not exactly a "migration", as the old & new networks will never be interconnected... rather ACI is being deployed in a brand new building, and once it's ready all the existing servers will be powered down simultaneously, transported to the new building, connected to ACI, and powered back up. So although I don't have to consider any transitional migration period, I am effectively just trying to provide exactly the same functionality with ACI on day 1 as the old network is currently providing.

Having read Cisco's "verified scalability guide" for ACI, one of the numbers in particular stood out to me... which is the maximum number of EPGs per tenant. Supposedly for a "single tenant" fabric, there can be up to 4000 EPGs (but I don't understand what a "single tenant" fabric even means... surely the built-in tenants count towards that limit, so even without any user-defined tenants there are 3 already?), but if there are multiple tenants then no single tenant can have more than 500 EPGs.

Is that supposed to be a "hard limit", or does it just mean that Cisco haven't tested with more than 500 EPGs per tenant, but it might work fine?

In my case, I'd ideally like to create 2 user-defined tenants, but one of those is likely to have around 1000 VLANs, each of which I intend to map to a separate EPG (ie I'll be taking a "network-centric" approach). Is that actually not possible? Do I need to subdivide that larger tenant into multiple smaller ones to make it work (in which case I guess those EPGs will also end up being split across separate VRFs, which I hadn't intended to do)?

I have actually tried to test whether 500 EPGs per tenant is a hard limit by creating a test tenant, and defining 501 EPGs in it. ACI was happy to accept the EPG #501, so I assume it's not a hard limit, but I still wonder whether it is a "bad idea" to plan to have ~1000 EPGs in a single tenant. Can anyone provide some advice? I just think it's odd that 4000 EPGs in a single tenant is fine, but as soon as a second tenant is added, neither of them should have more than 500 EPGs.

Also, I imagine an application-centric approach typically requires more EPGs than a network-centric approach (as there's likely to be more segmentation going on in such a network), so I find it a little surprising that I'm already at risk of bumping up against some of these numbers when just trying to re-implement an old existing network using a network-centric approach in ACI.

6askorobogatov · ‎01-29-2021

Alex, most (if not all) Cisco scalability numbers based on the test results, not a hard limit.

For your network centric design, start with network. Are your 1000 VLANs on the same VRF ? Probably not. How inter-VRF routing will be implemented in ACI ? You may conceder one VRF per tenant, or if there are more than one, you should have a good idea why.

Also, keep in mind, naming convention for fabric object (leaf, spines, oob IOPs, inb IPs, pools, AEPs, domains, tenant , vrf , BD, epg, contact etc) will have huge impact on future management and support.

Robert Burns · ‎01-29-2021

There are two types of QA'd (aka supported) limits. Fabric Capacity and Leaf Capacity. Various HW models will have varying supported limitations. It's always best to use the built-in Capacity Planner Dashboard.

Operations > Capacity Dashboard. There you'll find the limits for the fabric as well as individual Leaf scale limits. This is good for knowing what levels you're currently at, as well as what limits are. If you need to look forward (newer versions) you can leverage the Scale/Capacity documentation as well.

From your Tenant/EPG scale question above, its saying that any single user tenant within a fabric should be limited to a max of 4K EPGs. Again, these are soft limits, so even if you have multiple user tenants (which says limited to 500 EPGs each) I've seen customers go beyond this without issue. I wouldn't be concerned if you ran with two user tenants each with ~1K EPGs. I'll double check with Engineering just to be sure there shouldn't be an issue and respond back here.

Robert

David Yang · ‎02-12-2021

In a Multi-site ACI implementation, does the number of the shadow EPG's need to be taken into consideration of EPG scalability when site-local EPG's need to communicate with the stretched EPG's or site-local EPG's in another site?

Thanks,

Robert Burns · ‎02-12-2021

Yes EPGs are EPGs (Shadow or local) so they will count towards your scalability #s. Keep an eye on the Capacity Dashboard if you have any concerns about approaching these "soft" limitations.

Robert