Short version: In general, what's the best way to advertise an anycast address to ACI from multiple independent/geo-redundant platforms? For multi-pod ACI, is the only option to stretch a single L3Out across all pods and systems, since the anycast address can only be present as a subnet on one EEPG? This sounds like it would limit scalability, as floating L3Outs (floating SVI) only officially support up to 6 anchor nodes (ACI BGP nodes).
Long version (original):
Hi, we have a design problem around anycasts from a geo-redundant OpenStack cluster, or anycasts from separate L3Outs in general.
The cluster mainly consists of two geo-redundant locations (regions) and a third "half-location" for quorum reasons. Each OpenStack region lies in a separate ACI pod. We decided to use floating L3Outs (floating SVI) as the OpenStack cluster spans multiple racks at each region, and we don't want every involved leaf to need an IP address from each L3Out when we only plan to use two leaf pairs at each region for BGP.
For regional anycasts (separate anycast for each region), we use a separate L3Out for each region. For global anycasts (same anycast for all regions), we're unsure a bit unsure about the best approach wrt. geo-redundancy. We don't want to stretch the same subnet/external bridge domain across all regions if we don't have to.
Some ways we considered:
- Use separate floating L3Outs for each region, but add the anycast IP address to all EEPGs (all L3Outs). This isn't directly possible since subnets other than the default route can't be reused across multiple EEPGs. However, since EEPG lookup simply uses longest prefix match across all EEPGs, would it be possible to add an extra "common" L3Out consisting only of an EEPG with the anycast address and no peerings, so that EEPG would match traffic advertised from the real L3Outs from the regions? Would this be a supported design, or would we risk it breaking down the line when Cisco "fixes" it?
- Use a single floating L3Out, but split it up into node profiles, peering subnets, VLANs and PhysDoms (for the floating SVI) for each region. The problem here lies with the limit of at most 6 anchor nodes (BGP speakers) for floating L3Outs and that leaf pairs most both be anchor or non-anchor, meaning we only have enough anchor nodes for one rack (leaf pair) per region. We do not consider this enough "rack redundancy" within regions, when the regional OpenStack controllers are spread out in different racks.
- Use a single floating L3Out with separate region profiles/subnets/etc. and one "peering rack"/anchor node pair for each region (like above), but set up peering from ACI anchor nodes in all regions to OpenStack BGP speakers in all regions (i.e. cross-region peering). The individual floating SVI node profiles for anchor nodes (under "logical interface profiles") seem to be pretty locked to a single VLAN, so I suppose we can't easily add the regional subnets and VLANs from other regions as secondary addresses on the anchor nodes, meaning we need to use multi-hop BGP for the cross-region peerings. This feels messy ...
- Use a normal L3Out with SVI interfaces. Use bigger peering subnets such that all connected leafs can each get an IP address, even though only BGP speakers really need one. Configure BGP on two pairs in each region. Or is there any way to not configure IP addresses on every single member node?
What is the best approach for such anycast addresses, when trying to bear in mind geo-redundancy and not stretching things across all regions? Or are we being too difficult and should accept a stretched L3Out external bridge domain across all regions as sufficient?
