[Edit 2019-09-11: When you've read my answer, scroll down and check out @cweinhold's pictorial approach to answering this question]
Hi Marcelo,
This question has been around in various forms since the 1990s!
But to be specific to ACI, let me tell you about Application Centric Design and Network Centric Design.
Both are valid design strategies, and you can read more of the official Cisco view here. But basically, what you have described is an Application Centric approach. A Network Centric Design would put each EPG in its own Bridge Domain and own Subnet.
Now, back to your question.
What's the pros and cons for using only one Bridge Domain for multiple EPGs?
Pros of an Application Centric Design vs Network Centric Design
- Application developers don't need to be tied to the Network team for the allocation of new subnets and VLANs to be able to implement policy between groups of servers (EPGs). Once an initial Bridge Domain and Subnet has been allocated, they can peel off as many EPGs as they want un till they run out of IPs. And even then...
- ...if required, the same Bridge Domain can be allocated multiple IP Addresses across multiple subnets, and if this is done...
- ...there is no restriction about which servers are in which subnet - the only restrictions come via the Contracts between the EPGs.
Cons of an Application Centric Design vs Network Centric Design
- A single Bridge Domain is still a single Broadcast Domain, just as a single VLAN is a broadcast domain in a traditional network - apart from the fact that ACI can reduce ARP broadcast traffic and some BUM traffic (Broadcast, Unknown Unicast and Multicast traffic). So, you need to remember that...
- ...just like since forever, you should be wary of having broadcast domains that are too big. How big is too big? That's another hornet's nest that I won't get into. Ask 10 professional network engineers and you'll get 11 different answers. But in part it depends on how the following options are enabled on your Bridge Domain:
- L2 Unknown Unicast [Not flooded by default - instead sent to the proxy]
- L3 Unknown Multicast Flooding [Flooded by default. Can be Optimised so only Multicast Router ports receive L3 Unknown Multicast]
- Multi-Destination Flooding. [Flooded in BD by default]
- ARP Flooding [Disabled by default. ARP traffic is routed based on the Target IP address]
Assuming you have left the options above at their defaults (which is what happens if you specify Optimize as the Forwarding method when you create the bridge domain) then you will be able to create a larger Bridge Domain than the traditional Broadcast Domain you would have been able to create using traditional networking methods. (Switches/VLANs). But, there are some limitations on those optimised values, and if you need to change them in the future, you may find that your BD suddenly has to cope with more BUM traffic than before. Specifically:
- L2 Unknown Unicast - not much to worry about there, you should never have any valid Unknown L2 traffic, so let it be sent to the Proxy
- L3 Unknown Multicast Flooding - Not sure why you wouldn't set this to Optimized Flood, but again, you are unlikely to have to worry about too much unknown Multicast traffic. However, if you DO optimise L3 Unknown Multicast traffic, you are limited to optimising 50 BDs per switch. Which should be plenty in most cases but might a pain to troubleshoot if exceeded.
- Multi-Destination Flooding - This is one that can get a little out of control. Multi-Destination means L2 Multicasts and Broadcasts that aren't consumed by the Leaf Switch. (E.g. LLDP Multicasts are consumed by the Leaf Switch). Capture a few minutes of traffic on a given switch port on your own corporate network (if you are allowed to do that without breaching company policy) and look at all the weird traffic you capture that is NOT being sent to your PC. I did this recently and saw IPX broadcasts and other obsolete protocols floating around. This is your typical Multi-Destination traffic. Amongst it you will likely see some NetBIOS broadcasts as well as lots or ARP requests. (I'll deal with ARP later).
- If you tell ACI to Drop Mult-Destination frames, you may break some protocols - I can't think of any in particular at the moment, but my first suspects would be file sharing applications, like Windows Filesharing or Apple File Sharing and even DropBox.
- IMPORTANT In a single BD with Multiple EPGs, each EPG is defined by a different VLAN encapsulation. IF those VLANs extend into you legacy non-ACI network, then (by default) these frames will leak from one VLAN to another - which is probably NOT what you want, which is why there is a Multi-Destination Flooding option to Flood in Encapsulation - to prevent leakage from one EPG to another - but again, this may break some protocols. But before setting your BD to Flood in Encapsulation re-read the IF statement above.
[Edit - inspired by Manuel's answer regarding Flexpod: Another example of when you can get traffic appearing to leak from one VLAN to other is if a manufacturer uses the same MAC addresses on multiple NICs, then puts the NICs in multiple VLANs and then put those VLANs in the same BD. The Cisco official answer in this case is to use a Network Centric approach (put each VLAN in its own BD), but I don't see why Flood in Encapsulation wouldn't also work.]
- ARP Flooding is the most important of these options. There are times when you need to enable ARP flooding, such as if you have a silent host on a subnet that does not exist in the ACI environment (if your BD has an IP address on the same subnet as the silent host, ACI will encourage the silent host to speak by sending it an ARP request). The quintessential case for this is when one of you EPs is in fact a router port or a firewall interface. So, just remember that IF you have such a situation, it may be worth separating the BD where the Firewall/Router lives into a separate BD if possible.
- Application Centric Designs do not follow the normal expectations of network engineers who have to troubleshoot networking issues, so troubleshooting may be more difficult.
Pros of a Network Centric Design vs Application Centric Design
- If your situation is one of migrating an existing network to ACI, a Network Centric approach is much more aligned to the existing structure and therefore much easier to implement. This is pretty much an overriding reason why most ACI implementations end up with at least some part of the design being Network Centric.
- A Network Centric approach looks much more like a traditional design. This gives some people a lot of comfort, especially Network engineers who can't handle the new-fangled software-designed thingy that might threaten their life. The more like the old network it is, the better they like it.
- There are some cases, such as when leaking routes between VRFs, that are easier to implement in a network centric design.
- When working with Layer 4-7 devices that require traffic to directed to an IP address that is not part of the ACI Fabric (think Firewall), a Network Centric design is easier to work with.
Cons of a Network Centric Design vs Application Centric Design
- A network centric design requires more careful IP subnet planning - much the same as it is now.
- Application designers and writers of orchestration applications will probably find it harder to work within the restraints of a Network Centric Design. Think about it. If you are writing an Orchestration Application that is going to spin up half a dozen VMs with policies between them, is it easier to put all servers in the same subnet (Application Centric), or easier to find say 3 different subnets to put the servers in (Network Centric)?
Conclusion
One approach is not necessarily better than the other.
When migrating existing VLANs/Subnets, a Network Centric approach is what I would recommend.
When writing an Orchestration Application, an Application Centric approach is what I would recommend.
If neither is more important, and each is equally easy to implement, I'd recommend sticking with a Network Centric approach because it gives you better flexibility to later implement L4-7 services and control BUM traffic.
I hope this helps
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem
RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.
