cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
42954
Views
116
Helpful
6
Replies

Bridge Domain approach

What's the pros and cons for using only one Bridge Domain for multiple EPGs?
In the same tenant we’ve one AP, twenty or more EPGs and one BD. In this case all EPGs share the same BD and communicate between there is across contract. The question is. Do I have any problem or important limitation with this approach?

1 Accepted Solution

Accepted Solutions

RedNectar
VIP
VIP

[Edit 2019-09-11: When you've read my answer, scroll down and check out @cweinhold's pictorial approach to answering this question]

Hi Marcelo,

This question has been around in various forms since the 1990s!

But to be specific to ACI, let me tell you about Application Centric Design and Network Centric Design.

Both are valid design strategies, and you can read more of the official Cisco view here. But basically, what you have described is an Application Centric approach.  A Network Centric Design would put each EPG in its own Bridge Domain and own Subnet.

Now, back to your question.

What's the pros and cons for using only one Bridge Domain for multiple EPGs?

Pros of an Application Centric Design vs Network Centric Design

  1. Application developers don't need to be tied to the Network team for the allocation of new subnets and VLANs to be able to implement policy between groups of servers (EPGs).  Once an initial Bridge Domain and Subnet has been allocated, they can peel off as many EPGs as they want un till they run out of IPs. And even then...
  2. ...if required, the same Bridge Domain can be allocated multiple IP Addresses across multiple subnets, and if this is done...
  3. ...there is no restriction about which servers are in which subnet - the only restrictions come via the Contracts between the EPGs.

Cons of an Application Centric Design vs Network Centric Design

  1. A single Bridge Domain is still a single Broadcast Domain, just as a single VLAN is a broadcast domain in a traditional network - apart from the fact that ACI can reduce ARP broadcast traffic and some BUM traffic (Broadcast, Unknown Unicast and Multicast traffic).  So, you need to remember that...
  2. ...just like since forever, you should be wary of having broadcast domains that are too big. How big is too big? That's another hornet's nest that I won't get into. Ask 10 professional network engineers and you'll get 11 different answers.  But in part it depends on how the following options are enabled on your Bridge Domain:
    1. L2 Unknown Unicast [Not flooded by default - instead sent to the proxy]
    2. L3 Unknown Multicast Flooding [Flooded by default. Can be Optimised so only Multicast Router ports receive L3 Unknown Multicast]
    3. Multi-Destination Flooding. [Flooded in BD by default]
    4. ARP Flooding [Disabled by default. ARP traffic is routed based on the Target IP address]

Assuming you have left the options above at their defaults (which is what happens if you specify Optimize as the Forwarding method when you create the bridge domain) then you will be able to create a larger Bridge Domain than the traditional Broadcast Domain you would have been able to create using traditional networking methods. (Switches/VLANs).  But, there are some limitations on those optimised values, and if you need to change them in the future, you may find that your BD suddenly has to cope with more BUM traffic than before. Specifically:

    1. L2 Unknown Unicast - not much to worry about there, you should never have any valid Unknown L2 traffic, so let it be sent to the Proxy
    2. L3 Unknown Multicast Flooding - Not sure why you wouldn't set this to Optimized Flood, but again, you are unlikely to have to worry about too much unknown Multicast traffic. However, if you DO optimise L3 Unknown Multicast traffic, you are limited to optimising 50 BDs per switch. Which should be plenty in most cases but might a pain to troubleshoot if exceeded.
    3. Multi-Destination Flooding - This is one that can get a little out of control. Multi-Destination means L2 Multicasts and Broadcasts that aren't consumed by the Leaf Switch. (E.g. LLDP Multicasts are consumed by the Leaf Switch).  Capture a few minutes of traffic on a given switch port on your own corporate network (if you are allowed to do that without breaching company policy) and look at all the weird traffic you capture that is NOT being sent to your PC. I did this recently and saw IPX broadcasts and other obsolete protocols floating around.  This is your typical Multi-Destination traffic. Amongst it you will likely see some NetBIOS broadcasts as well as lots or ARP requests. (I'll deal with ARP later).
      1. If you tell ACI to Drop Mult-Destination frames, you may break some protocols - I can't think of any in particular at the moment, but my first suspects would be file sharing applications, like Windows Filesharing or Apple File Sharing and even DropBox.
      2. IMPORTANT In a single BD with Multiple EPGs, each EPG is defined by a different VLAN encapsulation. IF those VLANs extend into you legacy non-ACI network, then (by default) these frames will leak from one VLAN to another - which is probably NOT what you want, which is why there is a Multi-Destination Flooding option to Flood in Encapsulation - to prevent leakage from one EPG to another - but again, this may break some protocols. But before setting your BD to Flood in Encapsulation re-read the IF statement above.
        [Edit - inspired by Manuel's answer regarding Flexpod: Another example of when you can get traffic appearing to leak from one VLAN to other is if a manufacturer uses the same MAC addresses on multiple NICs, then puts the NICs in multiple VLANs and then put those VLANs in the same BD.  The Cisco official answer in this case is to use a Network Centric approach (put each VLAN in its own BD), but I don't see why Flood in Encapsulation wouldn't also work.]
    4. ARP Flooding is the most important of these options.  There are times when you need to enable ARP flooding, such as if you have a silent host on a subnet that does not exist in the ACI environment (if your BD has an IP address on the same subnet as the silent host, ACI will encourage the silent host to speak by sending it an ARP request).  The quintessential case for this is when one of you EPs is in fact a router port or a firewall interface.  So, just remember that IF you have such a situation, it may be worth separating the BD where the Firewall/Router lives into a separate BD if possible.
  1. Application Centric Designs do not follow the normal expectations of network engineers who have to troubleshoot networking issues, so troubleshooting may be more difficult.

Pros of a Network Centric Design vs Application Centric Design

  1. If your situation is one of migrating an existing network to ACI, a Network Centric approach is much more aligned to the existing structure and therefore much easier to implement.  This is pretty much an overriding reason why most ACI implementations end up with at least some part of the design being Network Centric.
  2. A Network Centric approach looks much more like a traditional design.  This gives some people a lot of comfort, especially Network engineers who can't handle the new-fangled software-designed thingy that might threaten their life. The more like the old network it is, the better they like it.
  3. There are some cases, such as when leaking routes between VRFs, that are easier to implement in a network centric design.
  4. When working with Layer 4-7 devices that require traffic to directed to an IP address that is not part of the ACI Fabric (think Firewall), a Network Centric design is easier to work with.

Cons of a Network Centric Design vs Application Centric Design

  1. A network centric design requires more careful IP subnet planning - much the same as it is now.
  2. Application designers and writers of orchestration applications will probably find it harder to work within the restraints of a Network Centric Design.  Think about it. If you are writing an Orchestration Application that is going to spin up half a dozen VMs with policies between them, is it easier to put all servers in the same subnet (Application Centric), or easier to find say 3 different subnets to put the servers in (Network Centric)?

Conclusion

One approach is not necessarily better than the other.

When migrating existing VLANs/Subnets, a Network Centric approach is what I would recommend.

When writing an Orchestration Application, an Application Centric approach is what I would recommend.

If neither is more important, and each is equally easy to implement, I'd recommend sticking with a Network Centric approach because it gives you better flexibility to later implement L4-7 services and control BUM traffic.

 

I hope this helps

 

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem


RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

View solution in original post

6 Replies 6

Manuel Velasco
Cisco Employee
Cisco Employee

In the legacy networks the vlan represents the forwarding domain for L2 traffic, in comparison with ACI the BD represents the forwarding domain for L2 traffic.  

 

Having said that, one limitation I can think of is when a single endpoint, is configured with multiple IPs to send traffic to multiple vlans, but it uses the same NIC (MAC address) to source the traffic.  In this situation endpoint will be flapping between EPGs and this can cause traffic interruptions.  To solve this problem you would need to deploy a separate BD per vlan used by this VM instead of a single BD.

 

 

RedNectar
VIP
VIP

[Edit 2019-09-11: When you've read my answer, scroll down and check out @cweinhold's pictorial approach to answering this question]

Hi Marcelo,

This question has been around in various forms since the 1990s!

But to be specific to ACI, let me tell you about Application Centric Design and Network Centric Design.

Both are valid design strategies, and you can read more of the official Cisco view here. But basically, what you have described is an Application Centric approach.  A Network Centric Design would put each EPG in its own Bridge Domain and own Subnet.

Now, back to your question.

What's the pros and cons for using only one Bridge Domain for multiple EPGs?

Pros of an Application Centric Design vs Network Centric Design

  1. Application developers don't need to be tied to the Network team for the allocation of new subnets and VLANs to be able to implement policy between groups of servers (EPGs).  Once an initial Bridge Domain and Subnet has been allocated, they can peel off as many EPGs as they want un till they run out of IPs. And even then...
  2. ...if required, the same Bridge Domain can be allocated multiple IP Addresses across multiple subnets, and if this is done...
  3. ...there is no restriction about which servers are in which subnet - the only restrictions come via the Contracts between the EPGs.

Cons of an Application Centric Design vs Network Centric Design

  1. A single Bridge Domain is still a single Broadcast Domain, just as a single VLAN is a broadcast domain in a traditional network - apart from the fact that ACI can reduce ARP broadcast traffic and some BUM traffic (Broadcast, Unknown Unicast and Multicast traffic).  So, you need to remember that...
  2. ...just like since forever, you should be wary of having broadcast domains that are too big. How big is too big? That's another hornet's nest that I won't get into. Ask 10 professional network engineers and you'll get 11 different answers.  But in part it depends on how the following options are enabled on your Bridge Domain:
    1. L2 Unknown Unicast [Not flooded by default - instead sent to the proxy]
    2. L3 Unknown Multicast Flooding [Flooded by default. Can be Optimised so only Multicast Router ports receive L3 Unknown Multicast]
    3. Multi-Destination Flooding. [Flooded in BD by default]
    4. ARP Flooding [Disabled by default. ARP traffic is routed based on the Target IP address]

Assuming you have left the options above at their defaults (which is what happens if you specify Optimize as the Forwarding method when you create the bridge domain) then you will be able to create a larger Bridge Domain than the traditional Broadcast Domain you would have been able to create using traditional networking methods. (Switches/VLANs).  But, there are some limitations on those optimised values, and if you need to change them in the future, you may find that your BD suddenly has to cope with more BUM traffic than before. Specifically:

    1. L2 Unknown Unicast - not much to worry about there, you should never have any valid Unknown L2 traffic, so let it be sent to the Proxy
    2. L3 Unknown Multicast Flooding - Not sure why you wouldn't set this to Optimized Flood, but again, you are unlikely to have to worry about too much unknown Multicast traffic. However, if you DO optimise L3 Unknown Multicast traffic, you are limited to optimising 50 BDs per switch. Which should be plenty in most cases but might a pain to troubleshoot if exceeded.
    3. Multi-Destination Flooding - This is one that can get a little out of control. Multi-Destination means L2 Multicasts and Broadcasts that aren't consumed by the Leaf Switch. (E.g. LLDP Multicasts are consumed by the Leaf Switch).  Capture a few minutes of traffic on a given switch port on your own corporate network (if you are allowed to do that without breaching company policy) and look at all the weird traffic you capture that is NOT being sent to your PC. I did this recently and saw IPX broadcasts and other obsolete protocols floating around.  This is your typical Multi-Destination traffic. Amongst it you will likely see some NetBIOS broadcasts as well as lots or ARP requests. (I'll deal with ARP later).
      1. If you tell ACI to Drop Mult-Destination frames, you may break some protocols - I can't think of any in particular at the moment, but my first suspects would be file sharing applications, like Windows Filesharing or Apple File Sharing and even DropBox.
      2. IMPORTANT In a single BD with Multiple EPGs, each EPG is defined by a different VLAN encapsulation. IF those VLANs extend into you legacy non-ACI network, then (by default) these frames will leak from one VLAN to another - which is probably NOT what you want, which is why there is a Multi-Destination Flooding option to Flood in Encapsulation - to prevent leakage from one EPG to another - but again, this may break some protocols. But before setting your BD to Flood in Encapsulation re-read the IF statement above.
        [Edit - inspired by Manuel's answer regarding Flexpod: Another example of when you can get traffic appearing to leak from one VLAN to other is if a manufacturer uses the same MAC addresses on multiple NICs, then puts the NICs in multiple VLANs and then put those VLANs in the same BD.  The Cisco official answer in this case is to use a Network Centric approach (put each VLAN in its own BD), but I don't see why Flood in Encapsulation wouldn't also work.]
    4. ARP Flooding is the most important of these options.  There are times when you need to enable ARP flooding, such as if you have a silent host on a subnet that does not exist in the ACI environment (if your BD has an IP address on the same subnet as the silent host, ACI will encourage the silent host to speak by sending it an ARP request).  The quintessential case for this is when one of you EPs is in fact a router port or a firewall interface.  So, just remember that IF you have such a situation, it may be worth separating the BD where the Firewall/Router lives into a separate BD if possible.
  1. Application Centric Designs do not follow the normal expectations of network engineers who have to troubleshoot networking issues, so troubleshooting may be more difficult.

Pros of a Network Centric Design vs Application Centric Design

  1. If your situation is one of migrating an existing network to ACI, a Network Centric approach is much more aligned to the existing structure and therefore much easier to implement.  This is pretty much an overriding reason why most ACI implementations end up with at least some part of the design being Network Centric.
  2. A Network Centric approach looks much more like a traditional design.  This gives some people a lot of comfort, especially Network engineers who can't handle the new-fangled software-designed thingy that might threaten their life. The more like the old network it is, the better they like it.
  3. There are some cases, such as when leaking routes between VRFs, that are easier to implement in a network centric design.
  4. When working with Layer 4-7 devices that require traffic to directed to an IP address that is not part of the ACI Fabric (think Firewall), a Network Centric design is easier to work with.

Cons of a Network Centric Design vs Application Centric Design

  1. A network centric design requires more careful IP subnet planning - much the same as it is now.
  2. Application designers and writers of orchestration applications will probably find it harder to work within the restraints of a Network Centric Design.  Think about it. If you are writing an Orchestration Application that is going to spin up half a dozen VMs with policies between them, is it easier to put all servers in the same subnet (Application Centric), or easier to find say 3 different subnets to put the servers in (Network Centric)?

Conclusion

One approach is not necessarily better than the other.

When migrating existing VLANs/Subnets, a Network Centric approach is what I would recommend.

When writing an Orchestration Application, an Application Centric approach is what I would recommend.

If neither is more important, and each is equally easy to implement, I'd recommend sticking with a Network Centric approach because it gives you better flexibility to later implement L4-7 services and control BUM traffic.

 

I hope this helps

 

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem


RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

Your Server or Application

garden-app.jpg

 

Your Current Data Center

garden-dc.jpg

 

English Garden (aka "Network-Centric")

garden-english.jpg

Trees remain in place. You have tools to add incremental security improvements around them.

 

French Garden (aka "Application-Centric")

garden-french.jpg

You cut down all your trees and plant new ones that are all exactly the same shape and height and placed in a prescribed location.

 

English Garden with some French accents (Hybrid?)

garden-hybrid.jpg

Legacy forest intermixed with some highly-structured areas. 95% of my ACI deployments are Hybrid.

 

Tenants - ACI's most powerful, unsung feature

garden-tenant.jpg

Tenants allow you to deploy as many gardens as you want across the same physical & virtual infrastructure with shared northbound networking. You can deploy whatever garden that networking green thumb of yours can dream up! 

 

You are awesome....... 

And what about L2 security ? I mean, with a single bridge domain (and a single large subnet over it) with many EPG, how does the fabric mitigates (or not) the risk of arp poisoning ?

Host1 in EPG1 is compromised and send a malicious garp/arp response for every address in the subnet. The arp table of the fabric and the arp table of every endpoint in the BD will be corrupted (if the arp processing mode is "flooding"), am i right ?

Every host in every EPG will be affected, EPG isolation and contracts don't come into play as it happens at a lower level, ie at bridge domain level.

Contracts should still prevent host1 from talking to other hosts in other EPG, but they will be affected as long as the invalid arp entries will be kept in their peers table (who won't be able to talk to the legitimate hosts as they will have the malicious entry sent by host1 in their table, disrupting communication for every affected host).

Am I my missing something because if i am not, application centric configuration (many EPG on same BD) sensibility to L2 attacks could be a big concern :(

 

Manuel Velasco
Cisco Employee
Cisco Employee
Hi Marcelo,

The Cisco document below, explains a similar scenario like the one I explained. See snip below

"Bridge Domain: A bridge domain represents a L2 forwarding construct within the fabric. One or more EPG can be associated with one bridge domain or subnet. A bridge domain can have one or more subnets associated with it. One or more bridge domains together form a tenant network. For FlexPod design, setting up a bridge domain is an important consideration. A bridge domain in ACI is equivalent to a broadcast layer-2 domain in traditional Ethernet networks. When a bridge domain contains endpoints belonging to different VLANs (outside of ACI fabric), a unique MAC address is required for every unique endpoint. NetApp controllers, however, use the same MAC address for an interface group and all the VLANs defined for that interface group. As a result, all the LIFs on NetApp end up sharing a single MAC address even though these LIFs belong to different VLANs.

network port show -fields mac
node port mac
------------ ---- -----------------
FLEXPODCL-01 a0a 02:a0:98:51:f6:a4
FLEXPODCL-01 a0a-317002:a0:98:51:f6:a4 (NFS)
FLEXPODCL-01 a0a-90102:a0:98:51:f6:a4 (iSCSI-A)
FLEXPODCL-01 a0a-90202:a0:98:51:f6:a4 (iSCSI-B)
To overcome potential issues caused by overlapping MAC addresses, multiple bridge domains need to be deployed for correct storage connectivity. The details of required bridge domains are covered in the design section below."

Cisco ACI and Flexpod white paper
https://www.cisco.com/c/dam/en/us/td/docs/unified_computing/ucs/UCS_CVDs/sharepoint2013_aci_flexpod_vmware.pdf

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Save 25% on Day-2 Operations Add-On License