ACI Spanning Tree Design

[pengus]

I want to connect ACI to two Nexus switches in Layer 2 mode. All documentation states that the interfaces on the Nexus switches facing ACI should be set to shared mode. However, I have a separate VLAN pool for L2Outs, and the EPGs extended with L2Out are in a different VLAN pool. Will BPDUs coming from the outside cause the ACI EPG table to be flushed?

[AshSe]

Hey @pengus , When connecting ACI to Nexus switches in Layer 2 mode, it's crucial to understand how ACI handles BPDUs (Bridge Protocol Data Units) and VLAN pools to ensure network stability and proper configuration.

1. Shared Mode on Nexus Switches: Setting the interfaces on the Nexus switches facing ACI to shared mode is recommended because it allows multiple VLANs to be carried over the same physical link. This is essential for extending Layer 2 domains across the ACI fabric.

2. Separate VLAN Pools: Having separate VLAN pools for L2Outs and EPGs is a common practice to segregate traffic and apply different policies. However, this segregation does not inherently protect against BPDU-related issues.

3. BPDU Handling in ACI: ACI fabric is designed to handle BPDUs in a specific way. When BPDUs are received on an interface, ACI typically processes them to prevent loops and ensure network stability. However, if BPDUs are not properly managed, they can cause issues such as flushing of the EPG table.

4. BPDU Guard and Filtering: To prevent BPDUs from causing disruptions, you can enable BPDU Guard or BPDU Filtering on the interfaces connecting to the Nexus switches. BPDU Guard will shut down the port if a BPDU is received, while BPDU Filtering will drop BPDUs and prevent them from being processed.

5. Configuration Recommendations:

   1. Enable BPDU Guard: On the ACI interfaces facing the Nexus switches, enable BPDU Guard to protect against potential loops and BPDU-related issues.
   2. Use Consistent VLAN Pools: Ensure that the VLAN pools used for L2Outs and EPGs are consistently applied and properly configured to avoid VLAN mismatches.
   3. Monitor BPDU Activity: Regularly monitor BPDU activity on the ACI fabric to detect and address any potential issues early.

   Would appreciate a diagram of your setup if you would like me to take a more better and deeper dive in to the issue.

   Hope This Helps!!!

   AshSe

   Forum Tips: 

   1. Paste images inline - don't attach.

   2. Always mark helpful and correct answers, it helps others find what they need.

[pengus]

ACI Border are in VPC groups. However, Nexus is not a VPC pair because of a version mismatch (an upgrade is planned to equalize the versions). The thing that confuses me, or that I don't fully understand, is this: FD VLAN seems to consist of two parameters. The first is the VLAN ID, and the second is the VLAN pool/domain (in the access policy). Also, BPDUs cause the deletion of entries with the incoming FD ID on the leaves. In this case, it seems to me that the EPGs within ACI can continue to communicate with each other without interruption.

[AshSe]

Hey @pengus , Before giving answer to your doubt; let me break your question in to smaller pieces for better understanding: 

• Nexus Shared Mode = Interface Trunk configuration = Allow multiple VLANs on an interface
• FD VLAN = Flood Domain VLAN

---

 this: FD VLAN seems to consist of two parameters. The first is the VLAN ID, and the second is the VLAN pool/domain (in the access policy). 

---

FD VLAN is a specific VLAN used within the ACI fabric to handle broadcast, unknown unicast, and multicast (BUM) traffic.  FD VLAN itself is a specific VLAN configured automatically and used internally by the ACI fabric. Below are the key components related to the FD VLAN:

1. Bridge Domain (BD):

   1. A bridge domain is a Layer 2 forwarding construct within ACI. It defines the scope of Layer 2 flooding and is associated with one or more subnets.
   2. Each bridge domain can have its own flood domain, which is managed using the FD VLAN.

2. Endpoint Groups (EPGs):

   1. Endpoint Groups are logical groupings of endpoints (such as virtual machines, physical servers, etc.) that share common policy requirements.
   2. EPGs are associated with bridge domains, and the traffic between EPGs within the same bridge domain can be subject to Layer 2 flooding.

3. Flood Domain:

   1. The flood domain defines the scope of BUM traffic within a bridge domain. It ensures that such traffic is contained within the specified boundaries.
   2. The FD VLAN is used internally by the ACI fabric to manage this flood domain.

4. VLAN Pools:

   1. VLAN pools are collections of VLANs that can be dynamically assigned to EPGs. These VLANs are used for encapsulating traffic within the ACI fabric.
   2. The FD VLAN is part of the internal VLAN pool used by ACI to manage flood domains.

5. Contracts:

   1. Contracts define the policies for communication between EPGs. They can include filters and actions that control the flow of traffic.
   2. While contracts primarily manage inter-EPG traffic, they can also influence how BUM traffic is handled within a bridge domain.

6. Multicast Groups:

   1. Multicast groups are used to manage multicast traffic within the ACI fabric. They are part of the broader flood domain management.
   2. The FD VLAN helps in isolating multicast traffic within the defined flood domain.

7. Spine and Leaf Switches:

   1. The ACI fabric is built using spine and leaf switches. These switches work together to forward traffic, including BUM traffic, based on the policies defined in the ACI fabric.
   2. The FD VLAN is used by these switches to manage the flood domain traffic.

8. Policy Enforcement:

   1. Policies in ACI define how traffic is handled, including BUM traffic. These policies are enforced at the leaf switches.
   2. The FD VLAN is part of the internal mechanisms that ensure these policies are applied correctly.

   ---

   Also, BPDUs cause the deletion of entries with the incoming FD ID on the leaves. In this case, it seems to me that the EPGs within ACI can continue to communicate with each other without interruption.

   ---

   In Cisco ACI, Bridge Protocol Data Units (BPDUs) are used for Spanning Tree Protocol (STP) operations, which are essential for preventing loops in traditional Layer 2 networks. However, ACI operates differently from traditional networks, and its handling of BPDUs is unique due to its fabric-based architecture.

   In ACI, the concept of Flood Domain VLAN (FD VLAN) is used to manage broadcast, unknown unicast, and multicast (BUM) traffic within the fabric. The handling of BPDUs in ACI is designed to ensure that the fabric can interoperate with traditional Layer 2 networks while maintaining its own internal mechanisms for traffic management.

   Regarding your statement that BPDUs cause the deletion of entries with the incoming FD ID on the leaves, here are some key points to consider:

   1. BPDU Handling in ACI:

      1. ACI can be configured to either filter or forward BPDUs. By default, ACI filters BPDUs to prevent external STP from influencing the internal fabric.
      2. When BPDUs are filtered, they are not forwarded within the ACI fabric, and the internal forwarding tables are not affected by these BPDUs.

   2. Flood Domain and FD VLAN:

      1. The FD VLAN is used internally by ACI to manage BUM traffic within a bridge domain. It is not directly influenced by external BPDUs.
      2. The flood domain ID (FD ID) is an internal identifier used by ACI to manage the scope of flooding within the fabric.

   3. Entry Deletion:

      1. In ACI, the deletion of entries in the forwarding tables (such as MAC address entries) is typically based on aging timers or specific network events, not directly on the reception of BPDUs.
      2. BPDUs themselves do not cause the deletion of entries with the incoming FD ID on the leaves. Instead, the handling of BPDUs is managed separately from the internal mechanisms that control the flood domain and FD VLAN.

   4. Interoperability with Traditional Networks:

      1. When ACI is connected to traditional Layer 2 networks, it can participate in STP to ensure loop-free topologies. However, this participation is controlled and does not directly impact the internal flood domain management.
      2. ACI can be configured to allow BPDUs to pass through specific ports if needed for interoperability, but this is done with careful consideration to avoid disrupting the internal fabric operations.

   In summary, BPDUs do not cause the deletion of entries with the incoming FD ID on the leaves in Cisco ACI. The handling of BPDUs is managed separately from the internal flood domain mechanisms, and ACI's architecture ensures that BUM traffic and flood domain management are not directly influenced by external STP operations.

   Hope This Helps!!!

   AshSe

   Forum Tips: 

   1. Insert photos/images inline - don't attach.

   2. Always mark helpful and correct answers, it helps others find what they need.

[pengus]

Nexus Shared Mode = Interface Trunk configuration = Allow multiple VLANs on an interface
Sorry for the confusion of concepts. Here I wanted to express that the spanning tree interface type should be shared instead of the default port type of the spanning tree, point to point. Generally recommended mode to use if there is a hub between the switches.

BPDUs do not cause the deletion of entries with the incoming FD ID on the leaves in Cisco ACI. Yes correct i want to express TCN packet.
https://unofficialaciguide.com/2019/03/28/spanning-tree-stp-and-aci/ i referenced this document.
"TCNs generated from switches running Spanning-tree will cause ACI to flush endpoints from EPGs in which they are received. This can result in intermittent traffic for devices on those EPGs. If you want to know more about this, check out this article – STP and ACI: Intermittent packet loss due to TCNs."

I have inferred from the above sentence as follows: I extended an EPG marked with VLAN 10 to a legacy network with a Layer 2 Out over ACI. A TCN (Topology Change Notification) from the legacy network only affects and flushes MAC addresses outside the ACI fabric within the scope of Layer 2 Out. Does this mean that the EPG marked with VLAN 10 on the ACI fabric will not be affected?

https://unofficialaciguide.com/2019/03/28/stp-and-aci-intermittent-packet-loss-due-to-tcns/ 

According to this document, if I connect a legacy network to the ACI fabric using Layer 2 Out, I understand that a TCN (Topology Change Notification) coming from the legacy environment will not affect communication between EPGs within the ACI fabric. However, if the gateway for these EPGs is in the legacy network, it seems there may only be interruptions in their communication with the outside world.

[AshSe · Accepted solution]

---

@pengus 

BPDUs do not cause the deletion of entries with the incoming FD ID on the leaves in Cisco ACI. Yes correct i want to express TCN packet.
https://unofficialaciguide.com/2019/03/28/spanning-tree-stp-and-aci/ i referenced this document.
"TCNs generated from switches running Spanning-tree will cause ACI to flush endpoints from EPGs in which they are received. This can result in intermittent traffic for devices on those EPGs. If you want to know more about this, check out this article – STP and ACI: Intermittent packet loss due to TCNs."

---

In Cisco Application Centric Infrastructure (ACI), the handling of Topology Change Notifications (TCNs) generated by switches running Spanning Tree Protocol (STP) can indeed impact endpoint learning and aging. However, the specific behavior depends on the configuration and integration of the ACI fabric with the traditional network.

When a TCN is received by an ACI leaf switch, it can trigger the ACI fabric to update its endpoint learning and aging processes. This is because a TCN indicates a change in the network topology, which could mean that endpoints have moved or that there are changes in the path to reach those endpoints.

---

I have inferred from the above sentence as follows: I extended an EPG marked with VLAN 10 to a legacy network with a Layer 2 Out over ACI. A TCN (Topology Change Notification) from the legacy network only affects and flushes MAC addresses outside the ACI fabric within the scope of Layer 2 Out. Does this mean that the EPG marked with VLAN 10 on the ACI fabric will not be affected?

---

Yes, you are right. a TCN from the legacy network will primarily affect the MAC addresses learned through the Layer 2 Out interface, causing them to be flushed and re-learned. The EPG marked with VLAN 10 within the ACI fabric itself will not be directly affected by the TCN, and its internal MAC addresses will remain stable. This ensures that the internal ACI fabric maintains its endpoint learning and forwarding stability while accommodating changes in the external legacy network.