APIC issues-Excessive STP TCN flushes

[faizal_vi]

Hi,

I am having few access switches uplinked to the APIC fabric. These access switches do not have any inter links between each other. The endpoints in each access switches communicate via APIC only. I have some issues with STP with this setup.

Whenever any STP event happens in the access switches, the complete APIC Fabric is affected bringing down the communication. Please refer the screenshot for the events from APIC. Is there any way that I can stop the STP frames from entering the FABIC.

Can Ienable BPDU-GUARD filter on the uplink ports to the APIC? I mean on FABRIC side or Access Switch uplink port?

Appreciate your inputs on this

[RedNectar · Accepted solution]

Hi faizal_vi,

Regarding your comments & question:

---

I am having few access switches uplinked to the APIC fabric. These access switches do not have any inter links between each other. The endpoints in each access switches communicate via APIC only. I have some issues with STP with this setup.

If the switches do not have any links to other switches, the only STP TCN events you should be seeing is when a port goes up or down. I suggest you check that topology. 

Whenever any STP event happens in the access switches, the complete APIC Fabric is affected bringing down the communication.

When an ACI switch sees a TCN, it dumps the MAC address table, and therefore the Local Station Table for that VLAN on that switch. This is what a normal switch would do too, so again it may NOT be the ACI fabric that is causing the loss of communication, but it certainly could cause some interruption. 

Please refer the screenshot for the events from APIC. Is there any way that I can stop the STP frames from entering the FABIC.

Yes, but be careful.

Can I enable BPDU-GUARD filter on the uplink ports to the APIC? I mean on FABRIC side or Access Switch uplink port?

Yes. You enable BPDU guard by creating a STP Interface Policy and adding that policy to the Interface Policy Group you use to connect your switches.

To create the STP Interface Policy:

Fabric > Access Policies > Interface Policies > Polices > Spanning Tree Interface

• (+) Create Spanning Tree Interface Policy
  • Name: Enable-BPDU.Guard
    • Interface Controls: (x) BPDU Guard Enabled

You might also consider creating STP Interface Policies for BPDU Filter and both BPDU Filter and BPDU Guard together as well, becasue in the end you probably don't want to apply a BPDU guard policy on interfaces connected to switches.

Fabric > Access Policies > Interface Policies > Polices > Spanning Tree Interface

• (+) Create Spanning Tree Interface Policy
  • Name: Enable-BPDU.Filter
    • Interface Controls: (x) BPDU Filter Enabled

Fabric > Access Policies > Interface Policies > Polices > Spanning Tree Interface

• (+) Create Spanning Tree Interface Policy
  • Name: Enable-BPDU.FilterAndGuard
    • Interface Controls: (x) BPDU Filter Enabled
      (x) BPDU Guard Enabled
      

To apply the STP Interface Policy to the Interface Policy Group:

I have no idea which Policy group you have applied to the interfaces but you'll find it here:

Fabric > Access Policies > Interface Policies > Policy Groups > YourInterfacePolicyGroup 

Locate the  Spanning Tree Interface policy for YourInterfacePolicyGroup and change it's value to Enable-BPDU.Guard (the policy you just created) then apply the policy.

But BPDU Guard may NOT be the policy you want to apply. Best practice is that you enable BPDU Guard on interfaces that are connected to servers/workstations only.

Once a BPDU Gurad policy has been applied, the interface where the BPDU arrives at the fabric will be shut down, again disrupting traffic.  Possibly what you want to do is enable a BPDU filter instead, but again I urge you to double check your topology to ensure that there is no possible loop on any VLAN.

Also, you proabably should make sure you understand how STP BPDUs are treated by ACI. Cisco proprietary PVSTP and RPVSTP BPDUs are treated differently to IEEE 802.1D MST BPDUs. (PVST/RPVSTP BPDUs are flooded within the End Point Group that is mapped to the VLAN encapsulation, for MST BPDUs it is considered best practice to create a special EPG for all untagged traffic per MST Domain to ensure the MST BPDUs are flooded where needed. Suggest reading BRKACI-3101 to learn more)

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[faizal_vi]

Hi Chris,

Thank you for your input on this.

I would like add few more points regarding the design.

We have mapped each vlan to Bridge Domain to EPG. Same VLAN/EPG are spread across the different access layer switch. What I understood is that ACI fabric do not participate in STP. I would like to know whether STP frames from one Access Layer switch are carried out to other access layer switch..? If I enable STP bpdu-filter on the up link to switches , will there be any issue considering the topology we have..

I am seeing in all access layer switch the STP root bridge for all the vans are single MAC. But I am unable to identify which is this device.? The below pic will provide you a better understanding of our topology. We have more access layer switches than what is mentioned in the diagram.

[RedNectar · Accepted solution]

Hi 

Beautiful pic. Thanks for taking the time to add it.

So long as those access switches are not connected to each other in any way other than via the ACI Fabric, and there is only one VPC per access switch* you will have no problem if you apply a BPDU Filter to the VPC Interface Policy Groups that define the VPCs in your diagram. This is what should happen.

BEFORE Applying the BPDU Filter:

BPDU frames arrive on one of the VPC links and are flooded throughout the EPG based on the VLAN tag on the BPDUs (For PVST/RPVST: VLAN tag=VLAN ID; for MST: VLAN tag=untagged)

That would explain why you see the same root switch for all VLANs - the BPDUs are getting trough as expected and so the remote switches can determine the root.

Although it worries me that you say you haven't yet identified which device is the root.

AFTER Applying the BPDU Filter to the VPC Interface Policy groups:

BPDU frames arriving on any of the VPC links are dropped.

Now each switch should see itself as the root of the spanning tree, because it is not seeing the DPDUs from the other switches.

One Final Note about the Baremetal hosts in the server farm:

Are these servers ESXi servers, or hypervisors of any kind?  If so, keep in mind that those two links go to a Virtual Switch, and should that virtual switch decide to send any frames up the redundant link (I know, it shouldn't unless there is a failure... but how does the vswitch detect a failure - if it is detecting failure based on beacon frames, the beacon frames may not be passing through the ACI fabric... Have a read of this vmware doc on NIC teaming) then you will have a problem.  I have a sneaking suspicion that the STP changes you see may be a result of some uninteded behaviour in your dual-attached Bare Metal hosts.  My advice (if possible and can be done in a suitable admistrative window) would be to shut down the redundant links to the bare metal servers before you try any BPDU filters. If the STP TCN messages stop, then it would indicate that there is a potential problem there.

---

* I have seen more than one VPC configured to the same access switch, one for each tenant. That can certainly cause STP problems.

[josecarlos]

Hello my frind, how are you? Can you share with me how you resolve this problem, please?

I have same question. You enable bpdu filter on interfaces of access switchs?

[faizal_vi]

Hi,

Its been an year ago. We applied  the BPDU filter on the APIC side. And the issue was resolved. 

[Timothy Riegert]

Do you have the portfast / port type edge configured for the end host ports on the access switches?

https://dpitaci.wordpress.com/2016/04/25/intermittent-traffic-due-to-tcn/