Understanding Traffic Interruptions in ACI caused by Excessive TCN/STP

[ayee]

Recently we had a loss of connectivity of a bunch of endpoints within an EPG, and using the troubleshooting steps found here we managed to track it down the cause of the TCN's due to a reboot of an ESX server.  The ESX server in question is connected to our legacy side 6509-VSS which is in-turn connected to a pair of leaf switches with a legacy L2 out.  The gateway for this particular EPG has already been migrated over with ACI handling all the routing.  All L2 links from ACI to the server are port-channels with the exception of the link for the server itself.  The server is connected via only a single 1Gb link to a 6509 switchport with portfast NOT enabled.

Is this expected behavior?  Will enabling portfast fix this?  Is this somehow caused by the vswitch in ESX?

[julian.bendix]

Hi!

ACI doesn't participate in the Spanning Tree process, but upon receiving a TCN, ACI flushes all Endpoints in the concerned EPG(s)..

So yes, reboots of an ESXi Server which is connected to a port on a Cat6509 which has no portfast configured can cause that..
But one reboot would only cause that two times, once the port goes down and once when the port comes back up.

I don't think one reboot of an ESXi has caused loss of connectivity severe enough to "feel" it..
Maybe one or more of such ports were flapping?

And yes, if that was indeed the root cause, setting portfast on the ports facing the ESXi will fix it.
Generally you should enable portfast on all ports facing endpoints that do not participate in Spanning Tree, also servers.
.. I mean outside of ACI of course.

Cisco Switches (non-ACI) sent STP TCNs every time a link changes, you are turning that of by setting concerned links to portfast enabled..
Only do that on ports facing endpoints of course!  

Best regards
Juls