ACI vPC to C2960X Stack err-disable ports/PO.

[SIMMN]

I have a two 2960x unit stack I need to uplink into the ACI fabric (via 10Gbps ports). 

On ACI (v4.2(5l)), I set the interface group policy with LACP mode Active (screenshot below); STP BPDU Guard off; STP BPDU Filter off. 

On the 2960x stack (v15.2.7E2), the configuration is below and I do not think it can be simpler...

interface Port-channel11
switchport trunk allowed vlan 10-20
switchport mode trunk
!
interface TenGigabitEthernet1/0/1
switchport trunk allowed vlan 10-20
switchport mode trunk
channel-group 11 mode active
!
interface TenGigabitEthernet2/0/1
switchport trunk allowed vlan 10-20
switchport mode trunk
channel-group 11 mode active
!

Once I have the cables plugged in, within about 30-sec to 1-min, the 2960 would err-disable the ports and PO...Log similiar below. BTW, I do have other working switches/stacks uplink into ACI with the same PO setting...

%PM-4-ERR_DISABLE: channel-misconfig (STP) error detected on Po11, putting Po11 in err-disable state

I tried to remove all the LACP controls and even tried static PO and LACP passive on ACI, made no difference. 2960 always err-disable the ports... The only way I found to keep the PO running is to STP BPDU Guard off; STP BPDU Filter on for the ACI vPC group policy... (but this makes the 2960X stack STP root for all the available VLANs which I do not like...)

Here below is the output of "show spanning-tree summary" from the 2960X.

EtherChannel misconfig guard is enabled
Extended system ID is enabled
Portfast Default is disabled
Portfast Edge BPDU Guard Default is disabled
Portfast Edge BPDU Filter Default is disabled
Loopguard Default is disabled
PVST Simulation Default is enabled but inactive in rapid-pvst mode
Bridge Assurance is enabled
UplinkFast is disabled
BackboneFast is disabled

Suggestions?

[julian.bendix]

Hey!

The log message on the Catalyst 2960X indicates, that different BPDUs are received on the two different interfaces of the port channel...
That should not happen with a port-channel.

Could you show the output of "show etherchannel summary" of the Catalyst?

Thanks and br

Juls

[SIMMN]

"That should not happen with a port-channel."

I would not post here if the issue isn't strange...)

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Te1/0/1(P)    Te2/0/1(P)

[julian.bendix]

Can you also share "show cdp neigh" from the Catalyst?

Seems really strange so far.

[Hector Gustavo Serrano Gutierrez]

Hi @SIMMN,

On your Catalyst Switch, could you please do no spanning-tree etherchannel guard misconfig?

I suspect this will solve this situation.

Best Regards.

[SIMMN]

Thanks! "no spanning-tree etherchannel guard misconfig" does keep the PO up on 2960 even the ACI side is configured for STP BPDU Guard off; STP BPDU Filter off...

But why? According to Cisco "The EtherChannel Guard feature is used to detect EtherChannel misconfigurations between the switch and a connected device....EtherChannel parameters must be the same on both sides for the guard to work."

I also have a two unit C9300 stack, N5Ks connected into ACI and none of those have this command in... So what parameters are mismatched?

[Hector Gustavo Serrano Gutierrez]

I'm glad it is working OK now.

ACI acts like a Hub when it comes to STP BPDUs, when received from an interface associated to an EPG, it just forwards those out all other interfaces associated with the same EPG (and encap vlan) on the same or other Leaf Switches. ACI Leaf switches do not generate STP BPDUs by themselves. When you are connecting the C2960X to ACI, I suspect that STP converges and your Catalyst ends up receiving the STP BPDUs generated by your different external Switches connected to ACI. Since the Catalyst's Port-channel interprets this as a mis-configuration as it has received different STP BPDUs coming its port-channel interface, the error happens. Etherchannel guard misconfig monitors for the Source Mac address of STP BPDUs received on any port of a port-channel, it expects not to receive BPDUs with different Source MACs as the port-channel is supposed to be connected to one and single Switch.

There are 2 recommendations on this scenario:

1) The command "no spanning-tree etherchannel guard misconfig" is required on this scenario as the C2960X is not really connecting to a one single device but to a "shared L2 segment".

2) On Switches connecting to ACI Leaf switches, the suggestion is to configure the Layer 2 interfaces with STP link-type shared.

https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/211236-ACI-operation-with-L2-switches-and-Spann.html

I hope this helps.

[SIMMN]

Thanks! But regarding “When you are connecting the C2960X to ACI, I suspect that STP converges and your Catalyst ends up receiving the STP BPDUs from your other Switches in short intervals of time. Since the Catalyst's Port-channel interprets this as a mis-configuration on the other side, the error happens.”, I am not sure i follow...is this an issue on the 2960 platform? Is the trunk link in 2960 by default STP Point-to-Point type? I would have to login to check...

I want to c2960 to receive the BPDU via ACI from other switches so it can calculate per vlan spanning tree properly. I have a C9300 stack, couple of N5ks configured/connected the same fashion with the same ACI fabric and they did not have the issue (without disabling the misconfig guard)...

[SIMMN]

Regarding the link shared, do you know which version the ACI was when the doc was written? Wonder if any behaviour change in ACI v4.2 or 5.x?

[Robert Burns]

This isn't an ACI issue - so don't expect any behavior to change with different ACI versions.  It's an external switch issue - at least how it's responding to ACI's expected behavior.  The reason you're likely not seeing this issue on your 5Ks is that command/feature doesn't exist in NXOS.  For the 9300's you likely didn't enable this on your C9300's.

Did you check the output of "show spanning summary" on your 9300s?

Robert

[SIMMN]

The c9300 also has etherchannel guard misconfigure turned on which is the default.

[Hector Gustavo Serrano Gutierrez]

Hi @SIMMN,

It is correct that the feature is enabled by default, although it is not necessary in this scenario. As Robert was suggesting, this has to do with the use of this feature and that the Catalyst does not really connect to another single Switch over EtherChannel as when the feature was originally conceived. If you would like to dig deeper, you may want to SPAN for STP BDPUs sent & received on all your Switches connected to ACI when you add the C2960X and its ports go err-disabled. Although this would be an interesting activity, you will find that disabling Etherchannel Guard Misconfig on the Catalyst is the permanent solution.

I hope this helps.

[SIMMN]

Thanks, so I guess it it recommended to do "no spanning-tree etherchannel guard misconfig" globally on the C9300 as well even its PO does not experience the err-disable issue? Also the C9300 has POs to other devices...