Spanning-tree - MST switches interaction with ACI

[rducombl]

Spanning-tree - MST switches interaction with ACI

Spanning-tree - Default Behavior of spanning tree  on ACI with MST connected switches

ACI fabric does not run spanning-tree. By default, an EPG in ACI will just flood BPDUs (even if Hardware proxy mode is chosen for the BD).

This is the topology we are using : 

The downstream Spanning-tree switches  (N7-50 and N7k-65) are running MST.

Here we assume 

- BD name - RD_BD_1101

- EPG in that BD with 4 paths all with encapsulation vlan-1110

-- on leaf1 port-channel to Nexus7k-50 (Node101/Policy_to_sw50) - port 101/1/11,13 to port-channel1 on nexus7k-50

-- on leaf1 port-channel to Nexus7k-65 (Node101/Policy_to_sw65) - port 101/1/17-18 to port-channel1 on nexus7k-65

-- on leaf3 port-channel to Nexus7k-50 (Node103/Policy_to_sw50) - port 103/1/11,13 to port-channel2 on nexus7k-50

-- on leaf3 port-channel to Nexus7k-65 (Node103/Policy_to_sw65) - port 103/1/17-18 to port-channel2 on nexus7k-65

The two  nexus 7000 switches are running MST 

and are configured with the following (in Nexus7000):

N7K-2-65# sh run spanning-tree
spanning-tree mode mst

spanning-tree mst configuration
  name mst-rd
  revision 1
  instance 1 vlan 1100-1110
 

The MST BPDU are not transmitted in the VLAN as with Rapid PVST+, but instead transmitted untagged in the native vlan of the switch.

ACI fabric does not run spanning-tree. By default, an EPG in ACI will just flood BPDUs (even if Hardware proxy mode is chosen for the BD). However it is likely that there won't be an EPG matching the untagged vlan encapsulation on ports connected to MST switches.

In that scenario,  the MST BPDU send by the nexus7000 will be dropped ingress of ACI fabric.

then, you would see for the topology here that all 4 ports in vlan 1110 (now MST instances 1) are forwarding and that both switches believes to be root.

N7K-65# sh spanning-tree Vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     001b.54c2.2641
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     001b.54c2.2641
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1000      128.4096 P2p
Po2              Desg FWD 1000      128.4097 P2p




N7K-50# sh spanning-tree vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     0026.980a.df41
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0026.980a.df41
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1000      128.4096 P2p
Po2              Desg FWD 1000      128.4097 P2p

This scenario will potentially easily introduce a layer 2 loop. With ACI loop may not occur if the BD do not allow flooding (which is the default).

However this is not a valid design.

Avoiding loop with MST connected switches

In order to avoid the loop, we need to make sure MST BPDU are propagated across ACI fabric. To do that we need to create an EPG containing all ports going to spanning-tree MST switches and with untagged native vlan.

Assuming on the nexus 7000 the native vlan is set to be vlan 1102. We will create a new EPG with the 4 port-channel in vlan 1102 and mark those as untagged (native).

Step 1 -  Create a new EPG.

- in tenant TAB --> Application Profiles --> Application EPGs : right click and create a new Application EPG

- In create application EPG window:

-- select Name for the new EPG (EPG-native-vlan)

-- select the bridge domain (here RD_BD_1101)

-- Associate with a physical domain that contains the port-channel

-- Click Finish

 

Step 2 - Add each Path to the new EPG

- in TENANT --> Application Profiles --> Application EPGs -->  --> Static bindings (Paths)

Right click and select "Deploy Static EPG on PC, VPC or interfaces.

- In Deploy Static EPG on PC, VPC or Interface window:

-- Select The path type (for this example it is all Direct Port Chanel)

-- Select the path to the MST switch

-- Write the native vlan in the Encap field (here vlan 1102)

-- Select the box "native"  in mode section

-- Click submit

Repeat those steps for all pathes going to the MST switches:

Finally you will have:

Now if you check spanning-tree status on MST switches, you will see that BPDU are going through ACI properly,

there are 2 blocked ports (one per MST switches) and only one root:

N7K-65# sh spanning-tree vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     001b.54c2.2641
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     001b.54c2.2641
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1000      128.4096 P2p
Po2              Back BLK 1000      128.4097 P2p


N7K-50# sh spanning-tree vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     001b.54c2.2641
             Cost        1000
             Port        4096 (port-channel1)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0026.980a.df41
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 1000      128.4096 P2p
Po2              Altn BLK 1000      128.4097 P2p

[rducombl]

Spanning-tree - Default Behavior of spanning tree  on ACI with MST connected switches

ACI fabric does not run spanning-tree. By default, an EPG in ACI will just flood BPDUs (even if Hardware proxy mode is chosen for the BD).

This is the topology we are using : 

The downstream Spanning-tree switches  (N7-50 and N7k-65) are running MST.

Here we assume 

- BD name - RD_BD_1101

- EPG in that BD with 4 paths all with encapsulation vlan-1110

-- on leaf1 port-channel to Nexus7k-50 (Node101/Policy_to_sw50) - port 101/1/11,13 to port-channel1 on nexus7k-50

-- on leaf1 port-channel to Nexus7k-65 (Node101/Policy_to_sw65) - port 101/1/17-18 to port-channel1 on nexus7k-65

-- on leaf3 port-channel to Nexus7k-50 (Node103/Policy_to_sw50) - port 103/1/11,13 to port-channel2 on nexus7k-50

-- on leaf3 port-channel to Nexus7k-65 (Node103/Policy_to_sw65) - port 103/1/17-18 to port-channel2 on nexus7k-65

The two  nexus 7000 switches are running MST 

and are configured with the following (in Nexus7000):

N7K-2-65# sh run spanning-tree
spanning-tree mode mst

spanning-tree mst configuration
  name mst-rd
  revision 1
  instance 1 vlan 1100-1110
 

The MST BPDU are not transmitted in the VLAN as with Rapid PVST+, but instead transmitted untagged in the native vlan of the switch.

ACI fabric does not run spanning-tree. By default, an EPG in ACI will just flood BPDUs (even if Hardware proxy mode is chosen for the BD). However it is likely that there won't be an EPG matching the untagged vlan encapsulation on ports connected to MST switches.

In that scenario,  the MST BPDU send by the nexus7000 will be dropped ingress of ACI fabric.

then, you would see for the topology here that all 4 ports in vlan 1110 (now MST instances 1) are forwarding and that both switches believes to be root.

N7K-65# sh spanning-tree Vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     001b.54c2.2641
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     001b.54c2.2641
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1000      128.4096 P2p
Po2              Desg FWD 1000      128.4097 P2p




N7K-50# sh spanning-tree vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     0026.980a.df41
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0026.980a.df41
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1000      128.4096 P2p
Po2              Desg FWD 1000      128.4097 P2p

This scenario will potentially easily introduce a layer 2 loop. With ACI loop may not occur if the BD do not allow flooding (which is the default).

However this is not a valid design.

Avoiding loop with MST connected switches

In order to avoid the loop, we need to make sure MST BPDU are propagated across ACI fabric. To do that we need to create an EPG containing all ports going to spanning-tree MST switches and with untagged native vlan.

Assuming on the nexus 7000 the native vlan is set to be vlan 1102. We will create a new EPG with the 4 port-channel in vlan 1102 and mark those as untagged (native).

Step 1 -  Create a new EPG.

- in tenant TAB --> Application Profiles --> Application EPGs : right click and create a new Application EPG

- In create application EPG window:

-- select Name for the new EPG (EPG-native-vlan)

-- select the bridge domain (here RD_BD_1101)

-- Associate with a physical domain that contains the port-channel

-- Click Finish

 

Step 2 - Add each Path to the new EPG

- in TENANT --> Application Profiles --> Application EPGs -->  --> Static bindings (Paths)

Right click and select "Deploy Static EPG on PC, VPC or interfaces.

- In Deploy Static EPG on PC, VPC or Interface window:

-- Select The path type (for this example it is all Direct Port Chanel)

-- Select the path to the MST switch

-- Write the native vlan in the Encap field (here vlan 1102)

-- Select the box "native"  in mode section

-- Click submit

Repeat those steps for all pathes going to the MST switches:

Finally you will have:

Now if you check spanning-tree status on MST switches, you will see that BPDU are going through ACI properly,

there are 2 blocked ports (one per MST switches) and only one root:

N7K-65# sh spanning-tree vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     001b.54c2.2641
             This bridge is the root
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     001b.54c2.2641
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Desg FWD 1000      128.4096 P2p
Po2              Back BLK 1000      128.4097 P2p


N7K-50# sh spanning-tree vlan 1110

MST0001
  Spanning tree enabled protocol mstp
  Root ID    Priority    32769
             Address     001b.54c2.2641
             Cost        1000
             Port        4096 (port-channel1)
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

  Bridge ID  Priority    32769  (priority 32768 sys-id-ext 1)
             Address     0026.980a.df41
             Hello Time  2  sec  Max Age 20 sec  Forward Delay 15 sec

Interface        Role Sts Cost      Prio.Nbr Type
---------------- ---- --- --------- -------- --------------------------------
Po1              Root FWD 1000      128.4096 P2p
Po2              Altn BLK 1000      128.4097 P2p

[Robert Rowland III]

I think somehow you are posting the images with internal hosts / urls.

I can not see the images so I after troubleshooting my ISP I looked at the http source:

http://ecats-uc-wtools.cisco.com/ecats-testmgr-files/TSI001/STP1.jpg

C:\Users\ME>nslookup ecats-uc-wtools.cisco.com  ns2.cisco.com
Server:  ns2.cisco.com
Address:  64.102.255.44

Name:    ecats-uc-web1.cisco.com
Address:  172.18.106.82
Aliases:  ecats-uc-wtools.cisco.com

I am not likely getting to your version of 172.18 network.

It is kind of interesting that your DNS server are "advertising" private address to Internet.

[rducombl]

Thanks a lot for letting me know, this should be fix now.

[lacirasella]

Hi Roland,

i have made the setup you explain, it is working as expected but i still have a convergence problem.

When i cut one of the Po on N7K-50 (Po1 for example), it take around 30s for the data traffic to switch to the other Po going to N7K-65 (listening-learning STP state can be visible... :-).

In your setup, i suppose N7K-50 and N7K-65 are in vPC, in my setup this is not the case, both are independant and making a STP triangle with ACI.

have you got a clue for this situation?

Thanks

Maurice

[rducombl]

Hi Maurice, 

I did the setup 2 years ago, but there was no VPC anywhere from what I remember. Hence essentially 4 spanning-tree port in the same MST instance (2 each nexus7k).

I just retest similar setup quickly in my lab and I never see slow convergence. 

(didn't test all scenario though, but shut/no shut of root port and shut/no shut of designated port) 

in both case BLK port converge immediately to forwarding. 

[Vladimir Stepanov]

Hello rducombl,

in addition to create vlan1, it's need to configure mapping on aci between mspt instanses and aci.

[tohlim]

Hi,

I encountered similar situation in my customer ACI deployment.
A Huawei 9306 switch is connected to a pair of leaf switches using vPC. ACI release is 2.2(2k).

The following Critical Faults were raised:

"Native vlan is not configured on interface po1, but it is receiving MST(802.1s) BPDUs. This could result in a layer2 topology with a loop". Please refer to attached screen captures.

I applied the solution in this forum - Created a new EPG and bind it to the VPC. I set "Access (Untagged)" mode. Native VLAN is 1 as informed by Huawei.

The faults did not clear although I can see the Native VLAN is applied to the physical interface and port-channel interface.

Thank you.

B.Rgds,
Lim TS

[GMetric]

Digging up an old post.

We have a back-to-back multipdd where the SPINES of 2 different PODS are directly connected with each other. It seems the MST BPDU is not being sent between the PODS. We have 1 switch with 2 vPC, 1 vPC to POD 1 and the other to POD 2. The issue what we see is that both go into forwarding when both vPC's are up. We did not expect his.

We have a separate EPG for the NATIVE VLAN, in our case VLAN 1.

This EPG has been statically deployed to both vPC and and their interfaces from both PODS.

What are we missing here ?