Understanding MST Convergence

[rasmus.elmholt]

Hi All,

I have the following setup in CML:

• SW1 is the Root Birdge
• I have VLANs 1-20
• All Switches have the same MST configuration and running in the same region.

When I shut down interface gi1/0/3 on SW3 the MST environment converges and uses Gi1/0/6 as the root port for some time until it falls back to Po1.

I cannot explain why as the internal cost is higher than using Po1.

Relevant configuration on SW3:

spanning-tree mode mst
spanning-tree extend system-id
spanning-tree mst configuration
 name CiscoSTD
 revision 1

interface GigabitEthernet1/0/2
 switchport mode trunk
 channel-group 1 mode active
interface GigabitEthernet1/0/3
 switchport mode trunk
 channel-group 1 mode active
interface GigabitEthernet1/0/6
 switchport access vlan 20
 switchport mode access
 spanning-tree portfast
interface GigabitEthernet1/0/7
 switchport access vlan 20
 switchport mode access
 spanning-tree portfast
interface Port-channel1
 switchport mode trunk
 spanning-tree mst 0 cost 10000

Debug when I shut down Gi1/0/3:

SW3(config-if)#do show span mst 0

##### MST0    vlans mapped:   1-4094
Bridge        address 5254.0002.8506  priority      32768 (32768 sysid 0)
Root          address 5254.0009.3a34  priority      24576 (24576 sysid 0)
              port    Po1             path cost     0        
Regional Root address 5254.0009.3a34  priority      24576 (24576 sysid 0)
                                      internal cost 30000     rem hops 18
Operational   hello time 2 , forward delay 15, max age 20, txholdcount 6 
Configured    hello time 2 , forward delay 15, max age 20, max hops    20

Interface                        Role Sts Cost      Prio.Nbr Type
----------------                 ---- --- --------- -------- --------------------------------
Gi1/0/1                          Desg FWD 20000     128.1    P2p 
Gi1/0/4                          Desg FWD 20000     128.4    P2p 
Gi1/0/5                          Desg FWD 20000     128.5    P2p 
Gi1/0/6                          Altn BLK 20000     128.6    P2p 
Gi1/0/7                          Altn BLK 20000     128.7    P2p 
Gi1/0/8                          Desg FWD 20000     128.8    P2p 
Po1                              Root FWD 10000     128.2281 P2p 

SW3(config-if)#int gi 1/0/3
SW3(config-if)#shut
SW3(config-if)#
*Jul 16 08:06:17.403: %LINK-5-CHANGED: Interface GigabitEthernet1/0/3, changed state to administratively down
*Jul 16 08:06:18.402: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to down
*Jul 16 08:06:19.764: MST[0]: Po1 state change forwarding -> blocking
*Jul 16 08:06:19.765: MST[0]: Gi1/0/1 state change forwarding -> blocking
*Jul 16 08:06:19.765: MST[0]: Gi1/0/4 state change forwarding -> blocking
*Jul 16 08:06:19.765: MST[0]: Gi1/0/5 state change forwarding -> blocking
*Jul 16 08:06:19.765: MST[0]: Gi1/0/8 state change forwarding -> blocking
*Jul 16 08:06:19.765: MST[0]: Gi1/0/6 state change blocking -> forwarding
*Jul 16 08:06:19.767: STP[0]: Generating TC trap for port GigabitEthernet1/0/6
SW3(config-if)#do show span mst 0

##### MST0    vlans mapped:   1-4094
Bridge        address 5254.0002.8506  priority      32768 (32768 sysid 0)
Root          address 5254.0009.3a34  priority      24576 (24576 sysid 0)
              port    Gi1/0/6         path cost     0        
Regional Root address 5254.0009.3a34  priority      24576 (24576 sysid 0)
                                      internal cost 50000     rem hops 17
Operational   hello time 2 , forward delay 15, max age 20, txholdcount 6 
Configured    hello time 2 , forward delay 15, max age 20, max hops    20

Interface                        Role Sts Cost      Prio.Nbr Type
----------------                 ---- --- --------- -------- --------------------------------
Gi1/0/1                          Desg BLK 20000     128.1    P2p 
Gi1/0/4                          Desg BLK 20000     128.4    P2p 
Gi1/0/5                          Desg BLK 20000     128.5    P2p 
Gi1/0/6                          Root FWD 20000     128.6    P2p 
Gi1/0/7                          Altn BLK 20000     128.7    P2p 
Gi1/0/8                          Desg BLK 20000     128.8    P2p 
Po1                              Desg BLK 10000     128.2281 P2p 

SW3(config-if)#
*Jul 16 08:06:34.765: MST[0]: Gi1/0/1 state change blocking -> learning
*Jul 16 08:06:34.765: MST[0]: Gi1/0/4 state change blocking -> learning
*Jul 16 08:06:34.765: MST[0]: Gi1/0/5 state change blocking -> learning
*Jul 16 08:06:34.765: MST[0]: Gi1/0/8 state change blocking -> learning
*Jul 16 08:06:34.765: MST[0]: Po1 state change blocking -> learning
SW3(config-if)#do show span mst 0

##### MST0    vlans mapped:   1-4094
Bridge        address 5254.0002.8506  priority      32768 (32768 sysid 0)
Root          address 5254.0009.3a34  priority      24576 (24576 sysid 0)
              port    Gi1/0/6         path cost     0        
Regional Root address 5254.0009.3a34  priority      24576 (24576 sysid 0)
                                      internal cost 50000     rem hops 17
Operational   hello time 2 , forward delay 15, max age 20, txholdcount 6 
Configured    hello time 2 , forward delay 15, max age 20, max hops    20

Interface                        Role Sts Cost      Prio.Nbr Type
----------------                 ---- --- --------- -------- --------------------------------
Gi1/0/1                          Desg LRN 20000     128.1    P2p 
Gi1/0/4                          Desg LRN 20000     128.4    P2p 
Gi1/0/5                          Desg LRN 20000     128.5    P2p 
Gi1/0/6                          Root FWD 20000     128.6    P2p 
Gi1/0/7                          Altn BLK 20000     128.7    P2p 
Gi1/0/8                          Desg LRN 20000     128.8    P2p 
Po1                              Desg LRN 10000     128.2281 P2p 

SW3(config-if)#
*Jul 16 08:06:49.765: MST[0]: Gi1/0/1 state change learning -> forwarding
*Jul 16 08:06:49.765: MST[0]: Gi1/0/4 state change learning -> forwarding
*Jul 16 08:06:49.765: MST[0]: Gi1/0/5 state change learning -> forwarding
*Jul 16 08:06:49.765: MST[0]: Gi1/0/8 state change learning -> forwarding
*Jul 16 08:06:49.765: MST[0]: Po1 state change learning -> forwarding
*Jul 16 08:06:49.767: STP[0]: Generating TC trap for port Port-channel1
*Jul 16 08:07:36.649: MST[0]: Gi1/0/6 state change forwarding -> blocking
*Jul 16 08:07:36.649: MST[0]: Gi1/0/1 state change forwarding -> blocking
*Jul 16 08:07:36.649: MST[0]: Gi1/0/4 state change forwarding -> blocking
*Jul 16 08:07:36.649: MST[0]: Gi1/0/5 state change forwarding -> blocking
*Jul 16 08:07:36.649: MST[0]: Gi1/0/8 state change forwarding -> blocking
*Jul 16 08:07:36.650: STP[0]: Generating TC trap for port Port-channel1
*Jul 16 08:07:51.651: MST[0]: Gi1/0/1 state change blocking -> learning
*Jul 16 08:07:51.651: MST[0]: Gi1/0/4 state change blocking -> learning
*Jul 16 08:07:51.651: MST[0]: Gi1/0/5 state change blocking -> learning
*Jul 16 08:07:51.651: MST[0]: Gi1/0/8 state change blocking -> learning
*Jul 16 08:08:06.652: MST[0]: Gi1/0/1 state change learning -> forwarding
*Jul 16 08:08:06.652: MST[0]: Gi1/0/4 state change learning -> forwarding
*Jul 16 08:08:06.652: MST[0]: Gi1/0/5 state change learning -> forwarding
*Jul 16 08:08:06.652: MST[0]: Gi1/0/8 state change learning -> forwarding
SW3(config-if)#
SW3(config-if)#do show span mst 0

##### MST0    vlans mapped:   1-4094
Bridge        address 5254.0002.8506  priority      32768 (32768 sysid 0)
Root          address 5254.0009.3a34  priority      24576 (24576 sysid 0)
              port    Po1             path cost     0        
Regional Root address 5254.0009.3a34  priority      24576 (24576 sysid 0)
                                      internal cost 30000     rem hops 18
Operational   hello time 2 , forward delay 15, max age 20, txholdcount 6 
Configured    hello time 2 , forward delay 15, max age 20, max hops    20

Interface                        Role Sts Cost      Prio.Nbr Type
----------------                 ---- --- --------- -------- --------------------------------
Gi1/0/1                          Desg FWD 20000     128.1    P2p 
Gi1/0/4                          Desg FWD 20000     128.4    P2p 
Gi1/0/5                          Desg FWD 20000     128.5    P2p 
Gi1/0/6                          Altn BLK 20000     128.6    P2p 
Gi1/0/7                          Altn BLK 20000     128.7    P2p 
Gi1/0/8                          Desg FWD 20000     128.8    P2p 
Po1                              Root FWD 10000     128.2281 P2p 

SW3(config-if)#

At first I thought the issues was that the cost on po1 changed and MST had to do a recalculation, but I have manually configured it to 10.000 in the lab. And I still see the same issue.

I know it seems strange that we have SW3 and SW4 connected directly but IRL we have them connected to a production ring of siemens switches.

Can someone explain why the MST tree does not stay stable when I shut down one port in the LAG?

[Enes Simnica]

Hello @rasmus.elmholt  First of all, respect for ur professionalism and ur contribution man, it’s crazy!

and Yes, this behavior is expected in MST when a Port-Channel member link goes down.

When uu shut down Gi1/0/3, even though Po1 stays up, MST detects a topology change and temporarily considers Po1 unstable. During that brief recalculation, MST selects Gi1/0/6 as the Root Port because it sees it as the next best available path.

Once updated BPDUs confirm that Po1 is still valid and has the lowest cost, MST reverts back to Po1 as the Root Port. So, this happens even if u’ve manually configured the cost on Po1, because MST reacts quickly to any link-state change, especially within EtherChannels.

To reduce this, consider using LACP instead of static channel mode and make sure all member interfaces are configured identically.

I’ll definitely lab this later  looks like a fun one to explore deeper! Hope it helps G

-Enes

[rasmus.elmholt]

Hi @Enes Simnica 

Thank you for the reply. I always thought that a LAG was seen as one port from STP's point of view, and that it did not consider the physical ports.

What puzzles me is that if I shut down Gi1/0/2 instead of Gi 1/0/3 I do not see the same convergence and the STP is stable.

And if I do not shut down the port, but disconnect it in CML I find that STP is stable as well.

In this setup I am using LACP. But I still think some of this could be caused by the virtual HW, next I will try to configure this using 9200CX switches.

[rasmus.elmholt]

@Enes Simnica 

I have now done the same test with 4 9200CX switches and whenever i shutdown one of the links on the LAG between SW2 and SW3 I do not see any change in the MST topology.

The only thing I see is "MST[0]: updt roles, non-tracked event" and then the interface Cost changes.

But no change in the STP topology. So I will count this issue as a virtualisation error.

[MHM Cisco World]

Totally correct 

L2 in vm have many issue 

MHM

[Enes Simnica]

G, I’ll lab this situation tomorrow morning and keep u posted..

[MHM Cisco World]

If your PO port members not "" P"" when you do show port channel summary

You face same issue as lab before

MHM

[rasmus.elmholt]

Hi @MHM Cisco World 

When the interface is shut down it changes in the LAG to "D".

SW3(config-if)#int gi 1/0/3
SW3(config-if)#shut
SW3(config-if)#do show etherchan summ | be Group
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP        Gi1/0/2(P)      Gi1/0/3(D)      

This is the same senario as before, but I get a different error. Last time I got the p2p dispute.

Now I have changed from vIOSL2 in EVE-NG to Cat9000v in CML, and the issue is different.

I will have to lab this up on real switches next.

[MHM Cisco World]

Check three PO and check PO in both SW sides

Do show spanning tree interface details 

MHM

[rasmus.elmholt]

I am not sure I understand what you want me to verify, do you want the stp interface details from the port channel?

[MHM Cisco World]

sw-right
show port-channel summary 
sw-left 
show port-channel summary 

check PO port member is down in both SW

MHM

[paul driver]

Hello
in the real world directly p2p links would disable at both ends, but as you are in simulation, and you've just shutdown a single port on Sw3 that is part of a PC, i would envisage that the PC between Sw3-Sw2 would go in dispute due to unidirectional issue because Sw3 port is still showing up even though its directly p2p link should be down.

Try the same test again but this time either remove the link or suspend it in your lab so it shows at both ends of that interconnect down physically I envisage the issue you experienced last time wont occur?

Also when disabling a link running MST, a topology change should not be incurred, a trap may be seen but its only when the link comes up will should a tc be incurred