Puzzle - Impossible MAC Flap and Random Forward/Blocking SPANTREE

mriksman · ‎03-29-2019

Hi,

Firstly, please bare with me - I am certainly no expert, but I am the custodian of our beastly network, which covers a large, remote area utilising fiber and Wi-Max.

The (simplified) diagram of the bits I am trying to understand is as follows;

Randomly, 3 or 4 times a day, I get the following events across the above switches;

SAEQL2501	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from forwarding to blocking	Khepri Link
SAEQL2501	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from blocking to forwarding	Khepri Link
SUCRL201	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi0/1 instance 0 moving from forwarding to blocking	Crossover SUCRL201 to SUCRL202
SUCRL201	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi0/1 instance 0 moving from blocking to forwarding	Crossover SUCRL201 to SUCRL202
SAEQL2501	26/03/2019 6:02:47	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a6.cc65 in vlan 195 is flapping between port Gi1/0/6 and port Gi1/0/1	SA-ESVT-SRV1A FTE Green
SACRL201	26/03/2019 6:02:47	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a6.cc65 in vlan 195 is flapping between port Fa0/5 and port Gi0/1	SA-ESVT-SRV1A FTE Green
SACRL201	26/03/2019 6:02:47	%SW_MATM-4-MACFLAP_NOTIF: Host 000c.2994.c8e2 in vlan 195 is flapping between port Fa0/5 and port Fa0/23	Zabbix
SAEQL2501	26/03/2019 6:02:49	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a6.cc65 in vlan 195 is flapping between port Gi1/0/6 and port Gi1/0/1	SA-ESVT-SRV1A FTE Green
SAEQL2502	26/03/2019 6:02:49	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a6.cc65 in vlan 195 is flapping between port Gi1/0/1 and port Gi1/0/6	SA-ESVT-SRV1A FTE Green
SACRL201	26/03/2019 6:02:51	%SW_MATM-4-MACFLAP_NOTIF: Host 000c.2994.c8e2 in vlan 195 is flapping between port Fa0/23 and port Fa0/5	Zabbix
SACRL201	26/03/2019 6:02:51	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a6.cc65 in vlan 195 is flapping between port Fa0/5 and port Gi0/1	SA-ESVT-SRV1A FTE Green
	26/03/2019 6:02:51	Timeout on PLC Comms (Alarmed on SA-ESVT-SRV1A)
SACRL201	26/03/2019 6:03:07	%SW_MATM-4-MACFLAP_NOTIF: Host 000c.2994.c8e2 in vlan 195 is flapping between port Fa0/5 and port Fa0/23	Zabbix

I enabled debugging on SAEQL2501, and gathered the following during on of these 'events';

Mar 27 16:46:27: STP: MST0 rx BPDU: config protocol = mstp, packet from GigabitEthernet1/0/4 , linktype IEEE_SPANNING , enctype 2, encsize 17
Mar 27 16:46:27: STP: enc 01 80 C2 00 00 00 D8 24 BD A3 33 01 00 69 42 42 03
Mar 27 16:46:27: STP: Data 000003027C8000D824BDA33300000000008000D824BDA3330080010000140001000F00
Mar 27 16:46:27: STP: MST0 Gi1/0/4:0000 03 02 7C 8000D824BDA33300 00000000 8000D824BDA33300 8001 0000 1400 0100 0F00
Mar 27 16:46:27: MST[0]: Gi1/0/4 state change forwarding -> blocking
Mar 27 16:46:27: STP SW: Gi1/0/4 new blocking req for 1 vlans
Mar 27 16:46:27: Found no corresponding dummy port for instance 0, port_id 128.4
Mar 27 16:46:27: %SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from forwarding to blocking
Mar 27 16:46:27: STP: MST0 rx BPDU: config protocol = mstp, packet from GigabitEthernet1/0/4 , linktype IEEE_SPANNING , enctype 2, encsize 17
Mar 27 16:46:27: STP: enc 01 80 C2 00 00 00 D8 24 BD A3 33 01 00 69 42 42 03
Mar 27 16:46:27: STP: Data 000003027D8000001A2FD6B380000000008000001A2FD6B38080010000140001000F00
Mar 27 16:46:27: STP: MST0 Gi1/0/4:0000 03 02 7D 8000001A2FD6B380 00000000 8000001A2FD6B380 8001 0000 1400 0100 0F00
Mar 27 16:46:28: STP: MST0 rx BPDU: config protocol = mstp, packet from GigabitEthernet1/0/4 , linktype IEEE_SPANNING , enctype 2, encsize 17
Mar 27 16:46:28: STP: enc 01 80 C2 00 00 00 D8 24 BD A3 33 01 00 69 42 42 03
Mar 27 16:46:28: STP: Data 00000302791000000AB84E4600000000001000000AB84E460080010000140001000F00
Mar 27 16:46:28: STP: MST0 Gi1/0/4:0000 03 02 79 1000000AB84E4600 00000000 1000000AB84E4600 8001 0000 1400 0100 0F00
Mar 27 16:46:28: MST[0]: Gi1/0/4 state change blocking -> forwarding
Mar 27 16:46:28: STP SW: Gi1/0/4 new forwarding req for 1 vlans
Mar 27 16:46:28: Found no corresponding dummy port for instance 0, port_id 128.4
Mar 27 16:46:28: %SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from blocking to forwarding
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: STP SW: VLAN1: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN8: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN16: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN24: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN192: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN195: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN210: topology change over - this bridge is root
Mar 27 16:46:28: MST[0]: flushing Gi1/0/1
Mar 27 16:46:28: MST[0]: flushing Gi1/0/2
Mar 27 16:46:28: MST[0]: flushing Gi1/0/5
Mar 27 16:46:28: MST[0]: flushing Gi1/0/6
Mar 27 16:46:28: MST[0]: flushing Gi1/0/8
Mar 27 16:46:28: MST[0]: flushing Gi1/0/9
Mar 27 16:46:28: MST[0]: flushing Gi1/0/10
Mar 27 16:46:28: MST[0]: flushing Gi1/0/12
Mar 27 16:46:28: STP: MST0 rx BPDU: config protocol = mstp, packet from GigabitEthernet1/0/4 , linktype IEEE_SPANNING , enctype 2, encsize 17
Mar 27 16:46:28: STP: enc 01 80 C2 00 00 00 D8 24 BD A3 33 01 00 69 42 42 03
Mar 27 16:46:28: STP: Data 00000302791000000AB84E4600000000001000000AB84E460080010000140001000F00
Mar 27 16:46:28: STP: MST0 Gi1/0/4:0000 03 02 79 1000000AB84E4600 00000000 1000000AB84E4600 8001 0000 1400 0100 0F00
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:29: STP: MST0 rx BPDU: config protocol = mstp, packet from GigabitEthernet1/0/4 , linktype IEEE_SPANNING , enctype 2, encsize 17
Mar 27 16:46:29: STP: enc 01 80 C2 00 00 00 D8 24 BD A3 33 01 00 69 42 42 03
Mar 27 16:46:29: STP: Data 00000302791000000AB84E4600000000001000000AB84E460080010000140001000F00
Mar 27 16:46:29: STP: MST0 Gi1/0/4:0000 03 02 79 1000000AB84E4600 00000000 1000000AB84E4600 8001 0000 1400 0100 0F00
Mar 27 16:46:29: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:29: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:29: MST[0]: flushing Gi1/0/1
Mar 27 16:46:29: MST[0]: flushing Gi1/0/2
Mar 27 16:46:29: MST[0]: flushing Gi1/0/5
Mar 27 16:46:29: MST[0]: flushing Gi1/0/6
Mar 27 16:46:29: MST[0]: flushing Gi1/0/8
Mar 27 16:46:29: MST[0]: flushing Gi1/0/9
Mar 27 16:46:29: MST[0]: flushing Gi1/0/10
Mar 27 16:46:29: MST[0]: flushing Gi1/0/12
Mar 27 16:46:30: %SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/6 and port Gi1/0/1
Mar 27 16:46:44: %SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/1 and port Gi1/0/6
Mar 27 16:47:03: MST[0]: tc timer expired

The two servers (SA-ESVT-SRV1A and SA-ESVT-SRV1B) are redundant, so only one is communicating with these external PLCs at one time. The PLC communications get disrupted during these 'events' - only briefly - but they log an alarm. When I fail over to server SA-ESVT-SRV1B I receive the same set of logs, but with the MAC address of the SA-ESVT-SRV1B instead of SA-ESVT-SRV1A;

SAEQL2501	27/03/2019 16:46:27	%SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from forwarding to blocking	Khepri Link
SAEQL2501	27/03/2019 16:46:27	%SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from blocking to forwarding	Khepri Link
SUCRL201	27/03/2019 16:46:27	%SPANTREE-6-PORT_STATE: Port Gi0/1 instance 0 moving from forwarding to blocking	Crossover SUCRL201 to SUCRL202
SUCRL201	27/03/2019 16:46:28	%SPANTREE-6-PORT_STATE: Port Gi0/1 instance 0 moving from blocking to forwarding	Crossover SUCRL201 to SUCRL202
SACRL201	27/03/2019 16:46:28	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Fa0/5 and port Gi0/1	SA-ESVT-SRV1B FTE Green
SAEQL2502	27/03/2019 16:46:32	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/1 and port Gi1/0/6	SA-ESVT-SRV1B FTE Green
	27/03/2019 16:46:34	Timeout on PLC Comms (Note there was none on the events below)
SACRL201	27/03/2019 16:46:37	%SW_MATM-4-MACFLAP_NOTIF: Host 000c.2994.c8e2 in vlan 195 is flapping between port Fa0/5 and port Fa0/23	Zabbix
SAEQL2502	27/03/2019 16:46:44	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/1 and port Gi1/0/6	SA-ESVT-SRV1B FTE Green
SACRL201	27/03/2019 16:46:44	%SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi0/1 and port Fa0/5	SA-ESVT-SRV1B FTE Green
SACRL201	27/03/2019 16:46:50	%SW_MATM-4-MACFLAP_NOTIF: Host 000c.2994.c8e2 in vlan 195 is flapping between port Fa0/5 and port Fa0/23	Zabbix

The PLCs are located on port Fa0/4 of SACRL201 (not shown in the architecture diagram attached).

I have spent days looking at this, but I just cannot work out why;

Is there a MAC Flap? There are definitely no loops. I can't see how it is physically possible for any of these devices to flap between these ports/
- On the VMware ESXi server, we have many VMs; two are Linux OS - built using the same template. Why is only one of them ("Zabbix") appear to have a MAC Flap issue?
- Why would having a server set as master (SA-ESVT-SRV1A/B) have any impact on whether it gets caught MAC Flapping?
Why does it all start with the fiber crossover link between SUCRL201 and SUCRL202 changing from Forwarding to Blocking and back again? And why does it cause SAEQL2501's port Gi1/0/4 to do the same? All switches run MST, and the Wi-Max "Khepri Link" has STP turned off. Only our Cisco switches have STP turned on.

I'm hoping someone will read this and have a 'Ah hah!' moment. Because I sure haven't...

Thanks. Let me know what config files you need (what command to run to get that) and which switches.

Deepak Kumar · ‎03-29-2019

Hi,

As per details, It is looking at that port Gi1/0/4 is showing an issue. Any other switch which is connected to remote end is making STP issue:

ar 27 16:46:28: MST[0]: Gi1/0/4 state change blocking -> forwarding
Mar 27 16:46:28: STP SW: Gi1/0/4 new forwarding req for 1 vlans
Mar 27 16:46:28: Found no corresponding dummy port for instance 0, port_id 128.4
Mar 27 16:46:28: %SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from blocking to forwarding
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: STP SW: VLAN1: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN8: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN16: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN24: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN192: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN195: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN210: topology change over - this bridge is root

Have you verified that what is switch priority at remote end switch and any loop in the network design over the wifi?

I would like to suggest some commands as:

interface Gi1/0/4
spanning-tree guard root

Sametime we need some more details to identify the current issue:

show spanning-tree mst 0

show spanning-tree mst configuration

show spanning-tree active

Please share above commands output from switch 201, 202 and 2501, 2502

Regards,

Deepak Kumar

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

mriksman · ‎03-29-2019

Hi Deepak,

The Wi-Fi (Wi-MAX) - the 'Khepri link' - is comprised of;

They have no other wireless clients connecting to them, and are just linked together, point-to-point.

The 'remote' switch (SUCRL201) shows the root switch to be SAEQL2501;

Spanning tree enabled protocol mstp
Root ID Priority 4096
Address 000a.b84e.4600
Cost 0
Port 1 (FastEthernet0/1)
Hello Time 1 sec Max Age 20 sec Forward Delay 15 sec

Bridge ID Priority 32768 (priority 32768 sys-id-ext 0)
Address d824.bda3.3300
Hello Time 1 sec Max Age 20 sec Forward Delay 15 sec

Regarding the 'root' switch. What happens if the link between SUCRL201 and SAEQL2501 is disrupted (the 'Khepri' Wi-Max link). If SUCRL201 can no longer see SAEQL2501 (which is the root), what does SUCRL201 do? Does it become the root? And then if the link is re-established just moments later, what happens then?

I have attached the output of the commands for each of those switches.

Thanks for taking the time to look at this for me; it's appreciated!

Mike

mriksman · ‎04-02-2019

I hate being 'that' guy, but... *Bump*

@Deepak Kumar - any further ideas based on the information I provided?

Thanks.

Deepak Kumar · ‎04-03-2019

Hi,

Here I am looking two issues in your network:

1. Mac address flapping

Mar 27 16:46:30: %SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/6 and port Gi1/0/1
Mar 27 16:46:44: %SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/1 and port Gi1/0/6

The Mac address "2880.23a4.f63d" is flapping between Gi1/0/6 and port Gi1/0/1 on the switch 2501 means "FTP Green" and "FTP Yellow" ports are not configured correctly on the Server host.

What is a configuration on the server for those ports? Is is in EtherChannel?

2. Wireless Link flapping. That may actual wireless link flapping or BDPU loss due to Timeout. Check the logs on all Wireless devices and both switches for any disconnection.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

mriksman · ‎04-03-2019

I'll take a look.
Any issue with the Root configuration? If the Khepri link goes down, will SUCRL201 try and become root?

Deepak Kumar · ‎04-03-2019

Hi,

 If the Khepri link goes down, will SUCRL201 try and become root?

Yes, if the current root bridge will not available then SUCRL201 or 202 anyone who will win the election process will become the root bridge. As soon as possible the link restored then it will revert to the current state. Because the current root bridge 2501 is having low priority compared to others.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

mriksman · ‎04-03-2019

Could this cause what we see at the start of the issue/event;

SAEQL2501	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from forwarding to blocking
SAEQL2501	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from blocking to forwarding
SUCRL201	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi0/1 instance 0 moving from forwarding to blocking
SUCRL201	26/03/2019 6:02:46	%SPANTREE-6-PORT_STATE: Port Gi0/1 instance 0 moving from blocking to forwarding

And if so, how do we prevent that from happening? Being a wireless link, the connection may get disrupted briefly.

Should SUCRL201 be made its own Root of that remote network? I think I remember reading about Regions...? The remote network is on its own VLAN 16.

Deepak Kumar · ‎04-03-2019

Hi,

Please share "Show logging" output from the 2501 switch for getting into the details.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

mriksman · ‎04-04-2019

Attached show logging

paul driver · ‎04-03-2019

Hello

@Deepak Kumar wrote:

Hi,

Here I am looking two issues in your network:

1. Mac address flapping
Mar 27 16:46:30: %SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/6 and port Gi1/0/1
Mar 27 16:46:44: %SW_MATM-4-MACFLAP_NOTIF: Host 2880.23a4.f63d in vlan 195 is flapping between port Gi1/0/1 and port Gi1/0/6
The Mac address "2880.23a4.f63d" is flapping between Gi1/0/6 and port Gi1/0/1 on the switch 2501 means "FTP Green" and "FTP Yellow" ports are not configured correctly on the Server host.

What is a configuration on the server for those ports? Is is in EtherChannel?

2. Wireless Link flapping. That may actual wireless link flapping or BDPU loss due to Timeout. Check the logs on all Wireless devices and both switches for any disconnection.

Note- This may not be an issue it could just mean a the wifi client has roamed over to an other ap and as such its mac is being seen on another switchport

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Deepak Kumar · ‎04-03-2019

Hi @paul driver

This is a Server and it is connected on two different switches. As in the diagram, he has shown that it is working with two different IP address but not sure.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

paul driver · ‎04-03-2019

Hello

SUCRL201
Fa0/1 Root FWD 200000 128.1 P2p <----- this is a stating a non edge root port that is forwaring
Fa0/6 Desg FWD 200000 128.6 P2p Edge
Fa0/8 Desg FWD 200000 128.8 P2p Edge
Fa0/10 Desg FWD 200000 128.10 P2p Edge
Fa0/14 Desg FWD 200000 128.14 P2p Edge
Fa0/15 Desg FWD 200000 128.15 P2p Edge
Fa0/16 Desg FWD 200000 128.16 P2p Edge
Gi0/1 Desg FWD 20000 128.25 P2p <----- this is a stating a non edge port that is forwarding

I would suggest to at least check FA0/1 why that would be a root port, check the what connected to these ports

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Deepak Kumar · ‎04-03-2019

Hi @paul driver

I would suggest to at least check FA0/1 why that would be a root port,  check the what connected to these ports

A Port FA0/1 on SUCRL201 is root port and it is correct as per his high-level network diagram and logs. This port is working as uplink (over wifi) and connected to Root Bridge Switch SACRL2501. It seems that his WIFI connection was dropped or BPDU was timeout due to network delay or congestion.

Regards,
Deepak Kumar,
Don't forget to vote and accept the solution if this comment will help you!

paul driver · ‎04-03-2019

Hello

@Deepak Kumar wrote:

Hi,

As per details, It is looking at that port Gi1/0/4 is showing an issue. Any other switch which is connected to remote end is making STP issue:

ar 27 16:46:28: MST[0]: Gi1/0/4 state change blocking -> forwarding
Mar 27 16:46:28: STP SW: Gi1/0/4 new forwarding req for 1 vlans
Mar 27 16:46:28: Found no corresponding dummy port for instance 0, port_id 128.4
Mar 27 16:46:28: %SPANTREE-6-PORT_STATE: Port Gi1/0/4 instance 0 moving from blocking to forwarding
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: MST[0]: port Gi1/0/4 received internal tc
Mar 27 16:46:28: STP SW: VLAN1: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN8: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN16: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN24: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN192: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN195: topology change over - this bridge is root
Mar 27 16:46:28: STP SW: VLAN210: topology change over - this bridge is root

Have you verified that what is switch priority at remote end switch and any loop in the network design over the wifi?

I would like to suggest some commands as:

interface Gi1/0/4
spanning-tree guard root

Suggest to refine from applying any stp guardroot at this time , you could only caused further problems and negate connection.

Are you using MST throughout the estate, is your MST region connecting to any non-mst regions (non mstp mode switches)?

Make sure you have stp portfast/bduguard on all edge ports
Make sure you don't have any bpdu-filtering on access/trunks ports or access ports that are set to dynamic-desirable

On your root switch is bridge priority manually set and is it lower than any other switch in the estate

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul