MAC Flapping with Incomplete MAC Addresses

[ThomasBarr]

Hello,

I am new to the Support Community, I joined to try and get some help resolving an ongoing issue with set of distribution level Cisco Catalyst 3850-24XS switches setup in a ring topology. 

Running rapid-pvst and I have verified that the VLANs are in a blocking state on the switch it is configured to. I am however seeing a rather strange MAC flapping event in the logs. Which shows as the following

Jan 29 03:39:41: %SW_MATM-4-MACFLAP_NOTIF: Host 004f.fc00.0000 in vlan 1 is flapping between port Te1/0/1 and port Te1/0/2
Jan 29 03:55:14: %SW_MATM-4-MACFLAP_NOTIF: Host 004f.fc00.0000 in vlan 1 is flapping between port Te1/0/1 and port Te1/0/2
Jan 29 03:58:10: %SW_MATM-4-MACFLAP_NOTIF: Host 0051.3300.0000 in vlan 1 is flapping between port Te1/0/1 and port Te1/0/2
Jan 29 04:00:44: %SW_MATM-4-MACFLAP_NOTIF: Host 0052.a900.0000 in vlan 1 is flapping between port Te1/0/1 and port Te1/0/2

I have tried following the mac-address table to identify the source but I am unable to identify the device, or configuration causing this because of the MAC address format ending in 0000.

Vlan 1 is in a administrative down state, but it seems the trunk ports between the switches were setup using native vlan 1.

Has anyone seen something like this before?

[ThomasBarr]

Bump. Any help?

[brselzer]

Hello Thomas,

If I do a vendor lookup on those macs, they don't return anything which is very strange. Have you tried to do a capture for these packets to see what they are? It seems strange that only packets with strange macs are flapping and not  "valid" traffic. It is possible the L2 headers of same packets are being corrupted or there is something on your network talking in a non-standard protocol. Either way, I would try to capture these packets to get a better understanding of what is going on. 

Hope that helps!

[Matt Delony]

Hello Thomas,

It's hard to say what exactly is going on without more information. I haven't personally seen this problem before.

Here's what the information you presented is telling me:

• MAC flaps are occurring. From the perspective of the switch, MAC flap occurs when MAC address is dynamically learned on one interface, but a different interface receives packet with same source MAC. This forces the switch to flush the MAC entry on first port, then populate MAC entry on second port. MAC flaps usually mean that the entry is flip-flopping, so to speak, between the first and second port, as indicated in the MAC flap log message.
• The MAC addresses in question are strange. The MAC address OUI doesn't map to any known vendor. Also, the last 3 bytes of MAC address are all 0's, which is strange and something I've never seen before.
• Supposed traffic is listed as coming in on VLAN 1. As you say in your post, trunk looks like it has VLAN 1 as native VLAN. If this is the case, then the MAC flaps in question are occurring with traffic that doesn't have 802.1Q VLAN tag.

If I were troubleshooting the issue, I would check the following outputs:

• show spanning-tree vlan 1
• show spanning-tree vlan 1 detail
• show spanning-tree interface T1/0/1
• show spanning-tree interface T1/0/2
• show interface trunk
• show mac address-table vlan 1

My goal would be to get a good idea of the current state of STP topology. Which interfaces are supposed be forwarding and which are supposed to be blocking, if any. I would check for any STP instability. Am I getting excessive TCN's? Is one of the interfaces going from blocking to forwarding when it shouldn't?

I would check the trunking status. What vlans we are passing across the trunks. Also, can I see the offending MAC addresses in MAC address table for VLAN 1?

I would also attempt packet capture on ingress of trunks. If it can't capture the desired traffic, then I would attempt to capture on the other end of the link (i.e. from the device connecting to the trunks), and see if I can capture the offending traffic which is being sent towards 3850.

[ThomasBarr]

Hello Matt,

Thanks for the detailed response.

As you and Bradley mention the MAC addresses do not resolve to a vendor and do not seem like valid addresses. Which is very confusing to me and is not something I have seen before either.

In response to your questions.

• show spanning-tree vlan 1
  • Show FWD on all ports on the root bridge, as I work around the ring I get to the switch I have configured to block port Te1/0/2 is in a Altn BLK state. All switches agree on the root bridge.
• show spanning-tree vlan 1 detail
  • Not sure what exact parameters you want to focus on here. I am seeing 10902 topology changes on the root bridge, and 10947 on the switch that is blocking. Blocking port shows the number of transitions to forwarding state as 1.
• show spanning-tree interface T1/0/1
  • shows FWD all vlans
• show spanning-tree interface T1/0/2
  • shows Altn BLK all vlans (on root bridge all are forwarding)
• show interface trunk
  • Not really pruning off vlans on these trunks as they are for the distribution ring. So all vlans are present.
• show mac address-table vlan 1
  • all the flapping malformed MAC addresses are shown as dynamic.

I will work to capture the traffic on these ports maybe that will provide some insight here. Also I am not sure if the number of TCN's on the trunks is abnormal or not.

[routerospf]

 I am encountering this exact issue, did you find a resolution?