Mac-Flapping Bug? Switch generates logs for a MAC that doesn't exist

[Matthias.G]

There is an odd problem on the core switch right now, its a C9500-48Y4C running  IOS XE 17.09.03

Core-Switch#show logging | inc flapping
%SW_MATM-4-MACFLAP_NOTIF: Host 0026.fc2a.0000 in vlan 1259 is flapping between port Po49 and port Po51
%SW_MATM-4-MACFLAP_NOTIF: Host 0026.fc2a.0000 in vlan 1259 is flapping between port Po49 and port Po51
%SW_MATM-4-MACFLAP_NOTIF: Host 0026.fc2a.0000 in vlan 1259 is flapping between port Po49 and port Po51
(...)

This MAC is apparently flapping between 2 portchannels, however upon further research on the distribution switches behind these port-channels, no MAC like that is known. I have yet to see a bug that causes a random adress to be detected on certain ports, however I dont know what would cause this adress to not be visible in the MAC table when it is obviously connected somehow.  

Distribution-1# show mac address-table | inc 0026.fc1f.0000
Distribution-1#

Distribution-2# show mac address-table | inc 0026.fc1f.0000
Distribution-2#

[balaji.bandi]

the mac address flapping between  0026.fc2a.0000

But you are mac address different show mac address-table | inc 0026.fc1f.0000

check the correct MAC address to trace - the mac coming from Po49 and Po51 you can check what devices connect to that port-channel and get on that device and do mac trace ?

Some time if this WLC - we do see some roaming clients 

AcSiP Technology Corp. - looking at MAC address vendor looks for me some IOT device.

[Matthias.G]

Hi Balaji, I meant to obfuscate the adress in Notepad over security/company policy concerns without changing the OUI but I made a mistake missed some of the output. Thats why you have 2 MACs there. Anyway we dont need the real mac, only to know that its the same in the show command i provided and that the OUI part was not tampered with for troubleshooting purposes. I also noticed that the company behind the OUI produces integrated circuits for all kinds of IOT capable devices so its hard to tell what type of device belongs to the mac. I wanted to gather the mac trace but it seems that its unavailable on this system:

Distribution-2# traceroute ?
A.B.C.D IP address of remote system
WORD Hostname of remote system

Distribution-2#

[Leo Laohoo]

I have seen this behaviour when we upgraded one of our 9500 to 17.9.3.

This was solved by checking the configuration of the switch at the other end of the EtherChannel.  Our switches had IP Device Tracking turned on and the IPDT policy used had "tracking enable".  When we turned it off ("tracking disable") the spamming stopped.

[Matthias.G]

Thanks for the advice Leo, I tried some show and global-config commands related to the feature but it doesnt seem like the switches use it, neither do they seem able to at this time, maybe due to the licenses in use.

Failed/unrecognized commands:

show ip device tracking

show run | inc tracking

switch(config)# ip device?

However, I can now rule this out as the cause which is nice 🙂

[Leo Laohoo]

Please try:  sh run | include tracking enable

[Matthias.G]

Looks like its not running any tracking:

Switch#sh run | include tracking enable
Switch#
Switch#

[Leo Laohoo]

Thanks for confirming. 

Can you try "show mac address-table | inc 0026.fc" regularly from both Distro-1 and Distro-2?  

(The MAC address in question is a randomized MAC address.)

What sort of switches are Distro-1 and Distro-2?  If they are Cisco, what are their firmware versions?

[Matthias.G]

Distro 1 and 2 are both NXOS 9.3(5) - C93360YC-FX2. The MAC was not seen on either switch again.

Dist-1# show mac address-table | inc 0026.fc
Dist-1# show mac address-table | inc 0026.fc
Dist-1# show mac address-table | inc 0026.fc
Dist-1# show mac address-table | inc 0026.fc
Dist-1# show mac address-table | inc 0026.fc

Dist-2# show mac address-table | inc 0026.fc
Dist-2# show mac address-table | inc 0026.fc
Dist-2# show mac address-table | inc 0026.fc
Dist-2# show mac address-table | inc 0026.fc
Dist-2# show mac address-table | inc 0026.fc

[Leo Laohoo]

1.  If Po49 or Po51 is disabled, does the error message stop?
2.  What happens if, using a MAC-based ACL, the MAC address is null-routed?
3.  What are the chances for rebooting the 9500?

[Matthias.G]

1 and 3 are not possible because its customer equipment in this case and the MAC address is kind of null routed as a workaround at this time:

mac address-table static 0026.fc1f.0000 vlan 1 drop
mac address-table static 0026.fc1f.0000 vlan 3 drop
mac address-table static 0026.fc1f.0000 vlan 15 drop
(and so on)

Result: Log message does not reappear. This indicates that the MAC does exist in the network but its highly strange that it cant be seen on either Distro.

[Leo Laohoo]

---

@Matthias.G wrote:
Result: Log message does not reappear. This indicates that the MAC does exist in the network but its highly strange that it cant be seen on either Distro.

---

Let's do a "scream test":  Leave that ACL alone and watch the counters (if they increment or not).  

Leave the ACL alone until someone screams.  

[Matthias.G]

A scream test is probably not what we need because no one is influenced by this other than the administrator who sees the logs filled with mac-flapping. I disabled the drop commands for a moment though and the logs started being flooded again immediatly.

I saw a pattern though in the minute where I had the drop-config removed: 11:02:09 - 11:02:24 - 11:02:39 (15 seconds between logs) I dont know if this is interesting to us at all but could be related to the timers of some protocol so im sharing it.

[MatthewRobinson5751]

We observed identical logs on our Catalyst 9606 core switch, flapping MAC between port-channels connected to a Nexus vPC pair:

%SW_MATM-4-MACFLAP_NOTIF: Host 0026.f1f4.0000 in vlan 80 is flapping between port Po25 and port Po26

Capturing traffic from this address showed that the frames with this MAC were spanning-tree BPDUs, which led me to this link:
Understand Source MAC Address Field in Spanning-Tree PDUs on Nexus Series Switches - Cisco

The link explains that the vPC ID is embedded in three nibbles of the source MAC for BPDUs - in our example the bold digits in 0026.f1f4.0000 correspond to the decimal vPC ID 500. The nuisance logs occur as spanning-tree timers trigger BPDUs to be sent from two different pairs of Nexus switches which have the same vPC IDs configured on port-channels connected to the Catalyst switch.

interface port-channel1
description VPC to 9606
switchport
switchport mode trunk
switchport trunk native vlan 80
vpc 500


Changing the vPC ID for both peers on one of the Nexus pairs is the fix, although it will result in brief disruption to traffic so best scheduled in a maintenance window.

[Mark Elsen]

  - @MatthewRobinson5751                              Following the Subject of the thread :
                                                             https://macaddress.io/mac-address-lookup/w56WAnqAko
                                                        >....Company name   ProCurve Networking by HP

                                                       If you are talking about another problem then  start a new thread
                                                      Besides replying to a very old thread is counter productive
                                                               because            no one may notice it

  M.