Mac-Flapping Bug? Switch generates logs for a MAC that doesn't exist

Matthias.G · ‎09-11-2023

There is an odd problem on the core switch right now, its a C9500-48Y4C running IOS XE 17.09.03

Core-Switch#show logging | inc flapping
%SW_MATM-4-MACFLAP_NOTIF: Host 0026.fc2a.0000 in vlan 1259 is flapping between port Po49 and port Po51
%SW_MATM-4-MACFLAP_NOTIF: Host 0026.fc2a.0000 in vlan 1259 is flapping between port Po49 and port Po51
%SW_MATM-4-MACFLAP_NOTIF: Host 0026.fc2a.0000 in vlan 1259 is flapping between port Po49 and port Po51
(...)

This MAC is apparently flapping between 2 portchannels, however upon further research on the distribution switches behind these port-channels, no MAC like that is known. I have yet to see a bug that causes a random adress to be detected on certain ports, however I dont know what would cause this adress to not be visible in the MAC table when it is obviously connected somehow.

Distribution-1# show mac address-table | inc 0026.fc1f.0000
Distribution-1#

Distribution-2# show mac address-table | inc 0026.fc1f.0000
Distribution-2#

balaji.bandi · ‎09-11-2023

the mac address flapping between 0026.fc2a.0000

But you are mac address different show mac address-table | inc 0026.fc1f.0000

check the correct MAC address to trace - the mac coming from Po49 and Po51 you can check what devices connect to that port-channel and get on that device and do mac trace ?

Some time if this WLC - we do see some roaming clients

AcSiP Technology Corp. - looking at MAC address vendor looks for me some IOT device.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Matthias.G · ‎09-12-2023

Hi Balaji, I meant to obfuscate the adress in Notepad over security/company policy concerns without changing the OUI but I made a mistake missed some of the output. Thats why you have 2 MACs there. Anyway we dont need the real mac, only to know that its the same in the show command i provided and that the OUI part was not tampered with for troubleshooting purposes. I also noticed that the company behind the OUI produces integrated circuits for all kinds of IOT capable devices so its hard to tell what type of device belongs to the mac. I wanted to gather the mac trace but it seems that its unavailable on this system:

Distribution-2# traceroute ?
A.B.C.D IP address of remote system
WORD Hostname of remote system

Distribution-2#

Leo Laohoo · ‎09-11-2023

I have seen this behaviour when we upgraded one of our 9500 to 17.9.3.

This was solved by checking the configuration of the switch at the other end of the EtherChannel. Our switches had IP Device Tracking turned on and the IPDT policy used had "tracking enable". When we turned it off ("tracking disable") the spamming stopped.

Matthias.G · ‎09-12-2023

Thanks for the advice Leo, I tried some show and global-config commands related to the feature but it doesnt seem like the switches use it, neither do they seem able to at this time, maybe due to the licenses in use.

Failed/unrecognized commands:

show ip device tracking

show run | inc tracking

switch(config)# ip device?

However, I can now rule this out as the cause which is nice

Leo Laohoo · ‎09-12-2023

Please try: sh run | include tracking enable

Matthias.G · ‎09-13-2023

Looks like its not running any tracking:

Switch#sh run | include tracking enable
Switch#
Switch#

Leo Laohoo · ‎09-13-2023

Thanks for confirming.

Can you try "show mac address-table | inc 0026.fc" regularly from both Distro-1 and Distro-2?

(The MAC address in question is a randomized MAC address.)

What sort of switches are Distro-1 and Distro-2? If they are Cisco, what are their firmware versions?

Matthias.G · ‎09-13-2023

Distro 1 and 2 are both NXOS 9.3(5) - C93360YC-FX2. The MAC was not seen on either switch again.

Leo Laohoo · ‎09-14-2023

1. If Po49 or Po51 is disabled, does the error message stop?
2. What happens if, using a MAC-based ACL, the MAC address is null-routed?
3. What are the chances for rebooting the 9500?

Matthias.G · ‎09-15-2023

1 and 3 are not possible because its customer equipment in this case and the MAC address is kind of null routed as a workaround at this time:

mac address-table static 0026.fc1f.0000 vlan 1 drop
mac address-table static 0026.fc1f.0000 vlan 3 drop
mac address-table static 0026.fc1f.0000 vlan 15 drop
(and so on)

Result: Log message does not reappear. This indicates that the MAC does exist in the network but its highly strange that it cant be seen on either Distro.

Leo Laohoo · ‎09-15-2023

@Matthias.G wrote:
Result: Log message does not reappear. This indicates that the MAC does exist in the network but its highly strange that it cant be seen on either Distro.

Let's do a "scream test": Leave that ACL alone and watch the counters (if they increment or not).

Leave the ACL alone until someone screams.

Matthias.G · ‎09-18-2023

A scream test is probably not what we need because no one is influenced by this other than the administrator who sees the logs filled with mac-flapping. I disabled the drop commands for a moment though and the logs started being flooded again immediatly.

I saw a pattern though in the minute where I had the drop-config removed: 11:02:09 - 11:02:24 - 11:02:39 (15 seconds between logs) I dont know if this is interesting to us at all but could be related to the timers of some protocol so im sharing it.