Cisco Nexus9000 many mac issue

[nazim@circlenetworkbd.net]

Dear Concern,

I face some issue for replace my switch. Here i share some log. 

LOG:

2020 Dec 31 03:49:55 cisco-9000 %L2FM-2-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan 1949

for 120s due to too many mac moves

sh mac address-table  notification mac-move

AM MAC Registration/Deletion Notifications: 3

  Number of MAC Addresses added: 16109

  Number of MAC Addresses moved: 10855

  Number of MAC Addresses removed: 8

version:

Software
BIOS: version 05.38
NXOS: version 7.0(3)I7(9)
Hardware
cisco Nexus9000

Thanks.

[marce1000]

 Ref : https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/93x/system_messages/n9k_syslog_9_3_5.html?dtid=osscdc000283

 >...

  Error Message  L2FM-2-L2FM_MAC_FLAP_DISABLE_LEARN: Disabling learning in vlan [dec] for 120s due to too many mac moves

Explanation  MAC(s) in vlan [dec] have moved too many times. Disabling learning for 120s.

Recommended Action  No action is required.

   - So Cisco seems rather optimistic on this message, I would advice however to check your network for loops (e.g.). In  that context use a syslog server to collect messages from the device in a continuous manner. Review the logs from the device at regular times.                                    Perhaps a broader problem such as a loop can be detected.

 M.

[Georg Pauwen]

Hello,

in addition to the other post, the document linked below explains why these MAC move messages occur. I don't think you can disable this control plane protection, as the 'no mac address-table loop-detect port-down' command just "reverts to the default action of disabling MAC learning for 180 seconds."

Nexus 9000 Mac move troubleshooting and preventive methods

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/nx-os-software/213906-nexus-9000-mac-move-troubleshooting-and.html

[Christopher Hart]

Hello!

This syslog indicates that one or more MACs in a VLAN are rapidly moving between two interfaces, which is typically indicative of a loop. As a self-protection mechanism, the switch will disable dynamic MAC learning in that VLAN for 120 seconds to reduce the impact the loop has on the switch.

When troubleshooting these issues, it can usually be helpful to know which MAC addresses are moving, as well as between which two interfaces the MAC is moving. A syslog can expose this information after increasing the logging level for the L2FM (Layer 2 Forwarding Manager) to a level of 5 through the below command:

switch# configure terminal
switch(config)# logging level l2fm 5
switch(config)# end
switch#

When the issue happens again, you should see syslogs similar to the following:

2018 Nov 14 16:04:23.881 N9K %L2FM-4-L2FM_MAC_MOVE2: Mac 0000.117d.e02e in vlan 741 has moved between Po6 to Eth1/3
2018 Nov 14 16:04:23.883 N9K %L2FM-4-L2FM_MAC_MOVE2: Mac 0000.117d.e02e in vlan 741 has moved between Po6 to Eth1/3

This will help you troubleshoot the issue further by identifying which specific MAC addresses are moving, as well as between what two ports the MACs are moving between. This will help you isolate the issue and determine where the loop may be coming from.

These commands are documented in the Nexus 9000 MAC Move Troubleshooting and Preventive Methods Troubleshooting TechNote.

I hope this helps - thank you!

-Christopher

[dwalker09DW]

Christopher - Thank you for sharing these commands.  They are very usefull.

[pHz]

Hello,

This mac move / mac block happens in our network when we perform a firewall failover. Our servers are connected behind a firewall, and when we perform a failover of that firewall, the MAC address table is updated to point to the interface of our 2nd firewall. Since we have thousands of servers, this triggers the behavior described in this article. The end result is that our network / our servers are essentially unreachable for 2+ minutes, causing an incident.

What can be done to solve this ? Is this inherent to the Nexus ? Do we really have no way of disabling this behavior, even temporarily when we know we're going to perform a firewall failover ?

[Christopher Hart]

Hello!

This behavior should only be observed if there's multiple moves that happen rapidly back and forth between two interfaces. In other words, a MAC address (or a large number of MAC addresses) moving from interface Ethernet1/1 to Ethernet1/2 should not cause this issue. However, if a MAC address moves from Ethernet1/1 to Ethernet1/2, then moves back to Ethernet1/1, that "bouncing back and forth" behavior is what eventually triggers this issue if it happens rapidly enough in a short period of time (which most typically happens with loops).

In the context of your scenario, I'd want to hear a few more details (and it may be worth starting your own thread to track this issue). Particularly, I'd be curious if your firewalls are routing traffic, or if they're L2/transparent. Specifically, I'm curious whether the Nexus switches are seeing the MAC addresses of downstream servers move, or if it's the MAC addresses of the firewalls themselves move. It sounds like the firewalls are operating in an L2/transparent mode based on the context of your reply, but it'd be good to confirm that fact.

To directly answer your question, there is no way to disable this behavior (at least, not that I am aware of, but I am highly confident it's not something that can be toggled). This is a self-protection mechanism that the switch must keep enabled to minimize control plane disruption in loop scenarios.

Thank you!

-Christopher

[tristanr]

This command is extremely valuable in finding out who is doing the flapping.  Thanks for sharing it!

[vuyisa100]

hi Christopher

thank you so much, this helped me too.

[gzigoto133]

Hello,

I would like to understand what is the threshold that triggers the MAC learning disable on a Nexus 9K.

So far i did not find this information in the Cisco documentation. I would expect it is triggered by something like "one unique MAC address moves more that 5 times during a 30 seconds duration" (just an example).

If someone knows the exact values and can share it, i would appreciate a lot

Best regards

Gabriel