MAC Flap / loop with Nexus vPC

[Smithsco]

Hi,

I am seeing an issue where what appears to be a loop forming over our vPC setup. Hopefully the attached topology makes sense but the issue we are seeing is when Po40 is connected from sw4 to both Nexus switches, I get the below mac flap warnings from sw4:
Sep 18 09:04:25.060 UTC: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0f09.0005 in vlan 672 is flapping between port Po40 and port Gi1/0/22
Sep 18 09:04:45.196 UTC: %SW_MATM-4-MACFLAP_NOTIF: Host 0009.0f09.0005 in vlan 674 is flapping between port Po40 and port Gi1/0/22

On top of the warnings, loss starts sporadically to the external firewall interfaces on 672 and 674 with a few pings dropped in sequence every minute or so to suddenly connectivity dropping off with only the odd response coming back as if a loop has formed. Connectivity has been tested remotely.

Everything returned to normal as soon as I shut down Po40 on Nexus1 almost as if the MAC address was being learnt on Nexus1 via port 1/0/4 on sw4 and being advertised back to sw4 on port 1/0/6 from Nexus2. But that breaks the vPC rule of not sending traffic traversing a vPC link down a vPC member port so I am at a bit of a loss as to how sw4 can be learning the same Mac address over the port channel and the port the firewall is directly connected to.

HSRP is set up on both Nexus switches for 674 and 672 with Nexus 1 currently being the primary although 'peer-gateway' is setup on both switches. We also have OSPF across 2 vlans/SVIs that exists on the vPC peer link only so that traffic can route between the Nexus switches incase of transit link failure. The transit vlans/SVIs do not traverse the vPC peer link. E1/48 is the keep alive link. The port channel between sw3 and sw4 is trunk all.

There is no direct connectivity from the nexus switches to either of the firewalls and Po30 is STP down so I cannot see how the MAC address can be learnt other than via Po40.

The nexus switches are N3Ks running 9.3(9). The other switches are Catalyst 9200s. I'm aware context is key and there is no config, but I can supply this if more info is required.

The current situation is Po40 is shutdown from Nexus1 and I am keen to bring this backup but it would be nice to have some additional input from the community on whether or not they have seen something like this before?

Thanks!

[Jitendra Kumar]

RESOLUTION

A MAC Flap is caused when a switch receives packets from two different interfaces with the same source MAC address.

If you are getting the behavior for a lot of other MACs, that most likely is a layer 2 loop.

You can do the following:

• Check the network switches for misconfigurations that might cause a data-forwarding loop.
• If you are not running spanning-tree, turn it on.
• To track down a loop, start with the following command: #show mac-address-table address [flapping mac] 
• We see that the MAC is coming in on port gi0/5 and gi0/16. One port will lead us to where that MAC is plugged in and the other will lead us to the loop. Pick a port and start working through.
• Or Some load balancing techniques can send traffic to both ports, and that would cause the switch to go crazy, since it is receiving traffic from the same MAC on two or more different ports.
• Fix this type of LB make it active/standby or make sure the server uses 2 different mac addresses, one per NIC.

[Jitendra Kumar]

You can refer the below discussion as well.

https://community.cisco.com/t5/switching/mac-flapping-between-multiple-ports/m-p/3062667

[MHM Cisco World]

Hi friend, 
can you share the config of NSK and SW?

[Smithsco]

Hi,

Apologies for the delay. Please find the config showing all ports these vlans are on across all 4 switches along with 'show spanning tree' for all 3 vlans. 2 additional things i didn't mention, sw4 is the stp root for vlans 670-674 and sw3 and sw4 run pvst where as the nexus run rpvst. The Nexus should be STP root for the vlans but never the less STP is working with the Po30 segment being blocked. Not sure if that makes a difference other than muddying the waters. I feel the next step is to bring up the shutdown Po40 on Nexus1 and find out where the Nexus switches may also be learning about the MAC address that is flapping, but before I do, another set of eyes to confirm the vPC config is correct and there is nothing obvious would be useful. Thanks.