MAC Flapping on 3750 Stack coming from Fabric Interconnects

[David Rodriguez]

Hey,

Have an issue that's doing my head in a bit. Here is the setup

Cisco UCS 2 x Blade Chassis consisting of two fabric extenders in each. Port 1 of each fabric extender going to Fabric Interconnect A, and Port 2 of each fabric extender going to Fabric Interconnect B.

2 x Fabric Interconnects ports 17 and 18 of Fabric Interconnect connecting to 3750 switch stack in etherchannel. Fabric A to Port Channel 13 and Fabric B to Port Channel 14. (see configs below)

Fabric Interconnects have been setup as port channels in UCS manager.

Cisco 3750 have been setup using LACP port channels. Here is a config of the ports on the 3750

switchport trunk encapsulation dot1q
switchport mode trunk
channel-protocol lacp
channel-group 13 mode active

Each server profile in UCS manager has been setup with 2 vNICS, 1 vNIC to Fabric A and the other vNIC to Fabric B. Each vNIC contains the VLANS that consist within the virtual infrastructure.

Cisco Nexus 1000v being used in vCenter as our VDS.

Port-profiles have been setup for each vm network. Each vmk has been migrated to Nexus.Port-profile ethernet is as follows for each blade.

port-profile type ethernet server1-uplink

  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1-3967,4048-4093
  mtu 9000
  channel-group auto mode on
  no shutdown
  system vlan 50-52,324,1500-1505
  state enabled

An example of a vm network in Nexus:

port-profile type vethernet vMotion
  vmware port-group
  switchport mode access
  switchport access vlan 20
  no shutdown
  state enabled

The problem is, when I log into my 3750 switch stack and issue a show log I see lots of:

%SW_MATM-4-MACFLAP_NOTIF: Host 0025.b530.007f in vlan 324 is flapping between port Po13 and port Po14

So the virtual machine MAC address is flapping between the port channels of the fabric interconnects. There is no vmotion happening either at the time of the alerts.

Any ideas ?

[Robert Burns]

David,

Hopefully I'm reading this wrong, but you say that you have "Port 1 of each fabric extender going to Fabric Interconnect A, and Port 2  of each fabric extender going to Fabric Interconnect B."

This is not supported.  A single Fabric Extender (aka IOM) should connect to only one Fabric Interconnect.

The second thing to note is that the only supported uplink configuration for Nexus 1000v Uplink Port Profile is MAC Pinning when the Host is a UCS blade.  The two Fabric Interconnects are NOT clustered Switches, only the Management functionality is clustered. The host facing connection on the blades must be treated as if they connect to separate upstream switches.  Channel group mode on (static Port channel) is not possible here.  MAC Pinning will provide all the failover redundancy you need.

Regards,

Robert

[David Rodriguez]

Hi Robert, thanks for your reply, yes you're right, I just checked our network diagram and I config is fabric extender 1 port 1 & 2 going directly to Fabric A and fabric extender 2 port 1 & 2 going directly to Fabric B. Same for the second chassis.

In regards to MAC pinning, I had this turned on and thought that I would try it turned off. Though I still get the flapping unfortunately. I will put it back on now.

Any other ideas ?

[Robert Burns]

Detail or provide a topology diagram for your Fabric Interconnects - 3750 connections.

Sounds like you have two port channels with members from each FI possibly.

Also provide from the 3750 "show cdp neighbor" and "show ip int brief"

Robert

[David Rodriguez]

Ok here we go:

Blade Chassis 1

IOM 1 port 1 goes to Fabric Interconnect 1 port 1

IOM 1 port 2 goes to Fabric Interconnect 1 port 2

IOM 2 port 1 goes to Fabric Interconnect 2 port 1

IOM 2 port 2 goes to Fabric Interconnect 2 port 2

Blade Chassis 2

IOM 1 port 1 goes to Fabric Interconnect 1 port 3

IOM 1 port 2 goes to Fabric Interconnect 1 port 4

IOM 2 port 1 goes to Fabric Interconnect 2 port 3

IOM 2 port 2 goes to Fabric Interconnect 2 port 4

Fabric Interconnect 1

Port 17 goes to 3750 stack switch 1 te1/0/1

Port 18 goes to 3750 stack switch 2 te2/0/1

Fabric Interconnect 2

Port 17 goes to 3750 stack switch 1 te1/0/2

Port 18 goes to 3750 stack switch 2 te2/0/2

Cisco 3750 Stack

Port Channel 13 contains te1/0/1 and 2/0/1

Port Channel 14 contains te2/0/2 and 2/0/2

both are LACP

CDP is not enabled on the 3750 switch stack

sh ip int brief

Port-channel13         unassigned      YES unset  up                    up
Port-channel14         unassigned      YES unset  up                    up

[Robert Burns]

That MAC address belongs to a UCS blade.  Can you determine which blade this is?  The MAC OUI shows this is a Cisco MAC.  Look through your MAC pools and you'll see which Service Profile has been assigned the MAC.

Which ever blade this MAC belongs to (this will NOT be a VMware host as VMware never uses the Host's NIC MACs for sourcing traffic) you've likely have some teaming happening to the two host facing Network interfaces at the OS level.  If so, remove the teaming.

Robert

[David Rodriguez]

Robert were you replaying to another thread ? Cause i'm wondering how you knew what the MAC address was

Anyhow let me give you an example, looking at the 3750 log I see this:

Mar 24 08:35:54.738: %SW_MATM-4-MACFLAP_NOTIF: Host 0002.3d40.0a0d in vlan 51 is flapping between port Po14 and port Po13
Mar 24 08:36:06.767: %SW_MATM-4-MACFLAP_NOTIF: Host 0002.3d40.0a0d in vlan 51 is flapping between port Po13 and port Po14
Mar 24 08:36:57.716: %SW_MATM-4-MACFLAP_NOTIF: Host 0002.3d40.0a0d in vlan 51 is flapping between port Po14 and port Po13

On the Nexus Switch if i issue the following command I get this output:

show mac address-table | i 0002.3d40.0a0d

51        0002.3d40.0a0d    dynamic 1         Po2                            3
51        0002.3d40.0a0d    dynamic 1         Po3                            4
51        0002.3d40.0a0d    dynamic 1         Po4                            5
51        0002.3d40.0a0d    dynamic 1         Po5                            6
51        0002.3d40.0a0d    dynamic 2         Po1                            7
51        0002.3d40.0a0d    dynamic 1         Po6                            8
51        0002.3d40.0a0d    dynamic 1         Po7                            9
51        0002.3d40.0a0d    dynamic 1         Po8                            10
51        0002.3d40.0a0d    static  0         N1KV Internal Port             14

Vlan 51 is not even used, it just exists as a port-profile vethernet at the moment. So not sure why anything is flapping in that vlan ?

The only vm's we have with dual NICs is to be used for data and ther othe nic for iSCSI

[Robert Burns]

The original MAC you posted "0025.b530.007f" uses the Cisco MAC OUI "00:25:b5:xx:xx:xx".

Those recent ones you've posted are N1K VEM MAC address.  Can you post a log file with your entire N1K running config.

Also post "show svs neighbors" from the VSM.

Thanks,

Robert

[David Rodriguez]

Hi Robert I just fixed this, I believe the problem was as follows:

When I created the first vmware hosts in the blade chassis 1 I created a host profile to accelerate the setup.

I used this same host profile for the blades on the second chassis. I think this screwed around with the network and port-channels.

Basically I removed the nics on the esxi hosts to vswitch along with the management interface. Removed the port-channel on the nexus switch, removed the port-profile for the uplink, as I did this I saw the in ethernet interface that were previously assigned moved to the Quarantine profile, re-created the port-profile, re-added the nic to the VDS and voila all working, no mac-flapping.

Thanks alot for your help and quick replies it's much appreciated, hopefully one day I can return the favour