Have an issue that's doing my head in a bit. Here is the setup
Cisco UCS 2 x Blade Chassis consisting of two fabric extenders in each. Port 1 of each fabric extender going to Fabric Interconnect A, and Port 2 of each fabric extender going to Fabric Interconnect B.
2 x Fabric Interconnects ports 17 and 18 of Fabric Interconnect connecting to 3750 switch stack in etherchannel. Fabric A to Port Channel 13 and Fabric B to Port Channel 14. (see configs below)
Fabric Interconnects have been setup as port channels in UCS manager.
Cisco 3750 have been setup using LACP port channels. Here is a config of the ports on the 3750
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 13 mode active
Each server profile in UCS manager has been setup with 2 vNICS, 1 vNIC to Fabric A and the other vNIC to Fabric B. Each vNIC contains the VLANS that consist within the virtual infrastructure.
Cisco Nexus 1000v being used in vCenter as our VDS.
Port-profiles have been setup for each vm network. Each vmk has been migrated to Nexus.Port-profile ethernet is as follows for each blade.
port-profile type ethernet server1-uplink
switchport mode trunk
switchport trunk allowed vlan 1-3967,4048-4093
channel-group auto mode on
system vlan 50-52,324,1500-1505
An example of a vm network in Nexus:
port-profile type vethernet vMotion
switchport mode access
switchport access vlan 20
The problem is, when I log into my 3750 switch stack and issue a show log I see lots of:
%SW_MATM-4-MACFLAP_NOTIF: Host 0025.b530.007f in vlan 324 is flapping between port Po13 and port Po14
So the virtual machine MAC address is flapping between the port channels of the fabric interconnects. There is no vmotion happening either at the time of the alerts.
Any ideas ?
Hopefully I'm reading this wrong, but you say that you have "Port 1 of each fabric extender going to Fabric Interconnect A, and Port 2 of each fabric extender going to Fabric Interconnect B."
This is not supported. A single Fabric Extender (aka IOM) should connect to only one Fabric Interconnect.
The second thing to note is that the only supported uplink configuration for Nexus 1000v Uplink Port Profile is MAC Pinning when the Host is a UCS blade. The two Fabric Interconnects are NOT clustered Switches, only the Management functionality is clustered. The host facing connection on the blades must be treated as if they connect to separate upstream switches. Channel group mode on (static Port channel) is not possible here. MAC Pinning will provide all the failover redundancy you need.
Hi Robert, thanks for your reply, yes you're right, I just checked our network diagram and I config is fabric extender 1 port 1 & 2 going directly to Fabric A and fabric extender 2 port 1 & 2 going directly to Fabric B. Same for the second chassis.
In regards to MAC pinning, I had this turned on and thought that I would try it turned off. Though I still get the flapping unfortunately. I will put it back on now.
Any other ideas ?
Detail or provide a topology diagram for your Fabric Interconnects - 3750 connections.
Sounds like you have two port channels with members from each FI possibly.
Also provide from the 3750 "show cdp neighbor" and "show ip int brief"
Ok here we go:
Blade Chassis 1
IOM 1 port 1 goes to Fabric Interconnect 1 port 1
IOM 1 port 2 goes to Fabric Interconnect 1 port 2
IOM 2 port 1 goes to Fabric Interconnect 2 port 1
IOM 2 port 2 goes to Fabric Interconnect 2 port 2
Blade Chassis 2
IOM 1 port 1 goes to Fabric Interconnect 1 port 3
IOM 1 port 2 goes to Fabric Interconnect 1 port 4
IOM 2 port 1 goes to Fabric Interconnect 2 port 3
IOM 2 port 2 goes to Fabric Interconnect 2 port 4
Fabric Interconnect 1
Port 17 goes to 3750 stack switch 1 te1/0/1
Port 18 goes to 3750 stack switch 2 te2/0/1
Fabric Interconnect 2
Port 17 goes to 3750 stack switch 1 te1/0/2
Port 18 goes to 3750 stack switch 2 te2/0/2
Cisco 3750 Stack
Port Channel 13 contains te1/0/1 and 2/0/1
Port Channel 14 contains te2/0/2 and 2/0/2
both are LACP
CDP is not enabled on the 3750 switch stack
sh ip int brief
Port-channel13 unassigned YES unset up up
Port-channel14 unassigned YES unset up up
That MAC address belongs to a UCS blade. Can you determine which blade this is? The MAC OUI shows this is a Cisco MAC. Look through your MAC pools and you'll see which Service Profile has been assigned the MAC.
Which ever blade this MAC belongs to (this will NOT be a VMware host as VMware never uses the Host's NIC MACs for sourcing traffic) you've likely have some teaming happening to the two host facing Network interfaces at the OS level. If so, remove the teaming.
Robert were you replaying to another thread ? Cause i'm wondering how you knew what the MAC address was
Anyhow let me give you an example, looking at the 3750 log I see this:
Mar 24 08:35:54.738: %SW_MATM-4-MACFLAP_NOTIF: Host 0002.3d40.0a0d in vlan 51 is flapping between port Po14 and port Po13
Mar 24 08:36:06.767: %SW_MATM-4-MACFLAP_NOTIF: Host 0002.3d40.0a0d in vlan 51 is flapping between port Po13 and port Po14
Mar 24 08:36:57.716: %SW_MATM-4-MACFLAP_NOTIF: Host 0002.3d40.0a0d in vlan 51 is flapping between port Po14 and port Po13
On the Nexus Switch if i issue the following command I get this output:
show mac address-table | i 0002.3d40.0a0d
51 0002.3d40.0a0d dynamic 1 Po2 3
51 0002.3d40.0a0d dynamic 1 Po3 4
51 0002.3d40.0a0d dynamic 1 Po4 5
51 0002.3d40.0a0d dynamic 1 Po5 6
51 0002.3d40.0a0d dynamic 2 Po1 7
51 0002.3d40.0a0d dynamic 1 Po6 8
51 0002.3d40.0a0d dynamic 1 Po7 9
51 0002.3d40.0a0d dynamic 1 Po8 10
51 0002.3d40.0a0d static 0 N1KV Internal Port 14
Vlan 51 is not even used, it just exists as a port-profile vethernet at the moment. So not sure why anything is flapping in that vlan ?
The only vm's we have with dual NICs is to be used for data and ther othe nic for iSCSI
The original MAC you posted "0025.b530.007f" uses the Cisco MAC OUI "00:25:b5:xx:xx:xx".
Those recent ones you've posted are N1K VEM MAC address. Can you post a log file with your entire N1K running config.
Also post "show svs neighbors" from the VSM.
Hi Robert I just fixed this, I believe the problem was as follows:
When I created the first vmware hosts in the blade chassis 1 I created a host profile to accelerate the setup.
I used this same host profile for the blades on the second chassis. I think this screwed around with the network and port-channels.
Basically I removed the nics on the esxi hosts to vswitch along with the management interface. Removed the port-channel on the nexus switch, removed the port-profile for the uplink, as I did this I saw the in ethernet interface that were previously assigned moved to the Quarantine profile, re-created the port-profile, re-added the nic to the VDS and voila all working, no mac-flapping.
Thanks alot for your help and quick replies it's much appreciated, hopefully one day I can return the favour