VMware ESX' MAC addresses flaps with Cisco 4506

ivarstrandberg · ‎03-17-2011

Hi.

We have a Cisco 4506 with a Supervisor V, running cat4500-ENTSERVICES-M, version 12.2(46)SG.

Several IBM x3850s running VMware ESX 4.0.0 332073 are connected to the 4506 with 4 traffic ports, 2 management ports and 1 RSA port each.

All of these ESX'es behave themselves, except one.

Mar 17 12:54:39.937 CET: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:5E:25:30 in vlan 90 is flapping between port Gi4/23 and port Gi3/32

Mar 17 12:54:25.761 CET: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:5E:25:31 in vlan 90 is flapping between port Gi2/23 and port Gi3/31

Mar 17 12:24:44.604 CET: %C4K_EBM-4-HOSTFLAPPING: Host 00:50:56:40:5B:06 in vlan 198 is flapping between port Gi3/4 and port Gi4/3

4506-config:

!
interface GigabitEthernet3/3
description ESX-004 RSA
switchport access vlan 198
switchport mode access
spanning-tree portfast

!
interface GigabitEthernet3/4
description ESX-004 NIC1
switchport access vlan 198
switchport mode access
spanning-tree portfast
!
interface GigabitEthernet4/3
description ESX-004 NIC2
switchport access vlan 198
switchport mode access
spanning-tree portfast

!

interface GigabitEthernet2/23
description ESX-004 traffic port
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 6,9,11-13,15,17,19,21,25,27,45,46,48,49,65-67
switchport trunk allowed vlan add 70-73,76-79,81-91,93,95-99,199
switchport mode trunk
!

interface GigabitEthernet3/23
description ESX-004 traffic port
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 6,9,11-13,15,17,19,21,25,27,45,46,48,49,65-67
switchport trunk allowed vlan add 70-73,76-79,81-91,93,95-99,199
switchport mode trunk
!

interface GigabitEthernet3/31
description ESX-004 traffic port
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 6,9,11-13,15,17,19,21,25,27,45,46,48,49,65-67
switchport trunk allowed vlan add 70-73,76-79,81-91,93,95-99,199
switchport mode trunk
!
interface GigabitEthernet4/23
description ESX-004 traffic port
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 6,9,11-13,15,17,19,21,25,27,45,46,48,49,65-67
switchport trunk allowed vlan add 70-73,76-79,81-91,93,95-99,199
switchport mode trunk
!

According to our VMware personell, ESX-004 is configured exactly like the other ESX'es. I can't say I trust that to be true, since comparing and verifying my 4-5 lines of config of each port is a fairly easy task to accomplish.

All ESX'es are set up with NIC1 and NIC2 in a Virtual Switch, and the 4 traffic ports in a Distributed Virtual Switch. If we power on a guest on the ESX, the ports start flapping.

Chad Peterson · ‎03-17-2011

Hello,

What sort of load-balancing/nic teaming do you have setup on your DVS? I'd assume you have some sort of hash load balancing selected (IP hash for example).

So whats happening then is sometimes egress traffic from a server goes out port 1, then next packet from same server may go out port 2. From the 4500's perspective, it looks like the host is moving between these two ports.

In this case you would need to configure an etherchannel on these ports (the ones for traffic and that are using a hash for load-balancing). This way traffic coming from any of these ports will be seen as coming into that port-channel.

You could also change the load-balancing to 'route based on the orgnating virtual port ID' as well. This will "pin" each host to a port. This way they don't move (unless a link failure occurs).

Take a look into that, hope it helps

Chad

ivarstrandberg · ‎03-18-2011

Finally, I found the reason for this conundrum.

I logged into the command line of ESX-004, and found the text file which contains the config of the ESX (/etc/vmware/esx.conf). I compared this to the esx.conf of two other ESX'es which behave normally, looking for differences in the config.

I only do network stuff, and I have asked the VMware guys for a text-config all the time, since I don't trust that cross-checking GUI settings between ESX'es shows us the whole truth.

Anyway, the esx.conf of ESX-004 showed me that it has the same MAC-addresses on some vmnics as ESX-005 has. After pointing this out to the VMware people, they said it might be possible that ESX-004, during installation, may have had the physical NIC that now resides in ESX-005. Apparently, ESX does not update its config when the NIC gets changed. Weird.

That explains the flapping, though it's a bit weird that the only ports mentioned in the HOSTFLAPPING-message of the 4506 are the traffic ports of ESX-004. I'll go to the datacenter and see if the cabling is correct.