C240 M3 with VIC 1225 VMware network issue

[msassemidwave]

Just got my hands on a few C240 M3′s with the VIC 1225 adapter and began setting them up as a VMware 5.1 U1 cluster. I’ve configured 8 vNICs per host: 2 for Management, 2 for VMotion, 2 for iSCSI, and 2 for VM traffic. Each pair of vNICs has one pinned to uplink 0 and the other to uplink 1. vNICs are set to trunk mode in the VIC since I am tagging multiple VLANs from VMware. The hosts are physically connected to a stacked pair of Cisco 4500-X switches; physical uplink 0 to one switch, uplink 1 to the other switch. Switch ports are trunked to allow the VLANs we plan to use. Portfast is enabled. VMware vSwitches using Source Port ID for Load Balancing and the vNICs are active/active for Management and VM traffic. No port channel or Etherchannel is configured on the swtiches.

After creating a VM for vCenter and joining the hosts to it, I started migrating existing VMs from an old cluster and saw odd network behavior. It seems whenever a VM or vmkernel interface are on any vNIC pinned to the same physical uplink on the same host, they can’t talk to each other.

Example: VM1 and VM2 are on host 1. VM1′s network adapter is pinned to vNIC1 using VLAN 10 which uses physical uplink 0. VM2′s network adapter is pinned to vNIC2 using VLAN 10 which uses physical uplink 1. VM1 and VM2 can ping each other. If I move VM2′s network adapter to any vNIC that uses physical uplink 0, just like VM1, then VM1 and VM2 can no longer ping each other. Or if host 1′s management IP is pinned to vNIC 1 on physical uplink 0, any VM on host 1 using a vNIC also pinned to physical uplink 0 cannot ping the host.

Anyone have any ideas?  Is something configured incorrectly on the switch?  On the VIC?

Thanks!

[padramas · Accepted solution]

Hello Michael,

I assume VIC is operating Classical Ethernet ( CE ) mode and VM1 and VM2 are in different vswitchs for the non-working scenario, ( where there vNICs used by vSwitch are pinned to physical uplink ) .

If this is the scenario, the frames have to be switched upstream switch and needs to be forwarded back on the same switch interface. Any IEEE standards based switch will not forward such traffic flow.

You might want to take advantage of Adapter-FEX or adapt changes to design to accommodate such traffic flows.

HTH

Padma

[msassemidwave]

This is exactly the issue.  We've also been told by Cisco support that a Nexus switch can accomodate multiple vNICs in an active/active configuration just like one would set up on the B-Series running VMware, but a Catalyst cannot.

To get around this we've pinned Management traffic to a vNIC on physical uplink 0 and VM traffic to physical uplink 1 so they'll have no issues talking to each other.

Thanks for the help!