With the vSwitch you can only use the default LB hashing with UCS (route based on virtual port ID). Using IP hashing is used only when you have both pnics (vmnics) port channeled, which you don't. UCS does not port channel the A & B fabric uplinks together, but rather the port channeling only occurs between each IOM and FI respectfully assuming you have 6200 series FI and 2200 series IOMs.
What's likely happening is you're two VMs having a problem communicating are getting pinned to different fabrics.
Ex. If you have two VMs on two different hosts, and the vSwitch hash happens to send both their traffic via the Fabric-A uplink - then traffic will hit the same FI, switch locally and come back down the same path. If the two VMs get hashed up different uplinks (one to FI-A, and the other to FI-B), then in order to reach each other (even within the same VLAN), they need to go the northbound LAN, switched in the upstream LAN and back down the other side. This is because there is no interconnection of dataplane traffic between the FI's, only control traffic is clustered.
When you Vmotion your VM back and forth the vSwitch hashing can change each time, so sometimes both VMs might end up hashing to the same fabric, and othertimes not. This would also explain the strange behavior when you have the LB hashing set to IP hash. Each flow has a 50/50 chance of working.
When the two VMs can't communicate, go to your upstream LAN switch and do a "show mac address vlan x" and see if you find both VM's MAC Addresses visible. If not you've likely either not created that VLAN on the upstream switches, or you haven't allowed it over one of the trunks in the path. Ensure the LB is set back to the default Virtual Port ID hash also.
Regards,
Robert
