Hi,
Wondering if anyone has seen this problem before:
We have quite a large distributed VMware environment where we have the multiple Virtual Centers located at HQ managing about 200 remote global sites where each site has 2xESX server running the 1000v (Some sites are terrestrial, some satellite links high latency/low bandwidth). Each server has 2 physical uplinks to 3750 stack, using LACP portchannels. About 2 times a month we deal with situations where a single VM loses network connectivity, other VMs on that server have no issues.
Checked everything on the 1000v and 3750, the configs all look good, VLANs are trunked properly. When I check the VEM itself. I saw something curious, I was seeing that one of the uplink ports of the etherchannel showed that there was some VLANs blocked "F/B: Port is BLOCKED on some of the vlan". When I displayed the port VLANs, I see VLANs missing. So based on the symptoms, I made the assumption that the VM that has lost communication is being hashed to the uplink port with the blocked VLANs, thus the reason why it has lost connectivity to the network.
jkrtav01# module vem 6 execute vemcmd show port vlans
Native VLAN Allowed
LTL VSM Port Mode VLAN State Vlans
18 Eth6/2 T 1 FWD 1,64,80,500
22 Eth6/6 T 1 FWD 500 <-------------missing VLAN 1, 64, 80 --- where 500 is a system VLAN
49 Veth2 A 500 FWD 500
50 Veth1 A 500 FWD 500
51 Veth13 A 80 FWD 80
52 Veth9 A 64 FWD 64
53 Veth5 A 64 FWD 64
54 Veth8 A 1 FWD 1
305 Po4 T 1 FWD 1,64,80,500
Any one of the following procedures fixes this issue:
1) reload the VSM
2) switchover the VSM to the secondary (or back to primary)
3) bounce the uplink port on the corresponding 3750 port.
Based on the fixes, I'm guessing that there is a synchronization issue between the VSM-> VC-> ESX Host. The fixes initiates a resync of the port-profile and the proper VLANs are configured on the port.
Running Version: 4.2(1)SV1(4a)
Sometimes this issues shows itself where VMs can't talk to each other between ESX hosts, but they can talk to devices throughout the network. Again I'm guessing that this is because the etherchannel hash pushes some traffic to the "bad" port.
Any insight would be helpful
Thanks,
Tho
