1000v in L3 mode issues

[miwitte]

Hey been working with  with the 1000v, we have it up and running fine in L3 mode, we have a VM in back of the 1000v all is good. The issue occurs when we try to migrate the actual VSM from the vSwitch onto the DVS. It appears we lose the L3 heartbeat between the VEM and the vmkernel interface, the VEM goes down and we lose management of the VSM ect. I haven’t done a whole lot of L3 so I am trying to go through docs to figure out what’s going on. Does anyone have any working configs of L3 where the vmkernel and the VSM management are on different subnets and both are managed by the DVS? we want the entire ESXi host to be on the 1000v.

Currently we have this and it works;

·        The vmkernel is on the DVS on VLAN 110

·        The vmkernel is in the DVS on a vethernet port profile with capability L3 and system VLAN 110

·        The VSM management control and packet are in the standard vSwitch on VLAN 101 in a separate subnet.

When I migrate the VSM management, control and packet to a port profile with VLAN 101 in the DVS, we lose the VEM and of course the port profiles and nay connectivity.

·        From what I just read it appears that the vmkernel and the VSM management should be in the same VLAN for L3 control to work but is it a requirement? That is the only thing I can see that would cause this.

·        Could it be that because before we migrate the VSM management port to the DVS, the default route is to the gateway in VLAN 101 now that it’s in the DVS would we have to then route that traffic to VLAN 110

when it fails, the Active VSM management is in a vethernet port profile on VLAN 101 on the DVS, the ESXi vmkernel is in a vethernet port profile onVLAN110. I am close I think it’s a routing issue as its no longer L2 and that’s breaking the VEM/VSM communication.   

• when the vmkernel is on the DVS the flow from VEM->VSM   VEM-> port-profile type vethernet with capability L3->vmkernel->pNIC->upstream physical switch on vlan 110->default gateway110->default gateway 101->vlan 101->pNIC->standard vSwitch on VLAN 101->VSM.
• when the vmkernel and the VSM is on the DVS the flow from VEM->VSM   VEM->port-profile type vethernet with capability L3->vmkernel->pNIC->upstream physical switch on vlan 110->default gateway110->default gateway 101->vlan 101->pNIC->DVS on VLAN 101->VSM. The issue is that we lose the VEM-VSM so we lose the vethernet ports that the vmkernel and the VSM are on..

[Robert Burns]

Please paste your VSM config.  The VSM doesn't care what gateway you use.  That's a VMware VMKernel setting - has nothing to do with the 1000v.

I can't tell if you're using Mgmt0 or Control0 for L3, so paste the config and I'll tell you what's wrong.  Likley a system or native vlan problem on your uplinks (guessing).

Thanks,

Robert

[miwitte]

waiting for customer to email it back. On the vethernet port-profile for the management of the VSM(and also for control and packet) do I need to add the system vlan command? I know for L2 when we sued to have seprate control and packet vlans you used the susytem vlan command on those port profiles. Since the control traffic in L3 is now going through the managemtn would you need that added? I have search and not found a full config for L3 where the vmkernel and management were on seperate VLANS, AND in the DVS. So if my management vlan was 101 woudl i need system vlan 101 under the managemtn port profile because the control traffic is going there as well?

[Robert Burns]

Yes your Management interfaces need to be set as system VLANs.  Make VLAN 101 a "system vlan" in the vEth port profile AND the Uplink Eth Port Profile.

It's well documented that Control, Packet, Management and IP Storage profiles all need to be made system vlans.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4/port_profile/configuration/guide/n1000v_portprof_4system.html#wp1113896

Regards,

Robert

[miwitte]

Yeah i was trying to go over the config in my head, I had one guy doing the 1000v config and the VMware guy doing his thing all via webex so looks like we missed that. We are supposed to take a look at it this afternoon.

[cweinhold]

Robert,

L3 control is recommended now by Cisco. It would be very nice if Cisco's Nexus 1000v troubleshooting documentation has a section on VSM/VEM steps with L3 mode. The current docs are heavily skewed towards L2 with checking MAC's, running vem-health, etc.

L3 challenges can be daunting (e.g., asymmetric routing, static routes) and there isn't much guidance.

-Craig