Nexus 1000v L3 design question

[tomazklancnik]

I have the following:

VSM - mgmt0 as SVS L3 control interface

mgmt IP - 10.1.2.1/24

mgmt vlan = 99

ESXi:

VMware ESXi host/VEM using separate Layer3 VMkernel interface for L3 control communication with VSM (Scenario 2 in Cisco Nexus 1000V Series Switches Deployment Guide Version 3 - Cisco)

vmk0 (on Switch0): 10.1.2.2/24 (ESXi mgmt, vlan 99)

vmk1 (on N1Kv): 192.168.10.2/24 (control, vlan 10)

Will communication between VSM and VEM (vmk1) work? How can I add a static route for traffic from vmk1 (VEM) to mgmt IP (VSM) since mgmt IP (VSM) is on the same subnet as vmk0 on ESXi?

[Robert Burns]

Tomaz,

Since you can only have one Default GW on your host (which belongs to your Mgmt vmk) you're limited to what you can do in regards to this scenario. I wouldn't advise using static routes myself, it gets really messy.

Supported options for your scenario are:

a.) Have separate vmknic interfaces for hypervisor and VEM management but

they would have to be same L3 Subnet as they would share the same default gateway

configuration. This leads to some unnecessary duplication of physical uplinks and not recommended.

b.) Have a single combined management vmknic for both hypervisor and VEM

management. It does require that you host your hypervisor management interface on the VEM rather

than a standard vSwitch.

c) Have separate vmknic for VEM Mgmt in different VLAN/subnet than Host Mgmt vmk, and installing static routes on the host and any intermediate devices (routers). VMware KB: Configuring static routes for vmkernel ports on an ESXi host

Option B is the recommended deployment and exactly what I would recommend.

Regards,

Robert

[tomazklancnik]

Hello Robert,

thank you for your quick reply. Customer would like to have out-of-band mgmt of ESXi so option b) is not suitable for them.

We decided to go with the option where VSM will use a dedicated Control 0 interface on VSM for VSM-to-VEM communication.

Something like that:

VSM:

mgmt 0: 10.1.2.1/24 (vlan 99)

control 0: 192.168.10.1/24 (vlan 10)

ESXi:

vmk0 (on Switch0): 10.1.2.2/24 (ESXi mgmt, vlan 99)

vmk1 (on N1Kv): 192.168.10.2/24 (control, vlan 10)

Maybe I don't even need to add any static routes now since control 0 (VSM) and vmk1 (VEM) are in the same subnet. Correct?

[Robert Burns · Accepted solution]

Correct.  This is pretty close to L2 mode, but will meet your requirements.

Cheers,

Robert

[Jeremy Waldrop]

So I have a customer looking to deploy N1KV in a similar fashion. They want ESX host mgmt vmkernel to stay on vSwitch0 and have a second vmkernel hosted on the N1KV for VEM-VSM communication.

I am doing some lab testing and have the VSM setup in L3 mode with an MGMT IP of 10.1.32.200. My ESXi host mgmt vmkernel that is living on vSwitch0 is 10.1.32.21. I have two dedicated vmnics for attaching to N1KV Uplink port-profile and an IP in a different VLAN of 10.1.38.121. I was thinking I would have to add a static route on the esx host for the 10.1.38.21 to route and talk to the VSM on 10.1.32.200 but it seem that the VSM is actually talking to 10.1.32.21 and even though the vmkernel that lives on the N1KV is 10.1.38.121.

I did some additional testing of removing the 10.1.38.121 vmkernel from the host and the VEM goes offline. If I add it back it comes back online. I can't ping 10.1.38.121 from the VSM but it still works just fine. It seem like as long as there is a vmkernel on a port-group that has capability l3control and a system vlan it works.

Does this sound right?