cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

Setting up L3 control on Nexus 1000V

196516
Views
16
Helpful
22
Comments
lwatta
Cisco Employee

In version 1.2 of the Nexus 1000V we introduced L3 based connectivity for control traffic between the VSM and its VEMs. We did this because in some environments it’s just not possible to have L2 connectivity between VSM and all the VEMs that will be attached.

It was still a best practice to use L2 based control but we offered L3 based connectivity for customers that needed it.

Going forward you will see situations where we need L3 based control over L2. Why? Well for one as we introduce integration with other products using L3 allows us to be more flexible in our deployments. L3 control is also easier to troubleshoot. You can use simple tools like ping to verify network connectivity between the VSM and VEM.

The downside is that it requires a VMK interface on the ESX host and an IP address. This is yet another management interface that you have to deal with.

The goal of this write up is to show you how to setup L3 control between the VSM and VEM.

The Nexus 1000V VSM VM has 3 network interfaces.

  • Control (control0)
  • Management (mgmt0)
  • Packet

When setting up L3 control you have two options.

  • L3 control through VSM mgmt0 interface (default)
  • L3 control through VSM control0 interface

Several important notes

  • When setting up L3 control both “control and packet” traffic are forwarded over the same interface.  In L3 control mode the packet nic does not get used on the VM, but don’t delete it. The VM still expects that nic to be present.
  • Also control0 and mgmt0 use a different VRF. If you use control0 you will need to setup a route in the default VRF.

n1kv-l3# show vrf interface

Interface                     VRF-Name                    VRF-ID

control0                      default                          1

mgmt0                         management                       2

When should I use dedicated VSM Control interface for L3?

  • If you want to isolate your control and management to the VSM VM.
  • If you want your control interface to be a in different subnet than the management interface

Sample topology

  • VSM is in a 172.18.217.x subnet in VLAN 2
  • My ESXi hosts are in 192.168.10.x in VLAN 10 and 192.168.11.x  in VLAN 11

Configure the VSM for L3 control

When running setup on a newly created VSM choose L3 when you get to the following question during setup.

setup.jpg

At this point the VSM is configured for L3 but it will be using the mgmt 0 interface.

n1kv-l3# show svs domain

SVS domain config:

Domain id:    43 

Control vlan: 1 

Packet vlan:  1 

L2/L3 Control mode: L3

L3 control interface: mgmt0

Status: Config push to VC failed (communication failure with VC).

In L3 mode control and packet vlans will show as vlan 1. No need to change these. You can also see that the L3 control interface is mgmt0.

Now that we have the L3 control configured on the VSM, continue with normal setup.

  • Configure VSM to vCenter connectivity
  • Configure uplink port-profiles

One note about your uplink port-profile: You need to create and allow the VLAN that your ESX hosts will use to communicate to the VSM.  My ESX hosts are going to communicate on VLAN 10 and VLAN 11 using a VMK interface in the 192.168.10.x and 192.168.11.x subnets. My uplink port-profile needs to look like the following

port-profile type ethernet uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 10,11,150-152

no shutdown

system vlan 10,11

state enabled

VLAN 10 and 11 are allowed and set as a system VLAN.

Next we need to create a veth port-profile for the VMK interfaces that will be used for L3 control. I need to create 2 veth port-profiles. One for hosts in VLAN 10 and another for hosts in VLAN 11.

N1kv-l3(config-port-prof)# port-profile type veth L3-control-vlan10

n1kv-l3(config-port-prof)# switchport mode access

n1kv-l3(config-port-prof)# switchport access vlan 10

n1kv-l3(config-port-prof)# vmware port-group

n1kv-l3(config-port-prof)# no shutdown

n1kv-l3(config-port-prof)# capability L3control

n1kv-l3(config-port-prof)# system vlan 10

n1kv-l3(config-port-prof)# state enabled

n1kv-l3(config-port-prof)# port-profile type vethernet L3-control-vlan11

n1kv-l3(config-port-prof)# switchport mode access

n1kv-l3(config-port-prof)# switchport access vlan 11

n1kv-l3(config-port-prof)# vmware port-group

n1kv-l3(config-port-prof)# no shut

n1kv-l3(config-port-prof)# capability L3control

n1kv-l3(config-port-prof)# system vlan 11

n1kv-l3(config-port-prof)# state enabled

Now comes the easy part. We need to make sure that ESXi hosts can ping the VSM so we know they have network connectivity.  With L2 control there was no easy way to verify network connectivity between the VSM and VEMs.

Next add the ESXi hosts to the Nexus 1000V in VMware vCenter. If you have VUM installed the VEM modules will automatically get loaded. If you don’t have VUM installed you will need to manually install the VEM modules on the ESXi hosts.

When you add the host, it will show up in VMware vCenter as added, but it will not show up as a module on the VSM until you add a VMK interface to the veth port-profile.

Do I need a dedicated VMK interface for control?

No. We are prefectly fine with customers migrating the mgmt vmk interface from the ESXi host to the VEM and using that interface. There have been changes on the VEM and VSM to better deal with higher latency networks and drops. Actually migrating the mgmt vmk interface makes life easier because there is no concern about needing to add static routes.

Create and add the VMK interface to the veth port-profile

Once you add the vmknic interface to either the L3-control-vlan10 or L3-control-vlan11 port-profile the VEM will know that it now has a way to communicate with the VSM. The VEM should show up on the VSM with a "show module" command and the VMK interface will show up with "show int virtual"

Note: We only support a single VMK interface per host to be assigned to the L3capability port-profile.

That's it for getting L3 control working with mgmt 0 on the VSM.

How about a quick example with control0

This time we will use

  • Mgmt0 172.18.217.245/24 on VLAN 2
  • Control0 192.168.150.10/24 on VLAN 150
  • ESX hosts on 192.168.10.x/24 on VLAN 10
  • ESX hosts on 192.168.11.x/24 on VLAN 11

Make sure control 0 NIC is on the right VMware port-group

vsm.jpg

Assuming a new install and you set L3 control in the setup script we pick up with configuring control 0.

n1kv-l3(config)# int control 0

n1kv-l3(config-if)# ip address 192.168.150.10 255.255.255.0

Next change the SVS domain to use control0 instead of mgmt0

n1kv-l3(config-if)# svs-domain

n1kv-l3(config-svs-domain)# svs mode L3 interface control0

See if you can ping the 192.168.150.10 interface from the ESXi hosts. You should not be able to since we have not yet set a default route for the default VRF.

n1kv-l3(config)# vrf context default

n1kv-l3(config)# ip route 0.0.0.0/0 192.168.150.1

Now add the hosts to the Nexus 1000V and you should be set.

You can verify with “show module” and “show module vem counters” to verify you are getting heartbeat messages between the VSM and VEM.

Troubleshooting:

Some quick troubleshooting tips for L3 control

First make sure that you have “capability L3control” in your veth port-profile that the VMK interface will be connected to. Without that the VSM does not know that L3 control is supposed to be used on that port-profile.

Second make sure you have the right system VLANs in the eth and veth port-profile.

On the ESXi hosts you can use "vemcmd show port-drops ingress/egress", "vemcmd show stats"  and "vemcmd show packets" to see if you are receiving packets on the control VMK interface. On the VSM you can run "show module vem counters" to see if you are receiving packets from the VEMs.

Remember that L3 controls uses UDP and port 4785 for both tx and rx. Keep this in mind for any firewalls that might be present.


22 Comments
Juan Olivares
Cisco Employee

Hi Louis,

I followed your recommendation enabling MAC-pinning, since the upstream FEX with NX-OS 5.0(3)N2(2b) doesn't support ACTIVE/ACTIVE vPC and we have only 2 NICs for MGMT.

For future deployment, we are planning to upgrade NX-OS to "Gold Coast 5.1".

Thanks again!

// Juan

prasad.gsmc
Beginner

Hi,

When you change the SVS mode to L3, dose the VSM to VEM communication will happen using Control 0 interface ? or always VSM to vCenter is using Mgmt 0 ?

Please help

regards

Prasad K

Juan Olivares
Cisco Employee

Hi Prasand,

Actually you can use mgmt0 or control0. For simplicity and best practice I'll recommend mgmt0.

svs-domain

domain id X

control vlan 1

packet vlan 1

svs mode L3 interface mgmt0

I hope this helps.

Regards!

// Juan

Luis Perez
Beginner

Hy everyone. Im going to deploy a Nexus1000v in an UCS environment with all of my servers with ESXi having 4 vNICs. Two of them are used for the VMware vSwitch for management, vMotion and Fault Tolerance; the other two interfaces are configured for Data Traffic using the VMware dVSwitch.

I was planning to migrate only the data interfaces and the vMotion and FT vmkernel interfaces to the Nexus 1000V using two of the vNICs, and left the other two interfaces with management of the ESXi in the vSwitch.

The management of the ESXi hosts is in VLAN 20 (10.1.15.X), but im going to put the VSM management interface in VLAN 99 (10.1.9.X). I have to use L3 mode in the N1000v deployment or can i use the L2 mode??? What will be your recomendation, if i use the L3 mode i can use the same management interface in the ESXi host or is mandatory to create another vmkernel interface???

Fernando Bello
Beginner

Hello Louis,

Since now we have L2 and L3 control mode, what is the best practice or recommendation, using L2 or using L3?

thanks in advance,

Fernando

Keny Perez
Collaborator

Fernando,

L3 is recommended and easier to troubleshoot.

-Kenny

asgar.shaikh
Beginner

Hi, admin

Please advise  if we use mgmt0  for L3 Communication

svs-domain

domain id X

control vlan 1

packet vlan 1

svs mode L3 interface mgmt0

do we need to add command

"capability l3control" ..?

if  yes then to which veth port-profile Please help Thanks

Content for Community-Ad