topic Re: VXLAN and Nexus 1000V L3 Control Mode in Data Center Switches

VXLAN and Nexus 1000V L3 Control Mode

trerober — Thu, 30 May 2013 13:50:00 GMT

Hello All,

Has anyone had any issues getting VXLAN to work with the Nexus 1000V in L3 Control Mode?

I followed the L3 Control instructions found here: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/guide_c07-556626.html#wp9000165

I included the step to use the control0 interface on the VSM for VEM communication.

However, the VMs on my bridge domain could not communicate. When I did show ip mroute on the L3 Switch, the VXLAN Multicast Group showed up briefly and would then disappear. Also, the hosts where my VMs were running did not show up as members of the IGMP Group.

When I switched back to L2 Control mode, suddenly, VXLAN communications began working between the VMs.

I used the same procedure to setup VXLAN communications that I posted on my blog: http://vmtrooper.com/vxlan-on-ucs-and-vsphere-from-l3-to-nexus-1000v/

I'm not sure why L3 Control Mode presented a problem. I may have missed something when configuring the VSM for L3 Control Mode, but I did verify that the VSM was able to see the ESXi hosts properly when I added them (i.e. they showed up properly with the show module command). Also, my test VMs communicated just fine on a standard VLAN.

I plan on trying this config again when I have test servers available. Unfortunately, I could not continue troubleshooting since this environment was due for another project.

Any insights would be appreciated!

-Trevor

Re: VXLAN and Nexus 1000V L3 Control Mode

lwatta — Thu, 30 May 2013 14:22:05 GMT

Trevor,

I would need a little more info to really troubleshoot the issue. Do you still have the running config from the N1Kv?

Also I looked at your blog and I see you set the MTU on the NIC but did you change the MTU in the QOS settings for the best effort network class? Also looking at your environment if all your VXLAN capable ESXi hosts are just in the UCS system and the vmk interfaces are in the same subnet you should be able to just get by with igmp querier enabled on the UCS.

On the ESXi side were you using the same VMK for both l3 control and vxlan? We recommend VMK interfaces for VXLAN be used just for VXLAN.

The good news is that the next version of N1KV will have a version of VXLAN that does not require multicast :-)

louis

Re: VXLAN and Nexus 1000V L3 Control Mode

trerober — Thu, 30 May 2013 15:23:42 GMT

Hello louis,

Thanks for responding, and sorry to be so vague. I will not be able to replicate the environment any time soon since I had to get VXLAN working for the team project. As such, I don't know if the show run will be useful, but I will include it below anyway.

I totally neglected updating the Best Effort Network Class. It was still set to "normal" Since I was able to get the setup working before, I didn't think to check that item. It is now at 1600 MTU to match what I set on the vNICs.

I saw that the UCS can now be the IGMP querier (I think that capability may not have been availalbe when I wrote the article). That is something I'd like try in version 2.0 of my VXLAN Setup post (if the VXLAN functionality update doesn't arrive first!).

I am using separate vmkernels for VXLAN and L3 Control, and they are on different subnets. One thing I noticed in the 1000V Deployment Guide this morning:

"This scenario requires an additional vmkernel interface with an IP address, and a separate uplink (VMNIC) interface on the VMware ESXi host for the Layer 3 (VSM-to-VEM) interface."

In that case, I'll need to assign additional vNICs to the Service Profile and create a separate Ethernet Uplink Port-Profile just for L3 Control when using the control0 interface? I missed this step during my setup last night, and I will try this out during Round 2 of my VXLAN-L3 Control Setup.

Yes, I heard about the VXLAN update at EMC World. I'm looking forward to no longer depending on multicast!

Thanks,

Trevor

--show run---

!Command: show running-config

!Time: Thu May 30 14:53:53 2013

version 4.2(1)SV2(1.1a)

svs switch edition essential

no feature telnet

feature segmentation

feature network-segmentation-manager

ssh key rsa 2048

ip domain-lookup

ip host sspN1kv 172.16.94.2

hostname sspN1kv

errdisable recovery cause failed-port-state

vem 3

host vmware id 00000000-0000-0020-0000-000000000006

vem 4

host vmware id 00000000-0000-0020-0000-000000000005

vem 5

host vmware id 00000000-0000-0020-0000-000000000003

vem 6

host vmware id 00000000-0000-0020-0000-000000000004

bridge-domain vxtest

segment id 5001

group 239.1.1.1

vrf context management

ip route 0.0.0.0/0 172.16.94.1

vlan 1,140,144,155-156,161-162

vlan 140

name Vblock_VXLAN

vlan 144

name Vblock_VM_Mgmt

vlan 155

name Vblock_ESXi_Mgmt

vlan 156

name Vblock_ESXi_vMotion

vlan 161

name Vblock_N1K_Pkt-Ctrl

vlan 162

name Vblock_N1K_L3_Control

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type vethernet NSM_template_vlan

no shutdown

description NSM default port-profile for VLAN networks. Do not delete.

state enabled

port-profile type vethernet NSM_template_segmentation

no shutdown

description NSM default port-profile for VXLAN networks. Do not delete.

state enabled

port-profile type ethernet Unused_Or_Quarantine_Uplink

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type ethernet Uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 140,144,155-156,161-162,180

mtu 1600

channel-group auto mode on mac-pinning

no shutdown

system vlan 140,144,155-156,161-162

state enabled

port-profile type vethernet VXLAN

vmware port-group

switchport mode access

switchport access vlan 140

capability vxlan

no shutdown

system vlan 140

state enabled

port-profile type vethernet tenant-vxlan

vmware port-group

switchport mode access

switchport access bridge-domain vxtest

no shutdown

state enabled

port-profile type vethernet ControlPacket

vmware port-group

switchport mode access

switchport access vlan 161

no shutdown

system vlan 161

state enabled

port-profile type vethernet SSP_Infrastructure_Management

vmware port-group

switchport mode access

switchport access vlan 144

no shutdown

system vlan 144

state enabled

port-profile type vethernet ESXi_Management

vmware port-group

switchport mode access

switchport access vlan 155

no shutdown

system vlan 155

state enabled

port-profile type vethernet ESXi_vMotion

vmware port-group

switchport mode access

switchport access vlan 156

no shutdown

system vlan 156

state enabled

vdc sspN1kv id 1

limit-resource vlan minimum 16 maximum 2049

limit-resource monitor-session minimum 0 maximum 2

limit-resource vrf minimum 16 maximum 8192

limit-resource port-channel minimum 0 maximum 768

limit-resource u4route-mem minimum 1 maximum 1

limit-resource u6route-mem minimum 1 maximum 1

interface port-channel1

inherit port-profile Uplink

vem 5

interface port-channel2

inherit port-profile Uplink

vem 3

interface port-channel3

inherit port-profile Uplink

vem 4

interface port-channel4

inherit port-profile Uplink

vem 6

interface mgmt0

ip address 172.16.94.2/24

interface Vethernet1

inherit port-profile ESXi_Management

description VMware VMkernel, vmk0

vmware dvport 320 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0025.B52A.0000

interface Vethernet2

inherit port-profile ESXi_vMotion

description VMware VMkernel, vmk2

vmware dvport 352 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.5664.7715

interface Vethernet3

inherit port-profile VXLAN

description VMware VMkernel, vmk3

vmware dvport 64 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.566A.1D28

interface Vethernet4

inherit port-profile ESXi_Management

description VMware VMkernel, vmk0

vmware dvport 321 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0025.B52A.0001

interface Vethernet5

inherit port-profile ESXi_vMotion

description VMware VMkernel, vmk2

vmware dvport 353 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.566E.3D73

interface Vethernet6

inherit port-profile VXLAN

description VMware VMkernel, vmk3

vmware dvport 65 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.5669.95E6

interface Vethernet7

inherit port-profile tenant-vxlan

description Test 2, Network Adapter 1

vmware dvport 385 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.56AB.5E9E

interface Vethernet10

inherit port-profile tenant-vxlan

description Test 1, Network Adapter 1

vmware dvport 386 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.56AB.1999

interface Vethernet12

inherit port-profile ESXi_Management

description VMware VMkernel, vmk0

vmware dvport 322 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0025.B52A.0004

interface Vethernet13

inherit port-profile ESXi_vMotion

description VMware VMkernel, vmk2

vmware dvport 354 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.5663.7674

interface Vethernet14

inherit port-profile VXLAN

description VMware VMkernel, vmk3

vmware dvport 67 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.5661.51CE

interface Vethernet16

inherit port-profile ESXi_Management

description VMware VMkernel, vmk0

vmware dvport 323 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0025.B52A.0003

interface Vethernet17

inherit port-profile VXLAN

description VMware VMkernel, vmk3

vmware dvport 66 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.5668.CA20

interface Vethernet18

inherit port-profile ESXi_vMotion

description VMware VMkernel, vmk2

vmware dvport 355 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

vmware vm mac 0050.5665.6567

interface Ethernet3/1

inherit port-profile Uplink

interface Ethernet3/2

inherit port-profile Uplink

interface Ethernet4/1

inherit port-profile Uplink

interface Ethernet4/2

inherit port-profile Uplink

interface Ethernet5/1

inherit port-profile Uplink

interface Ethernet5/2

inherit port-profile Uplink

interface Ethernet6/1

inherit port-profile Uplink

interface Ethernet6/2

inherit port-profile Uplink

interface control0

ip address 172.16.72.2/24

line console

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-1

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-2

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-2

ip route 0.0.0.0/0 172.16.72.1

ip route 0.0.0.0/0 172.16.73.1

svs-domain

domain id 559

control vlan 161

packet vlan 161

svs mode L2

svs connection vCenter

protocol vmware-vim

remote ip address 172.16.94.5 port 80

vmware dvs uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64" datacenter-name SSP

max-ports 8192

connect

vservice global type vsg

tcp state-checks invalid-ack

tcp state-checks seq-past-window

no tcp state-checks window-variation

no bypass asa-traffic

vnm-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level

network-segment manager switch

mgmt-server-uuid "2A9A0407-09F1-4DB6-9457-114DC3945C41"

dvs name sspN1kv

dvs uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

network-segment policy default_vlan_template

description Default template used for VLAN backed pools

type vlan

import port-profile NSM_template_vlan

network-segment policy default_segmentation_template

description Default template used for isolation backed pools

type segmentation

import port-profile NSM_template_segmentation

Re: VXLAN and Nexus 1000V L3 Control Mode

lwatta — Thu, 30 May 2013 16:43:11 GMT

Your config looks good.

That comment about the additional uplinks from the deployment guide is news to me.

I always just created one big port-channel and made VXLAN part of that PC.

I'll see if I can get clarification on it.

There is a great slide deck on VXLAN on Ciscolive365 that was done by Larry Kreeger. I'd highly recommend taking a look at it. VXLAN will try to do the right thing when it comes to MTU. It will try to do path discovery. When it comes to multicast in a lot of cases VXLAN looks like it's working but the switch is really just flooding.

louis

Re: VXLAN and Nexus 1000V L3 Control Mode

trerober — Thu, 30 May 2013 18:25:47 GMT

Thanks lous!

will check it out. That separate vmnic advice in the guide seemed to be specific to when the control0 interface is used explicitly for the VEM communication. I didn't see it mentioned elsewhere.

Either way, it would be good to know definitively.

-Trevor