VXLAN and Nexus 1000V L3 Control Mode

[trerober]

Hello All,

Has anyone had any issues getting VXLAN to work with the Nexus 1000V in L3 Control Mode?

I followed the L3 Control instructions found here: http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/guide_c07-556626.html#wp9000165

I included the step to use the control0 interface on the VSM for VEM communication.

However, the VMs on my bridge domain could not communicate.  When I did show ip mroute on the L3 Switch, the VXLAN Multicast Group showed up briefly and would then disappear.  Also, the hosts where my VMs were running did not show up as members of the IGMP Group.

When I switched back to  L2 Control mode, suddenly, VXLAN communications began working between the VMs.

I used the same procedure to setup VXLAN communications that I posted on my blog: http://vmtrooper.com/vxlan-on-ucs-and-vsphere-from-l3-to-nexus-1000v/

I'm not sure why L3 Control Mode presented a problem.  I may have missed something when configuring the VSM for L3 Control Mode, but I did verify that the VSM was able to see the ESXi hosts properly when I added them (i.e. they showed up properly with the show module command).  Also, my test VMs communicated just fine on a standard VLAN.

I plan on trying this config again when I have test servers available.  Unfortunately, I could not continue troubleshooting since this environment was due for another project.

Any insights would be appreciated!

-Trevor

[lwatta]

Trevor,

I would need a little more info to really troubleshoot the issue. Do you still have the running config from the N1Kv?

Also I looked at your blog and I see you set the MTU on the NIC but did you change the MTU in the QOS settings for the best effort network class? Also looking at your environment if all your VXLAN capable ESXi hosts are just in the UCS system and the vmk interfaces are in the same subnet you should be able to just get by with igmp querier enabled on the UCS.

On the ESXi side were you using the same VMK for both l3 control and vxlan? We recommend VMK interfaces for VXLAN be used just for VXLAN.

The good news is that the next version of N1KV will have a version of VXLAN that does not require multicast :-)

louis

[trerober]

Hello louis,

Thanks for responding, and sorry to be so vague.  I will not be able to replicate the environment any time soon since I had to get VXLAN working for the team project.  As such, I don't know if the show run will be useful, but I will include it below anyway.

I totally neglected updating the Best Effort Network Class.  It was still set to "normal"  Since I was able to get the setup working before, I didn't think to check that item.  It is now at 1600 MTU to match what I set on the vNICs.

I saw that the UCS can now be the IGMP querier (I think that capability may not have been availalbe when I wrote the article).  That is something I'd like try in version 2.0 of my VXLAN Setup post (if the VXLAN functionality update doesn't arrive first!).

I am using separate vmkernels for VXLAN and L3 Control, and they are on different subnets.  One thing I noticed in the 1000V Deployment Guide this morning:

"This scenario requires an additional vmkernel interface with an IP address, and a separate uplink (VMNIC) interface on the VMware ESXi host for the Layer 3 (VSM-to-VEM) interface."

In that case, I'll need to assign additional vNICs to the Service Profile and create a separate Ethernet Uplink Port-Profile just for L3 Control when using the control0 interface?  I missed this step during my setup last night, and I will try this out during Round 2 of my VXLAN-L3 Control Setup.

Yes, I heard about the VXLAN update at EMC World.  I'm looking forward to no longer depending on multicast!

Thanks,

Trevor

--show run---

!Command: show running-config

!Time: Thu May 30 14:53:53 2013

version 4.2(1)SV2(1.1a)

svs switch edition essential

no feature telnet

feature segmentation

feature network-segmentation-manager

ssh key rsa 2048

ip domain-lookup

ip host sspN1kv 172.16.94.2

hostname sspN1kv

errdisable recovery cause failed-port-state

vem 3

  host vmware id 00000000-0000-0020-0000-000000000006

vem 4

  host vmware id 00000000-0000-0020-0000-000000000005

vem 5

  host vmware id 00000000-0000-0020-0000-000000000003

vem 6

  host vmware id 00000000-0000-0020-0000-000000000004

bridge-domain vxtest

  segment id 5001

  group 239.1.1.1

vrf context management

  ip route 0.0.0.0/0 172.16.94.1

vlan 1,140,144,155-156,161-162

vlan 140

  name Vblock_VXLAN

vlan 144

  name Vblock_VM_Mgmt

vlan 155

  name Vblock_ESXi_Mgmt

vlan 156

  name Vblock_ESXi_vMotion

vlan 161

  name Vblock_N1K_Pkt-Ctrl

vlan 162

  name Vblock_N1K_L3_Control

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type vethernet NSM_template_vlan

  no shutdown

  description NSM default port-profile for VLAN networks. Do not delete.

  state enabled

port-profile type vethernet NSM_template_segmentation

  no shutdown

  description NSM default port-profile for VXLAN networks. Do not delete.

  state enabled

port-profile type ethernet Unused_Or_Quarantine_Uplink

  vmware port-group

  shutdown

  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

  vmware port-group

  shutdown

  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type ethernet Uplink

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 140,144,155-156,161-162,180

  mtu 1600

  channel-group auto mode on mac-pinning

  no shutdown

  system vlan 140,144,155-156,161-162

  state enabled

port-profile type vethernet VXLAN

  vmware port-group

  switchport mode access

  switchport access vlan 140

  capability vxlan

  no shutdown

  system vlan 140

  state enabled

port-profile type vethernet tenant-vxlan

  vmware port-group

  switchport mode access

  switchport access bridge-domain vxtest

  no shutdown

  state enabled

port-profile type vethernet ControlPacket

  vmware port-group

  switchport mode access

  switchport access vlan 161

  no shutdown

  system vlan 161

  state enabled

port-profile type vethernet SSP_Infrastructure_Management

  vmware port-group

  switchport mode access

  switchport access vlan 144

  no shutdown

  system vlan 144

  state enabled

port-profile type vethernet ESXi_Management

  vmware port-group

  switchport mode access

  switchport access vlan 155

  no shutdown

  system vlan 155

  state enabled

port-profile type vethernet ESXi_vMotion

  vmware port-group

  switchport mode access

  switchport access vlan 156

  no shutdown

  system vlan 156

  state enabled

vdc sspN1kv id 1

  limit-resource vlan minimum 16 maximum 2049

  limit-resource monitor-session minimum 0 maximum 2

  limit-resource vrf minimum 16 maximum 8192

  limit-resource port-channel minimum 0 maximum 768

  limit-resource u4route-mem minimum 1 maximum 1

  limit-resource u6route-mem minimum 1 maximum 1

interface port-channel1

  inherit port-profile Uplink

  vem 5

interface port-channel2

  inherit port-profile Uplink

  vem 3

interface port-channel3

  inherit port-profile Uplink

  vem 4

interface port-channel4

  inherit port-profile Uplink

  vem 6

interface mgmt0

  ip address 172.16.94.2/24

interface Vethernet1

  inherit port-profile ESXi_Management

  description VMware VMkernel, vmk0

  vmware dvport 320 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0025.B52A.0000

interface Vethernet2

  inherit port-profile ESXi_vMotion

  description VMware VMkernel, vmk2

  vmware dvport 352 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.5664.7715

interface Vethernet3

  inherit port-profile VXLAN

  description VMware VMkernel, vmk3

  vmware dvport 64 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.566A.1D28

interface Vethernet4

  inherit port-profile ESXi_Management

  description VMware VMkernel, vmk0

  vmware dvport 321 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0025.B52A.0001

interface Vethernet5

  inherit port-profile ESXi_vMotion

  description VMware VMkernel, vmk2

  vmware dvport 353 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.566E.3D73

interface Vethernet6

  inherit port-profile VXLAN

  description VMware VMkernel, vmk3

  vmware dvport 65 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.5669.95E6

interface Vethernet7

  inherit port-profile tenant-vxlan

  description Test 2, Network Adapter 1

  vmware dvport 385 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.56AB.5E9E

interface Vethernet10

  inherit port-profile tenant-vxlan

  description Test 1, Network Adapter 1

  vmware dvport 386 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.56AB.1999

interface Vethernet12

  inherit port-profile ESXi_Management

  description VMware VMkernel, vmk0

  vmware dvport 322 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0025.B52A.0004

interface Vethernet13

  inherit port-profile ESXi_vMotion

  description VMware VMkernel, vmk2

  vmware dvport 354 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.5663.7674

interface Vethernet14

  inherit port-profile VXLAN

  description VMware VMkernel, vmk3

  vmware dvport 67 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.5661.51CE

interface Vethernet16

  inherit port-profile ESXi_Management

  description VMware VMkernel, vmk0

  vmware dvport 323 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0025.B52A.0003

interface Vethernet17

  inherit port-profile VXLAN

  description VMware VMkernel, vmk3

  vmware dvport 66 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.5668.CA20

interface Vethernet18

  inherit port-profile ESXi_vMotion

  description VMware VMkernel, vmk2

  vmware dvport 355 dvswitch uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

  vmware vm mac 0050.5665.6567

interface Ethernet3/1

  inherit port-profile Uplink

interface Ethernet3/2

  inherit port-profile Uplink

interface Ethernet4/1

  inherit port-profile Uplink

interface Ethernet4/2

  inherit port-profile Uplink

interface Ethernet5/1

  inherit port-profile Uplink

interface Ethernet5/2

  inherit port-profile Uplink

interface Ethernet6/1

  inherit port-profile Uplink

interface Ethernet6/2

  inherit port-profile Uplink

interface control0

  ip address 172.16.72.2/24

line console

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-1

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-2

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-2

ip route 0.0.0.0/0 172.16.72.1

ip route 0.0.0.0/0 172.16.73.1

svs-domain

  domain id 559

  control vlan 161

  packet vlan 161

  svs mode L2 

svs connection vCenter

  protocol vmware-vim

  remote ip address 172.16.94.5 port 80

  vmware dvs uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64" datacenter-name SSP

  max-ports 8192

  connect

vservice global type vsg

  tcp state-checks invalid-ack

  tcp state-checks seq-past-window

  no tcp state-checks window-variation

  no bypass asa-traffic

vnm-policy-agent

  registration-ip 0.0.0.0

  shared-secret **********

  log-level

network-segment manager switch

  mgmt-server-uuid "2A9A0407-09F1-4DB6-9457-114DC3945C41"

  dvs name sspN1kv

  dvs uuid "63 33 2b 50 f8 79 3f 8d-a0 5d 9e f0 1e b6 63 64"

network-segment policy default_vlan_template

  description Default template used for VLAN backed pools

  type vlan

  import port-profile NSM_template_vlan

network-segment policy default_segmentation_template

  description Default template used for isolation backed pools

  type segmentation

  import port-profile NSM_template_segmentation

[lwatta · Accepted solution]

Your config looks good.

That comment about the additional uplinks from the deployment guide is news to me.

I always just created one big port-channel and made VXLAN part of that PC.

I'll see if I can get clarification on it.

There is a great slide deck on VXLAN on Ciscolive365 that was done by Larry Kreeger. I'd highly recommend taking a look at it. VXLAN will try to do the right thing when it comes to MTU. It will try to do path discovery. When it comes to multicast in a lot of cases VXLAN looks like it's working but the switch is really just flooding.

louis

[trerober]

Thanks lous!

will check it out.  That separate vmnic advice in the guide seemed to be specific to when the control0 interface is used explicitly for the VEM communication.  I didn't see it mentioned elsewhere.

Either way, it would be good to know definitively.

-Trevor