N1K Nexus1000v VSM Cannot See VEM

canero · ‎04-18-2012

Hello,

We are installing a N1K 4.2(1)SV1(5.1) and for this purpose installed VSM on a Seperate DataCenter Vsphere 5 Cluster onto a blank ESX host. Our problem is that VSM and VEM cannot see each other. What step may we missing?

Thanks in Advance,

Best Regards,

We followed the Installation and Getting started guides:

Cisco Nexus 1000V Installation and Upgrade Guide, Release 4.2(1)SV1(5.1)

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_5_1/install_upgrade/vsm_vem/guide/n1000v_installupgrade.html

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_2_1_s_v_1_4_a/getting_started/configuration/guide/n1000v_gsg.html

from other Discussions we understand that something must be missing with respect to "Control Vlan" which will provide L2 communication between VSM and VEM.

show svs connection shows that vcenter recognizes the VSM
vem status on the ESX host shows the VEM as ok
management, control and packet vlans are same vlan
show module sees only the active and ha-standby vsms, but no vem exists.
show module vem mapping indicates that somehow VSM recognizes the VEM, if the VEM 3 is absent, how could it be seen by VSM once?
One point to note is VSM and VEM are on the same host, can this be a difference between usual configuration? Will test a case when VSM and VEM are on seperate ESX hosts.
if we put Vmware dVS instead of N1K and put a host on this Control Vlan, it may ping to other vlans, and the VMs for port-profiles can communicate to the outside world,
We don't think this is a licensing problem because by default 16 CPUs are on the VSM, and it did not change after the licens is installed.

show module vem mapping

Mod Status UUID License Status

--- ----------- ------------------------------------ --------------

3 absent 34343335-3237-4742-3838-346337417236 unlicensed

Robert Burns · ‎04-18-2012

Can you provide the output of "show run" from the VSM.

Can you clarify the exact VC toplogy. Provide a screen shot if need be.

It's very likely this is a system vlan or other configuration related issue.

Thanks,

Robert

Joseph Ristaino · ‎04-20-2012

It seems pretty obvious that the control vlan is not set as a system vlan on your vethernet port profile as well as your uplink port profile.

You will need to purchase licenses for your modules as well as even if you able to get the module online you will not be able to pass traffic on Veth ports.

Please upload the "show run" as Rob requested and we will look at your configuration.

Joey

biaozhou · ‎04-24-2012

Hello,

I am having the exact same problem as Canero with the same software version. My setup is as follows:

1. L3 mode with VSM and VEM on different subnets/hosts

2. With the same set of port profiles and UCS service profile cloned, I have one host showing up in VSM and another not showing up. In my view, this rule out mis-configuration in

a. system vlan on veth port profile

b. vlan configuration in upstream switch and UCS fabric interconnect

3. Sometimes, the VEM did managed to show up but can drop without any reason

4. Sniffer trace indicates that there are bi-directional traffic between VSM and VEM

5. show logging logfile has the following error:

2012 Apr 24 15:21:47 VSM-PROD %VMS-5-DVS_HOSTMEMBER_INFO: A host with name=[172.16.105.131] and uuid=[a7f30c6a-6eb0-11e1-2000-000000000006] is added to the dvs. The host is not found as a module in the VSM configuration.

6. Opaque Data in VC shows the following:

"data-version 1.0

switch-domain 200

switch-name VSM-PROD

cp-version 4.2(1)SV1(5.1)

control-vlan 1

system-primary-mac 00:50:56:ab:44:01

active-vsm packet mac 00:50:56:ab:44:03

active-vsm mgmt mac 00:50:56:ab:44:02

standby-vsm ctrl mac 0050-56ab-4404

inband-vlan 1

svs-mode L3

l3control-ipaddr 192.168.1.200

upgrade state 0 mac 0050-56ab-4404 l3control-ipv4 null

profile dvportgroup-149 trunk 105-106,109,111

profile dvportgroup-149 mtu 1500

profile dvportgroup-151 access 105

profile dvportgroup-151 mtu 1500

profile dvportgroup-152 access 106

profile dvportgroup-152 mtu 1500

profile dvportgroup-155 access 111

profile dvportgroup-155 mtu 1500

profile dvportgroup-155 capability l3control

profile dvportgroup-161 access 109

profile dvportgroup-161 mtu 1500

end-version 1.0

"

In VSM:

VSM-PROD# sh svs domain

SVS domain config:

Domain id: 200

Control vlan: 1

Packet vlan: 1

L2/L3 Control mode: L3

L3 control interface: mgmt0

Status: Config push to VC successful.

VSM-PROD# sh mod vem missing

Mod Server-IP Server-UUID Server-Name

--- --------------- ------------------------------------ --------------------

4 172.16.105.31 a7f30c6a-6eb0-11e1-2000-000000000007 NA

5 172.16.105.131 a7f30c6a-6eb0-11e1-2000-000000000006 NA

VSM-PROD# sh mod vem map

Mod Status UUID License Status

--- ----------- ------------------------------------ --------------

3 powered-up a7f30c6a-6eb0-11e1-2000-000000000008 licensed

4 absent a7f30c6a-6eb0-11e1-2000-000000000007 n/a

5 absent a7f30c6a-6eb0-11e1-2000-000000000006 n/a

In host:

Below is "show run"

VSM-PROD# sh run

!Command: show running-config

!Time: Tue Apr 24 16:34:38 2012

version 4.2(1)SV1(5.1)

feature telnet

username admin password 5 $1$oCIIPBun$l0pIWvJ/OeQ3mNSY38OTG0 role network-admin

banner motd #Nexus 1000v Switch#

ssh key rsa 2048

ip domain-lookup

hostname VSM-PROD

vem 3

host vmware id a7f30c6a-6eb0-11e1-2000-000000000008

vem 4

host vmware id a7f30c6a-6eb0-11e1-2000-000000000007

vem 5

host vmware id a7f30c6a-6eb0-11e1-2000-000000000006

snmp-server user admin network-admin auth md5 0xd86407e6438a2a6179f6581e7d378d4b priv 0xd86407e6438a2a6179f6581

e7d378d4b localizedkey

vrf context management

ip route 0.0.0.0/0 192.168.1.1

vlan 1,100-103,105-106,109,111-112,200

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet storage

vmware port-group

switchport mode access

switchport access vlan 109

no shutdown

system vlan 109

state enabled

port-profile type ethernet n1kv-uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 105-106,109,111-112

channel-group auto mode on mac-pinning

no shutdown

system vlan 105-106,109,111

state enabled

port-profile type vethernet vm

vmware port-group

switchport mode access

switchport access vlan 112

no shutdown

state enabled

port-profile type vethernet esx_mgmt

vmware port-group

switchport mode access

switchport access vlan 105

no shutdown

system vlan 105

state enabled

port-profile type vethernet esx_vmotion

vmware port-group

switchport mode access

switchport access vlan 106

no shutdown

system vlan 106

state enabled

port-profile type vethernet n1kv-L3-dmz

capability l3control

vmware port-group

switchport mode access

switchport access vlan 111

no shutdown

system vlan 111

state enabled

vdc VSM-PROD id 1

limit-resource vlan minimum 16 maximum 2049

limit-resource monitor-session minimum 0 maximum 2

limit-resource vrf minimum 16 maximum 8192

limit-resource port-channel minimum 0 maximum 768

limit-resource u4route-mem minimum 1 maximum 1

limit-resource u6route-mem minimum 1 maximum 1

limit-resource m4route-mem minimum 58 maximum 58

limit-resource m6route-mem minimum 8 maximum 8

interface port-channel1

inherit port-profile n1kv-uplink

vem 3

interface mgmt0

ip address 192.168.1.200/24

interface Vethernet1

inherit port-profile esx_mgmt

description VMware VMkernel, vmk0

vmware dvport 320 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

vmware vm mac 0025.B500.A008

interface Vethernet2

inherit port-profile n1kv-L3-dmz

description VMware VMkernel, vmk2

vmware dvport 480 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

vmware vm mac 0050.5672.4908

interface Vethernet4

inherit port-profile esx_vmotion

description VMware VMkernel, vmk3

vmware dvport 352 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

vmware vm mac 0050.567B.E4B6

interface Vethernet7

inherit port-profile storage

description VMware VMkernel, vmk1

vmware dvport 577 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

vmware vm mac 0050.567D.5E12

interface Ethernet3/3

inherit port-profile n1kv-uplink

interface Ethernet3/4

inherit port-profile n1kv-uplink

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.5.1.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.5.1.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.5.1.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.5.1.bin sup-2

svs-domain

domain id 200

control vlan 1

packet vlan 1

svs mode L3 interface mgmt0

svs connection vcenter

protocol vmware-vim

remote ip address 192.168.1.252 port 80

vmware dvs uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2" datacenter-name Test-Servers

max-ports 8192

connect

vsn type vsg global

tcp state-checks

vnm-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level

Also the DVS screen shot for this host:

Any idea what might be wrong and where to look?

Thanks in advance!

Joseph Ristaino · ‎04-28-2012

At firt glance this all looks fine as far as configuration. Can you provide the following output from the problem host:

vemcmd show port

vemcmd show port vlans

vem version

vem status

I'm assuming the host is pingable the entire time?

Thanks,

Joey

biaozhou · ‎04-30-2012

Hi Joey,

Thanks for your response. The VSM eventually saw VEM after 3 hours, from the overnight log, there were a few add/remove events for the VEM. The host was pingable the entire time.

Unfortunately, we have to tear down the testbed for a totally different setup. I will update the thread if we see the same issue on this version of software.

Cheers,

Biao

canero · ‎05-04-2012

Hello All,

The problem turned out to be related with the VSM-VEM Communication (which is broadcast by default in L2 mode) when the SYSTEM-UPLINK port profile traffic passed through the HP Flex-10 switches. This is how we did the testing:

Connected the VSM Virtual Machine directly to the VEM, by this way it did not use any external switch uplink ports, but internal communication. When the VEM showed up in the "show module" it was clear that the problem was somewhere in the physical switches outside the ESXi host, not licensing or VSM, or Vcenter communication.
As a workaround for the only Control Vlan traffic for the VEM Module was allowed with a 1G physical mezzanine port on the ESXi host, which is connected to Cisco Catalyst 3120 Blade Switch, and the VSM was again connected to the Vmware Vswitch whose uplink is again the Catalyst 3120. We think this is a better design inorder to seperate the control and data plane. The VM Data traffic was left on the Flex-10 10G Ports. This worked as well.

So we used a SYSTEM-UPLINK port profile for only the control vlan, and a VM-UPLINK port profile for all VM Vlan

traffic, here are some useful commands that may be helpful

vemcmd show trunk

show port-profile virtual usage

show port-channel usage

show port-profile usage

show module vem counters

show module vem license-usage

show license usage Nexus1000v_PKG

Best Regards,