Showing results for 
Search instead for 
Did you mean: 

N1K Nexus1000v VSM Cannot See VEM

Level 1
Level 1


We are installing a N1K 4.2(1)SV1(5.1)   and for this purpose installed VSM on a Seperate DataCenter  Vsphere 5 Cluster onto a blank ESX host. Our problem is that VSM and VEM cannot see each other. What step may we missing?

Thanks in Advance,

Best Regards,

We followed the Installation and Getting started guides:

Cisco Nexus 1000V Installation and Upgrade Guide, Release 4.2(1)SV1(5.1)

from other Discussions we understand that something must be missing with respect to "Control Vlan"  which will provide L2 communication between VSM and VEM.

  1. show svs connection shows that vcenter recognizes the VSM
  2. vem status on the ESX host shows the VEM as ok
  3. management, control and packet vlans are same vlan
  4. show module sees only the active and ha-standby vsms, but no vem exists.
  5. show module vem mapping indicates that somehow VSM recognizes the VEM, if the VEM 3 is absent, how could it be seen by VSM once?
  6. One point to note is VSM and VEM are on the same host, can this be a difference between usual configuration?  Will test a case when VSM and VEM are on seperate ESX hosts.
  7. if we put Vmware dVS instead of N1K and put a host on this Control Vlan, it may ping to other vlans, and the VMs for port-profiles can communicate to the outside world,
  8. We don't think this is a licensing problem because by default 16 CPUs are on the VSM, and it did not change after the licens is installed.

show module vem mapping

Mod     Status          UUID                                    License Status

---     -----------     ------------------------------------    --------------

3       absent          34343335-3237-4742-3838-346337417236    unlicensed

6 Replies 6

Robert Burns
Cisco Employee
Cisco Employee

Can you provide the output of "show run" from the VSM.

Can you clarify the exact VC toplogy.  Provide a screen shot if need be.

It's very likely this is a system vlan or other configuration related issue.



Joseph Ristaino
Cisco Employee
Cisco Employee

It seems pretty obvious that the control vlan is not set as a system vlan on your vethernet port profile as well as your uplink port profile. 

You will need to purchase licenses for your modules as well as even if you able to get the module online you will not be able to pass traffic on Veth ports.

Please upload the "show run" as Rob requested and we will look at your configuration.



I am having the exact same problem as Canero with the same software version. My setup is as follows:

1. L3 mode with VSM and VEM on different subnets/hosts

2. With the same set of port profiles and UCS service profile cloned, I have one host showing up in VSM and another not showing up. In my view, this rule out mis-configuration in

a. system vlan on veth port profile

b. vlan configuration in upstream switch and UCS fabric interconnect

3. Sometimes, the VEM did managed to show up but can drop without any reason

4. Sniffer trace indicates that there are bi-directional traffic between VSM and VEM

5. show logging logfile has the following error:

2012 Apr 24 15:21:47 VSM-PROD %VMS-5-DVS_HOSTMEMBER_INFO: A host with name=[] and uuid=[a7f30c6a-6eb0-11e1-2000-000000000006] is added to the dvs. The host is not found  as a module in the VSM configuration.

6. Opaque Data in VC shows the following:

"data-version 1.0

switch-domain 200

switch-name VSM-PROD

cp-version 4.2(1)SV1(5.1)

control-vlan 1

system-primary-mac 00:50:56:ab:44:01

active-vsm packet mac 00:50:56:ab:44:03

active-vsm mgmt mac 00:50:56:ab:44:02

standby-vsm ctrl mac 0050-56ab-4404

inband-vlan 1

svs-mode L3


upgrade state 0 mac 0050-56ab-4404 l3control-ipv4 null

profile dvportgroup-149 trunk 105-106,109,111

profile dvportgroup-149 mtu 1500

profile dvportgroup-151 access 105

profile dvportgroup-151 mtu 1500

profile dvportgroup-152 access 106

profile dvportgroup-152 mtu 1500

profile dvportgroup-155 access 111

profile dvportgroup-155 mtu 1500

profile dvportgroup-155 capability l3control

profile dvportgroup-161 access 109

profile dvportgroup-161 mtu 1500

end-version 1.0



VSM-PROD# sh svs domain

SVS domain config:

  Domain id:    200

  Control vlan: 1 

  Packet vlan:  1 

  L2/L3 Control mode: L3

  L3 control interface: mgmt0

  Status: Config push to VC successful.

VSM-PROD# sh mod vem missing

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

4    a7f30c6a-6eb0-11e1-2000-000000000007  NA

5   a7f30c6a-6eb0-11e1-2000-000000000006  NA

VSM-PROD# sh mod vem map

Mod     Status          UUID                                    License Status

---     -----------     ------------------------------------    --------------

3       powered-up      a7f30c6a-6eb0-11e1-2000-000000000008    licensed

4       absent          a7f30c6a-6eb0-11e1-2000-000000000007    n/a

5       absent          a7f30c6a-6eb0-11e1-2000-000000000006    n/a

In host:

Below is "show run"

VSM-PROD# sh run

!Command: show running-config

!Time: Tue Apr 24 16:34:38 2012

version 4.2(1)SV1(5.1)

feature telnet

username admin password 5 $1$oCIIPBun$l0pIWvJ/OeQ3mNSY38OTG0  role network-admin

banner motd #Nexus 1000v Switch#

ssh key rsa 2048

ip domain-lookup

hostname VSM-PROD

vem 3

  host vmware id a7f30c6a-6eb0-11e1-2000-000000000008

vem 4

  host vmware id a7f30c6a-6eb0-11e1-2000-000000000007

vem 5

  host vmware id a7f30c6a-6eb0-11e1-2000-000000000006

snmp-server user admin network-admin auth md5 0xd86407e6438a2a6179f6581e7d378d4b priv 0xd86407e6438a2a6179f6581

e7d378d4b localizedkey

vrf context management

  ip route

vlan 1,100-103,105-106,109,111-112,200

vlan 1,100-103,105-106,109,111-112,200

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

  vmware port-group


  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

  vmware port-group


  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type vethernet storage

  vmware port-group

  switchport mode access

  switchport access vlan 109

  no shutdown

  system vlan 109

  state enabled

port-profile type ethernet n1kv-uplink

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 105-106,109,111-112

  channel-group auto mode on mac-pinning

  no shutdown

  system vlan 105-106,109,111

  state enabled

port-profile type vethernet vm

  vmware port-group

  switchport mode access

  switchport access vlan 112

  no shutdown

  state enabled

port-profile type vethernet esx_mgmt

  vmware port-group

  switchport mode access

  switchport access vlan 105

  no shutdown

  system vlan 105

  state enabled

port-profile type vethernet esx_vmotion

  vmware port-group

  switchport mode access

  switchport access vlan 106

  no shutdown

  system vlan 106

  state enabled

port-profile type vethernet n1kv-L3-dmz

  capability l3control

  vmware port-group

  switchport mode access

  switchport access vlan 111

  no shutdown

  system vlan 111

  state enabled

vdc VSM-PROD id 1

  limit-resource vlan minimum 16 maximum 2049

  limit-resource monitor-session minimum 0 maximum 2

  limit-resource vrf minimum 16 maximum 8192

  limit-resource port-channel minimum 0 maximum 768

  limit-resource u4route-mem minimum 1 maximum 1

  limit-resource u6route-mem minimum 1 maximum 1

  limit-resource m4route-mem minimum 58 maximum 58

  limit-resource m6route-mem minimum 8 maximum 8

interface port-channel1

  inherit port-profile n1kv-uplink

  vem 3

interface mgmt0

  ip address

interface Vethernet1

  inherit port-profile esx_mgmt

  description VMware VMkernel, vmk0

  vmware dvport 320 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

  vmware vm mac 0025.B500.A008

interface Vethernet2

  inherit port-profile n1kv-L3-dmz

  description VMware VMkernel, vmk2

  vmware dvport 480 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

  vmware vm mac 0050.5672.4908

interface Vethernet4

  inherit port-profile esx_vmotion

  description VMware VMkernel, vmk3

  vmware dvport 352 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

  vmware vm mac 0050.567B.E4B6

interface Vethernet7

  inherit port-profile storage

  description VMware VMkernel, vmk1

  vmware dvport 577 dvswitch uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2"

  vmware vm mac 0050.567D.5E12

interface Ethernet3/3

  inherit port-profile n1kv-uplink

interface Ethernet3/4

  inherit port-profile n1kv-uplink

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.5.1.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.5.1.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.5.1.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.5.1.bin sup-2


  domain id 200

  control vlan 1

  packet vlan 1

  svs mode L3 interface mgmt0

svs connection vcenter

  protocol vmware-vim

  remote ip address port 80

  vmware dvs uuid "a5 5d 2b 50 90 f4 f8 5a-7e 21 8b 8e f9 62 6e d2" datacenter-name Test-Servers

  max-ports 8192


vsn type vsg global

  tcp state-checks



  shared-secret **********


Also the DVS screen shot for this host:

Any idea what might be wrong and where to look?
Thanks in advance!

At firt glance this all looks fine as far as configuration.  Can you provide the following output from the problem host:

vemcmd show port

vemcmd show port vlans

vem version

vem status

I'm assuming the host is pingable the entire time?



Hi Joey,

Thanks for your response. The VSM eventually saw VEM after 3 hours, from the overnight log, there were a few add/remove events for the VEM. The host was pingable the entire time.

Unfortunately, we have to tear down the testbed for a totally different setup. I will update the thread if we see the same issue on this version of software.



Level 1
Level 1

Hello All,

The problem turned out to be related with the  VSM-VEM Communication (which is broadcast by default in L2 mode) when the SYSTEM-UPLINK port profile traffic passed through the HP Flex-10 switches. This is how we did the testing:

  1. Connected the VSM Virtual Machine directly to the VEM, by this way it did not use any external switch uplink ports, but internal communication. When the VEM showed up in the "show module" it was clear that the problem was somewhere in the physical switches outside the ESXi host, not licensing or VSM, or Vcenter communication.
  2. As a workaround for the only Control Vlan traffic for the VEM Module was allowed with a 1G physical mezzanine port on the ESXi host, which is connected to Cisco Catalyst 3120 Blade Switch, and the VSM was again connected to the Vmware Vswitch whose uplink is again the Catalyst 3120. We think this is a better design inorder to seperate the control and data plane. The VM Data traffic was left on the Flex-10 10G Ports. This worked as well.

So we used a SYSTEM-UPLINK port profile for only the control vlan, and a VM-UPLINK port profile for all VM Vlan

traffic, here are some useful commands that may be helpful

vemcmd show trunk

show port-profile virtual usage

show port-channel usage

show port-profile usage

show module vem counters

show module vem license-usage

show license usage Nexus1000v_PKG

Best Regards,