Solved: [Nexus 1000v] VEM can't be add into VSM

Doan Quang Hoa · ‎08-31-2011

hi all,

following my lab, i have some problems with Nexus 1000V when VEM can't be add into VSM.

+ on VSM has already installed on ESX 1 (standalone or ha) and you can see:

Cisco_N1KV# show module

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 0 Virtual Supervisor Module Nexus1000V active *

Mod Sw Hw

--- ---------------- ------------------------------------------------

1 4.2(1)SV1(4a) 0.0

Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA

Mod Server-IP Server-UUID Server-Name

--- --------------- ------------------------------------ -------------------

1 10.4.110.123 NA NA

+ on ESX2 that 's installed VEM

[root@esxhoadq ~]# vem status

VEM modules are loaded

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 128 3 128 1500 vmnic0

VEM Agent (vemdpa) is running

[root@esxhoadq ~]#

any advices for this,

thanks so much

Robert Burns · ‎08-31-2011

Doan,

Need more info.

Has the host been added through vCenter to the 1000v DVS successfully?

If so, there's likely an issue with your Control VLAN communication between the VSM and VEM. Start there and ensure the VLAN has been created on all intermediate switches and it's being allowed on every trunk end to end.

If you're still stuck, paste your running config from your VSM.

Regards,

Robert

View solution in original post

Robert Burns · ‎09-01-2011

Doan,

Can you please paste your running config from your VSM.

Before you can add the command "system vlan x" you have to ensure the VLAN is allowed. Start with the command "switchport trunk allowed vlan x".

Regards,

Robert

View solution in original post

mipetrin · ‎09-01-2011

Doan,

A few points that I would like to highlight from your configuration:

1) vlan 1-2

From the port-profile system-uplink, you want to allow vlans 1-3, so you will need to add the additional vlan, 3, to the Nexus 1000v

2) port-profile type vethernet system-uplink

This should be port-profile type ethernet instead of vethernet. Ethernet means that it will be applied to the physical nics (or vmnics in ESX/i) versus vethernet which will be applied to virtual NICs (such as a virtual machine NIC, or vmknic of the ESX/i host). When the Nexus 1000v is fully functional, you would have at least configured one port-profile type ethernet and one type vethernet. To begin with though and get initial communication, type ethernet is required.

3) system vlan 2

Damien is correct, you have misconfigured the system vlan list. From your svs-domain parameters, we see that you have configured a control vlan of 1 and a packet vlan of 2. Therefore, your system vlan should contain both vlan 1 and 2. You also want to ensure that these two vlans have been created throughout your layer 2 domain and trunking between all ESX/i hosts.

* From the 'show svs connection' we see that the communication between the VSM and vCenter is good.

* Next step would be to check the VEM to vCenter communication by verifying the output of "[root@esxhoadq ~]# vemcmd show card". If you see that it is correctly configured with the domain parameters, such as control/packet vlan, then this communication should be good too. You can past the output if you wish to have it double checked.

* If that shows up good, the last piece of the puzzle is the communication between the VSM to VEM. This communication takes place via the control vlan, which by your current configuration, is broken and will need to be rectified as per my earlier recommendations.

To answer Damien's question about licensing, the Nexus 1000v comes with a trial license, so you immediately do not need to enter a valid license.

Lastly, there guides and videos that you can watch that are available on the Nexus 1000v Install and Upgrade guides page. Also, don't forget the troubleshooting guide I linked to earlier.

Let us know how you go.

Cheers,

Michael

View solution in original post

Robert Burns · ‎08-31-2011

Doan,

Need more info.

Has the host been added through vCenter to the 1000v DVS successfully?

If so, there's likely an issue with your Control VLAN communication between the VSM and VEM. Start there and ensure the VLAN has been created on all intermediate switches and it's being allowed on every trunk end to end.

If you're still stuck, paste your running config from your VSM.

Regards,

Robert

mipetrin · ‎08-31-2011

Further to Rob's comments, you could also review the VSM and VEM module troubleshooting guide

Cheers,

Michael

Doan Quang Hoa · ‎09-01-2011

hi Robert, thanks u for supported,

i have already add ESX to dvs through vCenter, but you can see img below for my reason why i don't know to edit DVUplink port group

And on VSM, when i excuted system vlan [vlan-id], i had an error:

Cisco_N1KV(config)# port-profile system-uplink

Cisco_N1KV(config-port-prof)#

Cisco_N1KV(config-port-prof)# switchport mode trunk

Cisco_N1KV(config-port-prof)# switchport trunk allowed vlan add 1,2,3

Cisco_N1KV(config-port-prof)# no shutdown

Cisco_N1KV(config-port-prof)#

Cisco_N1KV(config-port-prof)# system vlan 1,2,3

ERROR: System vlan configuration requires defined set of allowed vlans. Please

configure allowed vlans set.

Inherited properties, if any, are not used for system vlan validation

Cisco_N1KV(config-port-prof)#vmware port-group

can u help me?

thanks so much

mipetrin · ‎09-01-2011

Hi Doan,

1) Ensure that your VSM is connected to vCenter with

Cisco_N1KV# show svs connection

2) Configure your port-profile in the following order:

Cisco_N1KV(config)# vlan 1,2,3

Cisco_N1KV(config-vlan)# exit

Cisco_N1KV(config)# port-profile type ethernet system-uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 1,2,3 //notice that I am not using the keyword add, as i am defining the initial set of vlans

# channel-group auto mode on mac-pinning // Not sure if you require any port-channels to be formed

no shutdown

system vlan 1,2,3

state enabled

* This should then create the corresponding port-group in vCenter

3) You will then be able to "manage host" in vCenter, then select the 10.4.36.36 ESX server, select which vmnics you want to assign to the Nexus 1000v by ticking the box and finally choose the "system-uplink" DVUplink port group

Let us know how it goes.

Thanks,

Michael

Doan Quang Hoa · ‎09-01-2011

hi Michael,

thanks for your supported,

i tried some times, but still add this command: system vlan x

any advices for this,

and anybody can share basic lab guide for nexus 1000v (VSM, VEM, and communication between them)

thanks so much

Robert Burns · ‎09-01-2011

Doan,

Can you please paste your running config from your VSM.

Before you can add the command "system vlan x" you have to ensure the VLAN is allowed. Start with the command "switchport trunk allowed vlan x".

Regards,

Robert

Doan Quang Hoa · ‎09-01-2011

thanks Robert,

here 's my n1000v configuration file:

Nexus 1000v Switch

login: admin

Password:

Cisco Nexus Operating System (NX-OS) Software

TAC support: http://www.cisco.com/tac

The copyrights to certain works contained in this software are

owned by other third parties and used and distributed under

license. Certain components of this software are licensed under

the GNU General Public License (GPL) version 2.0 or the GNU

Lesser General Public License (LGPL) Version 2.1. A copy of each

such license is available at

http://www.opensource.org/licenses/gpl-2.0.php and

http://www.opensource.org/licenses/lgpl-2.1.php

Cisco_N1KV#

Cisco_N1KV# show running-config

!Command: show running-config

!Time: Thu Sep 1 23:14:01 2011

version 4.2(1)SV1(4a)

feature telnet

username admin password 5 $1$FwzNBLnO$LEQNdoNSeJS8hzPwXt/bR/ role network-admi

banner motd #Nexus 1000v Switch#

ip domain-lookup

hostname Cisco_N1KV

snmp-server user admin network-admin auth md5 0x0662a4f05d994d91aa02b001d860575

priv 0x0662a4f05d994d91aa02b001d860575e localizedkey

vrf context management

ip route 0.0.0.0/0 10.4.110.254

vlan 1-2

port-profile default max-ports 32

port-profile default port-binding static

port-profile type ethernet Unused_Or_Quarantine_Uplink

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet system-uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 1-3

no shutdown

system vlan 2

state enabled

vdc Cisco_N1KV id 1

limit-resource vlan minimum 16 maximum 2049

limit-resource monitor-session minimum 0 maximum 2

limit-resource vrf minimum 16 maximum 8192

limit-resource port-channel minimum 0 maximum 768

limit-resource u4route-mem minimum 32 maximum 32

limit-resource u6route-mem minimum 16 maximum 16

limit-resource m4route-mem minimum 58 maximum 58

limit-resource m6route-mem minimum 8 maximum 8

interface mgmt0

ip address 10.4.110.123/24

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4a.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4a.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4a.bin sup-2

svs-domain

domain id 1

control vlan 1

packet vlan 2

svs mode L2

svs connection VC

protocol vmware-vim

remote ip address 10.4.110.10 port 80

vmware dvs uuid "90 aa 3a 50 54 d2 b1 25-b8 b7 a6 ae ca 70 de f1" datacenter-

ame LAB

max-ports 8192

connect

vsn type vsg global

tcp state-checks

vnm-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level info

Cisco_N1KV# show svs connections

connection VC:

ip address: 10.4.110.10

remote port: 80

protocol: vmware-vim https

certificate: default

datacenter name: LAB

admin:

max-ports: 8192

DVS uuid: 90 aa 3a 50 54 d2 b1 25-b8 b7 a6 ae ca 70 de f1

config status: Enabled

operational status: Connected

sync status: Complete

version: VMware vCenter Server 4.1.0 build-345043

Cisco_N1KV#

Cisco_N1KV# show module

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 0 Virtual Supervisor Module Nexus1000V active *

Mod Sw Hw

--- ---------------- ------------------------------------------------

1 4.2(1)SV1(4a) 0.0

Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------

1 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA

Mod Server-IP Server-UUID Server-Name

--- --------------- ------------------------------------ -------------------

1 10.4.110.123 NA NA

* this terminal session

Cisco_N1KV#

i have tried so many times but it seems that i had something wrong in some steps. but i don't know why?

Kindly to give me 1 lab guide for nexus 1000v installation (problem 's in VEM and VSM communication)

thanks so much

damien.raynal · ‎09-01-2011

sorry, but have you enter the license in the nexus 1000v ? because without license you cannot have module...

(sh license usage)

second point :

your system vlan in your port-profile is vlan 2 but in the svs connection i can see it's vlan 1... (control vlan)

mipetrin · ‎09-01-2011

Doan,

A few points that I would like to highlight from your configuration:

1) vlan 1-2

From the port-profile system-uplink, you want to allow vlans 1-3, so you will need to add the additional vlan, 3, to the Nexus 1000v

2) port-profile type vethernet system-uplink

This should be port-profile type ethernet instead of vethernet. Ethernet means that it will be applied to the physical nics (or vmnics in ESX/i) versus vethernet which will be applied to virtual NICs (such as a virtual machine NIC, or vmknic of the ESX/i host). When the Nexus 1000v is fully functional, you would have at least configured one port-profile type ethernet and one type vethernet. To begin with though and get initial communication, type ethernet is required.

3) system vlan 2

Damien is correct, you have misconfigured the system vlan list. From your svs-domain parameters, we see that you have configured a control vlan of 1 and a packet vlan of 2. Therefore, your system vlan should contain both vlan 1 and 2. You also want to ensure that these two vlans have been created throughout your layer 2 domain and trunking between all ESX/i hosts.

* From the 'show svs connection' we see that the communication between the VSM and vCenter is good.

* Next step would be to check the VEM to vCenter communication by verifying the output of "[root@esxhoadq ~]# vemcmd show card". If you see that it is correctly configured with the domain parameters, such as control/packet vlan, then this communication should be good too. You can past the output if you wish to have it double checked.

* If that shows up good, the last piece of the puzzle is the communication between the VSM to VEM. This communication takes place via the control vlan, which by your current configuration, is broken and will need to be rectified as per my earlier recommendations.

To answer Damien's question about licensing, the Nexus 1000v comes with a trial license, so you immediately do not need to enter a valid license.

Lastly, there guides and videos that you can watch that are available on the Nexus 1000v Install and Upgrade guides page. Also, don't forget the troubleshooting guide I linked to earlier.

Let us know how you go.

Cheers,

Michael

Doan Quang Hoa · ‎09-02-2011

thanks everybody very much,

i will try again

mta2891974 · ‎11-20-2011

Hi,

i'm having similar issue: the VEM insatlled on the ESXi is not showing up on the VSM.

please check from the following what can be wrong?

This is the VEM status:

~ # vem status -v

Package vssnet-esx5.5.0-00000-release

Version 4.2.1.1.4.1.0-2.0.1

Build 1

Date Wed Jul 27 04:42:14 PDT 2011

Number of PassThru NICs are 0

VEM modules are loaded

Switch Name Num Ports Used Ports Configured Ports MTU Uplinks

vSwitch0 128 4 128 1500 vmnic0

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

VSM11 256 40 256 1500 vmnic2,vmnic1

Number of PassThru NICs are 0

VEM Agent (vemdpa) is running

~ # vemcmd show port

LTL VSM Port Admin Link State PC-LTL SGID Vem Port

18 UP UP F/B* 0 vmnic1

19 DOWN UP BLK 0 vmnic2

* F/B: Port is BLOCKED on some of the vlans.

Please run "vemcmd show port vlans" to see the details.

~ # vemcmd show trunk

Trunk port 6 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(111) cbl 1, vlan(112) cbl 1, vlan(3968) cbl 1, vlan(3969) cbl 1, vlan(3970) cbl 1, vlan(3971) cbl 1,

Trunk port 16 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(111) cbl 1, vlan(112) cbl 1, vlan(3968) cbl 1, vlan(3969) cbl 1, vlan(3970) cbl 1, vlan(3971) cbl 1,

Trunk port 18 native_vlan 1 CBL 0

vlan(111) cbl 1, vlan(112) cbl 1,

~ # vemcmd show port

LTL VSM Port Admin Link State PC-LTL SGID Vem Port

18 UP UP F/B* 0 vmnic1

19 DOWN UP BLK 0 vmnic2

* F/B: Port is BLOCKED on some of the vlans.

Please run "vemcmd show port vlans" to see the details.

~ # vemcmd show port vlans

Native VLAN Allowed

LTL VSM Port Mode VLAN State Vlans

18 T 1 FWD 111-112

19 A 1 BLK 1

~ # vemcmd show port

LTL VSM Port Admin Link State PC-LTL SGID Vem Port

18 UP UP F/B* 0 vmnic1

19 DOWN UP BLK 0 vmnic2

* F/B: Port is BLOCKED on some of the vlans.

Please run "vemcmd show port vlans" to see the details.

~ # vemcmd show port vlans

Native VLAN Allowed

LTL VSM Port Mode VLAN State Vlans

18 T 1 FWD 111-112

19 A 1 BLK 1

~ # vemcmd show trunk

Trunk port 6 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(111) cbl 1, vlan(112) cbl 1, vlan(3968) cbl 1, vlan(3969) cbl 1, vlan(3970) cbl 1, vlan(3971) cbl 1,

Trunk port 16 native_vlan 1 CBL 1

vlan(1) cbl 1, vlan(111) cbl 1, vlan(112) cbl 1, vlan(3968) cbl 1, vlan(3969) cbl 1, vlan(3970) cbl 1, vlan(3971) cbl 1,

Trunk port 18 native_vlan 1 CBL 0

vlan(111) cbl 1, vlan(112) cbl 1,

~ # vemcmd show card

Card UUID type 2: ebd44e72-456b-11e0-0610-00000000108f

Card name: esx

Switch name: VSM11

Switch alias: DvsPortset-0

Switch uuid: c4 be 2c 50 36 c5 71 97-44 41 1f c0 43 8e 45 78

Card domain: 1

Card slot: 1

VEM Tunnel Mode: L2 Mode

VEM Control (AIPC) MAC: 00:02:3d:10:01:00

VEM Packet (Inband) MAC: 00:02:3d:20:01:00

VEM Control Agent (DPA) MAC: 00:02:3d:40:01:00

VEM SPAN MAC: 00:02:3d:30:01:00

Primary VSM MAC : 00:50:56:ac:00:42

Primary VSM PKT MAC : 00:50:56:ac:00:44

Primary VSM MGMT MAC : 00:50:56:ac:00:43

Standby VSM CTRL MAC : ff:ff:ff:ff:ff:ff

Management IPv4 address: 10.1.240.30

Management IPv6 address: 0000:0000:0000:0000:0000:0000:0000:0000

Secondary VSM MAC : 00:00:00:00:00:00

Secondary L3 Control IPv4 address: 0.0.0.0

Upgrade : Default

Max physical ports: 32

Max virtual ports: 216

Card control VLAN: 111

Card packet VLAN: 112

Card Headless Mode : Yes

Processors: 8

Processor Cores: 4

Processor Sockets: 1

Kernel Memory: 16712336

Port link-up delay: 5s

Global UUFB: DISABLED

Heartbeat Set: False

PC LB Algo: source-mac

Datapath portset event in progress : no

~ #

On VSM

VSM11# sh svs conn

connection vcenter:

ip address: 10.1.240.38

remote port: 80

protocol: vmware-vim https

certificate: default

datacenter name: New Datacenter

admin:

max-ports: 8192

DVS uuid: c4 be 2c 50 36 c5 71 97-44 41 1f c0 43 8e 45 78

config status: Enabled

operational status: Connected

sync status: Complete

version: VMware vCenter Server 4.1.0 build-345043

VSM11# sh svs ?

connections Show connection information

domain Domain Configuration

neighbors Svs neighbors information

upgrade Svs upgrade information

VSM11# sh svs dom

SVS domain config:

Domain id: 1

Control vlan: 111

Packet vlan: 112

L2/L3 Control mode: L2

L3 control interface: NA

Status: Config push to VC successful.

VSM11# sh port

^

% Invalid command at '^' marker.

VSM11# sh run

!Command: show running-config

!Time: Sun Nov 20 11:35:52 2011

version 4.2(1)SV1(4a)

feature telnet

username admin password 5 $1$QhO77JvX$A8ykNUSxMRgqZ0DUUIn381 role network-admin

banner motd #Nexus 1000v Switch#

ssh key rsa 2048

ip domain-lookup

hostname VSM11

snmp-server user admin network-admin auth md5 0x389a68db6dcbd7f7887542ea6f8effa1

priv 0x389a68db6dcbd7f7887542ea6f8effa1 localizedkey

vrf context management

ip route 0.0.0.0/0 10.1.240.254

vlan 1,111-112

port-channel load-balance ethernet source-mac

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type ethernet system-uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 111-112

no shutdown

system vlan 111-112

description "System profile"

state enabled

port-profile type vethernet servers11

vmware port-group

switchport mode access

switchport access vlan 11

no shutdown

description "Data Profile for VM Traffic"

port-profile type ethernet vm-uplink

vmware port-group

switchport mode access

switchport access vlan 11

no shutdown

description "Uplink profile for VM traffic"

state enabled

vdc VSM11 id 1

limit-resource vlan minimum 16 maximum 2049

limit-resource monitor-session minimum 0 maximum 2

limit-resource vrf minimum 16 maximum 8192

limit-resource port-channel minimum 0 maximum 768

limit-resource u4route-mem minimum 32 maximum 32

limit-resource u6route-mem minimum 16 maximum 16

limit-resource m4route-mem minimum 58 maximum 58

limit-resource m6route-mem minimum 8 maximum 8

interface mgmt0

ip address 10.1.240.124/24

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4a.bin sup-1

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4a.bin sup-2

boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4a.bin sup-2

svs-domain

domain id 1

control vlan 111

packet vlan 112

svs mode L2

svs connection vcenter

protocol vmware-vim

remote ip address 10.1.240.38 port 80

vmware dvs uuid "c4 be 2c 50 36 c5 71 97-44 41 1f c0 43 8e 45 78" datacenter-n

ame New Datacenter

max-ports 8192

connect

vsn type vsg global

tcp state-checks

vnm-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level

thank you

Michel

Robert Burns · ‎11-21-2011

Michel,

You have a problem with VLAN 111 (Your control VLAN) between your vSphere host and the VSM. Check all intermediate switches between these two endpoints and ensure VLAN 111 has been created and allowed on all switches in between. Since your 1000v control VLAN is likely a "new" VLAN in your infrastructure, I'd put my money it hasn't been created outside of the VSM.

vmnic2 is also showing as down. I hope that vmnic1 is assigned to your system-uplink profile in this case. I can't tell from the outputs above which vmnic is assigned to which uplink port profile but remember the following: If you plan to use more than 1 physical NIC on a host with the same uplink port profile, you must use a Channel-group command. See my post here for which mode to use.

https://communities.cisco.com/thread/22472?tstart=0

Also, not related to this problem, but VLAN 11 is references as an access VLAN in your VM-Uplink port profile, but that VLAN hasn't been created on the VSM (and elsewhere likely).

Regards,

Robert

mta2891974 · ‎11-21-2011

Thank you Robert for your answer.

Indeed, the problem was on the uplink switch. It was solved by configuring the uplink of the server on the upstream switch as Trunk! (i was mistakenly configured as access).

Thanks

Michel