How to add a new vem to the vsm?

jiangcaixia · ‎04-09-2010

Does anyone can tell me how to make vsm connect to the multi vems? I have made vsm connect the vem on the same host. The “show module” displays:

Nexus1000v# show module
Mod Ports Module-Type                      Model              Status
--- ----- -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
3    248    Virtual Ethernet Module          NA                 ok

Mod Sw               Hw
--- --------------- ------
1    4.0(4)SV1(2.175 0.0
3    4.0(4)SV1(3)     0.0

Mod MAC-Address(es)                         Serial-Num
--- -------------------------------------- ----------
1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA

Mod Server-IP        Server-UUID                           Server-Name
--- --------------- ------------------------------------ --------------------
1    10.117.6.172     NA                                    NA
3    10.117.5.239     44454c4c-4300-1051-8047-b5c04f4e3258 10.117.5.239

Then, I added another host into this datacenter in VC. And added one pnic of this host to the system-uplink port profile of Nexus1kv switch. Then I used esxupdate to install vem on that host. And I saw "vem status" as follows:

[root@localhost ~]# vem status
VEM modules are loaded

Switch Name      Num Ports   Used Ports Configured Ports MTU     Uplinks
vSwitch0         128         3           128               1500    vmnic0
DVS Name         Num Ports   Used Ports Configured Ports MTU     Uplinks
Nexus1000v       256         50          256               1500    vmnic1

VEM Agent (vemdpa) is running

But I can't find the new vem display after I used "show module" from Nexus vsm. The output of this command showed the exact as the above output.

I looked from "show logging logfile" and found a message showed:

2010 Apr 9 03:30:18 Nexus1000v %VMS-5-DVS_HOSTMEMBER_INFO: A host with name=[
10.117.5.171] and uuid=[564d5ca4-4aa9-c048-3473-02de91c6ce9d] is added to the
dvs. The host is not found as a module in the VSM configuration.

This host was the one I wanted to add to the vsm. The following are the output of vemcmd commands on this host:

[root@localhost datastore1 (2)]# vemcmd show trunk
Trunk port 17 native_vlan 1 CBL 0
vlan(3000) cbl 4, vlan(3002) cbl 4,

[root@localhost datastore1 (2)]# vemcmd show card
Card UUID type 2: 564d5ca4-4aa9-c048-3473-02de91c6ce9d
Card name:
Switch name: Nexus1000v
Switch alias: DvsPortset-0
Switch uuid: 4e a6 29 50 0c e7 db 96-7c 24 74 0b d3 c1 e2 56
Card domain: 141
Card slot: 1
VEM Tunnel Mode: L2 Mode
VEM Control (AIPC) MAC: 00:02:3d:10:8d:00
VEM Packet (Inband) MAC: 00:02:3d:20:8d:00
VEM Control Agent (DPA) MAC: 00:02:3d:40:8d:00
VEM SPAN MAC: 00:02:3d:30:8d:00
Management IPv4 address: 10.117.5.171
Management IPv6 address: 0000:0000:0000:0000:0000:0000:0000:0000
Max physical ports: 32
Max virtual ports: 216
Card control VLAN: 3000
Card packet VLAN: 3002
Processors: 4
Processor Cores: 4
Processor Sockets: 1
System Memory: 0

[root@localhost datastore1 (2)]# vemcmd show port
LTL    IfIndex   Vlan    Bndl SG_ID Pinned_SGID Type Admin State CBL Mode   Name
    8          0   3969       0     32          32 VIRT     UP DOWN    4 Access
    9          0   3969       0     32          32 VIRT     UP DOWN    4 Access
   10          0   3000       0     32          32 VIRT     UP DOWN    4 Access
   11          0   3968       0     32          32 VIRT     UP DOWN    4 Access
   12          0   3002       0     32          32 VIRT     UP DOWN    4 Access
   13          0      1       0     32          32 VIRT     UP DOWN    0 Access
   14          0   3967       0     32          32 VIRT     UP DOWN    4 Access
   15          0   3967       0     32          32 VIRT     UP DOWN    4 Access
   17          0      1 T     0     32          32 PHYS     UP    UP    0 Trunk vmnic1
   48          0      1       0     32          32 VIRT   DOWN    UP    0 Access testpxe.eth0

So, there must be something wrong. Are there other configurations I haven't done to add a new vem to the vsm? Does anyone have any ideas? Thanks in advance!

Caixia

abbharga · ‎04-10-2010

Hi Ciaxia,

The following URL will provide you some further details on adding the host to the n1k dvs and also on troublshooting this issue:

Adding a ESX host to the DVS:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0/install/software/guide/install_n1000v.html#wp39708

Troubleshooting Module issues:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0_4_s_v_1_2/troubleshooting/configuration/guide/trouble_5modules.html

In the details provided you mentioned addind the host to the datacenter in the VC -> adding the vminc from the host to the system-uplink -> installing the VEM on the ESX host. Ideally if you have the VUM running the VC should automatically install the vem on the ESX host when you add the host to the DVS in the networking view, else you should be installing the vem on the esx host first and then add the host to the n1k dvs.

In the output the slot number for the host is showing as 1 which means the VSM is not able to program it, this could be due to communication issues on the control vlan. Can you verify the control and packet vlans to be trunked on the interface on which vminc1 is connected to on the upstream switch, both vlans being defined on the upstream switch and if the upstream switch for this host and the host running the VSM is different make sure they are L2 connected for these vlans. You can refer to the module troubleshooting guide above.

Can you also verify the vem installed on the ESX host to be compatible with the N1K

http://www.cisco.com/en/US/docs/switches/datacenter/nexus1000/sw/4_0_4_s_v_1_2/compatibility/information/n1000v_compatibility.html

If this doesn't helps you can you share the running config from the VSM and the output of 'show module vem counters' from the VSM.

jiangcaixia · ‎04-11-2010

Thanks, abbharga.

I have tried your method to install vem before adding the host to the n1k dvs. However, it is still no display about the new vem. The new host I want to add to the dvs uses the same esx iso with the host running the vsm. So I think the compatibility is not the problem.

The situation in which I install my nexus n1k is a bit strange due to the limitation of the devices:

I don't have permissions to configure the physical switch to have their vlan ids. So, I just configure the vswitch of the host which will be running vem and vsm vm together to have a control port group of vlan id 3000 and a packet port group of vlan id 3002. Then I create a vm on this host to have three vmnics to connect to the control, packet and VM Network port group separately. VM Network port group doesn't have vlan id in order to connect to the outer network. In such situation, the vsm can connect the vem on the same host by showing module.
Then another host is intalled with the same esx iso. This host is installed in a vm above a esx. The vswitch of this below esx has been configured with promiscuous mode. Then I follow your instructions to add the above host to the n1k dvs. However, vsm seems it can't connect to this new vem.

So, I wonder why on the same host vsm can connect the vem? The vswitch and nexus dvs on this host use the different pnics. Does the command from the vsm vm with the vlan id 3000 go to the pnic of the vswitch of this host, then to the outer network, then to another pnic of dvs and configure the vem on this host? Or the vsm command directly go inside the host rather than transmit to the outer network on the same host?

What's the communication path between the vsm and the vem?

The output of show running-config is：
Nexus1000v# show running-config
version 4.0(4)SV1(3)
username admin password 5 $1$lJPeo6c8$04CbGUdmMC4fvT1C1Pjex1 role network-adm
in
username adminbackup password 5 $1$Oip/C5Ci$oOdx7oJSlBCFpNRmQK4na. role netwo
rk-operator
telnet server enable
ip domain-lookup
ip host Nexus1000v 10.117.6.172
kernel core target 0.0.0.0
kernel core limit 1
system default switchport
vem 3
host vmware id 44454c4c-4300-1051-8047-b5c04f4e3258
snmp-server user admin network-admin auth md5 0x92637dec10f2b3e3caccd6ad02cf8b
57 priv 0x92637dec10f2b3e3caccd6ad02cf8b57 localizedkey
snmp-server enable traps license
vrf context management
ip route 0.0.0.0/0 10.117.7.253
switchname Nexus1000v
vlan 1,3000,3002
no platform ip verify address source broadcast
no platform ip verify address source multicast
no platform ip verify address destination zero
no platform ip verify address source reserved
no platform ip verify checksum
no platform ip verify protocol
no platform ip verify fragment
no platform ip verify length minimum
no platform ip verify length maximum max-frag
vdc Nexus1000v id 1
limit-resource vlan minimum 16 maximum 513
limit-resource monitor-session minimum 0 maximum 64
limit-resource vrf minimum 16 maximum 8192
limit-resource port-channel minimum 0 maximum 256
limit-resource u4route-mem minimum 32 maximum 80
limit-resource u6route-mem minimum 16 maximum 48
port-profile type ethernet Unused_Or_Quarantine_Uplink
description Port-group created for Nexus1000V internal usage. Do not use.
description Port-group created for Nexus1000V internal usage. Do not use.
vmware port-group
shutdown
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
description Port-group created for Nexus1000V internal usage. Do not use.
vmware port-group
shutdown
state enabled
port-profile type ethernet system-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan all
no shutdown
system vlan 3000,3002
state enabled

port-profile type vethernet vm-data2
vmware port-group
switchport mode access
no shutdown
state enabled

interface Ethernet3/2
inherit port-profile system-uplink

interface mgmt0
ip address 10.117.6.172/22

interface Vethernet1
inherit port-profile vm-data2
description ubuntu, Network Adapter 1
vmware dvport 101

port-profile type vethernet vm-data2
vmware port-group
switchport mode access
no shutdown
state enabled

interface Ethernet3/2
inherit port-profile system-uplink

interface mgmt0
ip address 10.117.6.172/22

interface Vethernet1
inherit port-profile vm-data2
description ubuntu, Network Adapter 1
vmware dvport 101

interface Vethernet2
inherit port-profile vm-data2
description pxetest, Network Adapter 1
vmware dvport 102

interface control0
boot kickstart bootflash:/nexus-1000v-kickstart-mzg.4.0.4.SV1.2.175.bin sup-1
boot system bootflash:/nexus-1000v-mzg.4.0.4.SV1.2.175.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mzg.4.0.4.SV1.2.175.bin sup-2
boot system bootflash:/nexus-1000v-mzg.4.0.4.SV1.2.175.bin sup-2
svs-domain
domain id 141
control vlan 3000
packet vlan 3002
svs mode L2
svs connection vc
protocol vmware-vim
remote ip address 10.117.5.2 port 80
vmware dvs uuid "4e a6 29 50 0c e7 db 96-7c 24 74 0b d3 c1 e2 56" datacenter
-name nexus
connect

The output of show module vem counters is:
Nexus1000v# show module vem counters
------------------------------------------------------------------------------
--
Mod InNR OutMI InMI OutHBeats InHBeats InAipcMsgs OutTO OutTOC InsCnt RemCn
t
------------------------------------------------------------------------------
--
3 4 4 4 321712 321249 323717 0 0 4
3

So, can you give me some suggestions?

Thanks in advance!

Caixia

sachthak · ‎04-11-2010

Hi Caixia,

A couple quick things that might be worth checking if you haven't done so already. Since the host/vem that isn't coming up as a module is connected via different pnic, make sure that this interface on the upstream switch allows your vlans and the vlans are created on your vsm and on the upstream switch.

Hope this helps.

sachin

Robert Burns · ‎04-11-2010

Caixia

This issue is almost 100% related to Control VLAN mis-configuration (or lack thereof). When you add a host to the DVS the configuration for the VEM gets pushed from vCenter to each VEM host over the Service Console interface/VLAN. When the VEM then tries to talk back to the VSM it will use the Control VLAN. Once it has successfully reached the VSM over the Control VLAN it will be assigned a Slot # other than "1" and show up on your VSM "show mod" output. If you're only seeing Card Slot #1, your control VLAN is not reaching the VSM.

1. Did you create the Control & Packet VLANs on the upstream switch?

2. Did you allow VLANs 1, 3000 & 3002 on the upstream switch interface of the pnic you've assigned to the DVS?

An adapter can only belong to a vSwitch or the DVS, there's no "Sharing" adapters between these devices.

Can you post your upstream switchport configuration for the pnic assigned to the DVS for the VEM host having the problem - unless you missed step 1 or 2 above and that was your issue.

"show run interface x/y"

Robert

jiangcaixia · ‎04-12-2010

Thanks, Robert.

I am sorry that I don't configure the upstream switch with vlan ids because I have no permissions. So maybe I can't connect vem to the vsm through the upstream switch.

But I have problems in such a situation:

There is a host installing the vsm vm and the vem. The vem is displayed from show module in vsm vm. The vswitch of this host has four port groups: control port group/3000, packet port group/3002, VM Network port group and Service Console port group. The vem uses the first three port groups. Then I created a new vm on this host with two vmnics configured in VM Network port group. I installed the same iso esx in this vm. Then I installed the vem on this esx vm and attached one unused vmnic to the nexus dvs. But vsm still doesn't show this new vem from show module.

I think the upstream switch is not needed in such a situation. The packet with control vlan id from vem to vsm will be transmitted directly through the vswitch of the below esx host to the vsm vm.

Does anyone test this situation?

Thanks,

Caixia

Robert Burns · ‎04-12-2010

Installing the VSM and VEM in nested VMs is not supported.

Not to say it can't be done (I've have done this before but it requires a decent amount of VMware hacking). One clue I will give you is that you'll need to configure the vSwitch ports in Promiscous mode otherwise they will not pass tagged traffic downstream.

Unfortunately the only method Cisco will support officially and through this community is on physical hosts only.

Robert

jiangcaixia · ‎04-12-2010

Ok. I have configured the vswitch in Promiscous mode in the below host. However, vsm still can't connect the vem on the above esx vm.

So, Do you mean that definitely we need to install vem on the esx host(a physical host)? And we need a physical switch support to connect the vems to the vsm vm.

Thanks for your help.

Caixia

Robert Burns · ‎04-12-2010

Correct.

Robert