Yet another VEM / VSM issue

[cczcp1-uon]

Hi guys,

Sorry to bring this up yet again (considering the number of other discussions about it) however, I could do with some help getting the N1k DVS working.

My current test VMWare environment covers 3 ESX hosts (all on 4.0 Build 208167) with 8 nics each connected across 2 Cisco C2960 switches on Enterprise plus licences and a vCenter server (in a VM). On each ESX host, 6 physical NICs are assigned with the virtual standard switch and two are allocated for DVS uplinks (eventually this be moved over completely to DVS). I've ran through the the install pretty much along the guidelines of the official documentation, the VSM has been installed & port-profiles for uplinks configured and each ESX host has been added to the DVS where VUM has installed the VEM module.

Unfortunately this is the sticking point as none of the VEM's will register against the VSM and the svs show neighbor doesn't have any MAC addresses against it.

CCX1-VSM# show module

Mod  Ports  Module-Type                      Model              Status

---  -----  -------------------------------- ------------------ ------------

1    0      Virtual Supervisor Module        Nexus1000V         active *

Mod  Sw               Hw

---  ---------------  ------

1    4.0(4)SV1(2)     0.0

Mod  MAC-Address(es)                         Serial-Num

---  --------------------------------------  ----------

1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA

Mod  Server-IP        Server-UUID                           Server-Name

---  ---------------  ------------------------------------  --------------------

1    192.168.42.101   NA                                    NA

CCX1-VSM# show svs neighbor

Active Domain ID: 100

AIPC Interface MAC: 0050-56b8-0dc1

Inband Interface MAC: 0050-56b8-78e2

Src MAC           Type   Domain-id    Node-id     Last learnt (Sec. ago)

------------------------------------------------------------------------

From reading the troubleshooting documentation & other  discussions, this suggests that there is an issue with the VEM / VSM connectivity. Presumably caused by a VLAN misconfiguration however the control vlan has been assigned to 10, the packet vlan assigned to 11 and switch (which allows trunking on all ports which ESX hosts are connected to) allows for these VLANs plus the necessary ones for data traffic.

Does anyone have any suggestions where I can go from here as I'm completely stuck?

I've attached an output of show tech-support svs (if this helps).

Thanks for any help you can give in advance

[Robert Burns]

As with almost every single issue relating to modules not showing up on the VSM  - there's some misconfiguration with your Control VLAN.

- Have you created VLAN 10 and 11 on your upstream switches?  This is a common oversight.  You still need the vlan defined (in addition to be allowed over the trunks) on all upstream switches between your VSM and VEM hosts.

- How are you adding your hosts to the DVS - VMware VUM or manually .vib install method? Also, can you confirm that with the two phsyical links you're using to uplink each host to the DVS, you've assigned one to your "system uplink" and the other to your "VM Uplink" ?

On one of the hosts you're trying to add from it's console/cli, can you post the output of :

vemcmd show card

Thanks,

Robert

[cczcp1-uon]

Hi Robert,

Thanks for you quick reply.

Yes, the VLANs on both the physical upstream switches have been created.
The ESX hosts have been added via the vCenter / Home / Inventory / Networking -> CCX1-VSM -> Add host.
Once a host has been selected, 1x physical NIC is selected as a system-uplink  and 1x physical NIC is selected as a VM-Uplink.

vCenter then actions the request and installs the VEM module via the VUM automatically.

As requested heres a cut & paste of the vemcmd show card

[root@esx10 ~]# vemcmd show card

Card UUID type  2: 34393133-3135-435a-4339-333433544442

Card name: uievmesx10

Switch name: CCX1-VSM

Switch alias: DvsPortset-0

Switch uuid: 8c 0f 38 50 40 71 3f 15-65 b0 92 55 35 f8 23 e5

Card domain: 100

Card slot: 1

VEM Tunnel Mode: L2 Mode

VEM Control (AIPC) MAC: 00:02:3d:10:64:00

VEM Packet (Inband) MAC: 00:02:3d:20:64:00

VEM Control Agent (DPA) MAC: 00:02:3d:40:64:00

VEM SPAN MAC: 00:02:3d:30:64:00

Management IPv4 address: 192.168.42.77

Management IPv6 address: 0000:0000:0000:0000:0000:0000:0000:0000

Max physical ports: 32

Max virtual ports: 216

Card control VLAN: 10

Card packet VLAN: 11

       Processors: 16

  Processor Cores: 8

Processor Sockets: 2

  Kernel Memory:   4284739584

[Robert Burns]

The opaque data has been pushed successfully to your VEM host, but as you see the Slot # is still showing "1".  This means the VSM is not able to communicate and assign a proper slot # to your VEM using the Control VLAN.

Can you post the configuration of your upstream switch port (C2900) which this host is using for the System Uplink.  You can use the CDP information from within vCenter to verify the exact switchport if you're unsure.  This is always a goo sanity check.

Can you also confirm VLAN 10 and 11 are allowed on the interconnecting links between the two C2960 switches.  You may want to do a "show mac-address dynamic VLAN 10" on each switch.  You should see both the VSM and VEM MAC addresses on EACH C2960 switch.

Robert

[nenduri]

Can you send the output of the following commands?

vemcmd show profile

vemcmd show trunk

vemcmd show port

vemcmd show l2 <control vlan>

vemcmd show bd <control vlan>

Also, the configuration of the upstream switch ports that are connected down to VEM? what is the topology?

-Naren

[cczcp1-uon]

Sorry for the delay, other things took priority.

I've also been talking with our network team to go over a few things, to ensure the switches are configured correctly.
I've reshuffled the NICs in order for them to use the same switch.

[root@esx08 ~]# vemcmd show profile
Name                  ID  VLAN  Mode     Duplex Speed   Tagged VLANs
dvportgroup-210        3  4097  Trunk    Full   Auto    10-11

[root@esx08 ~]# vemcmd show trunk
Trunk port 18 native_vlan 1 CBL 0
vlan(10) cbl 4, vlan(11) cbl 4,
Trunk port 22 native_vlan 1 CBL 0
vlan(10) cbl 4, vlan(11) cbl 4,

[root@esx08 ~]# vemcmd show port
  LTL    IfIndex   Vlan    Bndl  SG_ID Pinned_SGID  Type  Admin State  CBL Mode   Name
    8          0   3969       0     32          32  VIRT     UP    UP    4 Access l20
    9          0   3969       0     32          32  VIRT     UP    UP    4 Access l21
   10          0     10       0     32          32  VIRT     UP    UP    4 Access l22
   11          0   3968       0     32          32  VIRT     UP    UP    4 Access l23
   12          0     11       0     32          32  VIRT     UP    UP    4 Access l24
   13          0      1       0     32          32  VIRT     UP    UP    0 Access l25
   14          0   3967       0     32          32  VIRT     UP    UP    4 Access l26
   15          0   3967       0     32          32  VIRT     UP    UP    4 Access l27
   18          0      1 T     0     32          32  PHYS     UP    UP    0  Trunk vmnic2
   22          0      1 T     0     32          32  PHYS     UP    UP    0  Trunk vmnic6


[root@esx08 ~]# vemcmd show l2 10
Bridge domain   10 brtmax 1024, brtcnt 1, timeout 300
Flags:  P - PVLAN  S - Secure
       Type         MAC Address   LTL   timeout   Flags    PVLAN
     Static   00:02:3d:40:64:00    10         0

[root@esx08 ~]# vemcmd show bd 10
BD 10, vdc 1, vlan 10, 3 ports
Portlist:
     10  l22
     18  vmnic2
     22  vmnic6

Looking at the DVS Configuration / Ports - the system uplink group has green NIC port icons but doesn't provide any CDP Info at all. As for the vm-uplink group only 1 of those hosts has a green NIC port and the remaining two are grey with X's on them (see attached picture), which also dont provide any CDP info.

Cheers guys.

[Robert Burns]

When you move these VMNICs (the one you're using for the DVS) to a vSwitch  - do you see CDP info there?  Are you sure that spanning tree isn't blocking these ports on your upstream switch?

+ From your upstream switch where these uplinks connect can you provide the output of "show spanning-tree int x/x"

Also, if you plan on adding multiple links to your uplink port profiles you'll need to add a channel-group command such as "channe-group auto mode on mac-pinning" (if running version 4.0.4.SV1.2) or "channel-group auto mode on sub-group cdp" (if running version 4.0.4.SV1.1).

It is interesting the CBL states of the vmnics is showing "0" rather than "1"...

Robert