N1KV VEM Module does not come online in VSM

[HUBERT RESCH]

Hi, we have on former working VEM which does not come online in VSM after reboot of VEM-ESX. Its VEM Module 5

SW3-LINZ-XDE5# sh svs connections

connection vcenter:
    ip address: 22.161.70.237
    remote port: 80
    protocol: vmware-vim https
    certificate: default
    datacenter name: NW-VIRT
    DVS uuid: ef 03 0b 50 af 04 16 1f-9a de 58 49 54 35 9d d0
    config status: Enabled
    operational status: Connected
    sync status: Complete
    version: VMware vCenter Server 4.1.0 Build-258902

!# VSM hear from Module 5

SW3-LINZ-XDE5# sh svs neighbors

Active Domain ID: 1

AIPC Interface MAC: 0050-568b-6cca
Inband Interface MAC: 0050-568b-3a4c

Src MAC           Type   Domain-id    Node-id     Last learnt (Sec. ago)
------------------------------------------------------------------------

0050-568b-13f7     VSM         1         0201     605520.16
0002-3d40-0103     VEM         1         0402     664779.10
0002-3d40-0104     VEM         1         0502       862.74
0002-3d40-0105     VEM         1         0602     1957112.55
0002-3d40-0106     VEM         1         0702     1712752.86

SW3-LINZ-XDE5#

!# but module 5 does not appear in VSM

SW3-LINZ-XDE5# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
2    0      Virtual Supervisor Module        Nexus1000V         ha-standby
4    248    Virtual Ethernet Module          NA                 ok
6    248    Virtual Ethernet Module          NA                 ok

Mod  Sw               Hw
---  ---------------  ------
1    4.0(4)SV1(3b)    0.0
2    4.0(4)SV1(3b)    0.0
4    4.0(4)SV1(3b)    2.0
6    4.0(4)SV1(3b)    2.0

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA
2    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA
6    02-00-0c-00-06-00 to 02-00-0c-00-06-80  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    22.161.70.233    NA                                    NA
2    22.161.70.233    NA                                    NA
4    22.161.70.34     001529b8-f87a-df11-8ae0-0022bdd7a96c  22.161.70.34
6    22.161.70.33     2023dbeb-7a67-df11-b476-8843e1c2d268  22.161.70.33

* this terminal session
SW3-LINZ-XDE5#

!# strange is that on VEM the

!#

!#

vem-health check 0050-568b-6cca
!#

!# shows that VEM, VSM conectivity seems ok

!#

!# also with

vem status -v

!#

!# we see that module is loaded and uplinks are present

!#

!# I also can see that the uplinkports are up with

vemcmd show port

!#

!# the only differnce to other working VEMs we can see with

vemcmd show card

!#

!# there the "Heartbeat set : False"  on working VEMs this value is true.

!#

!#

!# MAC-Adresses of VEM and VSM are seen on the phys Uplink-switches as well on the correct ports in the correct vlans

Also a additional reboot of the ESX does not help

But the Module does not come up !!!

Any Idea about this problem

btw

Nexus 1000 V : 4.0(4)SV1(3b)

VEM: 4.0.4.1.3.2.0-2.0.1

Thx in Advance

Hubert

[lwatta]

You are following the correct steps for troubleshooting the problem. If the MAC addresses are being seen on the VSM, VEM, and upstream switches than connectivity should work. The only thing left to check is to see if the VEM module is actually seeing the heartbeat messages. You can use vempkt on the VEM to see if you are getting heartbeat messages.

One other thing to try is to remove the ESX host from the N1KV in vCenter and add it back. I have seen times where vCenter doesn't always send all the right data to the host on the initial install.

If all the above fails, please open a TAC case. It might be a bug or some configuration issue having to do with your environment.

louis

[ababakha]

Hey All,

Just in regards to the  above posting. If the VEM is not seeing the right data you can validate this by running "vemcmd show card" this will validate if the vemis programmed correctly i.e. domain id etc.

[HUBERT RESCH]

@ Andrew, thx, I know, but we already doublechecked this

[HUBERT RESCH]

Yeah, I am going to open a SR