Problem with VEM UUID on different ESX hosts seen the same

[sampatel_presidio]

I was seeing a problem where on two different ESX hosts, VEM card show the same UUID. Because of this, when module gets inserted in VSM, they tried to take over each other's slot. This caused network disruption. I tried setting UUID manually to match Host's UUID (unique) for both VEMs but as soon as VEM is reloaded or restarted or ESX reboots, below UUID is set by default.

Has anyone seen this problem ? Is VEM UUID generated by VEM automatically ? Can we permanently set VEM UUID manually ? If yes, what is configuration file on ESX host that contains VEM's UUID ?

Outputs are attached as below. On two different servers, VEM UUIDs are seen the same. This was automatically generated when VEM starts.

[root@vsphere01 ~]# vemcmd show card
Card UUID type  2: 800c0000-0080-0c00-0000-800c00000080
Card name: vsphere01
Switch name: n1000vnew
Switch alias: DvsPortset-0
Switch uuid: 13 e3 19 50 1c da f6 7b-16 78 d3 e5 77 f3 06 ca Card domain: 1 Card slot: 4 VEM Control (AIPC) MAC: 00:02:3d:10:01:03 VEM Packet (Inband) MAC: 00:02:3d:20:01:03 VEM Control Agent (DPA) MAC: 00:02:3d:40:01:03 VEM SPAN MAC: 00:02:3d:30:01:03 Management IP address: 192.168.10.48 Max physical ports: 32 Max virtual ports: 216 Card control VLAN: 50 Card packet VLAN: 51
        Processors: 8
   Processor Cores: 8
Processor Sockets: 2
   Physical Memory: 3489333248
[root@vsphere01 ~]#

[root@vsphere02 ~]# vemcmd show card
Card UUID type  2: 800c0000-0080-0c00-0000-800c00000080
Card name: vsphere02
Switch name: n1000vnew
Switch alias: DvsPortset-0
Switch uuid: 13 e3 19 50 1c da f6 7b-16 78 d3 e5 77 f3 06 ca Card domain: 1 Card slot: 3 VEM Control (AIPC) MAC: 00:02:3d:10:01:02 VEM Packet (Inband) MAC: 00:02:3d:20:01:02 VEM Control Agent (DPA) MAC: 00:02:3d:40:01:02 VEM SPAN MAC: 00:02:3d:30:01:02 Management IP address: 192.168.10.49 Max physical ports: 32 Max virtual ports: 216 Card control VLAN: 50 Card packet VLAN: 51
        Processors: 8
   Processor Cores: 8
Processor Sockets: 2
   Physical Memory: 3489333248
[root@vsphere02 ~]#

[Robert Burns]

Was the second host a clone of the first?  (using something like Symantec Ghost, or other clong utilities)

Normally the UUID is randomly generated, and should almost always be unique.

Robert

[sampatel_presidio]

No. They were fresh install of vSphere4 GA release as required for N1000v. We were running in evaluation period. There were not cloned. On Nexus 1000v, we are using Evaluation license.

I checked yesterday and now the uuid that I pasted earlier here is no longer seen. Now both cards show unique uuid. They both are inserted ok in VSM and both are powered ON. But I had above observation two times. I re-installed N1000v, VEMs but problem was seen after couple of days. It should generate unique uuid when VEM module load into kernel. Than it remembers that uuid and stays tied with it.

Onc thing I noticed was when we execute "vem start", in verbose I could see a command running to set card uuid. "vemcmd card uuid vmware <string>. I wonder if we could manually set uuid which is permanent by editing some config file to avoid such a problem.

I still like to find out why it generated two identical IDs on two different ESX hosts having unique host UUIDs.

[Robert Burns]

The command you see in the verbose logging "vemcmd card uuid vmware <string>" is just setting the UUID which the VEM pulls from ESX (which in turn is pulled from the Server's BIOS).  We have heard of this case before, seeing duplicate UUIDs.

What is the exact make and model of the affected VEMs you've seen this duplicate behaviour on?  We'll ask VMware is this is a known issue on that hardware.

If possible can you also provide the output of "esxcfg-info" from each affected host and attach it here?

Thanks,

Robert

[sampatel_presidio]

Attached are esxcfg-info outputs from two ESX hosts. Hardware is Sun X4150 with Intel processors.

[Robert Burns]

Thanks for the info.

Can you also check the BIOS of these hosts and see if the UUID listed there matches the UUID displayed by the VEM?  Understandibly this will require a reboot. 

We're trying to determine if the host UUID is changing, or if the VEM is not reporting the correct UUID.

Thanks,

Robert

[thatkidee]

Robert, has there been any insight on why this happens.  I have 3 hosts all installed from cd and all of them have the same vem UUID.  All three hosts are exactly the same both in cpu and memory.  If I change the uuid prior to attaching the host the vem module shows up just fine but if I have to reboot the host or if I restart vem then it grabs the original UUID.

[p.verlee]

Hello,

I had the same problem with three Supermicro hosts that had the same UUID in the BIOS. The VEM takes the UUID from the host BIOS. There is a program called AMIDMI.EXE (put it on a boot floppy) to change the UUID in the BIOS (at least it works for AMI BIOS). With this tool I changed the UUID of two of my hosts and now "show module" shows the three VEM's.

Regards,

Patrick Verlee