1000v VSM lossing contact with VEMs after vMotion

[Patrick Colbeck]

Hi

We installed 1000v VSMs as VMs on a HP blade server with virtual connect modules.

It all worked fine and we migrated most of the VMs to 1000v. Even HA between the primary and backup VSMs worked.

Now we find that if you live migrate the active VSM is looses all contact with the VEMs but the backup VSM doesnt take over unless you power off the active one, The same thing happened when the customers Trend Deep Security Manager made soem update to the VSM VM.

We have the 1000v deployed in L3 mode using the control interface and the VEMs have their own vmkernel IP address which as it happens is on teh same IP subnet as the VSM control interface.

We have both the 1000v VSM control interface and managment interface configured as system VLANs

1000v version 4.2(1)sv2(2.1)ON VMware 5.1

Any ideas ? This kind of thing is usually a system VLAN issue.

Config info below.

Thanks

Pat

vlan 60

  name mgmt  <--- ssh to VSM etc

vlan 66

  name VEM-mgmt  <-- VSM to VEM

port-profile type vethernet n1kv_NEXUS_L3_CONTROL

  capability l3control

  vmware port-group

  switchport mode access

  switchport access vlan 66

  no shutdown

  system vlan 66

  state enabled

port-profile type vethernet n1kv_VLAN60

  vmware port-group

  switchport mode access

  switchport access vlan 60

  no shutdown

  system vlan 60

  state enabled

port-profile type ethernet n1kv_UPLINK

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 4,7,9,18,54,57-66,68,71,94-95

  service-policy type queuing output uplink-policy-map

  channel-group auto mode on mac-pinning relative

  no shutdown

  system vlan 60,66

  max-ports 32

  state enabled

svs-domain

  domain id 10

  control vlan 1

  packet vlan 1

  svs mode L3 interface control0

svs connection vcenter

  protocol vmware-vim

  remote ip address 172.30.60.10 port 80

  vmware dvs uuid "f7 29 21 50 82 15 50 7f-52 20 2d 6d 89 f1 6c a7" datacenter-name Gatwick

  admin user n1kUser

  max-ports 30000

  connect

The following are from when its working:

show svs dom

SVS domain config:

  Domain id:    10 

  Control vlan:  NA

  Packet vlan:   NA

  L2/L3 Control mode: L3

  L3 control interface: control0

  Status: Config push to VC successful.

  Control type multicast: No

show svs con

connection vcenter:

    ip address: 172.30.60.10

    remote port: 80

    protocol: vmware-vim https

    certificate: default

    datacenter name: Gatwick

    admin: n1kUser(user)

    max-ports: 30000

    DVS uuid: f7 29 21 50 82 15 50 7f-52 20 2d 6d 89 f1 6c a7

    config status: Enabled

    operational status: Connected

    sync status: Complete

    version: VMware vCenter Server 5.1.0 build-1064983

    vc-uuid: 8C96DBA4-C1AA-490E-86E4-50B76DFA31AC

[lwatta]

Patrick,

A couple of things to check.

When you vmotion the VSM can you run "show system internal redundancy trace" and see what mode you are in for HA. I believe you are in "degraded mode" which means that your VSMs are still talking to each other but over the mgmt interface. This is why the VEMs modules are not switching to the second VSM. Your control interface is dropping and not coming back. We need to figure out what is going on with your control 0 interface.

Which leads me to the second thing to check. Where is your control 0 interface connected to? is it connected to the VEM? or is it connected to a vSwitch. If its connected to the VEM which port-profile is it connected to?

louis

[Patrick Colbeck]

Hi

This is the profile the control interface should be using:

port-profile type vethernet n1kv_NEXUS_L3_CONTROL

  capability l3control

  vmware port-group

  switchport mode access

  switchport access vlan 66

  no shutdown

  system vlan 66

  state enabled

I will get the customer to get teh other information (I am not at that site anymore).

Thanks

[Patrick Colbeck]

Here you go with the "show system internal redundancy trace" and some "show  mod" sorry its a word doc but the customer pasted in screen caps rather than logging the terminal output.

The server is a HP C7000 blade server with HP Virtual Connect modules (hence the mac pinning) and the switches above that are Nexus 5548.

Thanks

Pat

[lwatta]

Your control 0 interface on the VSM is using the n1k_NEXUS_L3_CONTROL port-profile?

If thats true, then it's a problem.

Any port-profile with "capability l3control" can only have vmk interfaces assigned to it. You'll need to create a new port-profile exactly like n1kv_NEXUS_L3_CONTROL but without "capability l3control"

What version are you running? The document shows behavior that I would expect with either 1.5 code.

louis

[Patrick Colbeck]

Louis

The only thing on VLAN 66 is the vmk interfaces for the VEMs and VSM control interface.

Are you saying that the VSM control interface will need its own port profile without " capabilty L3 control" ?

If so thats a good one. I would normally have had the VEM vmk  interafces on a different subnet to the VSM interface (either control or management) so have never run into this one before as they would have had different port profiles.

Regards

Pat

NB It shoudl be 4.2(1)SV2(1.1a) but I will get them to check. It was supposed to be the new freemium 1000v as they only need basic features. I will be a bit cross if the engineer installed the old version that needs licensing.

[Patrick Colbeck]

Confirmed software version:

!Command: show running-config

!Time: Tue Jul  2 11:57:06 2013

version 4.2(1)SV2(1.1a)

svs switch edition essential

[lwatta · Accepted solution]

Yeah the problem is that any port-profile with "capability l3control" is meant to only work with vmk interfaces.

You never want to put a VM interface on port-profile with l3control. We used to allow this but for some reason it was pulled.

Create another port-profile just for control 0 and see if that fixes the problem. Again make it identical except for the "capability l3control"

Your version is good. I'll have to play in my lab. I'm kind of confused why your VSM2 gets hung at the loading prompt when you vmotion VSM1. That should not happen now.

louis

[Patrick Colbeck]

Louis

That did the trick. Both vMotion and Trend work fine now without causing loss of conectivity.

Thanks

Pat