VSM and VEM's losing connectivity

[daryn10]

I've got 3 test ESX hosts in my test lab.

I installed the VSM on my ESX1 host; setup the control/packet/mgmt vlans, the physical switches are all set correctly.  I'm running the control/packet data over the same pNics/port-profile as VM traffic BTW.

I then installed the VEM on ESX1 host and assigned one of the pNics to the Nexus vDS, leaving the other with the local vSwitch until my VM's were switched to the Nexus vDS, I then installed the VEM on my other 2 esx hosts and assigned the two pNics that were doing vm traffic, created the port groups and what not and everything seemed to working fine, the migrated vm's on those 2 hosts worked on the new Nexus vDS just fine.

So I then went back to my ESX1 host and migrated the remaining pNic from the local vSwitch over to the Nexus vDS and then the VSM lost connectivity to all of the VEM's, a show module command only shows the 2 VSM's I created.  If I move one of the pNics back to the local vSwitch, the VEM's start showing up again.

What am I missing here?  It seems like the control vlan only works when conected to the local vSwitch on ESX1, which has me kind of stumped as to why, any help would be much appreciated.

[estine]

Hello -

What does your topology look like? Are you placing the VSM on the VEM's that it is managing? Can you post the configuration of the VSM?

Thanks,

Liz

[daryn10]

Here you go, thanks for responding:

kernel core target 0.0.0.0
kernel core limit 1
system default switchport
vem 3
  host vmware id 44454c4c-3000-1052-8036-b8c04f503831
vem 4
  host vmware id 44454c4c-5200-1043-8035-b6c04f524231
vem 5
  host vmware id 44454c4c-4e00-1042-8036-c4c04f363931
snmp-server user admin network-admin auth md5 0xa54bb1631e3db394f6459672cf267fe1 priv 0xa54bb1631e3db394f6459672cf267fe1 localizedkey
snmp-server enable traps license
vrf context management
  ip route 0.0.0.0/0 10.1.117.1
switchname Nex1KV-VSM-Primary
vlan 1
vlan 604
  name Vlan-604
vlan 605
  name Vlan-605
vlan 607
  name Vlan-607
vlan 964
  name Vlan-964
vlan 1168
  name Vlan-1168
vlan 1169
vdc Nex1KV-VSM-Primary id 1
  limit-resource vlan minimum 16 maximum 513
  limit-resource monitor-session minimum 0 maximum 64
  limit-resource vrf minimum 16 maximum 8192
  limit-resource port-channel minimum 0 maximum 256
  limit-resource u4route-mem minimum 32 maximum 80
  limit-resource u6route-mem minimum 16 maximum 48
port-profile type ethernet Unused_Or_Quarantine_Uplink
  description Port-group created for Nexus1000V internal usage. Do not use.
  vmware port-group
  shutdown
  state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
  description Port-group created for Nexus1000V internal usage. Do not use.
  vmware port-group
  shutdown
  state enabled
port-profile type vethernet data604
  vmware port-group
  switchport mode access
  switchport access vlan 604
  no shutdown
  state enabled
port-profile type vethernet data605
  vmware port-group
  switchport mode access
  switchport access vlan 605
  no shutdown
  state enabled
port-profile type vethernet data607
  vmware port-group
  switchport mode access
  switchport access vlan 607
  no shutdown
  state enabled
port-profile type vethernet data964
  vmware port-group
  switchport mode access
  switchport access vlan 964
  no shutdown
  state enabled
port-profile type ethernet vm-sys-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 604-605,607,964,1168-1169
  no shutdown
  system vlan 964,1168-1169
  state enabled

interface Ethernet3/5
  inherit port-profile vm-sys-uplink

interface Ethernet3/9
  inherit port-profile vm-sys-uplink

interface Ethernet4/4
  inherit port-profile vm-sys-uplink

interface Ethernet5/5
  inherit port-profile vm-sys-uplink

interface Ethernet5/9
  inherit port-profile vm-sys-uplink

interface mgmt0
  ip address 10.1.117.104/24

interface Vethernet1
  inherit port-profile data607
  description sms-01.fsu-ad.edu, Network Adapter 1
  vmware dvport 101

interface Vethernet2
  inherit port-profile data607
  description uptime, Network Adapter 1
  vmware dvport 100

interface Vethernet3
  inherit port-profile data607
  description Hyperic, Network Adapter 1
  vmware dvport 102

interface Vethernet4
  inherit port-profile data607
  description SP-dev-app.fsu-ad.edu, Network Adapter 1
  vmware dvport 103

interface Vethernet5
  inherit port-profile data964
  description BaseWin2003_STD_SP2, Network Adapter 1
  vmware dvport 384

interface Vethernet6
  inherit port-profile data607
  description sp-dev-wfe.fsu-ad.edu, Network Adapter 1
  vmware dvport 104

interface control0
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3.bin sup-1
boot system bootflash:/nexus-1000v-mz.4.0.4.SV1.3.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3.bin sup-2
boot system bootflash:/nexus-1000v-mz.4.0.4.SV1.3.bin sup-2
svs-domain
  domain id 1
  control vlan 1168
  packet vlan 1169
  svs mode L2 
svs connection vc
  protocol vmware-vim
  remote ip address 146.201.7.237 port 10080
  vmware dvs uuid "01 5e 34 50 62 aa 55 3d-cb 7f ae c3 5c 63 9c ca" datacenter-name DEV-ITS
  connect

Nex1KV-VSM-Primary#

[daryn10]

Sorry forgot to mention my topology; basically got 3 esx hosts, got the vsm on one of those hosts, I'm wanting to have the Nexus vDS carry control/packet and vm traffic on the same port-profile, the VLAN964 is the mgmt vlan when looking at my config, that's the same VLAN as my esx hosts if that matters at all.

[estine]

Which VLAN are you using for control/packet? Is it the same as 964? You will need to make the control/packet vlan a "system vlan" in the vethernet port-profile as well.  Which port-profile are you using for your VSM?

Also, are you connecting your hosts to 2 different upstream switches?  If so, you will need to add the command "channel-group auto mode on mac-pinning" to the uplink port-profile.

Thanks,

Liz

[daryn10]

Control is 1168 and packet is 1169, the port-profile is vm-sys-uplink, is this how it should look?  Vlan 964 is the mgmt vlan.  I do have two upstream switches, basically on each host I have two pNics I'm attempting to use for the control/packet/data, one pNic goes to one switch and the other pNic to another switch.  Is this where the channel-group command might help?  Is this possibly why when I switch both pNics over I'm creating a spanning tree loop by chance?  Sorry I'm the vm guy here not a Cisco guy so I'm trying to learn enough to get this Nexus 1000v off the ground.  Thanks again.

port-profile type ethernet vm-sys-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 604-605,607,964,1168-1169
  no shutdown
  system vlan 964,1168-1169
  state enabled

[estine]

Hey Daryn -

Ryan mentioned it in his post about creating port-profiles for management/control/packet interfaces of the VSM.  This is assuming that you are putting the VSM behind the VEM on the ESX host which it looks like you are trying to do.

In order to do this, you need to create 3 vethernet port-profiles:

port-profile type vethernet control

vmware port-group

sw mode access

sw access vlan <control_vlan>

no shut

system vlan <control_vlan>

state enable

port-profile type vethernet packet

vmware port-group

sw mode access

sw access vlan <packet_vlan>

no shut

system vlan <packet_vlan>

state enable

port-profile type vethernet management

vmware port-group

sw mode access

sw access vlan <management_vlan>

no shut

system vlan <management_vlan>

state enable

once these are available in vCenter, you can migrate the VSM over to the 1000V.  To do this, go under "Edit Settings" and choose the appropriate port-groups for the network adapters.  Network adapter 1 = control, network adapter 2 = management, network adapter 3 = packet.

Also, you will want to add that "channel-group auto mode on mac-pinning" command to the "vm_sys_uplink" port-profile.

See if that helps any.

Thanks,

Liz

[daryn10]

Thanks for the reply, I'll give this all a try; just curious is what I'm doing common practice, I kind of feel like based on the feedback I'm getting that what I'm doing isn't really the way I should be doing this, even though it can be done.

Also, on a side note, how do you remove lines from the config file, I have an incomplete port-profile statement I'd like to remove but not sure how to remove it.  Thanks again.

[ryan.lambert]

Obviously just one guy's opinion here, but I think you'll find a variety of ways people approach this. I know some people who run hybrid vSwitch/1000v and leave things like Service Console, etc. off.

Personally, I dig the control/visibility I get into all of it as a network guy with limited/no server access, so we made the push to native 1000v. YMMV depending on where you stand. For me, I don't really feel like I am troubleshooting vSwitch issues in the dark now, because I can see exactly what's going on. Makes resolution to silly little things a bit quicker/easy to communicate.

You should be able to disassociate your empty port-profile with any VM/host, and no port-profile <name>. I believe it's case sensitive as well, if that helps any.

[estine]

Hey Daryn -

This configuration is consistent with Cisco's best practices when deploying the Nexus 1000V. 

Thanks,

Liz

[daryn10]

Just curious, after running the channel-group...l comman I get that "removing VEM 3..." error and VEM 3 drops out, I seemed to be able to get it back by removing it and adding it back to the dVS, but was wondering why that happens.  Sorry for the stupid questions, but not being a Cisco person, I like to keep track of errors I run accross in case they happen in production.  Thanks again.

Nex1KV-VSM-Primary(config-port-prof)# port-profile vm-sys-uplink
Nex1KV-VSM-Primary(config-port-prof)# vmware port-group
Nex1KV-VSM-Primary(config-port-prof)# switchport mode trunk
Nex1KV-VSM-Primary(config-port-prof)# sw trunk allowed vlan 604,605,607,964,1168,1169
Nex1KV-VSM-Primary(config-port-prof)# no shut
Nex1KV-VSM-Primary(config-port-prof)# system vlan 964,1168,1169
Nex1KV-VSM-Primary(config-port-prof)# channel-group auto mode on mac-pinning
Nex1KV-VSM-Primary(config-port-prof)# 2010 May  5 18:26:10 Nex1KV-VSM-Primary %PLATFORM-2-PFM_VEM_REMOVE_NO_HB: Removing VEM 3 (heartbeats lost)
2010 May  5 18:26:10 Nex1KV-VSM-Primary %PLATFORM-2-MOD_REMOVE: Module 3 removed (Serial number )
Nex1KV-VSM-Primary(config-port-prof)# state enabled

[nenduri]

This can happen if the upstream switches are not L2 connected on control VLAN. Can you please check the configuration on upstream switches?

-Naren

[ryan.lambert]

Wouldn't that have been broken on the vSwitch, too?

ie: When his VSM was on the single vSwitch uplink it would not have been able to see VEMs if there was no l2 path to control via the 1 uplink already moved to the 1000v. Since uplinks are supposedly on separate switches, it'd have to traverse some back-to-back or upstream path, I'd think...?

Could see why that might happen if 1 of the 2 uplinks had the Control VLAN and he had some bad luck with regards to which uplink he got pinned to, but seems like his original hybrid setup rules that out.

Might just have my thoughts twisted up... it's Friday (hooray). Still would be helpful to see upstream switch configs -- scrubbed if required.

[daryn10]

I ended up removing our esx1 VEM from the Nexus vDS and then added it back and it started showing up again on the VSM, so not sure what that means exactly; our upstream switches are two Foundry's and then those connect up to our core Cisco 6509's, I'm not sure if there is anything particuliar with the Foundry's that maybe be different than using an upstream Cisco, as far as configurating the Nexus.  I'll see if I can ge the configs for those Foundry's and post them and see  if they look ok, but right now for the most part everything seems to be working.

[ryan.lambert]

Daryn, couple of additional things to check...

- Switch ports connecting to your hosts are configured in portfast mode. I'm not sure the command syntax for Foundry, and/or if it will support that feature for a trunk, or just an access port.

- Switch ports connecting to your hosts should just be a trunk carrying VLANs identical to the uplink port-profile you created. No link aggregation or anything like that.