05-04-2010 01:02 PM
I've got 3 test ESX hosts in my test lab.
I installed the VSM on my ESX1 host; setup the control/packet/mgmt vlans, the physical switches are all set correctly. I'm running the control/packet data over the same pNics/port-profile as VM traffic BTW.
I then installed the VEM on ESX1 host and assigned one of the pNics to the Nexus vDS, leaving the other with the local vSwitch until my VM's were switched to the Nexus vDS, I then installed the VEM on my other 2 esx hosts and assigned the two pNics that were doing vm traffic, created the port groups and what not and everything seemed to working fine, the migrated vm's on those 2 hosts worked on the new Nexus vDS just fine.
So I then went back to my ESX1 host and migrated the remaining pNic from the local vSwitch over to the Nexus vDS and then the VSM lost connectivity to all of the VEM's, a show module command only shows the 2 VSM's I created. If I move one of the pNics back to the local vSwitch, the VEM's start showing up again.
What am I missing here? It seems like the control vlan only works when conected to the local vSwitch on ESX1, which has me kind of stumped as to why, any help would be much appreciated.
05-04-2010 01:07 PM
Hello -
What does your topology look like? Are you placing the VSM on the VEM's that it is managing? Can you post the configuration of the VSM?
Thanks,
Liz
05-04-2010 01:19 PM
Here you go, thanks for responding:
kernel core target 0.0.0.0
kernel core limit 1
system default switchport
vem 3
host vmware id 44454c4c-3000-1052-8036-b8c04f503831
vem 4
host vmware id 44454c4c-5200-1043-8035-b6c04f524231
vem 5
host vmware id 44454c4c-4e00-1042-8036-c4c04f363931
snmp-server user admin network-admin auth md5 0xa54bb1631e3db394f6459672cf267fe1 priv 0xa54bb1631e3db394f6459672cf267fe1 localizedkey
snmp-server enable traps license
vrf context management
ip route 0.0.0.0/0 10.1.117.1
switchname Nex1KV-VSM-Primary
vlan 1
vlan 604
name Vlan-604
vlan 605
name Vlan-605
vlan 607
name Vlan-607
vlan 964
name Vlan-964
vlan 1168
name Vlan-1168
vlan 1169
vdc Nex1KV-VSM-Primary id 1
limit-resource vlan minimum 16 maximum 513
limit-resource monitor-session minimum 0 maximum 64
limit-resource vrf minimum 16 maximum 8192
limit-resource port-channel minimum 0 maximum 256
limit-resource u4route-mem minimum 32 maximum 80
limit-resource u6route-mem minimum 16 maximum 48
port-profile type ethernet Unused_Or_Quarantine_Uplink
description Port-group created for Nexus1000V internal usage. Do not use.
vmware port-group
shutdown
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
description Port-group created for Nexus1000V internal usage. Do not use.
vmware port-group
shutdown
state enabled
port-profile type vethernet data604
vmware port-group
switchport mode access
switchport access vlan 604
no shutdown
state enabled
port-profile type vethernet data605
vmware port-group
switchport mode access
switchport access vlan 605
no shutdown
state enabled
port-profile type vethernet data607
vmware port-group
switchport mode access
switchport access vlan 607
no shutdown
state enabled
port-profile type vethernet data964
vmware port-group
switchport mode access
switchport access vlan 964
no shutdown
state enabled
port-profile type ethernet vm-sys-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 604-605,607,964,1168-1169
no shutdown
system vlan 964,1168-1169
state enabled
interface Ethernet3/5
inherit port-profile vm-sys-uplink
interface Ethernet3/9
inherit port-profile vm-sys-uplink
interface Ethernet4/4
inherit port-profile vm-sys-uplink
interface Ethernet5/5
inherit port-profile vm-sys-uplink
interface Ethernet5/9
inherit port-profile vm-sys-uplink
interface mgmt0
ip address 10.1.117.104/24
interface Vethernet1
inherit port-profile data607
description sms-01.fsu-ad.edu, Network Adapter 1
vmware dvport 101
interface Vethernet2
inherit port-profile data607
description uptime, Network Adapter 1
vmware dvport 100
interface Vethernet3
inherit port-profile data607
description Hyperic, Network Adapter 1
vmware dvport 102
interface Vethernet4
inherit port-profile data607
description SP-dev-app.fsu-ad.edu, Network Adapter 1
vmware dvport 103
interface Vethernet5
inherit port-profile data964
description BaseWin2003_STD_SP2, Network Adapter 1
vmware dvport 384
interface Vethernet6
inherit port-profile data607
description sp-dev-wfe.fsu-ad.edu, Network Adapter 1
vmware dvport 104
interface control0
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3.bin sup-1
boot system bootflash:/nexus-1000v-mz.4.0.4.SV1.3.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3.bin sup-2
boot system bootflash:/nexus-1000v-mz.4.0.4.SV1.3.bin sup-2
svs-domain
domain id 1
control vlan 1168
packet vlan 1169
svs mode L2
svs connection vc
protocol vmware-vim
remote ip address 146.201.7.237 port 10080
vmware dvs uuid "01 5e 34 50 62 aa 55 3d-cb 7f ae c3 5c 63 9c ca" datacenter-name DEV-ITS
connect
Nex1KV-VSM-Primary#
05-04-2010 01:35 PM
Sorry forgot to mention my topology; basically got 3 esx hosts, got the vsm on one of those hosts, I'm wanting to have the Nexus vDS carry control/packet and vm traffic on the same port-profile, the VLAN964 is the mgmt vlan when looking at my config, that's the same VLAN as my esx hosts if that matters at all.
05-04-2010 01:39 PM
Which VLAN are you using for control/packet? Is it the same as 964? You will need to make the control/packet vlan a "system vlan" in the vethernet port-profile as well. Which port-profile are you using for your VSM?
Also, are you connecting your hosts to 2 different upstream switches? If so, you will need to add the command "channel-group auto mode on mac-pinning" to the uplink port-profile.
Thanks,
Liz
05-04-2010 01:55 PM
Control is 1168 and packet is 1169, the port-profile is vm-sys-uplink, is this how it should look? Vlan 964 is the mgmt vlan. I do have two upstream switches, basically on each host I have two pNics I'm attempting to use for the control/packet/data, one pNic goes to one switch and the other pNic to another switch. Is this where the channel-group command might help? Is this possibly why when I switch both pNics over I'm creating a spanning tree loop by chance? Sorry I'm the vm guy here not a Cisco guy so I'm trying to learn enough to get this Nexus 1000v off the ground. Thanks again.
port-profile type ethernet vm-sys-uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 604-605,607,964,1168-1169
no shutdown
system vlan 964,1168-1169
state enabled
05-04-2010 07:21 PM
Hey Daryn -
Ryan mentioned it in his post about creating port-profiles for management/control/packet interfaces of the VSM. This is assuming that you are putting the VSM behind the VEM on the ESX host which it looks like you are trying to do.
In order to do this, you need to create 3 vethernet port-profiles:
port-profile type vethernet control
vmware port-group
sw mode access
sw access vlan <control_vlan>
no shut
system vlan <control_vlan>
state enable
port-profile type vethernet packet
vmware port-group
sw mode access
sw access vlan <packet_vlan>
no shut
system vlan <packet_vlan>
state enable
port-profile type vethernet management
vmware port-group
sw mode access
sw access vlan <management_vlan>
no shut
system vlan <management_vlan>
state enable
once these are available in vCenter, you can migrate the VSM over to the 1000V. To do this, go under "Edit Settings" and choose the appropriate port-groups for the network adapters. Network adapter 1 = control, network adapter 2 = management, network adapter 3 = packet.
Also, you will want to add that "channel-group auto mode on mac-pinning" command to the "vm_sys_uplink" port-profile.
See if that helps any.
Thanks,
Liz
05-05-2010 08:04 AM
Thanks for the reply, I'll give this all a try; just curious is what I'm doing common practice, I kind of feel like based on the feedback I'm getting that what I'm doing isn't really the way I should be doing this, even though it can be done.
Also, on a side note, how do you remove lines from the config file, I have an incomplete port-profile statement I'd like to remove but not sure how to remove it. Thanks again.
05-05-2010 08:12 AM
Obviously just one guy's opinion here, but I think you'll find a variety of ways people approach this. I know some people who run hybrid vSwitch/1000v and leave things like Service Console, etc. off.
Personally, I dig the control/visibility I get into all of it as a network guy with limited/no server access, so we made the push to native 1000v. YMMV depending on where you stand. For me, I don't really feel like I am troubleshooting vSwitch issues in the dark now, because I can see exactly what's going on. Makes resolution to silly little things a bit quicker/easy to communicate.
You should be able to disassociate your empty port-profile with any VM/host, and no port-profile <name>. I believe it's case sensitive as well, if that helps any.
05-05-2010 11:07 AM
Hey Daryn -
This configuration is consistent with Cisco's best practices when deploying the Nexus 1000V.
Thanks,
Liz
05-05-2010 03:45 PM
Just curious, after running the channel-group...l comman I get that "removing VEM 3..." error and VEM 3 drops out, I seemed to be able to get it back by removing it and adding it back to the dVS, but was wondering why that happens. Sorry for the stupid questions, but not being a Cisco person, I like to keep track of errors I run accross in case they happen in production. Thanks again.
Nex1KV-VSM-Primary(config-port-prof)# port-profile vm-sys-uplink
Nex1KV-VSM-Primary(config-port-prof)# vmware port-group
Nex1KV-VSM-Primary(config-port-prof)# switchport mode trunk
Nex1KV-VSM-Primary(config-port-prof)# sw trunk allowed vlan 604,605,607,964,1168,1169
Nex1KV-VSM-Primary(config-port-prof)# no shut
Nex1KV-VSM-Primary(config-port-prof)# system vlan 964,1168,1169
Nex1KV-VSM-Primary(config-port-prof)# channel-group auto mode on mac-pinning
Nex1KV-VSM-Primary(config-port-prof)# 2010 May 5 18:26:10 Nex1KV-VSM-Primary %PLATFORM-2-PFM_VEM_REMOVE_NO_HB: Removing VEM 3 (heartbeats lost)
2010 May 5 18:26:10 Nex1KV-VSM-Primary %PLATFORM-2-MOD_REMOVE: Module 3 removed (Serial number )
Nex1KV-VSM-Primary(config-port-prof)# state enabled
05-05-2010 04:21 PM
This can happen if the upstream switches are not L2 connected on control VLAN. Can you please check the configuration on upstream switches?
-Naren
05-07-2010 07:37 AM
Wouldn't that have been broken on the vSwitch, too?
ie: When his VSM was on the single vSwitch uplink it would not have been able to see VEMs if there was no l2 path to control via the 1 uplink already moved to the 1000v. Since uplinks are supposedly on separate switches, it'd have to traverse some back-to-back or upstream path, I'd think...?
Could see why that might happen if 1 of the 2 uplinks had the Control VLAN and he had some bad luck with regards to which uplink he got pinned to, but seems like his original hybrid setup rules that out.
Might just have my thoughts twisted up... it's Friday (hooray). Still would be helpful to see upstream switch configs -- scrubbed if required.
05-07-2010 07:44 AM
I ended up removing our esx1 VEM from the Nexus vDS and then added it back and it started showing up again on the VSM, so not sure what that means exactly; our upstream switches are two Foundry's and then those connect up to our core Cisco 6509's, I'm not sure if there is anything particuliar with the Foundry's that maybe be different than using an upstream Cisco, as far as configurating the Nexus. I'll see if I can ge the configs for those Foundry's and post them and see if they look ok, but right now for the most part everything seems to be working.
05-07-2010 07:49 AM
Daryn, couple of additional things to check...
- Switch ports connecting to your hosts are configured in portfast mode. I'm not sure the command syntax for Foundry, and/or if it will support that feature for a trunk, or just an access port.
- Switch ports connecting to your hosts should just be a trunk carrying VLANs identical to the uplink port-profile you created. No link aggregation or anything like that.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide