03-30-2010 01:36 AM
Hi,
since december we have the nexus 1000v in our company. Everything working fine and looks good. We have a ESX cluster with two hosts and 7 NICs uplink. Today after a long time i have a look in the logging and saw this:
Dropping received frames from duplicate VSM - kernel
How could i get rid of this? This message flooding the logging.
Here is the configuration.....
sh ver:
Software
loader: version 1.2(2) [last: image booted through mgmt0]
kickstart: version 4.0(4)SV1(1)
system: version 4.0(4)SV1(1)
kickstart image file is:
kickstart compile time: 4/2/2009 23:00:00
system image file is: bootflash:/nexus-1000v-mz.4.0.4.SV1.1.bin
system compile time: 4/2/2009 23:00:00 [04/26/2009 13:27:23]
and here the uplink profiles and other profiles for service console and so on:
port-profile system-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 2,51-52 (VLAN 2 is not in use at the moment)
channel-group auto mode on sub-group cdp
no shutdown
system vlan 51-52
state enabled
port-profile vm-uplink
capability uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1,10,101-200
channel-group auto mode on sub-group cdp
no shutdown
state enabled
port-profile nexus-control
vmware port-group
switchport mode access
switchport access vlan 51
no shutdown
system vlan 51
state enabled
port-profile nexus-packet
vmware port-group
switchport mode access
switchport access vlan 52
no shutdown
system vlan 52
state enabled
port-profile ServiceConsole
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 1-2 (maybe here ist the problem, this is configured from a guy before)
no shutdown
state enabled
port-profile VMotion
vmware port-group
switchport mode access
switchport access vlan 10
no shutdown
state enabled
We have vlan 1 as the defaultvlan for office network, service console belongs to vlan 1 (Yes i know this is not good but at this time there is no other choice :-( )
On the uplink switch a port-channel is created with five ports for vm-uplink and another port-channel with two nics for system-uplink, which connected with the vmnic`s.
If i have to reconfigure some points please let me know if i can make this changes while the system is running.....
03-30-2010 01:42 AM
Jason,
This message usually comes when there's multiple uplinks and the channel group is not correctly configured. If your upstream switches are not sending CDP information, the port channels will not form. In which case you can manually assign the sub-groups by entering into the configuration of your Ethernet Interfaces and assigning it. **This instruction is for the current version your running. In the present version 4.0.4.SV1.2 there's a different way to configure manual sub group IDs. Note at version 4.0.4.SV1.1 the max # of sub-groups is 2.
Ex.
conf t
ethernet3/1
sub-group-ID 0
ethernet3/2
sub-group-id 1
1. Port Profile "ServiceConsole" should be an access port, and assigned to one access VLAN. Make this VLAN a system vlan on both this Port Profile and the Uplink Port Profile (whichever you decide to use).
2. Ensure the CDP is working properly, and the Port Channels are forming correctly. Paste the output of the following command:
"show port-channel summary"
"module vem 3 execute vemcmd show pc"
"show interface brief"
Thanks,
Robert
03-30-2010 01:53 AM
CDP is working fine, in the dvs i see the details of the vmnic`s and on which switchports they connected..... is the the service console-misconfiguration the problem with the dropping frames or has this nothing to do with this?
Here is the output you want:
show port-channel summary
--------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
1 Po1(SD) Eth NONE --
2 Po2(SU) Eth NONE Eth3/2(P) Eth3/6(P)
3 Po3(SU) Eth NONE Eth3/3(P) Eth3/4(P) Eth3/5(P)
Eth3/7(P) Eth3/8(P)
4 Po4(SU) Eth NONE Eth4/2(P) Eth4/5(P) Eth4/6(P)
Eth4/7(P) Eth4/8(P)
5 Po5(SU) Eth NONE Eth4/3(P) Eth4/4(P)
------------------------------------------------------------------------------------------------------------------------------------
module vem 3 execute vemcmd show pc
pce_ind chan pc_ltl pce_in_pc SG_ID mbrs
------- ---- ------ --------- ----- ----
0 3 303 0 0 21,22,19,18,17,
1
1 2 304 1 0 20,16,
1
------------------------------------------------------------------------------------------------------------------------------------
sh interface brief
--------------------------------------------------------------------------------
Port VRF Status IP Address Speed MTU
--------------------------------------------------------------------------------
mgmt0 -- up 192.168.113.12 1000 1500
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port
Interface Ch #
--------------------------------------------------------------------------------
Eth3/2 1 eth trunk up none 1000(D) 2
Eth3/3 1 eth trunk up none 1000(D) 3
Eth3/4 1 eth trunk up none 1000(D) 3
Eth3/5 1 eth trunk up none 1000(D) 3
Eth3/6 1 eth trunk up none 1000(D) 2
Eth3/7 1 eth trunk up none 1000(D) 3
Eth3/8 1 eth trunk up none 1000(D) 3
Eth4/2 1 eth trunk up none 1000(D) 4
Eth4/3 1 eth trunk up none 1000(D) 5
Eth4/4 1 eth trunk up none 1000(D) 5
Eth4/5 1 eth trunk up none 1000(D) 4
Eth4/6 1 eth trunk up none 1000(D) 4
Eth4/7 1 eth trunk up none 1000(D) 4
Eth4/8 1 eth trunk up none 1000(D) 4
--------------------------------------------------------------------------------
Port-channel VLAN Type Mode Status Reason Speed Protocol
Interface
--------------------------------------------------------------------------------
Po1 1 eth access down No operational members auto(I) none
Po2 1 eth trunk up none a-1000(D) none
Po3 1 eth trunk up none a-1000(D) none
Po4 1 eth trunk up none a-1000(D) none
Po5 1 eth trunk up none a-1000(D) none
--------------------------------------------------------------------------------
Interface VLAN Type Mode Status Reason MTU
--------------------------------------------------------------------------------
Veth1 10 virt access up none 1500
Veth2 1 virt access up none 1500
Veth3 1 virt access up none 1500
Veth4 1 virt access nonPcpt nonParticipating 1500
Veth5 1 virt access up none 1500
Veth6 1 virt access up none 1500
Veth7 1 virt access up none 1500
Veth8 1 virt access nonPcpt nonParticipating 1500
Veth9 1 virt access up none 1500
Veth10 1 virt access up none 1500
Veth11 150 virt access up none 1500
Veth12 1 virt access up none 1500
Veth13 102 virt access up none 1500
Veth14 1 virt access nonPcpt nonParticipating 1500
Veth15 1 virt access up none 1500
Veth16 1 virt trunk up none 1500
Veth17 52 virt access up none 1500
Veth18 1 virt access up none 1500
Veth19 1 virt access up none 1500
Veth20 52 virt access up none 1500
Veth21 51 virt access up none 1500
Veth22 1 virt access up none 1500
Veth23 51 virt access up none 1500
Veth24 1 virt access nonPcpt nonParticipating 1500
Veth25 1 virt access nonPcpt nonParticipating 1500
Veth26 1 virt trunk up none 1500
Veth27 1 virt access nonPcpt nonParticipating 1500
Veth28 150 virt access nonPcpt nonParticipating 1500
Veth29 10 virt access up none 1500
Veth30 1 virt access up none 1500
Veth31 1 virt access up none 1500
Veth32 1 virt access up none 1500
Veth33 1 virt access up none 1500
Veth34 1 virt access up none 1500
Veth35 1 virt access up none 1500
Veth36 101 virt access up none 1500
Veth37 1 virt trunk up none 1500
Veth38 1 virt trunk up none 1500
Veth39 1 virt access up none 1500
Veth40 1 virt access up none 1500
Veth41 1 virt access up none 1500
Veth42 150 virt access nonPcpt nonParticipating 1500
Veth43 150 virt access up none 1500
Veth44 150 virt access up none 1500
Veth45 1 virt access up none 1500
Veth46 1 virt access up none 1500
Veth47 1 virt access up none 1500
Veth48 1 virt access nonPcpt nonParticipating 1500
Veth49 1 virt access nonPcpt nonParticipating 1500
Veth50 1 virt access nonPcpt nonParticipating 1500
Veth51 103 virt access up none 1500
Veth52 1 virt access up none 1500
Veth53 1 virt access nonPcpt nonParticipating 1500
Veth54 1 virt access nonPcpt nonParticipating 1500
Veth55 1 virt access nonPcpt nonParticipating 1500
Veth56 1 virt access nonPcpt nonParticipating 1500
Veth57 104 virt access up none 1500
Veth58 1 virt access nonPcpt nonParticipating 1500
Veth59 1 virt access nonPcpt nonParticipating 1500
Veth60 104 virt access up none 1500
Veth61 1 virt access up none 1500
Veth62 1 virt access nonPcpt nonParticipating 1500
Veth63 1 virt access nonPcpt nonParticipating 1500
Veth64 1 virt access nonPcpt nonParticipating 1500
Veth65 1 virt access up none 1500
03-30-2010 04:19 AM
Jason,
Can you post the upstream port (physical switch) config. If they're all the same, post one.
Thanks,
Robert
03-30-2010 06:08 AM
Robert,
hmmm the problem seems to be solved. I haven^t configured two ports from the upstream switch in a port-channel and these two ports are the system-uplink from nexus.
I have updated the nexus to v1.2 only raise the feature level is pending.
I have rebooted both nexus machines primary and secondary but if i now perform a show module i only see one VSM... where is my second?
And the second think are the VEMs.....they show an old version i think? Or do i first need to raise the feature level?
Here is a show module:
Mod Ports Module-Type Model Status
--- ----- -------------------------------- ------------------ ------------
2 0 Virtual Supervisor Module Nexus1000V active *
3 248 Virtual Ethernet Module NA ok
4 248 Virtual Ethernet Module NA ok
Mod Sw Hw
--- --------------- ------
2 4.0(4)SV1(2) 0.0
3 4.0(4)SV1(1) 1.11
4 4.0(4)SV1(1) 1.11
Mod MAC-Address(es) Serial-Num
--- -------------------------------------- ----------
2 00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8 NA
3 02-00-0c-00-03-00 to 02-00-0c-00-03-80 NA
4 02-00-0c-00-04-00 to 02-00-0c-00-04-80 NA
Mod Server-IP Server-UUID Server-Name
--- --------------- ------------------------------------ --------------------
2 192.168.100.12 NA NA
3 192.168.100.151 34393433-3239-4742-3839-3330584b5830 192.168.100.151
4 192.168.100.152 34393433-3239-4742-3839-3330584b5842 192.168.113.152
Hmmm if i look in the vSphere Networking i can see "An upgrade for the Distributed Virtual Switch in datacenter is in progress." Is there something wrong with the update ? Need urgent help. :-(
On the esx host if i perform a vem status -v i see this:
Package vssnet-esx4.1.2-00000-release
Version 4.0.4.1.1.31-1.11.11
Build 11
Date Sat Jan 30 22:53:30 PST 2010
Number of PassThru NICs are 0
VEM modules are loaded
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 32 2 32 1500 vmnic0
DVS Name Num Ports Used Ports Configured Ports Uplinks
nexus1kv-primary256 27 256 vmnic3,vmnic1,vmnic5,vmnic6,vmnic4,vmnic2,vmnic7
Number of PassThru NICs are 0
VEM Agent (vemdpa) is running
i think this is the newest vem version, isin`t it?
esx host build is 236512
Do you nee some other output?
03-30-2010 03:04 PM
Jason,
Version 4.0.4.1.1.31-1.11.11 <== Still looks like you haven't completed the VEM upgrade.
Did you follow the steps in the upgrade guide?
In short
1. Upgrade primary & secondary VSM
2. If using VUM, powerdown all your VMs and then from the VSM initiate the Upgrade notice
3. From vCenter - Accept the Upgrade
4. From the VSM, proceed with the upgrade
5. Raise the feature level of the VSM
you can verify if your VEM's are upgrade by issueing "vem version" from the CLI.
CDP sub-group support has been greatly improved in the new release which might explain your problem resolving.
Robert
03-31-2010 12:09 AM
Robert,
1. Upgrade primary & secondary VSM
Hmm this might be the Problem i`ve only updated one VSM, the guide says the other automaticaly update. But if i look both of the VSM shows the right image in bootflash but the formerly primary VSM has no connection to the vsphere center (svs). If i try to connect manually with "svs connection <name>" -> "connect"
This message appears:
ERROR: [VMWARE-VIM] Operation could not be completed due to connection failure. No route to host. connect failed in tcp_connect()
sh module shows only the VSM as active and no other modules are shown.
2. If using VUM, powerdown all your VMs and then from the VSM initiate the Upgrade notice
Yes, i used VUM but that was not a good idea. I cannot powerdown the VMs.
3. From vCenter - Accept the Upgrade
Yes, i have accepted the Upgrade
4. From the VSM, proceed with the upgrade
I procceed the upgrade but it ends with an error message -> "The Object or item referred to could not be found. Cannot get compliance status for host(s):host -795. This indicates that previous host operations might have failed for these hosts.
5. Raise the feature level of the VSM
not yet done.
How could i now update the VEM with no downtime of VMs and get back my secondary VSM. And in the networking TAB the "upgrade in progress" is still shown.
I figured out that i`m not able in vSphere connect the three NICs from the primary VSM. Error message "invalid configuration for device `0`."
03-31-2010 01:47 AM
Jason,
First you need to sort out your VSM upgrade. You need them both upgrade and showing the correct new version. Ensure the "show sys red sta" shows them as sync'd up before proceeding with the VEM upgrades. When you did the upgrade, did you copy the new image to the bootflash of each VSM? If having issues I suggest you do the following. Ensure your primary VSM is active and running correctly. Power down and blow away your secondary VSM. Delete it and re-deploy a fresh VSM using the .ova file downloaded from Cisco's software site. When it boots up you'll just assign the correct domain ID, set it to be a "Secondary" VSM and it will sync up with your primary by itself.
Next you need your VEM hosts connected to the VSM in order for the upgrade to proceed. If you need assistance here let me know but this can require a discuss offline.
You found my biggest pet peeve with the SV1.1 -> SV1.2 upgrade. VUM is a steam roller and will upgrade all your hosts sequentially without giving DRS (or you) a time to migrate running VMs around. It's this reason why we advise to power down all VMs before starting a "VUM" upgrade. This is a limiation of VMware Update Manager (not Cisco 1000v) which is fixed in Update Manager U1 and later (DRS will kick in and move your VMs around for you).
To workaround downtime, you'll need to do a controlled manual upgrade. First, find the correct .vib file for your ESX version (point your browser to the VSM IP for a matrix table). If you don't see your ESX version listed you'll need to find it from the VMware website. Ping me the build # and I'll tell you which one you need.
Next copy the .vib file to each of your ESX hosts, and manually install the new .vib file (again this is important you do this while the VEM host is attached to the VSM, otherwise VUM will revert it back to the old version. Once all your hosts have been upgraded and verified ("vem version"), you can then safely "Accept" the upgrade in vCenter, and issue the "vmware vem upgrade complete" command.
Once all VEM hosts and VSMs are showing the new version, raise the FL.
Regards,
Robert
03-31-2010 08:10 AM
The Problem is that the 2nd VSM is at the moment the active and i`m a litttle bit afraid of making changes because of the productive system The primary VSM coming up but with no network connectivity and if i connect the NICs from the primary VSM i got the error in my previous post.
#show system redundancy status
Redundancy role
---------------
administrative: secondary
operational: secondary
Redundancy mode
---------------
administrative: HA
operational: None
This supervisor (sup-2)
-----------------------
Redundancy state: Active
Supervisor state: Active
Internal state: Active with no standby
Other supervisor (sup-1)
------------------------
Redundancy state: Not present
When you did the upgrade, did you copy the new image to the bootflash of each VSM?
I copied the image only to the primary VSM.If i make a "dir bootflash:" on the secondary the new images on:
# dir bootflash:
77824 Mar 30 14:32:00 2010 accounting.log
16384 Apr 26 20:04:31 2009 lost+found/
21340160 Apr 26 20:04:33 2009 nexus-1000v-kickstart-mz.4.0.4.SV1.1.bin
21408768 Mar 30 12:22:24 2010 nexus-1000v-kickstart-mz.4.0.4.SV1.2.bin
49869376 Apr 26 20:04:36 2009 nexus-1000v-mz.4.0.4.SV1.1.bin
73068811 Mar 30 12:22:11 2010 nexus-1000v-mz.4.0.4.SV1.2.bin
4096 Apr 26 20:05:01 2009 vdc_2/
4096 Apr 26 20:05:01 2009 vdc_3/
4096 Apr 26 20:05:01 2009 vdc_4/
Ping me the build # and I'll tell you which one you need.
if i look in the vSphere client the Build is 236512 but if i perform a "vem version" from the CLI i git this:
Running esx version -208167 x86_64
VEM Version: 4.0.4.1.1.30-1.9.16
VSM Version: 4.0(4)SV1(2)
System Version: VMware ESX 4.0.0 Releasebuild-208167
What do you think? Everything destroyed or is there some light at the end of the tunnel
04-01-2010 10:08 AM
You should still be able to reinstall the second VSM and then worry about the VEM's. ON the second VSM, the VEM's are registered correct so re-installing the other VSM should be fine, just be sure it's set to secondary.
04-01-2010 10:09 AM
And use the same version as what you are running on the primary VSM.
04-04-2010 05:33 AM
But as you can see:
#show system redundancy status
Redundancy role
---------------
administrative: secondary
operational: secondary
Redundancy mode
---------------
administrative: HA
operational: None
This supervisor (sup-2)
-----------------------
Redundancy state: Active
Supervisor state: Active
Internal state: Active with no standby
Other supervisor (sup-1)
------------------------
Redundancy state: Not present
at the moment it seems i have only one VSM and this is a secondary. If i now install a new secondary i have two secondary, or i`m wrong?
04-04-2010 06:09 AM
It's important that your 1000V has the control, packet and management VLAN's correctly defined. I am going to lab it up but I know if you don't have that set up properly on the VSM install, it won't be able to know that there is another VSM in place..
Stay tuned.
04-04-2010 06:30 AM
After i set up the nexus and it worked well I`ve migrated all ports, including packet, control and management, to the nexus 1000v. In my first post you can see how i have defined these ports.
04-04-2010 06:35 AM
Yes but when you reinstall the VSM, it created another VM in VC right? before you go through the installation, you need to have those VLAN's defined correctly or else it won't work. If you go to do the install and it doesn't prompt you as to what role it shoule be in, then something's not right.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide