cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
17544
Views
0
Helpful
35
Replies

VEM problem ? Dropping received frames

born.jason
Level 1
Level 1

Hi,

since december we have the nexus 1000v in our company. Everything working fine and looks good. We have a ESX cluster with two hosts and 7 NICs uplink. Today after a long time i have a look in the logging and saw this:

Dropping received frames from duplicate VSM - kernel

How could i get rid of this? This message flooding the logging.

Here is the configuration.....

sh ver:

Software
  loader:    version 1.2(2) [last: image booted through mgmt0]
  kickstart: version 4.0(4)SV1(1)
  system:    version 4.0(4)SV1(1)
  kickstart image file is:
  kickstart compile time:  4/2/2009 23:00:00
  system image file is:    bootflash:/nexus-1000v-mz.4.0.4.SV1.1.bin
  system compile time:     4/2/2009 23:00:00 [04/26/2009 13:27:23]

and here the uplink profiles and other profiles for service console and so on:

port-profile system-uplink
  capability uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 2,51-52 (VLAN 2 is not in use at the moment)
  channel-group auto mode on sub-group cdp
  no shutdown
  system vlan 51-52
  state enabled

port-profile vm-uplink
  capability uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1,10,101-200
  channel-group auto mode on sub-group cdp
  no shutdown
  state enabled

port-profile nexus-control
  vmware port-group
  switchport mode access
  switchport access vlan 51
  no shutdown
  system vlan 51
  state enabled
port-profile nexus-packet
  vmware port-group
  switchport mode access
  switchport access vlan 52
  no shutdown
  system vlan 52
  state enabled

port-profile ServiceConsole
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1-2 (maybe here ist the problem, this is configured from a guy before)
  no shutdown
  state enabled
port-profile VMotion
  vmware port-group
  switchport mode access
  switchport access vlan 10
  no shutdown
  state enabled

We have vlan 1 as the defaultvlan for office network, service console belongs to vlan 1 (Yes i know this is not good but at this time there is no other choice :-( )

On the uplink switch a port-channel is created with five ports for vm-uplink and another port-channel with two nics for system-uplink, which connected with the vmnic`s.

If i have to reconfigure some points please let me know if i can make this changes while the system is running.....

35 Replies 35

Robert Burns
Cisco Employee
Cisco Employee

Jason,

This message usually comes when there's multiple uplinks and the channel group is not correctly configured.  If your upstream switches are not sending CDP information, the port channels will not form.  In which case you can manually assign the sub-groups by entering into the configuration of your Ethernet Interfaces and assigning it.  **This instruction is for the current version your running.  In the present version 4.0.4.SV1.2 there's a different way to configure manual sub group IDs.  Note at version 4.0.4.SV1.1 the max # of sub-groups is 2.

Ex.

conf t

ethernet3/1

     sub-group-ID 0

ethernet3/2

     sub-group-id 1

1. Port Profile "ServiceConsole" should be an access port, and assigned to one access VLAN.  Make this VLAN a system vlan on both this Port Profile and the Uplink Port Profile (whichever you decide to use).

2. Ensure the CDP is working properly, and the Port Channels are forming correctly.  Paste the output of the following command:

"show port-channel summary"

"module vem 3 execute vemcmd show pc"

"show interface brief"

Thanks,

Robert

CDP is working fine, in the dvs i see the details of the vmnic`s and on which switchports they connected..... is the the service console-misconfiguration the problem with the dropping frames or has this nothing to do with this?

Here is the output you want:

show port-channel summary
--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
1     Po1(SD)     Eth      NONE      --
2     Po2(SU)     Eth      NONE      Eth3/2(P)    Eth3/6(P)
3     Po3(SU)     Eth      NONE      Eth3/3(P)    Eth3/4(P)    Eth3/5(P)
                                     Eth3/7(P)    Eth3/8(P)
4     Po4(SU)     Eth      NONE      Eth4/2(P)    Eth4/5(P)    Eth4/6(P)
                                     Eth4/7(P)    Eth4/8(P)
5     Po5(SU)     Eth      NONE      Eth4/3(P)    Eth4/4(P)

------------------------------------------------------------------------------------------------------------------------------------

module vem 3 execute vemcmd show pc
pce_ind    chan   pc_ltl   pce_in_pc   SG_ID   mbrs
-------    ----   ------   ---------   -----   ----
      0       3      303           0       0    21,22,19,18,17,
                                           1
      1       2      304           1       0    20,16,
                                           1

------------------------------------------------------------------------------------------------------------------------------------

sh interface brief

--------------------------------------------------------------------------------
Port     VRF          Status IP Address                     Speed    MTU
--------------------------------------------------------------------------------
mgmt0     --           up     192.168.113.12                 1000     1500

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth3/2        1      eth  trunk  up      none                       1000(D) 2
Eth3/3        1      eth  trunk  up      none                       1000(D) 3
Eth3/4        1      eth  trunk  up      none                       1000(D) 3
Eth3/5        1      eth  trunk  up      none                       1000(D) 3
Eth3/6        1      eth  trunk  up      none                       1000(D) 2
Eth3/7        1      eth  trunk  up      none                       1000(D) 3
Eth3/8        1      eth  trunk  up      none                       1000(D) 3
Eth4/2        1      eth  trunk  up      none                       1000(D) 4
Eth4/3        1      eth  trunk  up      none                       1000(D) 5
Eth4/4        1      eth  trunk  up      none                       1000(D) 5
Eth4/5        1      eth  trunk  up      none                       1000(D) 4
Eth4/6        1      eth  trunk  up      none                       1000(D) 4
Eth4/7        1      eth  trunk  up      none                       1000(D) 4
Eth4/8        1      eth  trunk  up      none                       1000(D) 4

--------------------------------------------------------------------------------
Port-channel VLAN  Type Mode   Status  Reason                    Speed  Protocol
Interface
--------------------------------------------------------------------------------
Po1          1     eth  access down    No operational members      auto(I)  none
Po2          1     eth  trunk  up      none                      a-1000(D)  none
Po3          1     eth  trunk  up      none                      a-1000(D)  none
Po4          1     eth  trunk  up      none                      a-1000(D)  none
Po5          1     eth  trunk  up      none                      a-1000(D)  none

--------------------------------------------------------------------------------
Interface     VLAN   Type Mode   Status  Reason                   MTU
--------------------------------------------------------------------------------
Veth1         10     virt access up      none                     1500
Veth2         1      virt access up      none                     1500
Veth3         1      virt access up      none                     1500
Veth4         1      virt access nonPcpt nonParticipating         1500
Veth5         1      virt access up      none                     1500
Veth6         1      virt access up      none                     1500
Veth7         1      virt access up      none                     1500
Veth8         1      virt access nonPcpt nonParticipating         1500
Veth9         1      virt access up      none                     1500
Veth10        1      virt access up      none                     1500
Veth11        150    virt access up      none                     1500
Veth12        1      virt access up      none                     1500
Veth13        102    virt access up      none                     1500
Veth14        1      virt access nonPcpt nonParticipating         1500
Veth15        1      virt access up      none                     1500
Veth16        1      virt trunk  up      none                     1500
Veth17        52     virt access up      none                     1500
Veth18        1      virt access up      none                     1500
Veth19        1      virt access up      none                     1500
Veth20        52     virt access up      none                     1500
Veth21        51     virt access up      none                     1500
Veth22        1      virt access up      none                     1500
Veth23        51     virt access up      none                     1500
Veth24        1      virt access nonPcpt nonParticipating         1500
Veth25        1      virt access nonPcpt nonParticipating         1500
Veth26        1      virt trunk  up      none                     1500
Veth27        1      virt access nonPcpt nonParticipating         1500
Veth28        150    virt access nonPcpt nonParticipating         1500
Veth29        10     virt access up      none                     1500
Veth30        1      virt access up      none                     1500
Veth31        1      virt access up      none                     1500
Veth32        1      virt access up      none                     1500
Veth33        1      virt access up      none                     1500
Veth34        1      virt access up      none                     1500
Veth35        1      virt access up      none                     1500
Veth36        101    virt access up      none                     1500
Veth37        1      virt trunk  up      none                     1500
Veth38        1      virt trunk  up      none                     1500
Veth39        1      virt access up      none                     1500
Veth40        1      virt access up      none                     1500
Veth41        1      virt access up      none                     1500
Veth42        150    virt access nonPcpt nonParticipating         1500
Veth43        150    virt access up      none                     1500
Veth44        150    virt access up      none                     1500
Veth45        1      virt access up      none                     1500
Veth46        1      virt access up      none                     1500
Veth47        1      virt access up      none                     1500
Veth48        1      virt access nonPcpt nonParticipating         1500
Veth49        1      virt access nonPcpt nonParticipating         1500
Veth50        1      virt access nonPcpt nonParticipating         1500
Veth51        103    virt access up      none                     1500
Veth52        1      virt access up      none                     1500
Veth53        1      virt access nonPcpt nonParticipating         1500
Veth54        1      virt access nonPcpt nonParticipating         1500
Veth55        1      virt access nonPcpt nonParticipating         1500
Veth56        1      virt access nonPcpt nonParticipating         1500
Veth57        104    virt access up      none                     1500
Veth58        1      virt access nonPcpt nonParticipating         1500
Veth59        1      virt access nonPcpt nonParticipating         1500
Veth60        104    virt access up      none                     1500
Veth61        1      virt access up      none                     1500
Veth62        1      virt access nonPcpt nonParticipating         1500
Veth63        1      virt access nonPcpt nonParticipating         1500
Veth64        1      virt access nonPcpt nonParticipating         1500
Veth65        1      virt access up      none                     1500

Jason,

Can you post the upstream port (physical switch) config.  If they're all the same, post one.

Thanks,

Robert

Robert,

hmmm the problem seems to be solved. I haven^t configured two ports from the upstream switch in a port-channel and these two ports are the system-uplink from nexus.

I have updated the nexus to v1.2 only raise the feature level is pending.

I have rebooted both nexus machines primary and secondary but if i now perform a show module i only see one VSM... where is my second?

And the second think are the VEMs.....they show an old version i think? Or do i first need to raise the feature level?

Here is a show module:

Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
2    0      Virtual Supervisor Module        Nexus1000V         active *
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok

Mod  Sw               Hw
---  ---------------  ------
2    4.0(4)SV1(2)     0.0
3    4.0(4)SV1(1)     1.11
4    4.0(4)SV1(1)     1.11

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
2    00-19-07-6c-5a-a8 to 00-19-07-6c-62-a8  NA
3    02-00-0c-00-03-00 to 02-00-0c-00-03-80  NA
4    02-00-0c-00-04-00 to 02-00-0c-00-04-80  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
2    192.168.100.12   NA                                    NA
3    192.168.100.151  34393433-3239-4742-3839-3330584b5830  192.168.100.151
4    192.168.100.152  34393433-3239-4742-3839-3330584b5842  192.168.113.152

Hmmm if i look in the vSphere Networking i can see "An upgrade for the Distributed Virtual Switch in datacenter is in progress." Is there something wrong with the update ? Need urgent help. :-(

On the esx host if i perform a vem status -v i see this:

Package vssnet-esx4.1.2-00000-release
Version 4.0.4.1.1.31-1.11.11
Build 11
Date Sat Jan 30 22:53:30 PST 2010

Number of PassThru NICs are 0
VEM modules are loaded

Switch Name    Num Ports   Used Ports  Configured Ports  MTU     Uplinks
vSwitch0       32          2           32                1500    vmnic0
DVS Name       Num Ports   Used Ports  Configured Ports  Uplinks
nexus1kv-primary256         27          256               vmnic3,vmnic1,vmnic5,vmnic6,vmnic4,vmnic2,vmnic7

Number of PassThru NICs are 0
VEM Agent (vemdpa) is running

i think this is the newest vem version, isin`t it?

esx host build is 236512

Do you nee some other output?

Jason,

Version 4.0.4.1.1.31-1.11.11 <== Still looks like you haven't completed the VEM upgrade.

Did you follow the steps in the upgrade guide?

In short

1. Upgrade primary & secondary VSM

2. If using VUM, powerdown all your VMs and then from the VSM initiate the Upgrade notice

3. From vCenter - Accept the Upgrade

4. From the VSM, proceed with the upgrade

5. Raise the feature level of the VSM

you can verify if your VEM's are upgrade by issueing  "vem version" from the CLI.

CDP sub-group support has been greatly improved in the new release which might explain your problem resolving.

Robert

Robert,

1. Upgrade primary & secondary VSM

Hmm this might be the Problem i`ve only updated one VSM, the guide says the other automaticaly update. But if i look both of the VSM shows the right image in bootflash but the formerly primary VSM has no connection to the vsphere center (svs). If i try to connect manually with "svs connection <name>" -> "connect"

This message appears:

ERROR: [VMWARE-VIM] Operation could not be completed due to connection failure. No route to host. connect failed in tcp_connect()

sh module shows only the VSM as active and no other modules are shown.

2. If using VUM, powerdown all your VMs and then from the VSM initiate the Upgrade notice

Yes, i used VUM but that was not a good idea. I cannot powerdown the VMs.

3. From vCenter - Accept the Upgrade

Yes, i have accepted the Upgrade

4. From the VSM, proceed with the upgrade

I procceed the upgrade but it ends with an error message -> "The Object or item referred to could not be found. Cannot get compliance status for host(s):host -795. This indicates that previous host operations might have failed for these hosts.

5. Raise the feature level of the VSM

not yet done.

How could i now update the VEM with no downtime of VMs and get back my secondary VSM. And in the networking TAB the "upgrade in progress" is still shown.

I figured out that i`m not able in vSphere connect the three NICs from the primary VSM. Error message "invalid configuration for device `0`."


Jason,

First you need to sort out your VSM upgrade.  You need them both upgrade and showing the correct new version.  Ensure the "show sys red sta" shows them as sync'd up before proceeding with the VEM upgrades. When you did the upgrade, did you copy the new image to the bootflash of each VSM?  If having issues I suggest you do the following.  Ensure your primary VSM is active and running correctly.  Power down and blow away your secondary VSM.  Delete it and re-deploy a fresh VSM using the .ova file downloaded from Cisco's software site.  When it boots up you'll just assign the correct domain ID, set it to be a "Secondary" VSM and it will sync up with your primary by itself.

Next you need your VEM hosts connected to the VSM in order for the upgrade to proceed.  If you need assistance here let me know but this can require a discuss offline.

You found my biggest pet peeve with the SV1.1 -> SV1.2 upgrade. VUM is a steam roller and will upgrade all your hosts sequentially without giving DRS (or you) a time to migrate running VMs around.  It's this reason why we advise to power down all VMs before starting a "VUM" upgrade. This is a limiation of VMware Update Manager (not Cisco 1000v) which is fixed in Update Manager U1 and later (DRS will kick in and move your VMs around for you).

To workaround downtime, you'll need to do a controlled manual upgrade.  First, find the correct .vib file for your ESX version (point your browser to the VSM IP for a matrix table).  If you don't see your ESX version listed you'll need to find it from the VMware website.  Ping me the build # and I'll tell you which one you need.

Next copy the .vib file to each of your ESX hosts, and manually install the new .vib file (again this is important you do this while the VEM host is attached to the VSM, otherwise VUM will revert it back to the old version.  Once all your hosts have been upgraded and verified ("vem version"), you can then safely "Accept" the upgrade in vCenter, and issue the "vmware vem upgrade complete" command.

Once all VEM hosts and VSMs are showing the new version, raise the FL.

Regards,

Robert

The Problem is that the 2nd VSM is at the moment the active and i`m a litttle bit afraid of making changes because of the productive system The primary VSM coming up but with no network connectivity and if i connect the NICs from the primary VSM i got the error in my previous post.

#show system redundancy status
Redundancy role
---------------
      administrative:   secondary
         operational:   secondary

Redundancy mode
---------------
      administrative:   HA
         operational:   None

This supervisor (sup-2)
-----------------------
    Redundancy state:   Active
    Supervisor state:   Active
      Internal state:   Active with no standby

Other supervisor (sup-1)
------------------------
    Redundancy state:   Not present

When you did the upgrade, did you copy the new image to the bootflash of each VSM?

I copied the image only to the primary VSM.If i make a "dir bootflash:" on the secondary the new images on:

# dir bootflash:
      77824     Mar 30 14:32:00 2010  accounting.log
      16384     Apr 26 20:04:31 2009  lost+found/
   21340160     Apr 26 20:04:33 2009  nexus-1000v-kickstart-mz.4.0.4.SV1.1.bin
   21408768     Mar 30 12:22:24 2010  nexus-1000v-kickstart-mz.4.0.4.SV1.2.bin
   49869376     Apr 26 20:04:36 2009  nexus-1000v-mz.4.0.4.SV1.1.bin
   73068811     Mar 30 12:22:11 2010  nexus-1000v-mz.4.0.4.SV1.2.bin
       4096     Apr 26 20:05:01 2009  vdc_2/
       4096     Apr 26 20:05:01 2009  vdc_3/
       4096     Apr 26 20:05:01 2009  vdc_4/

Ping me the build # and I'll tell you which one you need.

if i look in the vSphere client the Build is 236512 but if i perform a "vem version" from the CLI i git this:

Running esx version -208167 x86_64
VEM Version: 4.0.4.1.1.30-1.9.16
VSM Version: 4.0(4)SV1(2)
System Version: VMware ESX 4.0.0 Releasebuild-208167

What do you think? Everything destroyed or is there some light at the end of the tunnel

You should still be able to reinstall the second VSM and then worry about the VEM's. ON the second VSM, the VEM's are registered correct so re-installing the other VSM should be fine, just be sure it's set to secondary.

And use the same version as what you are running on the primary VSM.

But as you can see:

#show system redundancy status
Redundancy role
---------------
       administrative:   secondary
         operational:   secondary

Redundancy  mode
---------------
      administrative:   HA
          operational:   None

This supervisor (sup-2)
-----------------------
     Redundancy state:   Active
    Supervisor state:   Active
       Internal state:   Active with no standby

Other supervisor (sup-1)
------------------------
     Redundancy state:   Not present

at the moment it seems i have only one VSM and this is a secondary. If i now install a new secondary i have two secondary, or i`m wrong?

It's important that your 1000V has the control, packet and management VLAN's correctly defined. I am going to lab it up but I know if you don't have that set up properly on the VSM install, it won't be able to know that there is another VSM in place..

Stay tuned.

After i set up the nexus and it worked well I`ve migrated all ports, including packet, control and management, to the nexus 1000v. In my first post you can see how i have defined these ports.

Yes but when you reinstall the VSM, it created another VM in VC right? before you go through the installation, you need to have those VLAN's defined correctly or else it won't work. If you go to do the install and it doesn't prompt you as to what role it shoule be in, then something's not right.

Review Cisco Networking for a $25 gift card