Re: VSUM and N1Kv for vSphere 6.0.0 Issue

alex_aasen · ‎11-09-2015

Hi all,

I just got a hold of the newest version of VSUM 1.5.5 to our vSphere 6.0.0 environment. This all seemed great until I tried adding a physical host to the DVS. The issue experienced is the following:

vSphere Web Client Home menue --> Cisco Virtual Switch Update Manager --> Nexus1000v --> Configure --> "Choose DC" --> "Choose DVS" --> Manage --> Add host --> "Try and add a host". And when I try accessing either choice, the "An internal error has happened". So the physical hosts cannot be joined to the N1Kv DVS from here.

Trying another approach to add hosts by "right clicking" the N1Kv SW under the Networking Tab gets me a bit further.

Network Tab --> Right click N1Kv DVS --> Add and Manage Hosts... --> Add host --> "Select host" --> Manage physical adapters --> Assign Uplink and choose the manually generated Ethernet Uplink group made in the VSM --> OK --> Then the error message "The number of physical network adapters per host exceeds the number of uplink ports" appears

Which does not make sense as the uplink port created has a upper limit of like 512 uplink ports.

The port profile has a very basic profile:

port-profile type ethernet UPLINK

switchport mode trunk

switchport trunk allowed vlan all

no sh

system enable

vmware port-group

system vlan 100

I tried installing different N1KV versions, but with no luck, got the same outcome.

According to the VSUM documentation this should have been really straight forward.

Any idea on what might have gone wrong?

rmxyz · ‎11-20-2015

We are experiencing this issue too. Brand new build of vCenter 6.0u1 (using vCSA), ESXi 6, and Cisco 1000v 5.2(1)SV3(1.5b). Standard VMware distributed switch works perfectly. No solution thus far.

faridsyusifov · ‎11-24-2015

Hello. The same issue. Do you find solution?

Here is config:

port-profile type ethernet UPLINKS

switchport mode trunk

switchport trunk allowed vlan 38,45,86,111-113,886

no shutdown

system vlan 38,45,111-113

state enabled

vmware port-group

Error message when adding physical network adapters: "the number of physical network adapters per host exceeds the number of uplink ports"

Output of "vem status -v"

*******************************************************

Package vssnet-esxesx2015-release

Version 5.2.1.3.1.5a.0-6.0.1

Build 1

Date Sat Aug 15 07:59:51 PDT 2015

VEM modules are loaded

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

Distrib-SW1 1792 9 512 1500 vmnic0

VEM Agent (vemdpa) is running

********************************************************

Tried to add using software client. Success. But I cannot see my VEM module in "sh module" output, plus no network connectivity for VMs.

Here is output after using software client:

********************************************************

Package vssnet-esxesx2015-release

Version 5.2.1.3.1.5a.0-6.0.1

Build 1

Date Sat Aug 15 07:59:51 PDT 2015

VEM modules are loaded

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

N1K-PRI 1024 12 1024 1500

DVS Name Num Ports Used Ports Configured Ports MTU Uplinks

Distrib-SW1 1792 7 512 1500 vmnic0

VEM Agent (vemdpa) is running

********************************************************

sh modules:

Mod Ports Module-Type Model Status

--- ----- -------------------------------- ------------------ ------------

1 0 Virtual Supervisor Module Nexus1000V active *

2 0 Virtual Supervisor Module Nexus1000V ha-standby

Mod Sw Hw

--- ------------------ ------------------------------------------------

1 5.2(1)SV3(1.5a) 0.0

2 5.2(1)SV3(1.5a) 0.0

Mod Server-IP Server-UUID Server-Name

--- --------------- ------------------------------------ --------------------

1 192.168.38.18 NA NA

2 192.168.38.18 NA NA

* this terminal session

Robert Burns · ‎11-25-2015

[Posting on behalf of colleague John Hibbs]:

We need the logs from the N1K Manager (VSUM) to further diagnose.

1. SSH into Nexus 1000v manager. Default username is root and password is cisco.

2. Once logged in, go to /etc/cisco/app_install/ and run ‘./bundleLogs.sh’

Copy off the generated log bundle using scp and attach to this thread.

Thanks,

Robert

niraj.kacha · ‎11-26-2015

Hi Robert,

I work with Robin Malik, who mentioned we are experiencing the same issue as describe in the original post.

I have generated the logs and attached.

Niraj

Robert Burns · ‎11-26-2015

Niraj,

Does your 1000v contain any PVLAN config by chance?

Robert

niraj.kacha · ‎11-26-2015

Hi Robert,

No private VLAN config, we don't use them in our environment.

Niraj

Robert Burns · ‎12-04-2015

Found an error in your Ajax logs:

com.cisco.n1kv.exception.PortProfileCreationTimeOut

at com.cisco.n1kv.migrate.MigrateHelper.waitForPortProfileCreation(MigrateHelper.java:1871)

at com.cisco.n1kv.migrate.MigrateHelper.migrateHosts(MigrateHelper.java:1757)

This typically points to a mis-match between the config on the N1K and VC. Can you provide the following outputs:

‘Show port-profile’ command output from VSM
Need a screenshot from vCenter > Home > Inventory > Networking > Select VSM(N1kv) > Networks.

Thanks,

Robert

niraj.kacha · ‎12-04-2015

Hi Robert,

Output from show port-profile

cisco-nexus-1000v-poc# show port-profile

port-profile system-uplink

type: Ethernet

description: system-uplink

status: enabled

max-ports: 512

min-ports: 1

inherit:

config attributes:

switchport mode trunk

switchport trunk native vlan 1034

switchport trunk allowed vlan 1031-1034,1099,1708

channel-group auto mode on mac-pinning

no shutdown

evaluated config attributes:

switchport mode trunk

switchport trunk native vlan 1034

switchport trunk allowed vlan 1031-1034,1099,1708

channel-group auto mode on mac-pinning

no shutdown

assigned interfaces:

port-channel1

port-group: system-uplink

system vlans: 1031-1033

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile Unused_Or_Quarantine_Uplink

type: Ethernet

description: Port-group created for Nexus 1000V internal usage. Do not use.

status: enabled

max-ports: 512

min-ports: 1

inherit:

config attributes:

shutdown

evaluated config attributes:

assigned interfaces:

port-group: Unused_Or_Quarantine_Uplink

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile Unused_Or_Quarantine_Veth

type: Vethernet

description: Port-group created for Nexus 1000V internal usage. Do not use.

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

shutdown

evaluated config attributes:

assigned interfaces:

port-group: Unused_Or_Quarantine_Veth

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vc03-poc-exchange

type: Vethernet

description:

status: enabled

max-ports: 16

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 1099

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 1099

no shutdown

assigned interfaces:

port-group: POC-Exchange

system vlans: 1099

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vc03-poc-server-1

type: Vethernet

description: POC-Server-1

status: enabled

max-ports: 128

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 1034

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 1034

no shutdown

assigned interfaces:

port-group: POC-Servers-1

system vlans: 1034

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vc03-poc-server-2

type: Vethernet

description:

status: enabled

max-ports: 256

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 1708

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 1708

no shutdown

assigned interfaces:

port-group: POC-Servers-2

system vlans: 1708

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vc03-poc-storage

type: Vethernet

description: Development storage

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 1033

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 1033

no shutdown

assigned interfaces:

port-group: POC-Storage

system vlans: 1033

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vc03-poc-vMotion

type: Vethernet

description: Development vMotion

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 1032

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 1032

no shutdown

assigned interfaces:

port-group: POC-vMotion

system vlans: 1032

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vc03-poc-vmsc

type: Vethernet

description: Development ESXi Management

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 1031

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 1031

no shutdown

assigned interfaces:

port-group: POC-VMSC

system vlans: 1031

capability l3control: yes

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

Need a screenshot from vCenter > Home > Inventory > Networking > Select VSM(N1kv) > Networks.

I cannot find this path within vSphere, I have uploaded a couple of screenshots in the hope they hold the information you require. We are using vSphere 6 web interface if that helps.

faridsyusifov · ‎12-04-2015

Hello, Robert.

Here is output:

sh port-profile

port-profile COMMON-86

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 86

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 86

no shutdown

assigned interfaces:

port-group: COMMON-86

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile CONTROL

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 111

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 111

no shutdown

assigned interfaces:

port-group: CONTROL

system vlans: 111

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile MNGMT

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 38

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 38

no shutdown

assigned interfaces:

port-group: MNGMT

system vlans: 38

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile PACKET

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 112

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 112

no shutdown

assigned interfaces:

port-group: PACKET

system vlans: 112

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile SECURE-886

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 886

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 886

no shutdown

assigned interfaces:

port-group: SECURE-886

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile Unused_Or_Quarantine_Uplink

type: Ethernet

description: Port-group created for Nexus 1000V internal usage. Do not use.

status: enabled

max-ports: 512

min-ports: 1

inherit:

config attributes:

shutdown

evaluated config attributes:

assigned interfaces:

port-group: Unused_Or_Quarantine_Uplink

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile Unused_Or_Quarantine_Veth

type: Vethernet

description: Port-group created for Nexus 1000V internal usage. Do not use.

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

shutdown

evaluated config attributes:

assigned interfaces:

port-group: Unused_Or_Quarantine_Veth

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile UPLINKS

type: Ethernet

description:

status: enabled

max-ports: 512

min-ports: 1

inherit:

config attributes:

switchport mode trunk

switchport trunk allowed vlan 38,45,86,111-113,886

no shutdown

evaluated config attributes:

switchport mode trunk

switchport trunk allowed vlan 38,45,86,111-113,886

no shutdown

assigned interfaces:

port-group: UPLINKS

system vlans: 38,45,111-113

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

port-profile vMotion

type: Vethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode access

switchport access vlan 45

no shutdown

evaluated config attributes:

switchport mode access

switchport access vlan 45

no shutdown

assigned interfaces:

port-group: vMotion

system vlans: none

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vservice: no

port-profile role: none

port-binding: static

*************************************************************

And one more thing. It is succesful, when you adding vmnic using software client. But no network connectivity. VMs are cut out from outside network. And againg no VEM module in sh modules output.

Thanks.

rmxyz · ‎12-07-2015

I'm not sure if this apparent UI bug is the Cisco plugin or something to do with VMware. Either way, I was able to circumvent the vSphere Web Client error using PowerCLI.

The code is below. It assumes the following:

Your host is imported to vCenter.
Your host has have 2 physical NICs.
vmnic0 is being used by the standard vSwitch for current management traffic.
You have VUM installed (if you do not, you'll need the VEM installed).
You are using PowerCLI 6.0 Release 2 build 3056836 (older versions may work).

This script will:

Add the host to the 1000v with vmnic1, which invokes VUM to install the VEM (if you have the VEM already installed the addition to the 1000v will be much quicker).
Migrate the management network from vSwitch on vmnic0, to 1000v with vmnic1
Remove vmnic0 from vSwitch
Add vmnic0 to 1000v (must be done by "adding" both vmnic0 and 1, even though vmnic1 is already added).

Hope this helps someone.

Please don't start pasting errors with this code into this thread. If you want, create a new thread and link to it.

$verbosepreference = 'continue'

$DebugPreference = 'continue'

$vcenterServer = 'myvcenter'

$esxihost = 'myhost.domain.com'

$shush = Connect-VIServer -Server $vcenterServer -Credential (Get-Credential) -Protocol Https -ErrorAction Stop -WarningAction SilentlyContinue

$esxiHostObj = Get-VMHost -Name $esxihost -ErrorAction Stop

# Define your 1000v name:

$ciscoVDSName = 'my1000v'

$ciscoVDSObj = Get-VDSwitch $ciscoVDSName

# Specify the 1000v uplink

$uplinkPortGroupName = 'system-uplink'

# Define management port group (port profile)

$vmkpg = 'POC-VMSC'

# Add vmnic1 to 1000v with system-uplink:

$vmnic = 'vmnic1'

$vmHost = Get-VMHost -Name $esxihost -Erroraction Stop | Get-View

$spec = New-Object -TypeName VMware.Vim.DVSConfigSpec

$targetHost = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberConfigSpec

$targetHost.operation = 'add'

$targetHost.backing = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicBacking

$pnic = $vmHost.Config.Network.Pnic | Where-Object -FilterScript {

$_.Device -eq $vmnic

}

$targetHost.Backing.PnicSpec = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

$targetHost.Backing.PnicSpec[0].pnicDevice = $pnic.Device

$uplinkObj = Get-VDPortgroup -Name $uplinkPortGroupName

$targetHost.Backing.PnicSpec[0].UplinkPortGroupKey = $uplinkObj.Key

$targetHost.host = $vmHost.MoRef

$spec.Host = $targetHost

$dvSwitch = Get-View -Id $ciscoVDSObj.ExtensionData.MoRef

$dvSwitch.UpdateViewData()

$spec.ConfigVersion = $dvSwitch.Config.ConfigVersion

$taskMoRef = $dvSwitch.ReconfigureDvs_Task($spec)

$taskID = 'Task-' + $taskMoRef.Value

while((Get-Task -Id $taskID).PercentComplete -lt "100")

{

$percentComplete = (Get-Task -Id $taskID).PercentComplete

Write-Verbose "Percent Complete: $percentComplete"

Start-Sleep -Seconds 3

}

# Move vmk management network to 1000v on vmnic1

$vNicManagement = Get-VMHostNetworkAdapter -VMHost $esxihost -Name vmk0

$vdPortgroupManagement = Get-VDPortgroup -VDSwitch $ciscoVDSName -Name $vmkpg

$pnicToUse = Get-VMHostNetworkAdapter -VMHost $esxihost -Physical | Where-Object { $_.Name -eq $vmnic }

Add-VDSwitchPhysicalNetworkAdapter -DistributedSwitch $ciscoVDSName -VMHostPhysicalNic $pnicToUse -VMHostVirtualNic $vNicManagement -VirtualNicPortGroup $vdPortGroupManagement -ErrorAction Stop -Confirm:$false

# Remove vmnic0 from vswitch

$esxihostobj | Get-VMHostNetworkAdapter -Physical -Name "vmnic0" | Remove-VirtualSwitchPhysicalNetworkAdapter -Confirm:$false

# add vmnic0 to 1000v

$config = New-Object -TypeName VMware.Vim.HostNetworkConfig

$config.proxySwitch = New-Object -TypeName VMware.Vim.HostProxySwitchConfig[] -ArgumentList (1)

$config.proxySwitch[0] = New-Object -TypeName VMware.Vim.HostProxySwitchConfig

$config.proxySwitch[0].changeOperation = 'edit'

$config.proxySwitch[0].uuid = $ciscoVDSObj.key

$config.proxySwitch[0].spec = New-Object -TypeName VMware.Vim.HostProxySwitchSpec

$config.proxySwitch[0].spec.backing = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicBacking

$config.proxySwitch[0].spec.backing.pnicSpec = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec[] -ArgumentList (2)

$config.proxySwitch[0].spec.backing.pnicSpec[0] = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

$config.proxySwitch[0].spec.backing.pnicSpec[0].pnicDevice = 'vmnic0'

$config.proxySwitch[0].spec.backing.pnicSpec[0].uplinkPortgroupKey = $uplinkObj.key

$config.proxySwitch[0].spec.backing.pnicSpec[1] = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

$config.proxySwitch[0].spec.backing.pnicSpec[1].pnicDevice = 'vmnic1'

$config.proxySwitch[0].spec.backing.pnicSpec[1].uplinkPortgroupKey = $uplinkObj.key

$vmhostRef = ($vmhost.MoRef.value).split('-')[1]

$_this = Get-View -Id "HostNetworkSystem-networkSystem-$vmhostRef"

$_this.UpdateNetworkConfig($config, 'modify')

faridsyusifov · ‎12-08-2015

Hello, Robin

Thank you very much for your post. I tried to add vmnics to N1K switch using PowerCLI, but unsuccessful.

romar · ‎02-07-2017

I am facing the issue in the thread and another stubborn one related to it. Cannot add the host to the NX1000v VDS - be it by WebClient, vSphere client or PowerCLI. Funny enough I am getting exactly same error when adding the host via PowerCLI or adding it via the vSphere "thick" client. And the error info points to an internal ESXi host SOAP call failure.

I am using latest and greatest vSphere 6.0.0.update02-3620759 patched with latest and greatest ESXi600-201611001.zip from VmWare, vSphere client downloaded from the patched ESXi host, Using Nexus1000v.5.2.1.SV3.2.8 and latest & greatest vCenter Server Appliance VMware-VCSA-all-6.0.0-3634788. I am starting "clean" each test i.e. in NX1000v I am doing in "svc connection <test-svc-conn>" "no vmware dvs datacenter-name <test-dc> ; no connect" then I recreate the dvs clean-slate.

Here is the error

vDS operation failed on host 192.168.198.98, Received SOAP response fault from [<cs p:00007f1425ae9990, TCP:192.168.198.98:443>]: invokeHostTransactionCall

Received SOAP response fault from [<cs p:1f37cb58, TCP:localhost:8307>]: invokeHostTransactionCall

An error occurred during host configuration. got (vim.fault.PlatformConfigFault) exception

I recall when I have upgraded from 5.1 to 5.5 upgraded using the latest & greatest 5.5 vmVisor image from vMware - and had the exact same issue with Nexus 1000v. Finally after many hours of shooting in the dark I have downgraded the 5.5. vmVisor a notch and the error disappeared. I am for now in the dark with this error and what is causing it.

Thanks in advance for any suggestion

p.s. the script I believe needs some adjustments - I have modified it a bit ( I am just doing "add" and for 2 vnics) but regardless some initializations might be required e..g. in my slightly modified version

###$targetHost.Backing.PnicSpec = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

$targetHost.Backing.PnicSpec = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec[] (2)

$targetHost.Backing.PnicSpec[0] = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

$targetHost.Backing.PnicSpec[1] = New-Object -TypeName VMware.Vim.DistributedVirtualSwitchHostMemberPnicSpec

romar · ‎02-07-2017

Did more testing and here is what I have found. The root cause of the issue seems to be the fact I have done an "upgrade" from 5.5 to 6.0 - it is not the vmVisor version. Dit a fresh install of 6.0 U2 (w/ preserve datastore) and LOL adding the host worked just fine (from the "thick" vSphere client). Now I recall that during the 5.1 => 5.5 upgrade I have also initially done an "upgrade". Then I have reverted to the 5.5. GA "standard" version but did a full install (w/ preserve datastore). Lesson learned - stay away from the vmVisor "upgrade" function - always do a fresh install to upgrade. At least one issue solved - unfortunately still not able to add the host from the web client getting same nonsensical error "The number of physical network adapters per host exceeds the number of uplink ports"

faridsyusifov · ‎11-26-2015

Hello. But I do not have N1K Manager (VSUM) installed in my production.