Re: Nexus1000V VEM ports are blocking

JonasNobs · ‎05-02-2013

Hi folks,

we do have a Nexus 1000V installation running and functional.

In order to do a proof of concept we started several worst case scenarios and are shutting down the VSM and ESX.

After fully shutting down everything in the order VSM-secondry, VSM-primary, the second ESX host and at last the first ESX host.

The next step is to start the systems and test everything for functionality or problems. Now we are facing the issue, that all ports of VEMs are blocked.

~ # vemcmd show port

LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type

18 UP UP F/B* 0 vmnic1

49 UP UP FWD 0 vmk0

50 DOWN UP BLK 0 centos-vm2.eth0

51 DOWN UP BLK 0 nexus-g4-vsm-2.eth2

52 DOWN UP BLK 0 nexus-g4-vsm-2.eth1

53 DOWN UP BLK 0 nexus-g4-vsm-2.eth0

* F/B: Port is BLOCKED on some of the vlans.

One or more vlans are either not created or

not in the list of allowed vlans for this port.

Please run "vemcmd show port vlans" to see the details.

~ # vemcmd show port vlans

Native VLAN Allowed

LTL VSM Port Mode VLAN State* Vlans

18 T 1 FWD 187,191

49 A 187 FWD 187

50 A 1 BLK 1

51 A 1 BLK 1

52 A 1 BLK 1

53 A 1 BLK 1

I am now not sure, is this normal behavior? I thought the configuration of VEMs should always be saved and in case of losing the VSM just new configuration of vethernet ports or port-profiles is not possible.

The following is the configuration of our N1Kv:

nexus-g4# show running-config

!Command: show running-config

!Time: Wed Apr 17 08:27:49 2013

version 4.2(1)SV2(1.1a)

svs switch edition essential

no feature telnet

feature netflow

username admin password 5 $1$vaQFlRGe$WmNylWhhNA6/B0/BlZ2Qe. role network-admin

banner motd #Nexus 1000v Switch#

ssh key rsa 2048

ip domain-lookup

ip host nexus-g4 10.10.10.50

hostname nexus-g4

errdisable recovery cause failed-port-state

ip access-list snmp-ro

10 permit ip 10.10.10.0/24 any

vem 3

host vmware id 34333535-3533-435a-4a37-323730325332

vem 4

host vmware id 34333535-3533-435a-4a37-323730325334

snmp-server source-interface inform mgmt0

snmp-server user admin network-admin auth md5 0x9c794b86442143780e0d0fef86f5e7a0 priv 0x9c794b864

42143780e0d0fef86f5e7a0 localizedkey

snmp-server community public group network-operator

snmp-server community public use-acl snmp-ro

vrf context management

ip route 0.0.0.0/0 10.10.10.1

vlan 1,181,187,191

vlan 181

name Clients

vlan 187

name Mgmt

vlan 191

name Control

port-channel load-balance ethernet source-virtual-port-id

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

vmware port-group

shutdown

description Port-group created for Nexus1000V internal usage. Do not use.

state enabled

port-profile type ethernet Uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 181,187,191

no shutdown

system vlan 187,191

state enabled

port-profile type vethernet VMkernel

capability l3control

vmware port-group

switchport mode access

switchport access vlan 187

no shutdown

system vlan 187

state enabled

port-profile type vethernet VLAN181-Clients

vmware port-group

switchport mode access

switchport access vlan 181

no shutdown

state enabled

port-profile type vethernet Control

capability l3control

vmware port-group

switchport mode access

switchport access vlan 191

no shutdown

system vlan 191

state enabled

system storage-loss log time 30

vdc nexus-g4 id 1

limit-resource vlan minimum 16 maximum 2049

limit-resource monitor-session minimum 0 maximum 2

limit-resource vrf minimum 16 maximum 8192

limit-resource port-channel minimum 0 maximum 768

limit-resource u4route-mem minimum 1 maximum 1

limit-resource u6route-mem minimum 1 maximum 1

interface mgmt0

ip address 10.10.10.50/24

interface Vethernet1

inherit port-profile VMkernel

description VMware VMkernel, vmk0

vmware dvport 32 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 001B.7830.52B4

interface Vethernet2

inherit port-profile VMkernel

description nexus-g4-vsm-1, Network Adapter 2

vmware dvport 35 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.3AB1

interface Vethernet3

inherit port-profile Control

description nexus-g4-vsm-1, Network Adapter 1

vmware dvport 160 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.7941

interface Vethernet4

inherit port-profile Control

description nexus-g4-vsm-1, Network Adapter 3

vmware dvport 162 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.7391

interface Vethernet5

inherit port-profile VLAN181-Clients

description centos-vm2, Network Adapter 1

vmware dvport 64 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.6A8C

interface Vethernet6

inherit port-profile VLAN181-Clients

description centos-vm3, Network Adapter 1

vmware dvport 65 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.041A

interface Vethernet7

inherit port-profile Control

description nexus-g4-vsm-2, Network Adapter 3

vmware dvport 163 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.0027

interface Vethernet8

inherit port-profile VMkernel

description nexus-g4-vsm-2, Network Adapter 2

vmware dvport 34 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.6DF9

interface Vethernet9

inherit port-profile Control

description nexus-g4-vsm-2, Network Adapter 1

vmware dvport 161 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 0050.5687.4380

interface Vethernet10

inherit port-profile VMkernel

description VMware VMkernel, vmk0

vmware dvport 33 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

vmware vm mac 001B.7830.128E

interface Ethernet3/2

inherit port-profile Uplink

interface Ethernet4/2

inherit port-profile Uplink

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-1

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-2

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-2

svs-domain

domain id 1

control vlan 1

packet vlan 1

svs mode L3 interface mgmt0

svs connection vcenter

protocol vmware-vim

remote ip address 10.10.10.98 port 80

vmware dvs uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d" datacenter-name test-center

admin user n1kUser

max-ports 8192

connect

vservice global type vsg

tcp state-checks invalid-ack

tcp state-checks seq-past-window

no tcp state-checks window-variation

no bypass asa-traffic

vnm-policy-agent

registration-ip 0.0.0.0

shared-secret **********

log-level

Robert Burns · ‎05-02-2013

The VEMs will retain their "last known" configuration pulled from a VSM until it's reboot. This is expected behavior. If you shut down both VSMs, the VEM will continue to operate fine, but if you then reboot the VEM, when it comes up, all interfaces that were not pre-configured as "system vlans" will stay down until the VEM inserts as a module to the VSM. This is a security mechanism - VEMs must check in with the VSM for any configuration changes before bringing any ports up.

It's a very unlikely scenario where you would have to reboot a VEM while both active & secondary VSMs are unreachable. We recommend configuring a DRS rule to separate the primary & secondary VSMs so they never reside on the same host improving redundancy.

Regards,

Robert

JonasNobs · ‎05-03-2013

Hi Robert,

thank you for your quick respond.

What you mean by 'all interfaces that were not pre-configured as "system-vlans"'?

The VSM's interfaces are configured as system-vlans as well, but are blocking after the shutdown of the ESX hosts.

In order to get VSM's connected to each of another and to the VEM's you have to manually put the ports into the specific VLAN's at the VEM's (at ESX console).

Is there probably any TechDoc discribing this behavior?

Thank in advance

Jonas

Robert Burns · ‎05-03-2013

While the system is in this state please provide the following output from the VSM:

"show svs connection"

"show int brief"

Robert

JonasNobs · ‎05-13-2013

Hi Robert,

finally I got the outputs;

nexus-g4# show svs connections

connection vcenter:

ip address: 10.10.10.98

remote port: 80

protocol: vmware-vim https

certificate: default

datacenter name: test-center

admin: n1kUser(user)

max-ports: 8192

DVS uuid: 8f 83 07 50 e3 81 90 81-a3 f0 x7 82 42 4b a2 1d

config status: Enabled

operational status: Disconnected

sync status: -

version: -

vc-uuid: -

nexus-g4#

nexus-g4# show interface brief

------------------------------------------------------------------------------------------------------

Port VRF Status IP Address Speed

------------------------------------------------------------------------------------------------------

mgmt0 -- up 10.10.10.50 1000

------------------------------------------------------------------------------------------------------

Vethernet VLAN Type Mode Status Reason Speed

------------------------------------------------------------------------------------------------------

Veth1 187 virt access down nonParticipating auto

Veth2 187 virt access down nonParticipating auto

Veth3 191 virt access down nonParticipating auto

Veth4 191 virt access down nonParticipating auto

Veth5 181 virt access down nonParticipating auto

Veth6 181 virt access down nonParticipating auto

Veth7 191 virt access down nonParticipating auto

Veth8 187 virt access down nonParticipating auto

Veth9 191 virt access down nonParticipating auto

Veth10 187 virt access down nonParticipating auto

Veth11 187 virt access down nonParticipating auto

------------------------------------------------------------------------------------------------------

Port VRF Status IP Address Speed

------------------------------------------------------------------------------------------------------

control0 -- up -- 1000

nexus-g4#

Jonas

Artem Gromov · ‎03-20-2014

Hi everybody,
I had exactly the same problem.
VSM vnics are in port-group with system vlan configured, but they were blocked after ESX reload.

I have solved that by moving pnic and primary VSM back to vswitch to restore connectivity between VSM and VEM and to bring up ports. And then went back to N1Kv switch.

Is there any easier way to recover the issue after shutdown by this scenario?

And I'd like to join the question about any TechDoc discribing this behavior.

JonasNobs,
could you please tell how did you do this at ESX console:

>> In order to get VSM's connected to each of another and to the VEM's you have to manually put the >> ports into the specific VLAN's at the VEM's (at ESX console).

Artem Gromov · ‎03-20-2014

I have found what caused the issue.

I was using the following port-profile for ESX vmk0, vcenter veth and vsm veths:

port-profile type vethernet system
capability l3control
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled

I found the following log message:

%VEM_MGR-SLOT11-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[50331670] DVPortGroup[dvportgroup-25]. L3 Control can be applied only on VMKernel port.

I created dedicated port-profile without capability l3control specially for vcenter and vsm veths (vsm already knows that it should use mgmt0 in L3 mode because that information is in svs domain configuration):

port-profile type vethernet 100-agromov
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled

After that everything was fine. No required veth ports was blocked.

So, command capability l3control at vm's veth (vcenter vm and vsm vm) makes veth port blocked after ESX + both vsm shutdown even if it was system vlan configured.

My working config:

N1K-1# sh run

!Command: show running-config
!Time: Wed Mar 19 14:36:50 2014

version 4.2(1)SV2(2.2)
svs switch edition advanced

no feature telnet

banner motd #Nexus 1000v Switch#

ip domain-lookup
ip host N1K-1 10.1.1.199
hostname N1K-1
errdisable recovery cause failed-port-state
vem 11
host id eabd8850-baa2-e311-af00-000000000001
vem 12
host id eabd8850-baa2-e311-af00-000000000002
vem 13
host id eabd8850-baa2-e311-af00-000000000003
vem 14
host id eabd8850-baa2-e311-af00-000000000004
localizedkey

vrf context management
ip route 0.0.0.0/0 10.1.1.1
vlan 1,100-101
vlan 100
name agromov
vlan 101
name agromov-vm1

port-channel load-balance ethernet source-mac
port-profile default max-ports 32
port-profile type ethernet Unused_Or_Quarantine_Uplink
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type vethernet 100-agromov
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled
port-profile type vethernet 101-agromov-vm1
vmware port-group
switchport mode access
switchport access vlan 101
no shutdown
state enabled
port-profile type ethernet management
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 100
no shutdown
system vlan 100
state enabled
port-profile type ethernet data
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 101
no shutdown
state enabled
port-profile type vethernet system
capability l3control
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled

system storage-loss log time 30
vdc N1K-1 id 1
limit-resource vlan minimum 16 maximum 2049
limit-resource monitor-session minimum 0 maximum 2
limit-resource vrf minimum 16 maximum 8192
limit-resource port-channel minimum 0 maximum 768
limit-resource u4route-mem minimum 1 maximum 1
limit-resource u6route-mem minimum 1 maximum 1

interface mgmt0
ip address 10.1.1.199/24

interface Vethernet1
inherit port-profile system
description VMware VMkernel, vmk0
vmware dvport 227 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0025.B5AF.0009

interface Vethernet2
inherit port-profile system
description VMware VMkernel, vmk0
vmware dvport 225 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0025.B5AF.0003

interface Vethernet3
inherit port-profile system
description VMware VMkernel, vmk0
vmware dvport 226 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0025.B5AF.0006

interface Vethernet4
inherit port-profile 100-agromov
description vsm2, Network Adapter 1
vmware dvport 35 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0050.5690.CEF9

interface Vethernet5
inherit port-profile 100-agromov
description vcenter, Network Adapter 1
vmware dvport 36 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 000C.29ED.B29E

interface Vethernet6
inherit port-profile 100-agromov
description vsm2, Network Adapter 2
vmware dvport 37 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0050.5690.4E09

interface Vethernet7
inherit port-profile 100-agromov
description vsm2, Network Adapter 3
vmware dvport 38 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0050.5690.2936

interface Vethernet8
inherit port-profile 100-agromov
description vsm1, Network Adapter 1
vmware dvport 39 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0050.5690.BD52

interface Vethernet9
inherit port-profile 100-agromov
description vsm1, Network Adapter 2
vmware dvport 32 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0050.5690.C461

interface Vethernet10
inherit port-profile system
description VMware VMkernel, vmk0
vmware dvport 224 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0025.B5AF.0000

interface Vethernet11
inherit port-profile 100-agromov
description vsm1, Network Adapter 3
vmware dvport 41 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
vmware vm mac 0050.5690.58D6

interface Ethernet11/1
inherit port-profile data

interface Ethernet11/2
inherit port-profile data

interface Ethernet11/3
inherit port-profile management

interface Ethernet12/1
inherit port-profile data

interface Ethernet12/2
inherit port-profile data

interface Ethernet12/3
inherit port-profile management

interface Ethernet13/1
inherit port-profile data

interface Ethernet13/2
inherit port-profile data

interface Ethernet13/3
inherit port-profile management

interface Ethernet14/1
inherit port-profile data

interface Ethernet14/2
inherit port-profile data

interface Ethernet14/3
inherit port-profile management

interface control0
line console
boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.2.2.bin sup-1
boot system bootflash:/nexus-1000v.4.2.1.SV2.2.2.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.2.2.bin sup-2
boot system bootflash:/nexus-1000v.4.2.1.SV2.2.2.bin sup-2
svs-domain
domain id 1
control vlan 1
packet vlan 1
svs mode L3 interface mgmt0
svs connection lab.local
protocol vmware-vim
remote ip address 10.1.1.200 port 80
vmware dvs uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce" datacenter-name lab
max-ports 8192
connect
vservice global type vsg
tcp state-checks invalid-ack
tcp state-checks seq-past-window
no tcp state-checks window-variation
no bypass asa-traffic
vnm-policy-agent
registration-ip 0.0.0.0
shared-secret **********
log-level