Nexus1000V VEM ports are blocking

[JonasNobs]

Hi folks,

we do have a Nexus 1000V installation running and functional.

In order to do a proof of concept we started several worst case scenarios and are shutting down the VSM and ESX.

After fully shutting down everything in the order VSM-secondry, VSM-primary, the second ESX host and at last the first ESX host.

The next step is to start the systems and test everything for functionality or problems. Now we are facing the issue, that all ports of VEMs are blocked.

~ # vemcmd show port

  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port  Type

   18                UP   UP    F/B*      0          vmnic1 

   49                UP   UP    FWD       0            vmk0 

   50              DOWN   UP    BLK       0        centos-vm2.eth0 

   51              DOWN   UP    BLK       0        nexus-g4-vsm-2.eth2 

   52              DOWN   UP    BLK       0        nexus-g4-vsm-2.eth1 

   53              DOWN   UP    BLK       0        nexus-g4-vsm-2.eth0 

* F/B: Port is BLOCKED on some of the vlans.

       One or more vlans are either not created or

       not in the list of allowed vlans for this port.

Please run "vemcmd show port vlans" to see the details.

~ # vemcmd show port vlans

                          Native  VLAN   Allowed

  LTL   VSM Port  Mode    VLAN    State* Vlans

   18              T          1   FWD    187,191

   49              A        187   FWD    187

   50              A          1   BLK    1

   51              A          1   BLK    1

   52              A          1   BLK    1

   53              A          1   BLK    1

I am now not sure, is this normal behavior? I thought the configuration of VEMs should always be saved and in case of losing the VSM just new configuration of vethernet ports or port-profiles is not possible.

The following is the configuration of our N1Kv:

nexus-g4# show running-config

!Command: show running-config

!Time: Wed Apr 17 08:27:49 2013

version 4.2(1)SV2(1.1a)

svs switch edition essential

no feature telnet

feature netflow

username admin password 5 $1$vaQFlRGe$WmNylWhhNA6/B0/BlZ2Qe.  role network-admin

banner motd #Nexus 1000v Switch#

ssh key rsa 2048

ip domain-lookup

ip host nexus-g4 10.10.10.50

hostname nexus-g4

errdisable recovery cause failed-port-state

ip access-list snmp-ro

  10 permit ip 10.10.10.0/24 any

vem 3

  host vmware id 34333535-3533-435a-4a37-323730325332

vem 4

  host vmware id 34333535-3533-435a-4a37-323730325334

snmp-server source-interface inform mgmt0

snmp-server user admin network-admin auth md5 0x9c794b86442143780e0d0fef86f5e7a0 priv 0x9c794b864

42143780e0d0fef86f5e7a0 localizedkey

snmp-server community public group network-operator

snmp-server community public use-acl snmp-ro

vrf context management

  ip route 0.0.0.0/0 10.10.10.1

vlan 1,181,187,191

vlan 181

  name Clients

vlan 187

  name Mgmt

vlan 191

  name Control

port-channel load-balance ethernet source-virtual-port-id

port-profile default max-ports 32

port-profile type ethernet Unused_Or_Quarantine_Uplink

  vmware port-group

  shutdown

  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type vethernet Unused_Or_Quarantine_Veth

  vmware port-group

  shutdown

  description Port-group created for Nexus1000V internal usage. Do not use.

  state enabled

port-profile type ethernet Uplink

  vmware port-group

  switchport mode trunk

  switchport trunk allowed vlan 181,187,191

  no shutdown

  system vlan 187,191

  state enabled

port-profile type vethernet VMkernel

  capability l3control

  vmware port-group

  switchport mode access

  switchport access vlan 187

  no shutdown

  system vlan 187

  state enabled

port-profile type vethernet VLAN181-Clients

  vmware port-group

  switchport mode access

  switchport access vlan 181

  no shutdown

  state enabled

port-profile type vethernet Control

  capability l3control

  vmware port-group

  switchport mode access

  switchport access vlan 191

  no shutdown

  system vlan 191

  state enabled

system storage-loss log time 30

vdc nexus-g4 id 1

  limit-resource vlan minimum 16 maximum 2049

  limit-resource monitor-session minimum 0 maximum 2

  limit-resource vrf minimum 16 maximum 8192

  limit-resource port-channel minimum 0 maximum 768

  limit-resource u4route-mem minimum 1 maximum 1

  limit-resource u6route-mem minimum 1 maximum 1

interface mgmt0

  ip address 10.10.10.50/24

interface Vethernet1

  inherit port-profile VMkernel

  description VMware VMkernel, vmk0

  vmware dvport 32 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 001B.7830.52B4

interface Vethernet2

  inherit port-profile VMkernel

  description nexus-g4-vsm-1, Network Adapter 2

  vmware dvport 35 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.3AB1

interface Vethernet3

  inherit port-profile Control

  description nexus-g4-vsm-1, Network Adapter 1

  vmware dvport 160 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.7941

interface Vethernet4

  inherit port-profile Control

  description nexus-g4-vsm-1, Network Adapter 3

  vmware dvport 162 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.7391

interface Vethernet5

  inherit port-profile VLAN181-Clients

  description centos-vm2, Network Adapter 1

  vmware dvport 64 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.6A8C

interface Vethernet6

  inherit port-profile VLAN181-Clients

  description centos-vm3, Network Adapter 1

  vmware dvport 65 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.041A

interface Vethernet7

  inherit port-profile Control

  description nexus-g4-vsm-2, Network Adapter 3

  vmware dvport 163 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.0027

interface Vethernet8

  inherit port-profile VMkernel

  description nexus-g4-vsm-2, Network Adapter 2

  vmware dvport 34 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.6DF9

interface Vethernet9

  inherit port-profile Control

  description nexus-g4-vsm-2, Network Adapter 1

  vmware dvport 161 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 0050.5687.4380

interface Vethernet10

  inherit port-profile VMkernel

  description VMware VMkernel, vmk0

  vmware dvport 33 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"

  vmware vm mac 001B.7830.128E

interface Ethernet3/2

  inherit port-profile Uplink

interface Ethernet4/2

  inherit port-profile Uplink

interface control0

line console

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-1

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-1

boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.1.1a.bin sup-2

boot system bootflash:/nexus-1000v.4.2.1.SV2.1.1a.bin sup-2

svs-domain

  domain id 1

  control vlan 1

  packet vlan 1

  svs mode L3 interface mgmt0

svs connection vcenter

  protocol vmware-vim

  remote ip address 10.10.10.98 port 80

  vmware dvs uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d" datacenter-name test-center

  admin user n1kUser

  max-ports 8192

  connect

vservice global type vsg

  tcp state-checks invalid-ack

  tcp state-checks seq-past-window

  no tcp state-checks window-variation

  no bypass asa-traffic

vnm-policy-agent

  registration-ip 0.0.0.0

  shared-secret **********

  log-level

[Robert Burns]

The VEMs will retain their "last known" configuration pulled from a VSM until it's reboot.  This is expected behavior.  If you shut down both VSMs, the VEM will continue to operate fine, but if you then reboot the VEM, when it comes up, all interfaces that were not pre-configured as "system vlans" will stay down until the VEM inserts as a module to the VSM.  This is a security mechanism - VEMs must check in with the VSM for any configuration changes before bringing any ports up.

It's a very unlikely scenario where you would have to reboot a VEM while both active & secondary VSMs are unreachable.  We recommend configuring a DRS rule to separate the primary & secondary VSMs so they never reside on the same host improving redundancy.

Regards,

Robert

[JonasNobs]

Hi Robert,

thank you for your quick respond.

What you mean by 'all interfaces that were not pre-configured as "system-vlans"'?

The VSM's interfaces are configured as system-vlans as well, but are blocking after the shutdown of the ESX hosts.

In order to get VSM's connected to each of another and to the VEM's you have to manually put the ports into the specific VLAN's at the VEM's (at ESX console).

Is there probably any TechDoc discribing this behavior?

Thank in advance

Jonas

[Robert Burns]

While the system is in this state please provide the following output from the VSM:

"show svs connection"

"show int brief"

Robert

[JonasNobs]

Hi Robert,

finally I got the outputs;

nexus-g4# show svs connections

connection vcenter:

     ip address: 10.10.10.98

     remote port: 80

     protocol: vmware-vim https

     certificate: default

     datacenter name: test-center

     admin: n1kUser(user)

     max-ports: 8192

     DVS uuid: 8f 83 07 50 e3 81 90 81-a3 f0 x7 82 42 4b a2 1d

     config status: Enabled

     operational status: Disconnected

     sync status: -

     version: -

     vc-uuid: -

nexus-g4#

nexus-g4# show interface brief

------------------------------------------------------------------------------------------------------

Port          VRF                    Status IP Address                    Speed

------------------------------------------------------------------------------------------------------

mgmt0     --                         up          10.10.10.50               1000

------------------------------------------------------------------------------------------------------

Vethernet     VLAN     Type Mode Status     Reason          Speed

------------------------------------------------------------------------------------------------------

Veth1          187          virt access down     nonParticipating     auto

Veth2          187          virt access down     nonParticipating     auto

Veth3          191          virt access down     nonParticipating     auto

Veth4          191          virt access down     nonParticipating     auto

Veth5          181          virt access down     nonParticipating     auto

Veth6          181          virt access down     nonParticipating     auto

Veth7          191          virt access down     nonParticipating     auto

Veth8          187          virt access down     nonParticipating     auto

Veth9          191          virt access down     nonParticipating     auto

Veth10          187          virt access down     nonParticipating     auto

Veth11          187          virt access down     nonParticipating     auto

------------------------------------------------------------------------------------------------------

Port          VRF                    Status IP Address                    Speed

------------------------------------------------------------------------------------------------------

control0     --                         up          --                              1000

nexus-g4#

Jonas

[Artem Gromov]

Hi everybody,
I had exactly the same problem.
VSM vnics are in port-group with system vlan configured, but they were blocked after ESX reload.

I have solved that by moving pnic and primary VSM back to vswitch to restore connectivity between VSM and VEM and to bring up ports. And then went back to N1Kv switch.

Is there any easier way to recover the issue after shutdown by this scenario?

And I'd like to join the question about any TechDoc discribing this behavior.

JonasNobs,
could you please tell how did you do this at ESX console:

>> In order to get VSM's connected to each of another and to the VEM's you have to manually put the >> ports into the specific VLAN's at the VEM's (at ESX console).

[Artem Gromov]

I have found what caused the issue.

I was using the following port-profile for ESX vmk0, vcenter veth and vsm veths:

port-profile type vethernet system
  capability l3control
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  system vlan 100
  state enabled

I found the following log message:

%VEM_MGR-SLOT11-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[50331670] DVPortGroup[dvportgroup-25]. L3 Control can be applied only on VMKernel port.

I created dedicated port-profile without capability l3control specially for vcenter and vsm veths (vsm already knows that it should use mgmt0 in L3 mode because that information is in svs domain configuration):

port-profile type vethernet 100-agromov
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  system vlan 100
  state enabled

After that everything was fine. No required veth ports was blocked.

So, command capability l3control at vm's veth (vcenter vm and vsm vm) makes veth port blocked after ESX + both vsm shutdown even if it was system vlan configured.

My working config:

N1K-1# sh run

!Command: show running-config
!Time: Wed Mar 19 14:36:50 2014

version 4.2(1)SV2(2.2)
svs switch edition advanced

no feature telnet

banner motd #Nexus 1000v Switch#

ip domain-lookup
ip host N1K-1 10.1.1.199
hostname N1K-1
errdisable recovery cause failed-port-state
vem 11
  host id eabd8850-baa2-e311-af00-000000000001
vem 12
  host id eabd8850-baa2-e311-af00-000000000002
vem 13
  host id eabd8850-baa2-e311-af00-000000000003
vem 14
  host id eabd8850-baa2-e311-af00-000000000004
 localizedkey

vrf context management
  ip route 0.0.0.0/0 10.1.1.1
vlan 1,100-101
vlan 100
  name agromov
vlan 101
  name agromov-vm1

port-channel load-balance ethernet source-mac
port-profile default max-ports 32
port-profile type ethernet Unused_Or_Quarantine_Uplink
  vmware port-group
  shutdown
  description Port-group created for Nexus1000V internal usage. Do not use.
  state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
  vmware port-group
  shutdown
  description Port-group created for Nexus1000V internal usage. Do not use.
  state enabled
port-profile type vethernet 100-agromov
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  system vlan 100
  state enabled
port-profile type vethernet 101-agromov-vm1
  vmware port-group
  switchport mode access
  switchport access vlan 101
  no shutdown
  state enabled
port-profile type ethernet management
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 100
  no shutdown
  system vlan 100
  state enabled
port-profile type ethernet data
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 101
  no shutdown
  state enabled
port-profile type vethernet system
  capability l3control
  vmware port-group
  switchport mode access
  switchport access vlan 100
  no shutdown
  system vlan 100
  state enabled

system storage-loss log time 30
vdc N1K-1 id 1
  limit-resource vlan minimum 16 maximum 2049
  limit-resource monitor-session minimum 0 maximum 2
  limit-resource vrf minimum 16 maximum 8192
  limit-resource port-channel minimum 0 maximum 768
  limit-resource u4route-mem minimum 1 maximum 1
  limit-resource u6route-mem minimum 1 maximum 1

interface mgmt0
  ip address 10.1.1.199/24

interface Vethernet1
  inherit port-profile system
  description VMware VMkernel, vmk0
  vmware dvport 227 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0025.B5AF.0009

interface Vethernet2
  inherit port-profile system
  description VMware VMkernel, vmk0
  vmware dvport 225 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0025.B5AF.0003

interface Vethernet3
  inherit port-profile system
  description VMware VMkernel, vmk0
  vmware dvport 226 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0025.B5AF.0006

interface Vethernet4
  inherit port-profile 100-agromov
  description vsm2, Network Adapter 1
  vmware dvport 35 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0050.5690.CEF9

interface Vethernet5
  inherit port-profile 100-agromov
  description vcenter, Network Adapter 1
  vmware dvport 36 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 000C.29ED.B29E

interface Vethernet6
  inherit port-profile 100-agromov
  description vsm2, Network Adapter 2
  vmware dvport 37 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0050.5690.4E09

interface Vethernet7
  inherit port-profile 100-agromov
  description vsm2, Network Adapter 3
  vmware dvport 38 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0050.5690.2936

interface Vethernet8
  inherit port-profile 100-agromov
  description vsm1, Network Adapter 1
  vmware dvport 39 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0050.5690.BD52

interface Vethernet9
  inherit port-profile 100-agromov
  description vsm1, Network Adapter 2
  vmware dvport 32 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0050.5690.C461

interface Vethernet10
  inherit port-profile system
  description VMware VMkernel, vmk0
  vmware dvport 224 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0025.B5AF.0000

interface Vethernet11
  inherit port-profile 100-agromov
  description vsm1, Network Adapter 3
  vmware dvport 41 dvswitch uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce"
  vmware vm mac 0050.5690.58D6

interface Ethernet11/1
  inherit port-profile data

interface Ethernet11/2
  inherit port-profile data

interface Ethernet11/3
  inherit port-profile management

interface Ethernet12/1
  inherit port-profile data

interface Ethernet12/2
  inherit port-profile data

interface Ethernet12/3
  inherit port-profile management

interface Ethernet13/1
  inherit port-profile data

interface Ethernet13/2
  inherit port-profile data

interface Ethernet13/3
  inherit port-profile management

interface Ethernet14/1
  inherit port-profile data

interface Ethernet14/2
  inherit port-profile data

interface Ethernet14/3
  inherit port-profile management

interface control0
line console
boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.2.2.bin sup-1
boot system bootflash:/nexus-1000v.4.2.1.SV2.2.2.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart.4.2.1.SV2.2.2.bin sup-2
boot system bootflash:/nexus-1000v.4.2.1.SV2.2.2.bin sup-2
svs-domain
  domain id 1
  control vlan 1
  packet vlan 1
  svs mode L3 interface mgmt0
svs connection lab.local
  protocol vmware-vim
  remote ip address 10.1.1.200 port 80
  vmware dvs uuid "17 09 10 50 52 bd 12 fc-d8 55 a6 66 6b af 30 ce" datacenter-name lab
  max-ports 8192
  connect
vservice global type vsg
  tcp state-checks invalid-ack
  tcp state-checks seq-past-window
  no tcp state-checks window-variation
  no bypass asa-traffic
vnm-policy-agent
  registration-ip 0.0.0.0
  shared-secret **********
  log-level