- Cisco Community
- Technology and Support
- Data Center and Cloud
- Data Center and Cloud Knowledge Base
- Nexus 1000v: VEM ports in blocking state after reboot of ESX host
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
- Subscribe to RSS Feed
- Mark as New
- Mark as Read
- Bookmark
- Subscribe
- Printer Friendly Page
- Report Inappropriate Content
04-07-2014 10:52 PM - edited 08-29-2017 04:46 AM
- Introduction
- Nexus 1000v VSM and VEM
- Problem
- Nexus 1000v Configuration
- Explanation
- Resolution
- Related Information
Introduction
This document describes a problem where all VEM ports of a Nexus 1000v switch move to blocking state after a proper reboot of ESX host. The external Virtual Supervisor Module (VSM) manages the Cisco Nexus 1000V Series switch VEMs. The Virtual Ethernet Module (VEM) executes inside the Cisco Nexus 1000V Series switch hypervisor.
Nexus 1000v VSM and VEM
Cisco Nexus 1000V has 2 parts:
Virtual supervisor module (VSM) - This is the control software of the Cisco Nexus 1000V distributed virtual switch. It runs on a virtual machine (VM) and is based on NX-OS software.
Virtual Ethernet module (VEM) - This is the part of Cisco Nexus 1000V that actually switches data traffic. It runs on a VMware ESX 4.0 host. Several VEMs are controlled by one VSM. All the VEMs that form a switch domain should be in the same virtual Data Center as defined by VMware VirtualCenter..
Problem
The system was properly shutdown in the order VSM-secondry, VSM-primary, the second ESX host and at last the first ESX host. After the system was brought back up it is seen that all VEM ports are blocked.
~ # vemcmd show port
LTL VSM Port Admin Link State PC-LTL SGID Vem Port Type
18 UP UP F/B* 0 vmnic1
49 UP UP FWD 0 vmk0
50 DOWN UP BLK 0 os-vm2.eth0
51 DOWN UP BLK 0 nexus1kv-vsm-2.eth2
52 DOWN UP BLK 0 nexus1kv-vsm-2.eth1
53 DOWN UP BLK 0 nexus1kv-vsm-2.eth0
* F/B: Port is BLOCKED on some of the vlans.
One or more vlans are either not created or
not in the list of allowed vlans for this port.
Please run "vemcmd show port vlans" to see the details.
~ # vemcmd show port vlans
Native VLAN Allowed
LTL VSM Port Mode VLAN State* Vlans
18 T 1 FWD 189,191
49 A 189 FWD 189
50 A 1 BLK 1
51 A 1 BLK 1
52 A 1 BLK 1
53 A 1 BLK 1
Nexus 1000v Configuration
nexus1kv# show running-config
version 4.2(1)SV2(1.1a)
svs switch edition essential
========= snip ==================
ip access-list snmp-ro
10 permit ip 10.10.10.0/24 any
vem 3
host vmware id 34333535-3533-435a-4a37-323730325332
vem 4
host vmware id 34333535-3533-435a-4a37-323730325334
========= snip ==================
vrf context management
ip route 0.0.0.0/0 10.10.10.1
vlan 1,181,189,191
vlan 181
name Clients
vlan 189
name Mgmt
vlan 191
name Control
port-channel load-balance ethernet source-virtual-port-id
port-profile default max-ports 32
port-profile type ethernet Unused_Or_Quarantine_Uplink
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
vmware port-group
shutdown
description Port-group created for Nexus1000V internal usage. Do not use.
state enabled
port-profile type ethernet Uplink
vmware port-group
switchport mode trunk
switchport trunk allowed vlan 181,187,191
no shutdown
system vlan 187,191
state enabled
port-profile type vethernet VMkernel
capability l3control
vmware port-group
switchport mode access
switchport access vlan 187
no shutdown
system vlan 187
state enabled
port-profile type vethernet VLAN181-Clients
vmware port-group
switchport mode access
switchport access vlan 181
no shutdown
state enabled
port-profile type vethernet Control
capability l3control
vmware port-group
switchport mode access
switchport access vlan 191
no shutdown
system vlan 191
state enabled
system storage-loss log time 30
vdc nexus1kv id 1
limit-resource vlan minimum 16 maximum 2049
limit-resource monitor-session minimum 0 maximum 2
limit-resource vrf minimum 16 maximum 8192
limit-resource port-channel minimum 0 maximum 768
limit-resource u4route-mem minimum 1 maximum 1
limit-resource u6route-mem minimum 1 maximum 1
interface mgmt0
ip address 10.10.10.50/24
interface Vethernet1
inherit port-profile VMkernel
description VMware VMkernel, vmk0
vmware dvport 32 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 001B.7830.52B4
interface Vethernet2
inherit port-profile VMkernel
description nexus1kv-vsm-1, Network Adapter 2
vmware dvport 35 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.3AB1
interface Vethernet3
inherit port-profile Control
description nexus1kv-vsm-1, Network Adapter 1
vmware dvport 160 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.7941
interface Vethernet4
inherit port-profile Control
description nexus1kv-vsm-1, Network Adapter 3
vmware dvport 162 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.7391
interface Vethernet5
inherit port-profile VLAN181-Clients
description centos-vm2, Network Adapter 1
vmware dvport 64 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.6A8C
interface Vethernet6
inherit port-profile VLAN181-Clients
description centos-vm3, Network Adapter 1
vmware dvport 65 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.041A
interface Vethernet7
inherit port-profile Control
description nexus1kv-vsm-2, Network Adapter 3
vmware dvport 163 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.0027
interface Vethernet8
inherit port-profile VMkernel
description nexus1kv-vsm-2, Network Adapter 2
vmware dvport 34 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.6DF9
interface Vethernet9
inherit port-profile Control
description nexus1kv-vsm-2, Network Adapter 1
vmware dvport 161 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 0050.5687.4380
interface Vethernet10
inherit port-profile VMkernel
description VMware VMkernel, vmk0
vmware dvport 33 dvswitch uuid "8f 83 07 50 e3 81 90 81-a3 f0 c7 82 42 4b a2 1d"
vmware vm mac 001B.7830.128E
interface Ethernet3/2
inherit port-profile Uplink
interface Ethernet4/2
inherit port-profile Uplink
============= snip =================
Explanation
VEMs will retain their "last known" configuration pulled from a VSM until it's reboot. This is expected behavior. If you shut down both VSMs, the VEM will continue to operate fine, but if you then reboot the VEM, when it comes up, all interfaces that were not pre-configured as "system vlans" will stay down until the VEM inserts as a module to the VSM. This is a security mechanism - VEMs must check in with the VSM for any configuration changes before bringing any ports up.
Resolution
Checked the logs and found the following message:
%VEM_MGR-SLOT11-1-VEM_SYSLOG_ALERT: sfswitchdata : L3 Control and System VLAN configurations not applied on vethernet port. VMware Port[50331670] DVPortGroup[dvportgroup-25]. L3 Control can be applied only on VMKernel port.
Originally was using the following port profile
port-profile type vethernet system
capability l3control
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled
Created dedicated port-profile without capability l3control specially for vcenter and vsm veths
port-profile type vethernet 100-agromov
vmware port-group
switchport mode access
switchport access vlan 100
no shutdown
system vlan 100
state enabled
No ports were blocked after applying this port profile.
This document is created from following discussion
https://supportforums.cisco.com/discussion/11808356/nexus1000v-vem-ports-are-blocking
Related Information
Nexus 1000v Switch - VSM and VEM Connectivity Issues
Nexus 1000V Recovery from Failed VEM
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:
