only system vlans forward traffic on 1000v

[bhatok]

I am trying to migrate to a Nexus 1000v vDS but only VM's in the system VLAN can forward traffic. I do not want to make my voice vlan a system VLAN but that is the only way I can get a VM in that VLAN to work properly. I have a host with its vmk in the L3Control port group. From the VSM, a show module shows the VEM 3 with an "ok" status. I currently only have 1 NIC under the vDS control. My VM's using the VM_Network port group work fine and can forward traffic normally. When I put a VM in the Voice_Network port group I lose communication with it. If I add vlan 5 as a system vlan to my Uplink port profile then the VM's in the Voice_Network work properly. I thought you shouldn't create system vlans for each vlan and only use it for critical management functions so I would rather not make it a system vlan. Below is my n1k config. The upstream switch is a 2960X with the "switchport mode trunk" command. Am I missing something that is not allowing VLAN 5 to communicate over the Uplink port profile?

port-profile type ethernet Unused_Or_Quarantine_Uplink
  vmware port-group
  shutdown
  description Port-group created for Nexus1000V internal usage. Do not use.
  state enabled
port-profile type vethernet Unused_Or_Quarantine_Veth
  vmware port-group
  shutdown
  description Port-group created for Nexus1000V internal usage. Do not use.
  state enabled

port-profile type vethernet VM_Network
  vmware port-group
  switchport mode access
  switchport access vlan 1
  no shutdown
  system vlan 1
  max-ports 256
  description VLAN 1
  state enabled
port-profile type vethernet L3-control-vlan1
  capability l3control
  vmware port-group L3Control
  switchport mode access
  switchport access vlan 1
  no shutdown
  system vlan 1
  state enabled
port-profile type ethernet iSCSI-50
  vmware port-group "iSCSI Uplink"
  switchport mode trunk
  switchport trunk allowed vlan 50
  switchport trunk native vlan 50
  mtu 9000
  channel-group auto mode active
  no shutdown
  system vlan 50
  state enabled
port-profile type vethernet iSCSI-A
  vmware port-group
  switchport access vlan 50
  switchport mode access
  capability iscsi-multipath
  no shutdown
  system vlan 50
  state enabled
port-profile type vethernet iSCSI-B
  vmware port-group
  switchport access vlan 50
  switchport mode access
  capability iscsi-multipath
  no shutdown
  system vlan 50
  state enabled
port-profile type ethernet Uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 1,5
  no shutdown
  system vlan 1
  state enabled
port-profile type vethernet Voice_Network
  vmware port-group
  switchport mode access
  switchport access vlan 5
  no shutdown
  max-ports 256
  description VLAN 5
  state enabled

[sprasath]

Please get the output of the following from the ESX host (without configuring VLAN 5 as system VLAN):

- vemcmd show card

- vemcmd show port

- vemcmd show port vlans

- vemcmd show bd

Thanks,

Shankar

[bhatok]

Below is the output you requested. Thank you.

~ # vemcmd show card
Card UUID type  2: 4c4c4544-004c-5110-804a-b9c04f564831
Card name: synergvm5
Switch name: synergVSM
Switch alias: DvsPortset-0
Switch uuid: 7d e9 0d 50 b3 3b 25 47-64 14 61 c0 3f c0 7b d9
Card domain: 4094
Card slot: 3
VEM Tunnel Mode: L3 Mode
L3 Ctrl Index: 49
L3 Ctrl VLAN: 1
VEM Control (AIPC) MAC: 00:02:3d:1f:fe:02
VEM Packet (Inband) MAC: 00:02:3d:2f:fe:02
VEM Control Agent (DPA) MAC: 00:02:3d:4f:fe:02
VEM SPAN MAC: 00:02:3d:3f:fe:02
Primary VSM MAC : 00:50:56:aa:70:b9
Primary VSM PKT MAC : 00:50:56:aa:70:bb
Primary VSM MGMT MAC : 00:50:56:aa:70:ba
Standby VSM CTRL MAC : 00:50:56:aa:70:b6
Management IPv4 address: 172.30.2.64
Management IPv6 address: 0000:0000:0000:0000:0000:0000:0000:0000
Primary L3 Control IPv4 address: 172.30.100.1
Secondary VSM MAC : 00:00:00:00:00:00
Secondary L3 Control IPv4 address: 0.0.0.0
Upgrade : Default
Max physical ports: 32
Max virtual ports: 216
Card control VLAN: 1
Card packet VLAN: 1
Control type multicast: No
Card Headless Mode : No
       Processors: 16
  Processor Cores: 8
Processor Sockets: 2
  Kernel Memory:   62904468
Port link-up delay: 5s
Global UUFB: DISABLED
Heartbeat Set: True
PC LB Algo: source-mac
Datapath portset event in progress : no
Licensed: Yes
~ # vemcmd show port
  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port  Type
   24     Eth3/8     UP   UP    FWD       0          vmnic7
   49      Veth1     UP   UP    FWD       0            vmk1
   50      Veth2     UP   UP    FWD       0        XP-Voice.eth0
   51      Veth3     UP   UP    FWD       0        synergPresence.eth0
~ # vemcmd show port vlans
                          Native  VLAN   Allowed
  LTL   VSM Port  Mode    VLAN    State* Vlans
   24     Eth3/8   T          1   FWD    1
   49      Veth1   A          1   FWD    1
   50      Veth2   A          1   FWD    1
   51      Veth3   A          5   FWD    5

* VLAN State: VLAN State represents the state of allowed vlans.
~ # vemcmd show bd
Number of valid BDS: 10
BD 1, vdc 1, vlan 1, swbd 1, 5 ports, ""
Portlist:
BD 2, vdc 1, vlan 3972, swbd 3972, 0 ports, ""
Portlist:
BD 3, vdc 1, vlan 3970, swbd 3970, 0 ports, ""
Portlist:
BD 4, vdc 1, vlan 3969, swbd 3969, 2 ports, ""
Portlist:
      8
      9

BD 5, vdc 1, vlan 3968, swbd 3968, 3 ports, ""
Portlist:
      1  inban
      5  inband port securit
     11

BD 6, vdc 1, vlan 3971, swbd 3971, 2 ports, ""
Portlist:
     14
     15

BD 7, vdc 1, vlan 5, swbd 5, 1 ports, ""
Portlist:
     51  synergPresence.eth0

BD 8, vdc 1, vlan 50, swbd 50, 0 ports, ""
Portlist:
BD 9, vdc 1, vlan 77, swbd 77, 0 ports, ""
Portlist:
BD 10, vdc 1, vlan 199, swbd 199, 0 ports, ""
Portlist:
~ #

[sprasath · Accepted solution]

Please get these from the VSM:

- show module

- show int eth3/8

- show run int eth3/8

For some reason, VLAN 5 is missing from your uplink interface (Eth3/8):

~ # vemcmd show port vlans

                          Native  VLAN   Allowed

  LTL   VSM Port  Mode    VLAN    State* Vlans

   24     Eth3/8   T          1   FWD    1   <<<<<

   49      Veth1   A          1   FWD    1

   50      Veth2   A          1   FWD    1

   51      Veth3   A          5   FWD    5

Also, get 'vemcmd show port vsm' from the ESX host

Thanks,

Shankar

[bhatok]

I ended up failing over to my other VSM and then I did a shutdown / no shutdown on ethernet3/8 and it started working. I am not sure if it was the failover or the shut/no shut that actually did it but everything is working now. Thanks again for helping with this.

[Peter Koltl]

Can you reproduce the issue by reloading the upstream physical switch? I have an open support case at TAC linked to CSCuj82788 bug. The main issue is that the vmnic (Ethx/y) is reported as DOWN both in vCenter and in Nexus 1000V 'show interface ethx/y' output. The consequence is the same as in your case: no VLANs are forwarded except system VLANs. However, the link seems to be UP in  'esxcli network nic list' and on the physical switch side. A simple shut / no shut on the physical switch fixes the situation.

What is the host NIC model and firmware version?