system vlan on Nexus 1000v

[minghui.qi]

Hi all,

I understand that system vlan allows traffic flowing for those vlan even if VSM is not reachable, and system vlan should NOT be normal virtual machine traffic vlan. In my deployment of a normal vSphere environment with N1kv, I am setting these vlans as system vlan: ESXi Mgmt, N1kv mgmt, control & packet, VMotion, IP Storage.

I am setting those vlans as system vlan on both uplink port profiles and indivdual port profiles for each of the VLAN. Correct me if this is wrong.

what else should be system vlan, or what of those should NOT be system vlan? VMotion vlan? what are the cons for specifying all vlans as system vlan? Isn't it better because even if VSM is down for any reason, VEMs will still forward traffic for all VMs?

thanks,

ming

[Robert Burns · Accepted solution]

Ming,

Your understanding of system VLANs is not fully accurate.  All VLANs will remain forwarding in the event your VSM is not reachable.  Each VEM module will continue to pass both system and non-system vlan traffic if the VSM goes offline.  EACH VEM will keep its current programming, but will not accept any changes until the VSM is back online.  System VLANs behave differently in that they will always remain in a forwarding state.  Systems VLANs will forward traffic even before a VEM is programmed by the VSM.  This is why certain system profiles require them - ie. Control/Packet etc.  These VLANs need to be forwarding IN ORDER for the VEM to talk to the VSM.

As for your list of "what should be system vlans" - remove VMotion.  There's no reason your VMotion network would need to be defined as a system VLAN.  All others are correct.

Also remember you can ONLY define a VLAN on one uplink port profile.   So if you're using one uplink for "system" type traffic and another for "VM Data" type traffic, you would only have any single VLAN "allowed" on one uplink - not both.  Allowing them on both will cause issues.   The only case you need to keep in mind is that for a "system vlan" to be applied, it must be defined on both the vEthernet Port profile and an Uplink Port Profile.

Ex.

Let's say my Service Console uses VLAN 10, and my VMs also use VLAN 10 for their data traffic.  (Bad design, but just to illustrate a point).

Having to define the system VLAN in "two places" would allow you to treat ONLY your "Service Console" traffic as a system traffic, and still enforce programming/security for your "VLAN Data" traffic.  Following a reboot, you Service Console traffic would flow immediately, but your VM Data would not until the VEM had pulled programming from the VSM.

port-profile type vethernet dvs_ServiceConsole
  vmware port-group
  switchport mode access
  switchport access vlan 10
  no shutdown
  system vlan 10  <== Defined as System VLAN
  state enabled

port-profile type vethernet dvs_VM_Data_VLAN10
   vmware port-group
   switchport mode access

  switchport access vlan 10 <== No System VLAN
   no shutdown
   state enabled

port-profile type ethernet system-uplink
  vmware port-group
  switchport mode trunk
  switchport trunk allowed vlan 10,3001-3002
  channel-group auto mode active
  no shutdown
  system vlan 10,3001-3002 <== System VLAN 10 Defined
  state enabled

Hope this clears up your understanding.

Regards,

Robert

[minghui.qi]

HI Rob,

Thanks a lot for your clear explanation, it does clerify things for me.

I agree and will remove VMotion from system vlan.

I understand if VSM goes down, VEM will still forward all traffic. What I meant was that when ESX reboots up and VSM VM is still down, if we have all VLANs specified as system vlan, then normal VMs will have connectivity immediately as they boot up, rather than having to wait for VSM to come up,  VSM is a VM itself, if VSM mgmt vlan can be system vlan, why other VMs' vlan cannot? There must be some bad implications on specifying normal VM traffic vlans as system vlans, but I just don't know what they are. Can someone tell me?

thanks

ming

[Robert Burns · Accepted solution]

When you specify a VLAN to be system vlan it will ALWAYS be forwarded.  This pretty much circumvents a profile "checking-in" with the VSM before forwarding traffic for VMs assigned to it.

The reason why your Mgmt VLAN is normally a system vlan will ensure you always have access.  If you have a VM assigned to a VLAN that was a system vlan you're circumventing all programming & security that you may want to apply to that VM.  Let's say for example you define QoS, ACLs and other security settings on your VMs port profile - they will not get enforced on your VM should you reboot a VEM host, and the VSM is not reachable.  Since your VLAN is a system VLAN it will have open access to forward any/all traffic when you might not want that.

Comes down to enforcing settings & security on Port Profiles, vs. ensuring access to system interfaces for managing tasks.

Robert

[minghui.qi]

Thanks Rob, that makes sense. I thought too much about ensuring VM connectivity when recovering from shutdown (power outage of things like that), and ignored the security or qos features.

[David Niemann]

I wish I had seen this earlier.  I was thinking the same as the original post and added all my VM data port-profiles as system vlans.  When I try to remove them it complains that I can't remove the system vlan as it is assigned to a veth interface.  Is there a way around this other than removing all VMs from the port-profile?

my example

port-profile type vethernet DMZ-ACE-Backside

  description "DMZ load balanced servers VLAN351"

  vmware port-group

  switchport mode access

  switchport access vlan 351

  no shutdown

  system vlan 351

  state enabled

Tried to remove system vlan from port-profile

no system vlan 351

This will remove all system vlans from this port profile. Do you really want to proceed(yes/no)? [yes]

ERROR: Cannot remove system vlans, port-profile currently in use by interface Veth7

[minghui.qi]

Hi David, it doesn't sound like you have other options. You can create another port profile with the same config without the Sys vlan, move all the VMs into it, then you should be able to delete the old port profile, you won't have any outage this way.

Let us know if you find better ways.

Ming

[Robert Burns]

Ming is correct.

You can't "remove" system vlans once configured.  As system VLANs are considered critical for operation this prevents any "whoops" mistakes from happening.  Your only option as Ming pointed out, is to create a "new" PP and migrate your interfaces from the old -> new, then delete the old.  This will not cause any service disruption.  

The same applies to Uplink (Eth) port profiles & physical interfaces. 

Regards,

Robert

[David Niemann]

While it won't cause a service disruption it will require the VMware administrator to associate each VM with the new port profile in the vcenter, correct?

[Robert Burns]

Correct.     Your VMware Admin can use the "Migrate Virtual Networking" wizard to make the process quick & painless - rather than manually changing the network binding of each VM.

See the screenshot below as an example.

Let me know if you have any other questions Dave.

Regards,

Robert