Unable to migrate network to Nexus 1000V

[APrassas7]

Greetings

I'm unable to migrate the ESXi hosts' management network to the N1KV DS as network connectivity is lost when I commence the migration. When I start the migration I leave one pNIC assigned to the vSphere standard switch to maintain network connection and the other is assigned to the N1KV uplink port group. I use the N1KV latest version 4.2(1)SV1(5.1a).

I have successfully completed the installation of the N1KV on my vSphere environment, both the VSM (via the installer app) and the VEM (via the CLI). My vShpere deployment comprises of the following components:

2 ESXi hosts

N1KV VSM installed on host 1

vCenter server VM installed on host 2

Other VMs installed on both hosts

I have created and configured the following port groups in the N1KV DS:

Uplink

port-profile type ethernet VM_uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 9-10

no shutdown

system vlan 9-10

state enabled

ESXi management

port-profile type vethernet ESXimgmt

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 9-10

no shutdown

system vlan 9-10

state enabled

N1KV control-management-packet

port-profile type vethernet N1KVctrl

vmware port-group

switchport mode access

switchport access vlan 9

no shutdown

system vlan 9

state enabled

VM traffic

port-profile type vethernet VMnetwork

vmware port-group

switchport mode access

switchport access vlan 10

no shutdown

system vlan 10

state enabled

I enabled the above vlans before using them. After that I migrated as follows:

vCenter server VM

vNIC1 assigned to VMnetwork (vSphere standard switch)

vNIC2 assigned to ESXimgmt (N1KV DS)

N1KV VSM VM

vNIC1 assigned to VMnetwork (vSS)

vNIC2 assigned to N1KVctrl (N1KV DS)

vNIC3 assigned to ESXImgmt (N1KV DS)

All other VMs

all vNICs assigned to VMnetwork (N1KV DS)

I made the following observations that I would like to share:

• Even though I set the allowed vlans in VM_uplink 9-10 in vSphere client VLAN ID has the value of VLAN access: 6-7. The same is happening with ESXimgmt which is also a mode trunk port prolfile.
• I noticed that there is no assigned interface when I input port-profile show name <profile name> and as far as I've looked into it in the documentation i couldn't find the command

port-profile VM_uplink

type: Ethernet

description:

status: enabled

max-ports: 32

min-ports: 1

inherit:

config attributes:

switchport mode trunk

switchport trunk native vlan 1

switchport trunk allowed vlan 9-10

channel-group auto mode on mac-pinning

no shutdown

evaluated config attributes:

switchport mode trunk

switchport trunk native vlan 1

switchport trunk allowed vlan 9-10

channel-group auto mode on mac-pinning

no shutdown

assigned interfaces:

port-group: VM_uplink

system vlans: 9-10

capability l3control: no

capability iscsi-multipath: no

capability vxlan: no

capability l3-vn-service: no

port-profile role: none

port-binding: static

I've been through most of the documentation about port profiles, interfaces and L2 switching. What is the difference between a port profile, a port group and an interface? Do I need to assign an interface? Can you please help me find out the correct configuration for my network? I'd really like to understand how all this works and set it up myself.

Thank you in advance

[Robert Burns]

Couple of points:

port-profile type vethernet ESXimgmt

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 9-10

no shutdown

system vlan 9-10

state enabled

This should be an access port, not a trunk and only set to the correct vlan for your managmeent VMK subnet.  This is why you're losing access each time you migrate your vmk port to the DVS.  Check the management VMK first while its on the vSwitch, get the vlan, then reconfigure the vEther port profile on the N1K to match this with an access vlan X.  System VLAN needs to contain just this access vlan.

Note: You can't remove a system vlan once assgined to a profile.  suggest you just create a new vEth profile for your ESX managment correct then delete the old one.

Second, your VSM interfaces are wrong.

N1KV VSM VM

vNIC1 assigned to VMnetwork (vSS)

vNIC2 assigned to N1KVctrl (N1KV DS)

vNIC3 assigned to ESXImgmt (N1KV DS)

If you're referring to the VSM VM's virtual NICs they need to be in this order  Should be:

vNic1 - Control

vNIc2 - Management

vNIC3 - Packet

If you're referring to the physical NICs of the host then disregard the above second point.

Regards,

Robert

[APrassas7]

Thank you for your quick reply Robert.

I haven't set any vlan values for the Management network in the vSwitch so it's the default value 0 which means no vlan is set. If I change the vlan value then I will lose connectivity to the management network from the vSwitch as well unless I set it to 4095 which is all vlans.

An ESXi host can be configured from vlan value 0-4095. 0 stands for no vlan and 4095 stands for all vlans.

I obviously can't set a value of 0 or 4095 to the N1KV access vlan. What should I do?

Also, I deleted the previous system vlans with the command "no system vlan 9-10"

[Robert Burns · Accepted solution]

So you need to find out what the native VLAN is on your upstream switch.  Once you know this you need to set it on your 1000v profiles

Ex.  Checking the upstream switch you see the interface has vlan 1 set as the native vlan. The command would look something like:

"switchport trunk native vlan x"

So you need to set the native vlan on your Port Profiles.  Replace "X" with which ever VLAN ID # is native on your upstream switch(es).

port-profile type ethernet VM_uplink

vmware port-group

switchport mode trunk

switchport trunk allowed vlan 9-10, x

switchport trunk native vlan x

no shutdown

system vlan 9-10,x

state enabled

port-profile type vethernet ESXimgmt

vmware port-group

switchport mode access

switchport access vlan x

no shutdown

system vlan x

state enabled

Regards,

Robert

[APrassas7]

Thank you Robert, this worked perfectly.

However, I'm trying to understand why it worked now, so please, bare with me a while longer.

On the layer 2 switching configuration document for N1KV I read what the VEM does when it encounters vlan tagged or untagged frames depending on the type of the port (access or trunk). 

So I believe what happens is that ESXimgmt which I configured to be an access port sees that the incoming frames from the management network are untagged (VLAN 0) and tags them with the access vlan x, which in this case is set to 1. The trunk uplink port takes no action on the frame and allows the traffic to pass as it determines that the frame is tagged with vlan #1.

[Robert Burns]

Your vEthernet service profile was incorrectly configured as a trunk.  Since this vEth port profile was also setup to only allow vlan 9 & 10, the packets were likely dropped as they would have arrived with no VLAN tag, and no native vlan was defined either.

There is no VLAN "0".  This is only used by VMware to set traffic as "untagged".  There are no "Access Port" uplinks on a standard vSwitch.  So if the upstream switch doesn't support 802.1q tagging then this allows the packet to be passed along to any other port groups without a VLAN tag. 

On the 1000v we don't say "VLAN 0" but rather "native vlan x" instead.  We have to designate a VLAN to put traffic on that arrives without any 802.1q tag.  Otherwise the packet would just be dropped by the 1000v.

Robert

[APrassas7]

Robert Burns wrote:
There is no VLAN "0".  This is only used by VMware to set traffic as "untagged".  There are no "Access Port" uplinks on a standard vSwitch.  So if the upstream switch doesn't support 802.1q tagging then this allows the packet to be passed along to any other port groups without a VLAN tag.  

By upstream switch, do you mean the physical switch? The physical switch I use doesn't have any vlan capabilities. This means, like you mentioned, that all the traffic passes through untagged. Does this mean that I can only use the native vlan 1 for all the traffic in the N1KV like I used vlan 0 in the vSwitch? If that's the case does this mean that I can't successfully deploy the N1KV DS configuration for vSphere?

Robert Burns wrote:
Your vEthernet service profile was incorrectly configured as a trunk.  Since this vEth port profile was also setup to only allow vlan 9 & 10, the packets were likely dropped as they would have arrived with no VLAN tag, and no native vlan was defined either.

Regarding this, I set up the port profile n1kv-control mode access like you suggested for my previous problem for the control-management-packet traffic of the N1KV VSM that can be on the same vlan. The access vlan is 9, system vlan is 9. So what should happen is that the access port receives an untagged frame and tags it with the access vlan tag. Then the frame which should now be tagged with vlan id 9 arrives on the uplink which is mode trunk and allows vlans 1,9,10 to pass and the system vlans are 1,9,10 as well. Since I lose connectivity to the N1KV VSM management something is obviously wrong and the frames are either not tagged as they should before they leave the access port or the uplink doesn't allow them to pass. It's working with the management network which is set to access vlan 1 thanks to your instructions, so it should work with other allowed vlans as well. VSM's vNIC2 is assigned to the n1kv-control port-profile as it should.

Something is also very odd. When I check the uplink port group via the vSphere client the allowed vlans are 1,6-7 while I set them to be 1,9-10. I tried setting the allowed vlans to be 13-14 then the vSphere client sees them as it should 1,9-10 but it's still not working.

Should I set up more uplink port profiles and set their native vlans to 9 and 10 for this to work?

[Robert Burns]

Yes by upstream switch I'm referring to the physical switch your ESX hosts connect into.  If you don't have any VLAN capability in your switch, then there's really no point in using different VLANs on your 1000v.  You have what we'd call a flat network.  You should set everything on the 1000v as VLAN 1 and set VLAN 1 to also be your native VLAN on your uplink port profiles.  This will function fine, it's just not a best practice as all traffic can be seen by all hosts and devices. 

On all your vEther port profiles set the VLAN to be VLAN 1 as detailed above.  Otherwise you'll have to deal with VLAN mismatching and it's a bugger to troubleshoot.

Regards,

Robert