09-06-2011 04:57 PM - edited 03-07-2019 02:04 AM
This is my first foray into the Nexus 1000v, and I'm having some issues.
So my understanding is that each server has one NIC that's system-uplink which is the equivilant to the backplane in the Nexus.
If I have 4 servers, each with a system-uplink NIC, is there a need to dump them each into a port-channel?
Also, should they be trunk ports if all three of my nexus VLANS are using the same vlan?
Second question...the vm-uplink ports...those are, from my understanding, for VM traffic traversing the VM to the physical NICs.
That said, again, if I have 4 servers, each with 2 NICs to use for this...how would I port-channel them? I originally tested without port-channeling, and my other switch (a 3750G stack) listed several errors with vlan/host flapping between the ports...
So how would I port-channel them? Or is there a better way to aggregate the bandwidth between the NICs? It would seem, with a distributed switch, that there would be some bandwidth aggregation available.
Also, concept-wise...if I have my 4 hosts, each with 2 NICs...since it's a distributed switch, in theorey, I could disable 7 of the NICs, and the traffic from all 4 hosts would still use the backplane NICs and use that last nic for egress, right?
Solved! Go to Solution.
09-07-2011 07:12 AM
Each ESX host will have a VEM module. The VEM module will have one or more uplinks. Each uplink will have one or more physical nics. If an uplink has more than one nic (recommended for redundancy), a portchannel for that uplink is required. If you have more than one uplink, you need to ensure they do not have overlapping VLAN.
In your statement, you stated you have a system and vmware uplink with one link each. You would not port-channel these uplinks together. You may wish to collapse the system and vm uplinks into a single uplink and create a port channel for the single uplink.
The recommended means of creating the portchannel is using LACP where supported. If not supported, MAC PINNING is the recommend means of creating the port channel.
If you have 4 hosts with 2 nics each, you will want to use all eight nics. Each ESX host will need to have access to the network. Either with two uplinks with a single nic each or a single uplink with two nics each.
The following documents may help:
Cisco Nexus 1000V Series Switches Deployment Guide Version 2
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/guide_c07-556626.html
Best Practices in Deploying Cisco Nexus 1000V Series Switches on Cisco UCS B and C Series Cisco UCS Manager Servers
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-558242.html
09-07-2011 07:45 AM
Because VM traffic and vCenter activities (ie vMotion) can be bandwidth intensive, it is preferred to have a dedicated port channel for VSM-VEM traffic to eliminate any contention for network bandwidth. With your limitation on NICs, it may want to collapse to single uplink.
You are ontrack with your uplink configuration. If you are using LACP on 4.2(1)SV1(4) or newer, you will want to review 'feature lacp' and 'lacp offload'
Cisco Nexus 1000V Command Reference, Release 4.2(1)SV1(4)
09-06-2011 06:47 PM
Hi There
Nexus 1000v is just like a physical switch logically but is is software only, server ports in vm terms vmnic connect to N1K ports by creating a port-group type vm port and in vm level you allocate the port-group to those VMs
again from N1K point of view you create port profile ( to be used as port-group in VM/ESXi ) once you assign a vm to that port group it will be allocated to vNIC in N1K
for th euplinks from N1K you create a port-prfile with ether-channel/trucking
and from VM side you chose this a port-group and you cn do the NIC teaming there as well
again this is if you are using the N1K VEM ( like Ethernet switch module in a normal switch ) and you configure the CLI and the control via VSM ( like supervisor modul in physical siwtch like 6500 )
the VEM is per ESXi Host, while the VSM can control all of your VEMs ( to some extent )
if you are using Cisco UCS B series with bypassing the hypervisore this is a bit different setup where you have VEM of N1K in each Host and no VSM anymore you will have the UCSManager to manager and populate the port-profiles to the vCenter of VM environment
and there is no backplane concept, VEM communicate with the VSM using special vlan for managment and control trafic need to be created
and for aggregation it depends on what physical system you are using as you can have 2 levels of it in the UCS level and VM level if you are using UCS here
HTH
if helpful rate
09-07-2011 06:01 AM
So the system-uplink NICs are, as vmware support explained it to me, the backplane used for internal VEM/VSM communication. Since each is in a different physical host, they cann't be port-channeled, which I guess makes sense.
For the VM traffic...I can only port-channel NICs from the same physical host to the switch?
I had a host flapping between ports before, but I will need to go back and confirm where it was happening. I will report back.
09-07-2011 07:12 AM
Each ESX host will have a VEM module. The VEM module will have one or more uplinks. Each uplink will have one or more physical nics. If an uplink has more than one nic (recommended for redundancy), a portchannel for that uplink is required. If you have more than one uplink, you need to ensure they do not have overlapping VLAN.
In your statement, you stated you have a system and vmware uplink with one link each. You would not port-channel these uplinks together. You may wish to collapse the system and vm uplinks into a single uplink and create a port channel for the single uplink.
The recommended means of creating the portchannel is using LACP where supported. If not supported, MAC PINNING is the recommend means of creating the port channel.
If you have 4 hosts with 2 nics each, you will want to use all eight nics. Each ESX host will need to have access to the network. Either with two uplinks with a single nic each or a single uplink with two nics each.
The following documents may help:
Cisco Nexus 1000V Series Switches Deployment Guide Version 2
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/guide_c07-556626.html
Best Practices in Deploying Cisco Nexus 1000V Series Switches on Cisco UCS B and C Series Cisco UCS Manager Servers
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/white_paper_c11-558242.html
09-07-2011 07:22 AM
I had set up both uplink port profiles based on Cisco's documentation (the getting started guide) as well as a call with a VMWare engineer...it did seem a little unexpected to have to dedicate a hardware NIC in each server to just management.
So...based on what you're saying...
I could collapse the two uplink groups into one, as long as its a trunk with all of my VLANS allowed, make sure i have the system VLAN assigned (that's what the VEM/VSM communicates on?), and dump each hardware NIC into the uplink group? Keep my existing vethernet profiles for my vlan tagging...
And set up separate port-channels on my 3750G stack for each host...
That, to me, makes a lot of sense...
Am I on the right track?
09-07-2011 07:45 AM
Because VM traffic and vCenter activities (ie vMotion) can be bandwidth intensive, it is preferred to have a dedicated port channel for VSM-VEM traffic to eliminate any contention for network bandwidth. With your limitation on NICs, it may want to collapse to single uplink.
You are ontrack with your uplink configuration. If you are using LACP on 4.2(1)SV1(4) or newer, you will want to review 'feature lacp' and 'lacp offload'
Cisco Nexus 1000V Command Reference, Release 4.2(1)SV1(4)
09-07-2011 02:50 PM
Hi Dan
Just i want ot be clear here regarding the uplinks the NIC teaming,, and port channel has to be done per physical server/VEW
can not be cross server nic teaming/portchanneling ? is that right
with you UCS B we can do it deffrently for example using the Palo adapter we can present several vNICs and get the VM team them while in the UCSM we pin them to diffrent FI
09-07-2011 03:24 PM
Correct, port-channels are done on a per physical server/VEM. The port channel does not extend beyond the ESX. Where a port-profile is applied to two ESX hosts were port channels are needed, two port channels are created, one for each host.
One's configuration options are expanded with the palo adapter because of its ability to create multiple vnics.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide