UCS - ESXi setup

[Kassim Ismail]

Hi All,

UCS newbie here.

Is there a  document (or can someone give pointers) that shows a "standard" configuration of ESXi 4.1 on UCS in the context  of networking. Primarily around number of nics, vlan configuration,  splitting out management,vmotion & data traffic etc.

Thanks

Kassim

[Robert Burns]

Here's a few guides on general best practices with UCS.

Robert

[simon.geary]

Here are a couple more documents that talk about VMware in particular.

When deciding on networking configuration of ESXi on UCS much will depend on the adapter type. If you have the Palo virtual interface card you have the option of splitting out services such as vMotion, FT Logging, IP Storage, VM Port Groups etc. on to their own virtualised NIC. If you do have Palo then separate all these services onto different NICs but still give each service a dedicated VLAN (Avoid untagged traffic with VMware in general). Without Palo, you will "only" have two 10 GbE ports to play with, so you will likely separate the different services with VLANs but all running over the same two NICs. The one configuration you want to avoid is lumping all these services together on the same VLAN.

[Kassim Ismail]

Thanks Robert & Simon for the pdfs.

Simon,

We have the Palo card. on the issue of Vlan, we are looking at separating out the traffic onto different vlans (speaking to our network team about this). the one thing i am not sure about is vmotion traffic. Would i be correct in assuming that the vmotion traffic does not need to go upto our switch (Nexus 7K) & back down to another host, but is "routed" within the 6120's?

This is where i am looking for information that will help in understanding the vlan concepts & also configuring it.

Thanks

[Jeremy Waldrop]

Our standard config has 6 vNIC for ESX/ESXi hosts:

• 2 for management, 1 on Fabric A, 1 on Fabric B
• 2 for vMotion, 1 on Fabric A, 1 on Fabric B
• 2 for VM networks,  1 on Fabric A, 1 on Fabric B

On the VMware side:

• put the first 2 NICs in vSwitch0 for management
• the 2 NICs for vMotion are in vSwitch1 and we change the NIC teaming and failover so that vmnic2 is standby and vmnic3 is active. With this configuration we force vMotion traffic on Fabric Interconnect B unless it is down and then the standby vmnic2 becomes active for Fabric A.
• the last 2 vmnics are put into either a VMware Distributed vSwitch, Nexus 1000v or a UCS VN-Link dvSwitch unless the customer doesn't have vSphere Enterprise Plus and then we have to put those into a standard vSwitch.

[simon.wright]

It would be simpler to do the following with the same results:

4 nics in total

• 2 for management and vMotion, 1 on Fabric A, 1 on Fabric B
• 2 for VM networks,  1 on Fabric A, 1 on Fabric B

On the VMware side:

• put the first 2 NICs in vSwitch0 for management, the 2 NICs for Management and vMotion change the NIC teaming and failover.
• Management, vmnic1 is standby and vmnic0 is active.  then flip it for vMotion. With this configuration we force vMotion traffic on Fabric Interconnect B unless it is down and then the standby becomes active for Fabric A.
• Still make sure QoS is used where possible as vMotion can now use up to 8Gb per vMotion !!

• the last 2 vmnics are put into either a VMware Distributed vSwitch, Nexus 1000v or a UCS VN-Link dvSwitch unless the customer doesn't have vSphere Enterprise Plus and then we have to put those into a standard vSwitch.

[Jeremy Waldrop]

It would be a little bit simpler but the other reason we break everthing up is to set different QoS policies. For example we give vMotion less priority than VM traffic. With ESX 4.1 VMware improved vMotion so that it can now saturate an entire 10G link. In a shared environment if you don't put some QoS in place vMotion could potentialy take bandwidth away from FC and VM networking traffic.

[Kassim Ismail]

Thanks Jeremy & Simon for the suggestions on the setup.

We  were looking at setting up our environment based on what Jeremy  suggested with 6 nics before this posting.  Your suggestion is also  something we will consider & as you said simpler to implement.

The other thing i am not fully sure about is the vlan setup for this within UCS. I can understand setting up a vlans for vm traffic & management (out to Nexus 7K), but not clear on the vlan setup for vmotion. Any comments on this would be appreciated.

Thanks

[Jeremy Waldrop]

We usually create a dedicated layer 2 vlan for vmotion in both UCS and on the uplink ethernet switches and keep vmotion on standard vSwitch1

[Kassim Ismail]

Jeremy,

Thanks for the info, that helps.

Kassim

[KRIS PATE]

Jeremy,

   When you build these 6 vNics what do configure for the MTU for each?   Do you set only the vMotion vNics to jumbo or all of them?

We are using ESXi 4.1 on B200M2 with Palo.

Kris