Port-groups, vSphere 5 and Jumbo frames (iSCSI)

[ADAman]

We will setup a UCS system with EMC iSCSI storage. Since this is my first time, I am a little insecure in the design, although I have aquired a lot of knowledge from reading in this forum and doing various courses.

We will be using the 1000V.

1. Is it ok to just use one uplink portgroup with the following portgroups: mgmt, vmotion, iscsi, vm network, external network?

My confusion here is what about jumboframes? Shouldn't this be a separate uplink for this? In this design all the frames are using jumboframes (or is this set per portgroup?)

I read something about using Class of Service for jumbo frames. Maybe this is the clue here.

2. I read in a thread to not include mgmt and VMotion in the 1000V and put it on a VSS. Is this correct?

In this case, the uplink design would be:

1: Mgmt + vMotion (2 vnics, VSS)

2:  iSCSi (2 vnics, 1000v)

3. VM data, external traffic (2 vnics, 1000v)

All vnics set up as Active-Active, Virtual port-id teaming

[Robert Burns]

Atle,

I'm a little confused.  In your uplink design you mention VSS, but the first sentence mentions UCS.  As you talking UCS-B or C-series?

I'll assume UCS-B and respond on that assumption.

With jumbo frames its a must to be configured end-to-end.  A Jumbo MTU is simply the largest packet it can TRANSMIT.  It's up to the sending device to send the larger packet, and the infrastructure just has to accomodate it.  Keeping this in mind there's no problem with putting iSCSI along with other traffic on your N1K uplink port profile.  The iSCSI VMK interface has to be configured with jumbo frames in order for it to send larger packets.  Any other traffic will send at the regular 1500 byte size. 

So from a start to finish you have the following touch points to configure jumbo frames.

1. iSCSI VMK Interface

2. 1000v Port Profiles

3. UCS QoS Policy/CoS

4. Upstream switches

5. iSCSI Target interface

I'm sure you understand how to configure the first two points (heavily documented).  As for the UCS QoS policy here's the catch.  We use a QoS policy to assign a CoS with a larger MTU - usually 9000.  This CoS policy is mapped to the QoS policy which gets assigned to your host facing vNIC.  The traffic from your host vNIC is marked with the assigned CoS value, put into the correct queue and sent unfragmented up to the point where it exits the UCS Interconnects.  Keep in mind this will affect OUTGOING traffic only.  Assuming you've taking care of jumbo MTU on your upstream switches and iSCSI target, it should reach its destination as a fat 9000B packet (give/take).  CoS values are assigned on a per-switch basis.  On the return, if the iSCSI target interface does not tag the return traffic with the same CoS value as UCS has configured for Jumbo frames, it will be passed through your switching infrastructure (still as jumbo since your switches allow it) but when it reaches the UCS Interconnects with no CoS marking - it will get dumped into the "Best Effort" CoS queue by default.  If your Best Effort class isn't configured for jumbo frames, you'll take a nasty performance hit.  So you have two options here.

1. Set your Best Effort CoS MTU to 9000

or

2. Ensure the switch north of the UCS Interconnects are marking the iSCSI target return traffic with the same CoS marking as UCS has configured for jumbo MTU.  You can use one of the other available classes on UCS for this - Bronze, Silver, Gold, Platinum.

Other than that, your vNIC breakdown configuration looks good.  Ensure on the 1000v uplinks you're using MAC Pinning for the Channeling mode. 

If you have any other questions let me know.

Regards,

Robert

[ADAman]

Hi Robert,

And thanks for a fast reply as always.

Yes, I was talking about UCS B-series, and I should have written vSS and not VSS, as VSS is normally referred to a Cisco-mode switch.

I have 2 follow-up questions:

1. What is the reason I cannot use a 1000V uplink profile for the vMotion and management? Is it just for simplicity people do it that way? Or can I do it if I want? What do you do?

2. Shouldn't I use MTU size 9216?

3. How do I do this step: "

Ensure the switch north of the UCS Interconnects are marking the iSCSI target return traffic with the same CoS marking as UCS has configured for jumbo MTU.  You can use one of the other available classes on UCS for this - Bronze, Silver, Gold, Platinum."

Does the Cisco switch also use the same terms "Bronze", Silver", "Gold" or "Platimum" for the classes? Should I configure the trunk with the same CoSes?

4. Concerning Nk1: Jason Nash has said to include vMotion in the System VLANs. You do not recommend this in previous threads. Why?

5. Do I have to set spanning-tree commands and to enable global BPDU Filter/Guard on both the 1000V side and the uplink switch?

Thanks,

Atle, Norway

Edit:

Do you have some recommendations on the weigting of the CoS?

E.g:

IP storage: 35

Vmotion: 35

Vmdata: 30

and I can then assign management VM-kernels to the Vmdata Cos.

Message was edited by: Atle Dale

[Robert Burns · Accepted solution]

Answers inline.

Regards,

Robert

Atle Dale wrote:
 
I have 2 follow-up questions:
1. What is the reason I cannot use a 1000V uplink profile for the vMotion and management? Is it just for simplicity people do it that way? Or can I do it if I want? What do you do?
[Robert] There is no reason.  Many customers run all their virtual networking on the 1000v.  This way they don't need vmware admins to manage virtual switches - keeps it all in the hands of the networking team where it belongs.  Management Port profiles should be set as "system vlans" to ensure access to manage your hosts is always forwarding.  With the 1000v you can also leverage CBWFQ which can auto-classify traffic types such as "Management", "Vmotion", "1000v Control", "IP Storage" etc.
2. Shouldn't I use MTU size 9216?
[Robert] UCS supports up to 9000 then assumed overhead.  Depending on the switch you'll want to set it at either 9000 or 9216 (whichever it supports).
3. How do I do this step: "
 Ensure the switch north of the UCS Interconnects are marking the iSCSI target return traffic with the same CoS marking as UCS has configured for jumbo MTU.  You can use one of the other available classes on UCS for this - Bronze, Silver, Gold, Platinum."
Does the Cisco switch also use the same terms "Bronze", Silver", "Gold" or "Platimum" for the classes? Should I configure the trunk with the same CoSes?
[Robert] The Plat, Gold, Silver, Bronze are user friendly words used in UCS Classes of Service to represent a defineable CoS value between 0 to 7 (where 0 is the lowest value and 6 is  highest value). COS 7 is reserved for internal traffic. COS value "any"  equals to best effort.  Weight values range from 1 to 10. The bandwidth percentage can be  determined by adding the channel weights for all channels then divide  the channel weight you wish to calculate the percentage for by the sum  of all weights.  
Example.  You have UCS and an upstream N5K with your iSCSI target directly connected to an N5K interface. If your vNICs were assigne a QoS policy using "Silver" (which has a default CoS 2 value), then you would want to do the same upstream by a) configuring the N5K system MTU of 9216 and tag all traffic from the iSCSI Array target's interface with a CoS 2.  The specifics for configuring the switch are specific to the model and SW version.  N5K is different than N7K and different than IOS.  Configuring Jumbo frames and CoS marking is pretty well documented all over.
Once UCS receives the traffic with the appropriate CoS marking it will honor the QoS and dump the traffic back into the Silver queue. This is the "best" way to configure it but I find most people just end up changing the "Best Effort" class to 9000 MTU for simplicity sake - which doesn't require any upstream tinkering with CoS marking.  Just have to enable Jumbo MTU support upstream.
4. Concerning Nk1: Jason Nash has said to include vMotion in the System VLANs. You do not recommend this in previous threads. Why?
[Robert] You have to understand what a system vlan is first.  I've tirelessly explained this on vaiours posts .  System VLANs allow an interface to always be forwarding.  You can't shut down a system vlan interface.  Also, when a VEM is reboot, a system vlan interface will be FWDing before the VEM attaches to the VSM to securely retrieve it's programming.  Think of the Chicken & Egg scenario.  You have to be able to FWD some traffic in order to reach the VSM in the first place - so we allow a very small subnet of interfaces to FWD before the VSM send the VEM's programming - Management, IP Storage and Control/Packet only.  All other non-system VLANs are rightfully BLKing until the VSM passes the VEM its policy.  This secures interfaces from sending traffic in the event any port profiles or policies have changed since last reboot or module insertion.  Now keeping all this in mind, can you tell me the instance where you've just reboot your ESX and need the VMotion interface fowarding traffic BEFORE communicating with the VSM?  If the VSM was not reachable (or both VSMs down) the VMs virtual interface would not even be able to be created on the receiving VEM.  Any virtual ports moved or created require VSM & VEM communication.  So no, the vMotion interface vlans do NOT need to be set as system VLANs.  There's also a max of 16 port profiles that can have system vlans defined, so why chew up one unnecessarily?
5. Do I have to set spanning-tree commands and to enable global BPDU Filter/Guard on both the 1000V side and the uplink switch? 
[Robert] The VSM doesn't participate in STP so it will never send BPDU's.  However, since VMs can act like bridges & routers these days, we advise to add two commands to your upstream VEM uplinks - PortFast and BPDUFilter.  PortFast so the interface is FWD faster (since there's no STP on the VSM anyway) and BPDUFilter to ignore any received BPDU's from VMs.  I prefer to ignore them then using BPDU Gaurd - which will shutdown the interface if BPDU's are received.
Thanks,
Atle, Norway
Edit: 
Do you have some recommendations on the weigting of the CoS?
[Robert] I don't personally.  Others customer can chime in on their suggestions, but each environement is different.  VMotion is very bursty so I wouldn't set that too high.  IP storage is critical so I would bump that up a bit.  The rest is up to you.  See how it works, check your QoS & CoS verification commands to monitor and adjust your settings as required.
E.g:
IP storage: 35
Vmotion: 35
Vmdata: 30
and I can then assign management VM-kernels to the Vmdata Cos.
Message was edited by: Atle Dale

[ADAman]

Thanks, Robert. So you are using BPDU Guard instead of the Prtfast and BPDUsFilter on your VEM uplink port profile?

Concerning the CoS: You said this: "

he traffic from your host vNIC is marked with the assigned CoS value, put into the correct queue and sent unfragmented up to the point where it exits the UCS Interconnects.  Keep in mind this will affect OUTGOING traffic only.  Assuming you've taking care of jumbo MTU on your upstream switches and iSCSI target, it should reach its destination as a fat 9000B packet (give/take).  CoS values are assigned on a per-switch basis.  On the return, if the iSCSI target interface does not tag the return traffic with the same CoS value as UCS has configured for Jumbo frames, it will be passed through your switching infrastructure (still as jumbo since your switches allow it) but when it reaches the UCS Interconnects with no CoS marking - it will get dumped into the "Best Effort" CoS queue by default.  If your Best Effort class isn't configured for jumbo frames, you'll take a nasty performance hit.  So you have two options here."

If I set the "Best effort" CoS queu to jumbo frames, what about returning traffic from non-iSCSI sources, such as traffic to VM from the uplink switch. This traffic won't be classified with a CoS number, so it will go to the Best Effort queue and then the packets will get Jumbo frame configuration. Is this correct? Will it still work?

[Robert Burns]

In regards to your last question about return traffic - Other non-iSCSI traffic may also end up in the Best-Effort queue, however it will remain with its default MTU.  Remember, the MTU that UCS defines has defined in the CoS policy only affects traffic it passes outgoing.  On the return, UCS will always accept any sized frame (as will any L2 device). The fact it will land in the Best Effort queue is fine.  The actualy frame size is set by the source device (NIC or Router);  which would be the iSCSI target interface in your case.  All other VM traffic going northbound to App server etc, will be using will have return traffic with the default MTU of 1500 as set by the northbound server's NIC, unless you purposely changed this. 

Robert

[ADAman]

Hi again,

Any thoughts on this N1kv design?

Port Profile 1: iSCSI + Mgmt Vmkernel port groups, 2 VMnics for uplink group

Port Profile 2: vMotion + FT Vmkernel port groups, 2 VMnics for uplink group

Port Profile 3: VM data + Mgmt  VMkernel port groups, 2 VMnics for uplink group

I will also be using CoS and Qos as you said to prioritize the traffic to the different interfaces. This is more tidy for troubleshooting purposes, because we can isolate the problem. But other than that, might as well have it all in one big Port Profile for all the veth interfaces connected to one single uplink port profile?

Something else: Native VLANs

Is it important to have the same native VLAN on the UCS and the Cisco switch? And not to use the default native VLAN 1?   I read somewhere that the native VLAN is used for communication between the switches and CDP amongst others. I know the native VLAN is for all untagged traffic. I see many people set the ESXi management VLAN as native also.

Example:Will I be able to access a VM set with VLAN 0 (native) if the native VLAN is the same in UCS and the Cisco switch (Eg. VLAN 2)? Can I just configure a access port with the same VLAN ID as the native VLAN, i.e 2 and connect to it with a PC using the same IP network address?

And is it important to trunk this native VLAN? I see in a Netapp Flexpod config they state this: "This configuration also leverages the native VLAN on the trunk ports to discard untagged packets, by setting the native VLAN on the port channel, but not including this VLAN in the allowed VLANs on the port channel". But I don't understand it...

What about the downlinks from the FI to the chassis. Do you configure this as a port channel also in UCS? I think this is possible.

--

[Robert] The VSM doesn't participate in STP so it will never send BPDU's.  However, since VMs can act like bridges & routers these days, we advise to add two commands to your upstream VEM uplinks - PortFast and BPDUFilter.  PortFast so the interface is FWD faster (since there's no STP on the VSM anyway) and BPDUFilter to ignore any received BPDU's from VMs.  I prefer to ignore them then using BPDU Gaurd - which will shutdown the interface if BPDU's are received.

-Are you thinking of the upstream switch here (Nexus, Catalyst) or the N1kV uplink profile config?

Message was edited by: Atle Dale