Ben,
Great to see you excited about UCS. You have a great looking setup in the works!
First observation, you said you have B200 M3 with the VIC 1280. Did you order your B200 M3's with the mLOM (VIC 1240) also?. If you're trying to get the best bang for your buck we have to be sure you're using the correct configuration.
Let me try to explain.
The VIC 1280 itself is "capable" of handling 4 x 10G lanes per port (80G total for the adapter)
The 2208 FEX provides 4 x 10G backplane lanes to each blade.
The B200 M3 has two IO slots, one for the mLOM specifically, and a "mezz" slot for the VIC/3rd Gen Adapters.
On the B200 M3 blades, these 4 x 10G backplane lanes from each IOM are divided between the mLOM and Mezz slots. This means if you ONLY have a VIC 1280 in your B200 M3 with a 2208 FEX, you will only get a max bandwidth of 20G per fabric. Though the VIC 1280 is "capable" of 4 x 10G lanes per fabric, the B200 M3 only presents 2 x10G lanes per fabric to the Mezz slot. If you want to get up to that 40G per fabric max, you need to utilize the mLOM.
The B200 M3 has an optional module LOM (mLOM) aka VIC 1240 onboard. The 1240 mLOM uses a specific slot purposed only for the VIC 1240 mLOM. The 1240 does not populate the Mezz slot.
With an add-on IP expander card in the Mezz slot, the VIC 1240 + Expander utilizes the remaining backplane lanes to get you up to that 40G per fabric mark. The VIC 1240 + Expander also aggregates the interface so they operate as 2 x 40G interface. The IO Expander is not visible to the PCI bus. All it really does is transparently enables the remaining backlanes to the VIC 1240.
So what if you have mLOM VIC 1240 + VIC 1280? Well in short its overkill. There's two main things to consider here:
1. The VIC 1280 though capable of 40G per fabric, when used in the B200 M3 mezz slot it will give you a max of 20G per fabric. The appropriate configuration if you want to max out your bandwidth is to use the VIC 1240 + IO Expander.
2. As I said previously using the VIC 1240 + Expander aggregates the bandwidth into 2 x 40G lanes transparently. Using the VIC 1240 + VIC 1280 will render 4 x 10G distinct lanes. You host OS would see min. of 4 interfaces, where as with the IO Expander option it would only see 2.
I know its a bit confusing, but this is how the B200 M3 is designed.
********
Now that we've cleared that up let's look at your design question.
If you're looking for design guides or recommendations there's plenty on this forum. Just search for the original VIC M81KR or "Palo". I know Jeremy Waldrop has posted counteless examples of his VIC deployments.
As long as you understand the benefits of each capability you should be able to decide on a design. As you know any Cisco VIC (M81KR, 1240, 1280, P81e) will allow you to virtualize the adapter and present distinct PCI devices to the host. Each vNIC/vHBA can have unique policies assigned to it such as QoS, CDP, VLANs, Fabric Failover, Adapter Policies etc. For each virtual interface you create, its one more point of management to the host, so keep that in mind. Also if your model VIC provides say 10G per physical port, then no matter how many virtual 10G vNICs you present to the host, the underlying 10G bandwidth limit will also be present.
Where we see most people dividing up virtual interface with ESX is for Management, VMotion, FT, VM Data, IP Storage. Basically these are "categories" of virtual traffic you may want different UCS QoS policies applied to. A good example is IP storage. Since you're only utilizing IP storage, you may want to utilize Jumbo Frames. In this instance you could create one vNIC on each fabric with an MTU of 9000. This requires some additional configuration (detailed in numerous posts on here) but you can see where I may want different policies applied to certain vNICs. If you're not planning on applying different policies to each vNIC, then there's no point of dividing them up virtually. They will just become another vmnic/vSwitch/vDS uplink to manage.
Regards,
Robert
