Bobby,
Every environment and specific requirements are different, but, not knowing anything about what you specifically need, I can offer some best practices. Sorry in advance if I'm too long winded on this topic.
Your FCoE - vlan ID's is fine. Pick vlan ID's outside of the range of all of your possible network vlans as the system will not allow you to overlap.
From your diagram and notes, you have an excellent looking setup. "Best practices" would keep the SAN switches and VSANs both separate and use a unique numbering scheme just like you have done. The reasons behind this are:
1) SAN masking, zoning and failure mechanism don't really lend themselves to a full mesh topology like you might have in the IP world. Multipathing in SAN pretty much relies on multiple paths and masking and most importantly a multipath driver on your host to recognize that it is talking to the same lun via different paths.
2) By creating the dual, physically isolated SAN fabric as you have done, you have limited the amount of damage that may result in the complete failure of one side of the SAN fabric, either due to some component failure (sfp, cable, switch) or misconfiguration of a zoneset on a single fabric. Again, this only works if the fc multipathing driver is properly installed. That driver is normally provided by the storage OSM (EMC, HDS, Netapp, HP, IBM).
3) By utilizing different VSAN numbering schemes as you have done (100's for fabA, 200's for fabB) you provide the following:
a) a visual cue to the operator that they are making changes to either fabA or fabB.
b) prevent an inadvertent active zoneset merge if staff manages to connect a cable between the two 9148's by mistake. While not necessarily catastrophic, if does require some effort to untangle the resulting active zoneset.
Regarding SAN f port-channel, f port-trunk. Both SAN port-channel and trunk have been available for ISLs (E-port) on the Cisco MDS since initial release (8 years) and provide similar capabilities as are found in the IP world. Those being link aggregation (san port-channel) and multiple VSANs ( trunk). Those features have now been added to f ports (NPV) for the MDS in NX-OS 4.1 and in UCS in 1.4 (Balboa code).
In your design, if you wanted to provide link aggregation and link failure protection between the UCS and MDS on one fabric, you could enable SAN f port-channels and SAN f port-trunking to carry multiple VSANs. Keep in mind that if created a single SAN f port-channel/trunk, all of the VSANs would share the port-channel. In your design, I can't see this being an issue, as you'll have 16Gb of bandwidth available to share and this is how I would configure it. But, you would have better knowledge of your requirements. You can mix and match the number of san port-channels and VSAN allows on the MDS side. For example you could create a 8G SAN port channel and only allow VSAN 100 on the MDS side to ensure 8 G of bandwidth for those hosts in VSAN 100 and just have single links for VSANs 101, 102, 103, again depending of what you think are the bandwidth requirements.
If you have any questions, please let me know.
