1. We have set up 3 VSAN's for the customer here (100-102). Our design has connectivity from the MDS 9148 to the Fabric Interconnect 6120's, however they are not redundant connections (4 FC connections each). Because of this, I created VSAN's 200-202 to go up the right hand side connection (see visio, VSAN topology tab). I did this b/c I read that Cisco recommends that you use a different VSAN and a different FCoE VLAN to keep your fabrics separated. Each of these VSAN's are mapped to FCoE VLANS with the same ID #, and they are configured on our Ethernet switches. Is this the proper way to have the VSAN's configured?
Or do I just create VSAN 100-102 on both sides? If there is a simpler way, let me know...
2. Between our FI's and our 9148's I have the following connectivity? On the UCS we created a Fiber Channel with the 4 FC ports. On the MDS 9148 we created a port-channel with trunking on its 4 ports. Is this the proper way to have these 2 connected? Should we be trunking? Everything is up and up at 16Gbps on the UCS, but from a design perspective, i'm not sure if this is the way to do it? The reason I ask is b/c it seems weird to have a port-channel connect to a fiber channel...
Thanks in advance for your input.
Every environment and specific requirements are different, but, not knowing anything about what you specifically need, I can offer some best practices. Sorry in advance if I'm too long winded on this topic.
Your FCoE - vlan ID's is fine. Pick vlan ID's outside of the range of all of your possible network vlans as the system will not allow you to overlap.
From your diagram and notes, you have an excellent looking setup. "Best practices" would keep the SAN switches and VSANs both separate and use a unique numbering scheme just like you have done. The reasons behind this are:
1) SAN masking, zoning and failure mechanism don't really lend themselves to a full mesh topology like you might have in the IP world. Multipathing in SAN pretty much relies on multiple paths and masking and most importantly a multipath driver on your host to recognize that it is talking to the same lun via different paths.
2) By creating the dual, physically isolated SAN fabric as you have done, you have limited the amount of damage that may result in the complete failure of one side of the SAN fabric, either due to some component failure (sfp, cable, switch) or misconfiguration of a zoneset on a single fabric. Again, this only works if the fc multipathing driver is properly installed. That driver is normally provided by the storage OSM (EMC, HDS, Netapp, HP, IBM).
3) By utilizing different VSAN numbering schemes as you have done (100's for fabA, 200's for fabB) you provide the following:
a) a visual cue to the operator that they are making changes to either fabA or fabB.
b) prevent an inadvertent active zoneset merge if staff manages to connect a cable between the two 9148's by mistake. While not necessarily catastrophic, if does require some effort to untangle the resulting active zoneset.
Regarding SAN f port-channel, f port-trunk. Both SAN port-channel and trunk have been available for ISLs (E-port) on the Cisco MDS since initial release (8 years) and provide similar capabilities as are found in the IP world. Those being link aggregation (san port-channel) and multiple VSANs ( trunk). Those features have now been added to f ports (NPV) for the MDS in NX-OS 4.1 and in UCS in 1.4 (Balboa code).
In your design, if you wanted to provide link aggregation and link failure protection between the UCS and MDS on one fabric, you could enable SAN f port-channels and SAN f port-trunking to carry multiple VSANs. Keep in mind that if created a single SAN f port-channel/trunk, all of the VSANs would share the port-channel. In your design, I can't see this being an issue, as you'll have 16Gb of bandwidth available to share and this is how I would configure it. But, you would have better knowledge of your requirements. You can mix and match the number of san port-channels and VSAN allows on the MDS side. For example you could create a 8G SAN port channel and only allow VSAN 100 on the MDS side to ensure 8 G of bandwidth for those hosts in VSAN 100 and just have single links for VSANs 101, 102, 103, again depending of what you think are the bandwidth requirements.
If you have any questions, please let me know.
Thanks very much for your in depth response. Next question....
So if I create a fiber port channel on the UCS, what else do I need to do on
the MDS switches? Here is what I think needs to be done, if you can add to
or correct me:
1. Create a port-channel on the interfaces to link with the UCS Fiber
2. Add the ports to this port channel
3. Enable trunking on those ports
On the fiber channel itself in UCS, i'm not sure what to do about the VSAN
drop down? What VSAN would I specify?
Thanks again for your valuable input.
On Mon, Jan 24, 2011 at 3:05 PM, damcfarl <
I've just posted to the documents section a guide that covers configuring f port-channel and trunk. It covers configration for both the MDS and UCS side. It is not an exhaustive guide, although at 25 pages it should cover most aspects.
I hope you find it useful.
Thanks again for your help. That document was exactly what I needed and was
an easy read.
Now that I have the SAN switch to Fabric Interconnect connectivity nailed
down, what do you think is the best way to configure the connectivity
between the MDS SAN Switch and the EMC Celerra Nx4 SAN? We will have 2 4Gbps
connections going to each SAN switch. I was thinking to do a port-channel
for each side, making it 8Gbps, with trunking. How will the EMC boxes work
with this? Do you know the best practice?
On Tue, Jan 25, 2011 at 11:18 AM, damcfarl <
Storage ports will login to the MDS 9148 as f-ports. I'm not aware of any storage vendor that supports VSAN trunking or SAN port-channeling from their adapter. Port-Channeling is generally a switch to switch feature, or in this case a NPV to switch feature. VSAN trunking is a standard, but, I have not seen it implemented by any of the storage vendors.
At the MDS you'ld put each storage port into a seperate VSAN. This will limit your current design to only 2 VSANs per fabric. You'd have to change your 4 VSAN per fabric design down to 2 VSANs if you only have 2 storage ports available per VSAN.
I'd still f port-channel/trunk between the UCS fc modules to the 9148s.
Can you please send the link to this document or the document name. I would like to setup this up.
They're posted in the "Documents" section of this forum.
Do we need to change FC mode in FI from 'end-host' to 'switch' if we are using fc port channels with trunck enabled? (post 1.4 versions)?
You do not need to be in Switch mode for FC Trunking and SAN port channels. You can do all this in EHM.