cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements

9714
Views
0
Helpful
10
Replies
Highlighted
Beginner

UCS Connectivity - Fiber channel or port channel?

Hello all,

1. We have set up 3 VSAN's for the customer here (100-102). Our design  has connectivity from the MDS 9148 to the Fabric Interconnect 6120's,  however they are not redundant connections (4 FC connections each).  Because of this, I created VSAN's 200-202 to go up the right hand side  connection (see visio, VSAN topology tab).  I did this b/c I read that  Cisco recommends that you use a different VSAN and a different FCoE VLAN  to keep your fabrics separated. Each of these VSAN's are mapped to FCoE  VLANS with the same ID #, and they are configured on our Ethernet  switches. Is this the proper way to have the VSAN's configured?

Or do I just create VSAN 100-102 on both sides? If there is a simpler way, let me know...

2. Between our FI's and our 9148's I have the following connectivity? On  the UCS we created a Fiber Channel with the 4 FC ports. On the MDS 9148  we created a port-channel with trunking on its 4 ports. Is this the  proper way to have these 2 connected? Should we be trunking? Everything  is up and up at 16Gbps on the UCS,  but from a design perspective, i'm  not sure if this is the way to do it? The reason I ask is b/c it seems  weird to have a port-channel connect to a fiber channel...

Thanks in advance for your input.

Bobby Grewal

10 REPLIES 10
Highlighted
Cisco Employee

Re: UCS Connectivity - Fiber channel or port channel?

Bobby,

Every environment and specific requirements are different, but, not knowing anything about what you specifically need, I can offer some best practices. Sorry in advance if I'm too long winded on this topic.

Your FCoE - vlan ID's is fine. Pick vlan ID's outside of the range of all of your possible network vlans as the system will not allow you to overlap.

From your diagram and notes, you have an excellent looking setup. "Best practices" would keep the SAN switches and VSANs both separate and use a unique numbering scheme just like you have done. The reasons behind this are:

1) SAN masking, zoning and failure mechanism don't really lend themselves to a full mesh topology like you might have in the IP world. Multipathing in SAN pretty much relies on multiple paths and masking and most importantly a multipath driver on your host to recognize that it is talking to the same lun via different paths.

2) By creating the dual, physically isolated SAN fabric as you have done, you have limited the amount of damage that may result in the complete failure of one side of the SAN fabric, either due to some component failure (sfp, cable, switch) or misconfiguration of a zoneset on a single fabric. Again, this only works if the fc multipathing driver is properly installed. That driver is normally provided by the storage OSM (EMC, HDS, Netapp, HP, IBM).

3) By utilizing different VSAN numbering schemes as you have done (100's for fabA, 200's for fabB) you provide the following:

a) a visual cue to the operator that they are making changes to either fabA or fabB.

b) prevent an inadvertent active zoneset merge if staff manages to connect a cable between the two 9148's by mistake. While not necessarily catastrophic, if does require some effort to untangle the resulting active zoneset.

Regarding SAN f port-channel, f port-trunk. Both SAN port-channel and trunk have been available for ISLs (E-port) on the Cisco MDS since initial release (8 years) and provide similar capabilities as are found in the IP world. Those being link aggregation (san port-channel) and multiple VSANs ( trunk). Those features have now been added to f ports (NPV) for the MDS in NX-OS 4.1 and in UCS in 1.4 (Balboa code).

In your design, if you wanted to provide link aggregation and link failure protection between the UCS and MDS on one fabric, you could enable SAN f port-channels and SAN f port-trunking to carry multiple VSANs. Keep in mind that if created a single SAN f port-channel/trunk, all of the VSANs would share the port-channel. In your design, I can't see this being an issue, as you'll have 16Gb of bandwidth available to share and this is how I would configure it. But, you would have better knowledge of your requirements. You can mix and match the number of san port-channels and VSAN allows on the MDS side. For example you could create a 8G SAN port channel and only allow VSAN 100 on the MDS side to ensure 8 G of bandwidth for those hosts in VSAN 100 and just have single links for VSANs 101, 102, 103, again depending of what you think are the bandwidth requirements.

If you have any questions, please let me know.

Highlighted
Beginner

Re: UCS Connectivity - Fiber channel or port channel?

David,

Thanks very much for your in depth response. Next question....

So if I create a fiber port channel on the UCS, what else do I need to do on

the MDS switches? Here is what I think needs to be done, if you can add to

or correct me:

1. Create a port-channel on the interfaces to link with the UCS Fiber

Channel

2. Add the ports to this port channel

3. Enable trunking on those ports

On the fiber channel itself in UCS, i'm not sure what to do about the VSAN

drop down? What VSAN would I specify?

Thanks again for your valuable input.

Bobby Grewal

On Mon, Jan 24, 2011 at 3:05 PM, damcfarl <

Highlighted
Cisco Employee

Re: UCS Connectivity - Fiber channel or port channel?

Bobby,

I've just posted to the documents section a guide that covers configuring f port-channel and trunk. It covers configration for both the MDS and UCS side. It is not an exhaustive guide, although at 25 pages it should cover most aspects.

I hope you find it useful.

Highlighted
Beginner

Re: UCS Connectivity - Fiber channel or port channel?

Thank you for your help David! Greatly appreciated...

On Tue, Jan 25, 2011 at 11:18 AM, damcfarl <

Highlighted
Beginner

Re: UCS Connectivity - Fiber channel or port channel?

David,

Thanks again for your help. That document was exactly what I needed and was

an easy read.

Now that I have the SAN switch to Fabric Interconnect connectivity nailed

down, what do you think is the best way to configure the connectivity

between the MDS SAN Switch and the EMC Celerra Nx4 SAN? We will have 2 4Gbps

connections going to each SAN switch. I was thinking to do a port-channel

for each side, making it 8Gbps, with trunking. How will the EMC boxes work

with this? Do you know the best practice?

Thanks again,

Bobby

On Tue, Jan 25, 2011 at 11:18 AM, damcfarl <

Highlighted
Cisco Employee

Re: UCS Connectivity - Fiber channel or port channel?

Bobby,

Storage ports will login to the MDS 9148 as f-ports. I'm not aware of any storage vendor that supports VSAN trunking or SAN port-channeling from their adapter. Port-Channeling is generally a switch to switch feature, or in this case a NPV to switch feature. VSAN trunking is a standard, but, I have not seen it implemented by any of the storage vendors.

At the MDS you'ld put each storage port into a seperate VSAN. This will limit your current design to only 2 VSANs per fabric. You'd have to change your 4 VSAN per fabric design down to 2 VSANs if you only have 2 storage ports available per VSAN.

I'd still f port-channel/trunk between the UCS fc modules to the 9148s.

Highlighted

Re: UCS Connectivity - Fiber channel or port channel?

Hi David,

Can you please send the link to this document or the document name. I would like to setup this up.

Regards

Highlighted
Cisco Employee

Re: UCS Connectivity - Fiber channel or port channel?

Sanjesh,

They're posted in the "Documents" section of this forum.

https://supportforums.cisco.com/community/netpro/data-center/unified-computing?view=documents

Regards

Robert

Highlighted
Beginner

UCS Connectivity - Fiber channel or port channel?

Hi David,

Do we need to change FC mode in FI from 'end-host' to 'switch' if we are using fc port channels with trunck enabled? (post 1.4 versions)?

Thanks

Shiva

Highlighted
Cisco Employee

UCS Connectivity - Fiber channel or port channel?

Shiva,

You do not need to be in Switch mode for FC Trunking and SAN port channels.  You can do all this in EHM.

Regards,

Robert

CreatePlease to create content
Content for Community-Ad
Cisco Community May 2020 Spotlight Award Winners

Cisco COVID-19 Survey