Cisco Supported Architecture for FCoE

[visitor68]

Do these two architectures support FCoE from the server all the way to the N5K??

First

• An IBM chassis with a Nexus 4000 FCoE pass-through blade switch uplinked to a Nexus 2232 top-of-rack switch, and then uplinked to a Nexus 5020 end-of-row switch, where the IP and FC traffic split off (FCF)?

Thats 3 hops before the converged traffic reaches the FCF. Is that supported?

Second,

• This time instead of a blade chassis, I have stand alone rack mount servers with CNAs. Those will be uplinked to the 2232 FEX and then uplinked to the N5K.

I got the partial answer to this one from a Cisco white paper on the 5000s.

It says that FEXs connected to the N5k in straight through mode DO support FCoE, but FEXs connected in active/active DO NOT.

Whats the difference? Im confused about this.

[Robert Burns]

Ex-man

[Updated]

Multihop FCoE with N4K -> N2K is not currently supported.  Mainly due to FLOGI scaling limitations as I found out from a colleague.  Currently we support up to 7 FCF's.  N4K and N2K are not considered FCFs, only the N5K would be in this case.  This follows what we support from a pure FC perpective also.  (Thanks Liz!)

Second option is defaniately supported.  Straight through connects a single FEX to a single N5K.  Whereas Active/Active connects a FEX to two upstream N5Ks in VPC.

The diagram below shows Straight-through vs. A/A in regards to N2K - N5K connectivity.  Disregard the host connections.

Regards,

Robert

[visitor68]

Robert, my man! Im gonna have to start paying you consultative fees lol

Lets forget about the IBM chassis scenario. What I really wan tto understand is how the FEXs (2232PPs, lets say), fit into an FCoE architecture. What the implications are of having them in the design...

Do they count as a "hop"?

Do they have an impact on having multihop FCoE?

Are they FIP snooping Bridges?

Do trhey create any virtual port FCoE instantiations with the upstream N5K? Perhaps they are just a bump in the wire...?

Also, I know what you mean abount the act/act and stright through set ups....Why is it that FCoE is not supported when FEXs are connected in act/act?

Robert, if you ever come to NY, theres a Steak dinner with all the fixins waitin for ya.

[Robert Burns]

Thanks - Credit is due to many of the other smart folks I work with who I do confirm facts I'm fuzzy with.

Let's address your questions.

Do they count as a "hop"?

- No.  Fabric Extenders are line cards and do not count as an FCoE "hop".

Do they have an impact on having multihop FCoE?

- Not that I'm aware of.  They're just an extension of the N5K.  Certain series (2232) will support full FCoE functionality just as any fixed N5K port would.

Are they FIP snooping Bridges?

- Unsure.  I'll let Liz (she's an FCoE/N5K queen) jump in here who can better answer this.

Do they create any virtual port FCoE instantiations with the upstream N5K? Perhaps they are just a bump in the wire...?

- A 2232 port operating with FCoE behaves the same was as the fixed N5K port.  They'll have a VFC bound to the physcial interface, but other than that it's just another interface/bump.

Why is it that FCoE is not supported when FEXs are connected in  act/act?

- I believe the limitations are more due to the FC protocol/behavior side.  Though we can multipath ethernet, I don't think FC has the same luxury.

Robert, if you ever come to NY, theres a Steak dinner with all the fixins waitin for ya.

- Deal!

http://www.cisco.com/en/US/docs/solutions/Enterprise/Data_Center/UF_FCoE_final.pdf

Liz - Feel free to correct me or add in here.  If you have any other design guides in regards to FCoE & N5K/N2K design, please shoot us a link.

Thanks!

Rob

[visitor68]

Thanks, Robert.

Yes, Liz, please join us. :-)

The  crux of my line of questioning is the desire to understand the  implications of introducing the FEX 2232 (or any other ToR FEX) into an FCoE environment. The Ethernet part is easy to understand; it's the FCoE/FC requirements that I want to understand. It's reference architectures for FCoE that involve the FEX modules that I need to know about.....

All I seem to find with regard to FEX white papers is the speeds and feeds and how to connect them to the parent N5K (basic topology information); I know all that like the back of my hand by now. I dont see anything about the FEXs and FCoE - ie, whther its a FIP Snooping Bridge, whether it counts as an FCoE hop (which I know Robert says it doesnt),  whether it creates any virtual port instantiation with the N5Ks, etc.

By the way, all I can seem to find on the Data Center Access Design Guide Using the Nexus 5000 and Nexus 2000 are chapters 1 and 6....why is that? What do the other chapters cover?

Heres Chapter 6 of the Guide Im Talking About:

http://www9.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572829-01_Design_N5K_N2K_vPC_DG.pdf

I will be going to the N7K school in 2 weeks, so I am excited about that! :-)

[estine]

Hey ex-engineer -

I believe we exchanged a thread last week as well

Think of the FEX (2232) as a line card to the Nexus 5000 just as you would think of a line card in an MDS chassis.  It has 32 host facing ports and 8 fabric uplinks (which equates to an "80 gig backplane fabric") to the Nexus 5000 parent switch -- or -- "supervisor."

The 2232 is NOT an FCoE hop - its a line card in the FCoE 5k/2k "chassis," NOR is the 2232 a FIP Snooping Bridge. Its solely a line card

Just like when you insert a 48 port 4Gig Fibre Channel module into an MDS chassis (which is 4:1 over subscribed to the MDS fabric) -- the 2232 is the exact same thing.

Let me know if that helps or feel free to reach out to me : estine@cisco.com

Thanks,

Liz

[visitor68]

Ah, yes, we did. You are "estine"?? Where does the "Liz" come in? :-) You are truly a scholar! Thanks for all your help. I know I ask tough questions and bug you, but you know, this is new technology and there is a lot written about it, but only in general terms. Right now, these questions are specifically about Cisco's solutions. Sorry to bug you.

Anyway, I got it. So, the 2232 FEX modules do not have any implications at all when deploying FCoE, right? No virtual port instantiations - nada, because its nothing but a logical line card physically deployed at a top of rack. got it!

But there is one caveat, at least one I know of....the restriction on running FEXs in act/act when wanting to do FCoE. Why is that? I mention this a few times in my posts above. I guess it may have something to do with the point-to-point nature of FC and that multipathing isnt allowed in FC or FCoE for that matter....any thoughts on that?

Also, there is the 4K to 2K limitation...so perhaps its not that clear cut. See my EDIT add-on below.

[EDIT] By the way, Liz, can you elaborate a bit more on the limitation regarding the 4K--->2K that Robert spoke about above. I dont have an IBM chassis with a 4K, but I do have a Dell m1000 blade chassis and Dell will be coming out with a 10G DCB-capable FSB (FIP Snooping Bridge). It will support PFC, DCBx and ETS, as well as FIP snooping.My vision is to place that in the chassis and uplink it to a 2K.

Of course I cant ask you to speak about a Dell blade, but just to comment on the concept of having a blade switch that can act as an FCoE pass-through (FSB) being uplinked to the 2K, then to the 5K. If the IBM with the 4K to 2K scenario is not supported, then I am wondering where exactly the limitation is and whether that limitation will also applyto other chassis and other I/O FSB modules. [EDIT]

Thanks

[estine · Accepted solution]

Hey Ex-Engineer -

estine = Elizabeth Stine = Liz for short

First off - Fex Active/Active

- we dont support FCoE over Fex A/A because it breaks the idea of SAN A/SAN B isolation.  When a single FEX is connected via a vPC to 2 upstream 5ks, the traffic going through that FEX is hashed between the upstream 5ks.  In the SAN world, we would never have this. Devices aren't "meshed" or "dual-homed" and rather we deploy two physical isolated fabrics for HA.  Make sense?

Now, if we move away from the necessity of deploying two physical fabrics for SAN redundancy, and rely on a single highly available fabric (much like we do in Ethernet today) for our SAN traffic, then this may change...however, I don't believe we are there yet

Second - 4k-->2k limitations

- The Nexus 2000 was designed to be spanning-tree edge devices with HOST facing ports.  We didnt design the Nexus 2000 to have network ports. In fact, we don't suggest you plug any networking device into a Nexus 2000 (though it can be done with native Ethernet traffic if we follow some special rules regarding spanning-tree behavior).  For FCoE, the limitation is in the number of FLOGIs we support on a Nexus 5000.  IF we did allow a Nexus 4000 to connect to a Nexus 2000 with FCoE, technically up to 512 FLOGIs (16 servers connections per N4k and 32 X N4k per Nexus 2000) could be logging into a single Nexus 5000 port.  Obviously this would be dependent on how you designed your network -- but it could happen.  We don't support that many FLOGIs into a single port.  So for FLOGI scalability reasons, we don't support a Nexus 4000 connecting to a Nexus 2000 with FCoE.

For FIP Snooping Bridges, like the Nexus 4000 or the Dell Blade you are referencing, it is best practice and supported to plug the FIP Snooping device directly into the Nexus 5000 FCF. 

Let me know if there is anything else

Thanks,

Liz

[visitor68]

Liz, that was beautiful! You answered all the questions with detailed answers. I am grateful.

Thank you so much.

You are AWESOME!