FEX Questions

[visitor68]

Need two things, please:

1.) a Link to a design guide that shows all the combinations of how I can connect a FEX to a 5K/7K

2.) The highest number of 2232 10G FEXs I can connect to a pair of 5/N7ks.

        Thank you

[Reza Sharifi]

Hi,

2.) The highest number of 2232 10G FEXs I can connect to a pair of 5/N7ks.

If 5k is layer-2 only, the max number of FEX is 24

if 5h is layer-2 and layer-3 (has a layer-3 card installed) the max is 8

for 7k the max is 32

1.) a Link to a design guide that shows all the combinations of how I can connect a FEX to a 5K/7K

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus2000/sw/configuration/guide/rel_4_0_1a/FEX-overview.html

HTH

[visitor68]

Reza, I signed in with my CCO account, but I couldnt access the document.

Anyway, is it possible to dual-home a FEX to 2 separate 5500s or 7Ks? Thats why I wanted that document....

[ameya_oke]

Hi,

Yes it is possible to dual-home fex to two Nexus5K or 7K by creating VPC on Nexus5 or 7K.

Below is the config to be done on each 5K box.

Example:

fex 112
  pinning max-links 1
  description
  type "Nexus 2148T"

interface port-channel112
  description
  switchport mode fex-fabric
  vpc 112
  fex associate 112

interface Ethernet1/12
  description
  switchport mode fex-fabric
  fex associate 112
  channel-group 112

Warm Regards,

Ameya Oke

[visitor68]

OK, let me cut to the chase...

lets say I have a requirement to build a network with 1500 or so 10G server ports....distributed...top-of-rack design.

what is the CHEAPEST way to do it? I need a 3 or 4:1 os ratio.

One way I can do it is to use 32 Nexus 3064 switches at the ToR. 48 10G server ports with 4 40G uplinks to a couple of Nexus 7Ks loaded with M2 40G cards with vPC running between them.

But that is EXPENSIVE! Each Nexus 3064 is 52,000! I would need 32 of them. Then I would need two 7K chassis with 64 40G ports each....thats 11 40G cards at $70,000 EACH!....That comes out to about $900,000 per Nexus for a total of about $3.5 MILLION dollars!

I am trying to find a cheaper way to do it with FEX modules....but there are so many damn caveats and limitations........

[ameya_oke]

Hi,

Let me understand the requirement.

-Your are planning for 1500 server.

-The OS ratio[3:1] that you are referring to is 3 virtual servers per physical server, right?

-If that is the case it comes around 500 physical servers.

-If we provide 2 * 10G uplink to server for redundancy we need 1000 * 10G ports

-1000/48[port nexus] = 21 Nexus switches

Please let me know if i am on right track.

Ameya Oke

[visitor68]

Ameya, no, that is not correct. I am talking about 1,500 10G physical server ports. I am not talking about virtualization at all.

A Cisco Nexus 3064 has 48 10G ports PLUS 4 40G ports, correct? OK, so using the 48 10G ports for servers and the 40G ports for uplinks, you get a 3:1 oversubscription ratio. 480G offered load from the servers and 160G uplinks to the core - that is 3:1.

Also, the core must NOT impose any oversubscription at all. So, what you get is a 3:1 ratio from any server to any server. A frame is generated by a server, it is sent to the 3064 and goes up to the core with a 3:1 os ratio imposed...then the core forwards the frame back down to another 3064 that has the destination server attached.

Think of a leaf and spine architecture - a CLOS. I would use the 3064 but it is insanely expensive. I just gave it as an example so you can understand one POSSIBLE solution.....I would rather use a FEX-based solution, but I dont understand all the caveats and different ways to connect ...and limitations....etc.

Makes sense?

[Leo Laohoo]

lets say I have a requirement to build a network with 1500 or so 10G server ports....distributed...top-of-rack design.

Sooooo, I guess talking about the coming-soon-to-a-store-near-you Nexus 6000 (that's not a type-o) is out of the question?

Ok, so you have 1500 physical servers that require 10 Gbps bandwidth.  My questions are:

1.  How are they cabled up?  Copper? Fibre?

2.  Are they all on ONE DC or multiple DC?

3.  Do you intend to run Layer 2 or Layer 3 to the servers?

The Nexus 2K and B22 FEX are not ideal for your scenario.  The 2232PP and 2232TM have 4:1 oversubscription rate.

The Nexus 5596UP or 5596T might be a contender.  If you are running Layer 2 only, these two models have a bandwidth of 1,920 Gbps.  If you want to run Layer 3, the 5596UP or 5596T requires a Daughter Card and can push up to 160 Gbps (4:1 oversubscription rate).

Now, let's talk big-daddy here.

Depending on whether you want to run Layer 2 or Layer 3 with your servers, the Nexus 7K is the perfect fit to run 10-, 40- or 100 Gbps.  This will boil down to what Supervisor Card, Fabric Card, what line cards or whether or not you want to Layer 2 or Layer 3.

I'm not talking about the Nexus 3K because this model is geared up for ultra-low latency and was specifically designed for large financial institutions (such as stock brokerage houses).

The N7K-F248XP-25 is a 48-port "F2" card that can do Layer 2/Layer 3.  It will only support fibre optics (SFP/SFP+) and has a bandwidth of 480 Gbps (or a 1:1 oversubscription rate).  This card can also support FEX:

Sup1 - 32 Nexus 2000 Fabric Extender (Layer 2)

Sup2 - 45 Nexus 2000 Fabric Extender (Layer 2)

For this card to push 480 Gbps you need a minimum of 5 of the newer Fabric-2 cards.

However, you have reiterated several times that you have a budget problem.  That'll be sad because the Nexus family is geared up for dense 10-, 40- and 100 Gbps traffic.  Unfortunately, recommending 6500 with Sup2T would be irresponsible for me.  So, I won't.

Can you have a look at the Excel attachment which will give you some statistics of what each can/can't do.

[sajid_m123]

Oke,

What is the best way to connect FEX to 5K, straight fiber connection from N2k to N5k or dual connections from one Fex2k to two N5k?

Suppose, I have a setup where, I need to connect two FEXs to two N5k, and I need to use pure 10gig ethernet on some servers, FCoE on few servers, and some of my storage is native Fibre Channel, other is FCoE, and I also like to connect two Fibre switches with N5k.

All servers will be connected to FEXs and storage and other Fibre switches will connect directly to two N5Ks.

[ameya_oke]

Hi Sajid,

It totally depends on how much redundancy you want to exten till your server and also more importantly availability of 10G fiber ports on servers.

My recommedation wud be for you to connect FEX to 5K in a straight fashion if you dont want VPC configuration extended to servers.Have 40gig straight cables bundled in portchannel to your 5K.

If you want VPC to be extended to your blade servers connect the FEX in dual fashion to both 5K.

Fex2232P is a good option for 10G fiber uplink to servers.

Warm Regards,

Ameya Oke.

[sajid_m123]

Oke,

Thanks for the update.

[visitor68]

LEO:

Im sorry, maybe I am not being clear...

I am talking about a Top-of-Rack model....simple..L2 or 3 doesnt matter. I am talking about connectivity only now..

Imagine 32 racks, each with a Nx3064 on top. Simple...thats 48x32=1,536. And then they can uplink to Nexus 7Ks with 40G cards ..core must be NON BLOCKING with NO oversubscription. Server to server must be 3:1  oversubscription....period....so I cant have traffic going through the crosslink...has to be server to fex to core switch back down to fex to destination server.....no core switch to core switch cross traffic so no oversub is added...so that would mean I need vPC between cores

All in All..VERY EXPENSIVE! I would rather do it using FEX modules at the top of rack....that will give 4:1, but not bad...acceptable....

So, again, I am asking very straightforward questions...

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 7K with vPC between them.

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 7K without vPC between them -single homed

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 5596 with vPC between them

[Leo Laohoo]

2232 has eight 10 Gbps uplinks and 32 x SFP/SFP+ ports.

So let's say you have all 8 uplinks connected.  Next you want to know how many SFP+ you can use per 2232.  Well, that depends upon your "tolerable" amount of over-subscription.  If you want 3:1 then you can have up to 24 SFP+.

Please understand that your over-subscription rate need to correspond to what kind of line card and Fabric card you have on your 7K chassis.  For example, let's take the above example.  Let's say that you've connected the 2232 (with 8 SFP+) to an M132XP ... This blade has an 80 Gbps bandwidth.  Ok, so you can only connect one 2232.  Fine, but what if you only have ONE Fabric-1 card?

When you say vPC a single 2232 to TWO Nexus 7K ... are you sure you want to do this?  Cisco has slowly re-drawn their recommendation about this.  Some whispers are starting to emerge that doing this scenario may "work" in paper but not recommandable.  They recommend, instead, to vPC a single 2232 chassis to a single 5K/7K and let the server become dual-homed.

[visitor68]

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 7K with vPC between them.

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 7K without vPC between them -single homed

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 5596 with vPC between them

[Leo Laohoo]

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 7K with vPC between them.
How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 7K without vPC between them -single homed

What line card?

Like I said in my previous post, let's say we go cheap.  You have ONE Fabric-1 card (46 Gbps bandwidth per slot).  You use M132XP card.  You want to connect 2232.  2232 has EIGHT 10 Gbps uplink ports.

You can connect up to four 2232 but YOU WILL get an oversubscription rate of 8:1.

Your question is very "misleading".  What is your line card to connect the 2232 with?  What fabric card are you using and how many.  I can't read your mind, I can't read your design doco.

I have given you an Excel file that will give you a heads-up of what you can/can't do.

How many 10g ports MAXIMUM can I get if I use 2232 FEX modules with the Nexus 5596 with vPC between them

Ok, let's presume you are talking about Layer 2.

5596 has 1920 Gbps bandwidth.  So you can connect up to 48 SFP+/copper with 1:1 oversubscription rate.

Your first two questions is like the analogy of showing up in the Le Mans 24-hour endurance race with a custom car.  And you ask the question:  Without inspecting anything, can my car win the Le Mans 24-Hour Endurance Race.