Nexus 7010 and maximum number of FEX

[michgri]

Apparently there is a maximum of 32 FEX per Nexus 7000 switch. I saw this spelled out for the Nexus 5000, but I couldn't find anything equivalent for the 7000. Does the same limitation apply if the FEX are dual homed to two Nexus 7000 switches?

I also wanted to make sure there is no per-module limitation. Assuming one FEX per shared port group in one of the 32-port M1 modules, I mean. That would mean that 4 modules on a 7010 would support the maximum 32 FEX.

Also, does anyone know the reason behind this limitation? Based purely on the 32-port M1 line cards, you would think that you could attach a maximum of 64 FEX on a 7010. Obviously you can't, but I'm curious as to why.

[Ruhann Du Plessis]

Very interesting question. It is a safegaurd limitation to protect the N7K.

Remember that all traffic between FEX ports traverse the parent Nexus switch.

So the Exert from CCO:

Information About Using a Fabric Extender with a Cisco Nexus 7000 Series Switch

#

You  can extend the Cisco Nexus 7000 Series switch architecture by  connecting up to 32 Cisco Nexus 2248 FEXs as remote I/O modules. Each  FEX provides top-of-the-rack connectivity for up to 48 hosts. Each FEX  becomes an extension of the parent Cisco Nexus 7000 Series switch  fabric, with the FEX and the switch becoming a virtual modular system.  The FEX forwards all 1-Gigabit Ethernet traffic from the hosts to the  switch over 10-Gbps uplinks. Traffic flows from the switch to the FEX  over the 10-Gbps uplinks and to the individual hosts over 100/1000 Mb  Ethernet downlinks.

#

You  connect a FEX to the Cisco Nexus 7000 Series switch through the 32-port  M1 series Ethernet I/O modules installed in the switch. We recommend  that you connect each FEX to one port in a set of four shared ports to  get the fully dedicated 10-Gb bandwidth.

If we do the math, we do hit the N7K hardware limits of the first generation linecards and fabric modules.

N2248 FEX @ 48x 1GbE ports = 48GB per 4x 10GbE uplinks  = 40GbE per fex (1:1.2 oversubscription on the FEX.)

Each FEX without oversubscription on the N7K would need 4x 10 GbE ports.

Each N7K-M132XP-12, has 8x 10 GbE Dedicated ports, which can take 2x FEX, which would require 16x N7K-M132XP-12 linecards going upto the 32 FEX limit. This requires a fully populate N7018 chassis.

Look at it the other way.

32  * 40 Gb = 1.25 terabytes which close the the first generation box limits.

I am sure this configuration limitation will be raised with the 2nd generation hardware coming soon.

Hth

[michgri]

Thanks, that makes sense. I hadn't even begun to take oversubscription into account at this stage.