Nexus, why and what is fabric module

[Jonn cos]

Hi all ,

I am having difficulty understanding what is a fabric module. I tried reading the datasheet but i couldnt find any specific reason except that it increases line cards bandwidth. But when under CCW i tried configure a N7k 7009 chassis with 2 Fabric modules, it denied and asked me to enter minimum of 3. So i am really confused about all this.

Any doc or pdf that anyone can share or if possible kindly guide me

[InayathUlla Sharieff]

Hi John,

Each slot on N7K is capable of switching upto 550 Gbps per slot. This is available with the Generation 2 fabric module (FAB 2), which supports 110 GBPS per slot capacity.You can use 5 of these in a N7K chassis which has upto 5 Switch fabric slots. 110*5 = 550 Gbps per slot capacity.In a full-duplex mode, N7K with FAB 2 will support 15+ Tbps switching throughput.

Each fabric has two physical traces to each line card, each trace capable of 23Gbps. Which means each fabric can deliver 46Gbps to each module. With 5 fabric modules installed that allows you 5*46 Gpbs = 230Gbps per slot. The current capabilities of the M-series of line cards is that they can handle a maximum of 80Gbps (this is a limitation of the linecard, not the fabric, so this is very seperate to what the fabrics can deliver). These figures are current, based upon what you can purchase today. This means with your current line cards you are using 80Gbps of a possible 230Gbps.

In the future Cisco will be releasing M1 Series 16 port 10G modules capable of 160Gbps/ slot (16 ports @ 10Gbps full line rate). The F1 series cards capable of 230Gbps/ slot, and the F2s will be capable of upto 480Gbps (48x 10Gbps ports). Future fabrics will support up to 550Gbps/ slot. Which means they will more than satisfy the needs of the future F2 modules. To fully utilize the M1 or F1 modules you can get away with 5 of the current fabric modules, but the F2s will require the new fabric modules.

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/ps9512/Data_Sheet_C78-437760.html

https://www.ciscolive365.com/connect/sessionDetail.ww?SESSION_ID=2129&backBtn=true

HTH

Regards

Inayath

*Plz rate all usefull posts.

[Jonn cos]

that means using fabric module is mandatory ?

[Reza Sharifi]

The max number of Fabric Modules you can put in the chassis is 5, but you need a minimum of 3 to start with.

That is most likely the reason the system asked you to put 3.

HTH

[Jonn cos]

Basically i am confused why do we need minimum of 3 ??? why cant we simply not use it ?

[Reza Sharifi]

You have to have fabric modules.  The switch will not work without them.

the Cisco Nexus 7000 Series Chassis are separate fabric modules  that provide parallel fabric channels to each I/O and supervisor module  slot. Up to five simultaneously active fabric modules work together  delivering up to 230 Gbps per slot. Through the parallel forwarding  architecture, a system capacity of more than 8 Tbps is achieved with the  five fabric modules. The fabric module provides the central switching  element for fully distributed forwarding on the I/O modules.

[Jonn cos]

Thanks alot,

Kindly tell me why we need minimum 3 of them ? pls let me break it down

1) Why we need minimum 3 fabric modules

2) if 1 of them fails in production, will nexus stop forwarding the traffic ? what will be the impact ?

Kindly guide me

[Steve Fuller · Accepted solution]

Hi John

As the posts above state, you have to have the Fabric Modules in the Nexus 7009 chassis as this is the interconnect between the I/O modules.

In terms of your specific questions here:

1) The Fabric-1 modules are able to provide 46 Gbit/s per Fabric module, per slot. The original M1 series I/O modules e.g., the N7K-M132XP-12 have 80 Gbit/s of capacity into the switch fabric, and so two Fabric-1 modules providing 92 Gbit/s are required to provide sufficient fabric capacity for the M1 series 10GE I/O modules. Although the configuration with two Fabric modules actually works, it provides no redundancy for 10GE I/O modules and so Cisco mandated that a third fabric module be used such that the chassis was still able to provide sufficient fabric capacity in the event of a fabric module failure.

In my opinion, the percentage cost increase for a third fabric module is minimal when compared to what you've already invested in the other components, and so I don't believe it makes any sense to not have redundancy in the Fabric modules.

2) If you have a failure of a Fabric module, then the switch will continue to operate, albeit with reduced capacity. If you only have M1 10GE I/O modules, and the minimum of three Fabric modules, then a failure of one will have no impact as the remaining two will provide 92 Gbit/s, with the M1 10GE I/O modules only requiring 80 Gbit/s. If on the other hand you have F1 10GE I/O modules that have 230 Gbit/s fabric capacity, then you actually need five fabric modules to provide the required capacity, and a failure of one of the fabric modules would mean the F1 10GE I/O modules would not be able to operate at their maximum capacity.

Regards

[Jonn cos]

Dear Steve

I want to read/understand more about it. I have partner access so kindly tell me from where did you get to know this stuff. Is it all experience ?

I will really appreciate if you can guide me where to study for it

[Steve Fuller]

Hi John,

As with most things, there's no easy way other than lots of reading, playing and time.

As per my other post to you, if you have Cisco Partner access then a good place to start would be the Partner Data Centre Community which has a wealth of information, not only for the Nexus 7000, but across other technologies and products. If you're new to Nexus then I'd recommend reviewing the material from the Nexus 7K/5K/2K Bootcamp held in August 2012.

If you need kit to practice, then check out the Data Center Virtual Remote Labs which is available via PEC.

Regards

[pritam.singha@global.ntt]

excellent

[pritam.singha@global.ntt]

see the answer above, explained by

Steve Fuller

[akhil kamalakaran]

---

A fabric module connects the line-cards and the supervisors together in a Nexus.

[Joseph W. Doherty]

Why 3, because Cisco says so. ;)

It's determined by the design of the device.

For example, using a different Cisco switch design, the Catalyst 6500 chassis provided a "shared bus" used by the line cards and "sup1" supervisors. Then with the "sup2", Cisco provided a series of line cards that had both a "shared bus" and "fabric" connection. They also provided an optional "fabric" card (which, if I recall correctly, you could install two for redundancy, but only one was used). If you didn't have the fabric card, the line cards and sup2 would use the "shared bus". If you installed the fabric card, then the sup2 and fabric capable cards could use either the bus connection or the fabric connection (there was a configuration option to choose which you wanted to use - if you had some bus only line cards in the chassis, you might continue to use the bus on the "dual" connections cards, but if you had all fabric capable cards, you would want to use the fabric. [The deal was, dual cards could "talk" directly to bus only cards, but if they used their fabric connection, to "talk" to a bus only card, traffic had to pass through the supervisor.]) Then, Cisco provided fabric only cards (which now had two fabric connections), but this did require a fabric in the chassis. Then Cisco provided the sup720, which had the fabric on the supervisor card itself (if also had much more bandwidth capacity); Cisco then provided a new series of the fabric only line cards. Lastly, Cisco provided the sup2T, which provided even more overall fabric capacity, twice the bandwidth to slot, to all slots, including those on the 6513-E had the full capacity.  Cisco also provided a new series of line cards to take advantage of the additional fabric bandwidth.  (BTW, there's even more to the  evolution of 6500 then the foregoing; just trying to show changes due to "fabric".)

Hopefully, the above shows it's up to Cisco to determine what a switch's operational requirements and options are.