Nexus 9336C-FX2 breakout support

[Alex Moore]

We are looking to use Nexus 9336C-FX2 switches as ACI leaves, but are unsure whether there are limits on the use of breakout cables with this switch. Specifically, how many 10G ports can be presented on a single switch in ACI mode...

I have found the following 4 relevant documents:

• The datasheet says: "Breakout supported on all ports, 1-36: 100G, 2x50G NRZ, 40G native, 4x10/25G (10G w/QSA), 1G w/QSA except ports 1-6 and 33-36"
• The transceiver compatibility matrix says that 40G->10G breakout cables are support on ports 1-30.
• The NX-OS mode switch hardware installation guide says (in the first paragraph): "The ports support breakout and 10-Gigabit with a QSA adapter on ports 1-36 and 1-Gigabit with QSA adapter on ports 7-32"
• The ACI mode switch hardware installation guide says (in the first paragraph): "All ports support 10-Gigabit with a QSA adapter (CVR-QSF-SFP10G). By default, ports 1-30 support 40/100-Gigabit breakout. Ports 1-34 support 4x25-Gigabit, and 4x100-Gigabit breakout starting from ACI release 4.2(4) and 5.0(1). This switch supports 20 breakout ports. From ACI release 3.2(3i) onwards, ports 31-34 are convert-able to downlink ports. Ports 7-32 support 1-Gigabit with QSA adapter"

Firstly, the documentation is not entirely clear about whether breakout to 10 Gbps is supported, or whether 10 Gbps support requires a QSA. I'm pretty sure it is saying that you can use either breakout cables or a QSA to achieve 10 Gbps, but it would be good to have confirmation.

Secondly, the ACI install guide says the switch supports 20 breakout ports. But there's no mention of that in the NX-OS install guide or anywhere else. Is that indeed an ACI-specific limitation? I assume it means up to 20 breakout cables can be used (but no more), resulting in a maximum of 80x 10G ports resulting from those breakouts, plus the remaining 16x 100G non-broken-out switchports. Can anyone confirm whether this is correct?

I also assume there is a typo in the ACI install guide... ie instead of "Ports 1-34 support 4x25-Gigabit, and 4x100-Gigabit breakout" it should say "Ports 1-34 support 4x25-Gigabit, and 4x10-Gigabit breakout"

P.S. I'm not really bothered about clarification on whether ports 31-36 support breakout (as we'll use those for uplinks). Although it is another ambiguity (the transceiver compatibility matrix says those ports don't support breakout; the ACI install guide says ports 31-34 do, but 35-36 don't support breakout; whereas the other two documents say all ports do).

[Orlando_Native]

Yes, there is a *lot* to be desired with Cisco documentation. 

I have those same model switches; but I don't run in ACI mode; so I can't really say anything about ACI mode and breakout; but in my environment I use the first 31 ports as 4 x 10g with (non Cisco) MPO/MPT to 4 LC splitter cables; and that works fine.  I *have* some MPO/MPT to 2 LC splitter cables; but I've never tried to see if that would work in a 100g -> 2 50g environment.  I don't think anyone makes 50g SFP+ transceivers. 

[Brian Dean]

I'm looking to do the same with the 9336C-FX2 in NX-OS mode for an upcoming project. What transceiver did you use for your 4x10G breakout? 

[Orlando_Native]

Brian,

  I work for Oracle; we have our own part # for QSFP's for this switch (since we use it in some of the engineered systems products we sell.  And the breakout cables that work with them.  If you are an Oracle customer; I'd suggest consulting your sales rep.

  That said; we also use a lot of "native" Cisco transceivers; and I've not found any yet that don't work just as well.  Their QSFP-40G-SR4 should work fine.  They also make QSFP-40G to 4xSFP-10G cables.  If you need transceivers at both ends;  that might be worth looking into; if you need LC on the other end; then you'll probably need to look elsewhere; I haven't run across any genuine Cisco optical cables that do that.

[hatman]

hi ,

qsfp40g break to 4x10g lr via mpo cable with cmd: interface breakout module 1 port xxxx map 10g-4x

on nx mode is ok for me.

here is an example port 10G break.
Eth1/18/1 eth 10G
Eth1/18/2 eth 10G
Eth1/18/3 eth 10G
Eth1/18/4 eth 10G

[Alex Moore · Accepted solution]

Responding to my own question, having now completed the associated project, I can confirm that I have some 9336C-FX2 ACI leaf switches with as many as 28x breakout cables attached (most of which are 40G->10G, but some 100G->25G too). So at least with current ACI software releases (5.1) there is no limitation preventing the use of more than 20x breakout cables on a single leaf, which had been my main concern.

[Orlando_Native]

I've found differing information in the various manuals and articles too.  I don't use those switches in ACI mode; and I generally use ports 32-36 as uplinks or VPC connections; but on the rest breakout seems to work.  At least for 4 10G breakouts; anyway.  I've had some issues with 4 25G breakouts; but I'm not sure (as yet) whether the issue is on the switch side or the host side at this point.