Editor’s Note: TS Newsletter readers get a 35% discount on Ron Fuller’s book, NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures. See discount code at the end of this article.
The new F3 series modules for the Nexus 7000 and Nexus 7700 series switches are amazing components of technology and the number of customers investing in them is very positive re-enforcement of that fact. With all of the capabilities available now and a rich roadmap for exciting new features, it’s no wonder customers are ordering them it like crazy. Whether it is fundamental L2/L3 switching, FabricPath or Overlay Transport Virtualization (OTV) or the other capabilities this module can do, there’s no end to the use cases. One of the less discussed features though is one we’ll focus on now, called the breakout feature.
“What is breakout?” you ask? Glad you asked, otherwise this would be a short article. Breakout is the ability to take a 40G port and configure it as four independent interfaces.
“What sorcery is this?! How can one interface become four? Are these sub-interfaces or some other similar idea?” First, remain calm - it’s not sorcery, it’s engineering and it provides a great migration path from 10G to 40G. Let’s dig into the details and start with the OSI system model and go to layer 1, the physical layer. To appreciate how breakout works let’s look at how 40G works over multi-mode fiber(MMF). The industry standard way to deliver 40G is to use a 12 stranded MMF cable with a multiple push on (MPO) end. Eight of the 12 strands are lit up making four pairs of 10G that are multiplexed together at the optical level to achieve a 40G link. It looks like the image below:
Breakout takes advantage of this physical layer requirement and allows the network administrator to configure a port to be used as 4 discrete, independent 10G interfaces. This allows you to connect 10G devices to your new F3 line card without needing a different line card to provide 10G support. These could be routers, switches, servers or any other device; as long as it’s 10G, it’ll work! So now you can buy a 40G module, run some of the ports in 10G mode and others in 40G mode. You can mix and match the configurations because F3 lets you do breakout on a port-by-port basis! An example is in the image below.
Now the big question, how do I get this feature on my switch? A couple of requirements are in order:
- NX-OS 6.2(6) or higher
- F3 or M2 module
- If using F3, SUP2 or SUP2e are required
After you have that, it’s very easy to configure. Let’s start by taking a look at a switch with F3, but no breakout configured.
N7K-1-F3# show int brief
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port Ch #
Interface --------------------------------------------------------------------------------
Eth5/1 -- eth routed down XCVR not inserted auto(D) --
Eth5/2 -- eth routed down XCVR not inserted auto(D) --
Eth5/3 -- eth routed down XCVR not inserted auto(D) --
Eth5/4 -- eth routed down Administratively down auto(D) --
Eth5/5 -- eth routed down XCVR not inserted auto(D) --
Eth5/6 -- eth routed down XCVR not inserted auto(D) --
Eth5/7 -- eth routed down Administratively down auto(D) --
Eth5/8 -- eth routed down XCVR not inserted auto(D) --
Eth5/9 -- eth routed down XCVR not inserted auto(D) --
Eth5/10 -- eth routed down XCVR not inserted auto(D) --
Eth5/11 -- eth routed down XCVR not inserted auto(D) --
Eth5/12 1 eth access down XCVR not inserted auto(D) --
N7K-1-F3#
We see the interfaces are “normal” and that two of them have an optic installed, Ethernet 5/4 and Ethernet 5/5. Now let’s apply the breakout command and see how it looks.
N7K-1-F3# config
Enter configuration commands, one per line. End with CNTL/Z.
N7K-1-F3(config)# interface breakout module 5 port 7 map 10g-4x
N7K-1-F3(config)# show int brief
--------------------------------------------------------------------------------
Ethernet VLAN Type Mode Status Reason Speed Port Ch #
Interface --------------------------------------------------------------------------------
Eth5/1 -- eth routed down XCVR not inserted auto(D) --
Eth5/2 -- eth routed down XCVR not inserted auto(D) --
Eth5/3 -- eth routed down XCVR not inserted auto(D) --
Eth5/4 -- eth routed down Administratively down auto(D) --
Eth5/5 -- eth routed down XCVR not inserted auto(D) --
Eth5/6 -- eth routed down XCVR not inserted auto(D) --
Eth5/7/1 -- eth routed down Administratively down auto(D) --
Eth5/7/2 -- eth routed down Administratively down auto(D) --
Eth5/7/3 -- eth routed down Administratively down auto(D) --
Eth5/7/4 -- eth routed down Administratively down auto(D) --
Eth5/8 -- eth routed down XCVR not inserted auto(D) --
Eth5/9 -- eth routed down XCVR not inserted auto(D) --
Eth5/10 -- eth routed down XCVR not inserted auto(D) --
Eth5/11 -- eth routed down XCVR not inserted auto(D) --
Eth5/12 1 eth access down XCVR not inserted auto(D) --
N7K-1-F3(config)#
We can see that with a single command we created 4 new interfaces – Ethernet 5/7/1 through 4! It’s also important to note that we didn’t need to reload or reset the module or any other component. NX-OS takes care of all the requirements. These interfaces can be configured as routed ports, switchports, port channels or Fabric Extender (FEX) interfaces. As shown below, we have a few different devices connected to these breakout ports.
N7K-1-F3# show int brief
<snip>
Eth5/7/1 -- eth routed down Link not connected auto(D) --
Eth5/7/2 1 eth trunk up none 10G(D) --
Eth5/7/3 1 eth fabric up none 10G(D) 101
Eth5/7/4 -- eth routed up none 10G(D) --
N7K-1-F3# show cdp ne
<snip>
Device-ID Local Intrfce Hldtme Capability Platform Port ID
N5K-1(SSI160905AL)
Eth5/7/2 171 S I s N5K-C5548UP Eth1/29
N3K-1(FOC1636R05P)
Eth5/7/4 131 R S I s N3K-C3064PQ-1 Eth1/1
N7K-1-F3# show fex
FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2232TM-10GE SSI15310HXS
<snip>
N7K-1-F3#
Eth101/1/1 -- eth routed down Administratively down auto(D) --
Eth101/1/2 -- eth routed down Administratively down auto(D) --
Eth101/1/3 -- eth routed down Administratively down auto(D) --
Can breakout mode solve all problems? Unfortunately, no it cannot. It can help you migrate from 10G to 40G over time at your pace in a very cost effective manner though! Following are answers to some common questions about breakout that I’d like to address, based on conversations with many customers are below.
Can I connect a 1G device to a 10G breakout port?
No, the device must be 10G, but a 10G FEX can be connected to a breakout port and provide 1G connectivity.
Can I connect LX/LH/ER/ZR/DWDM oo other optics to a 10G breakout port?
No, the optical QSFPs that support breakout (QSFP-40G-SR, QSFP-40G-CSR, FET-40G) require you to use the same type of optic on both ends. For example, FET to FET, CSR to CSR, SR to SR, etc.
Is breakout the same as BiDi optics?
No, BiDi solve the challenge by reducing the cabling required for 40G to 40G connections, while breakout focuses on 10G to 40G connectivity. BiDi optics are also supported on the Nexus 7000 and 7700 on both the F3 and M2 series modules.
Can I run breakout through my structured wiring system?
Certainly! It might look like the equipment below.
Hopefully you see the benefit of the breakout feature with the capability to run both 10G and 40G connectivity on the same module on a port by port basis with the F3 or M2 module. Routers, firewalls, switches, NAS appliances or servers can be connected to breakout ports without any issues. Breakout is just one of many great capabilities of the F3 module.
Ron Fuller, CCIE No. 5851 (Routing and Switching/Storage Networking), is a technical marketing engineer (TME) on the Nexus 7000 team for Cisco. He has 21 years of experience in the industry and has held certifications from Novell, HP, Microsoft, ISC2, SNIA, and Cisco. Ron's focus is working with customers worldwide to address their challenges with comprehensive end-to-end data center architectures and how they can best use Cisco technology to their advantage.
Ron has had the opportunity to speak at Cisco Live on VDCs, NX-OS Multicast, and general design. He lives in Ohio with his wife and four wonderful children and enjoys travel and auto racing. He can be found on Twitter: @ccie5851.
NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures, 2nd Edition
By Ron Fuller, David Jansen, Matthew McPherson.
Series: Networking Technology
Published: March 15, 2013
SBN-10: 0-13-288356-2
ISBN-13: 978-0-13-288356-6
Published by Cisco Press.
DISCOUNT: 35% off ciscopress.com for Cisco Technical Newsletter customers; use code CISCOTECH at checkout.
This article is featured in the April 2014 issue of the Cisco TS Newsletter. Are you subscribed?
