cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
9604
Views
5
Helpful
4
Comments
Julie Burruss
Level 4
Level 4

Editor’s Note: TS Newsletter readers get a 35% discount on Ron Fuller’s book, NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures. See discount code at the end of this article.

 

The new F3 series modules for the Nexus 7000 and Nexus 7700 series switches are amazing components of technology and the number of customers investing in them is very positive re-enforcement of that fact. With all of the capabilities available now and a rich roadmap for exciting new features, it’s no wonder customers are ordering them it like crazy. Whether it is fundamental L2/L3 switching, FabricPath or Overlay Transport Virtualization (OTV) or the other capabilities this module can do, there’s no end to the use cases.  One of the less discussed features though is one we’ll focus on now, called the breakout feature.

“What is breakout?” you ask? Glad you asked, otherwise this would be a short article.  Breakout is the ability to take a 40G port and configure it as four independent interfaces.

“What sorcery is this?!  How can one interface become four? Are these sub-interfaces or some other similar idea?”  First, remain calm - it’s not sorcery, it’s engineering and it provides a great migration path from 10G to 40G.  Let’s dig into the details and start with the OSI system model and go to layer 1, the physical layer. To appreciate how breakout works let’s look at how 40G works over multi-mode fiber(MMF). The industry standard way to deliver 40G is to use a 12 stranded MMF cable with a multiple push on (MPO) end. Eight of the 12 strands are lit up making four pairs of 10G that are multiplexed together at the optical level to achieve a 40G link. It looks like the image below:

 

 

Breakout takes advantage of this physical layer requirement and allows the network administrator to configure a port to be used as 4 discrete, independent 10G interfaces.  This allows you to connect 10G devices to your new F3 line card without needing a different line card to provide 10G support.  These could be routers, switches, servers or any other device; as long as it’s 10G, it’ll work! So now you can buy a 40G module, run some of the ports in 10G mode and others in 40G mode. You can mix and match the configurations because F3 lets you do breakout on a port-by-port basis!  An example is in the image below.

 

 

 

 

Now the big question, how do I get this feature on my switch? A couple of requirements are in order:

  1. NX-OS 6.2(6) or higher
  2. F3 or M2 module
  • If using F3, SUP2 or SUP2e are required

 

After you have that, it’s very easy to configure. Let’s start by taking a look at a switch with F3, but no breakout configured.

 

N7K-1-F3# show int brief

--------------------------------------------------------------------------------

Ethernet   VLAN    Type Mode   Status   Reason                 Speed  Port Ch #

Interface                                                                  --------------------------------------------------------------------------------

Eth5/1      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/2      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/3      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/4      --     eth  routed down    Administratively down   auto(D) --

Eth5/5      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/6      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/7      --     eth  routed down    Administratively down   auto(D) --

Eth5/8      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/9      --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/10     --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/11     --     eth  routed down    XCVR not inserted       auto(D) --

Eth5/12     1      eth  access down    XCVR not inserted       auto(D) --

N7K-1-F3#

 

We see the interfaces are “normal” and that two of them have an optic installed, Ethernet 5/4 and Ethernet 5/5.  Now let’s apply the breakout command and see how it looks.

N7K-1-F3# config

Enter configuration commands, one per line.  End with CNTL/Z.

N7K-1-F3(config)# interface breakout module 5 port 7 map 10g-4x

 

N7K-1-F3(config)# show int brief

--------------------------------------------------------------------------------

Ethernet   VLAN  Type Mode   Status  Reason                Speed     Port Ch #

Interface                                                                   --------------------------------------------------------------------------------

Eth5/1     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/2     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/3     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/4     --    eth  routed down    Administratively down   auto(D) --

Eth5/5     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/6     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/7/1   --    eth  routed down    Administratively down   auto(D) --

Eth5/7/2   --    eth  routed down    Administratively down   auto(D) --

Eth5/7/3   --    eth  routed down    Administratively down   auto(D) --

Eth5/7/4   --    eth  routed down    Administratively down   auto(D) --

Eth5/8     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/9     --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/10    --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/11    --    eth  routed down    XCVR not inserted       auto(D) --

Eth5/12    1     eth  access down    XCVR not inserted       auto(D) --

N7K-1-F3(config)#

 

We can see that with a single command we created 4 new interfaces – Ethernet 5/7/1 through 4!  It’s also important to note that we didn’t need to reload or reset the module or any other component. NX-OS takes care of all the requirements.  These interfaces can be configured as routed ports, switchports, port channels or Fabric Extender (FEX) interfaces.  As shown below, we have a few different devices connected to these breakout ports.

 

N7K-1-F3# show int brief

<snip>

Eth5/7/1      --      eth  routed down    Link not connected         auto(D) --

Eth5/7/2      1       eth  trunk  up      none                        10G(D) --

Eth5/7/3      1       eth  fabric up      none                        10G(D) 101

Eth5/7/4      --      eth  routed up      none                        10G(D) --

N7K-1-F3# show cdp ne

<snip>

Device-ID          Local Intrfce  Hldtme Capability  Platform      Port ID

N5K-1(SSI160905AL)

                    Eth5/7/2       171    S I s     N5K-C5548UP   Eth1/29

N3K-1(FOC1636R05P)

                    Eth5/7/4       131    R S I s   N3K-C3064PQ-1 Eth1/1

 

N7K-1-F3# show fex

  FEX         FEX           FEX                       FEX

Number    Description      State            Model            Serial

------------------------------------------------------------------------

101        FEX0101                Online    N2K-C2232TM-10GE   SSI15310HXS

<snip>

N7K-1-F3#

Eth101/1/1     --      eth  routed down    Administratively down     auto(D) --

Eth101/1/2     --      eth  routed down    Administratively down     auto(D) --

Eth101/1/3     --      eth  routed down    Administratively down     auto(D) --

 

Can breakout mode solve all problems? Unfortunately, no it cannot.  It can help you migrate from 10G to 40G over time at your pace in a very cost effective manner though!  Following are answers to some common questions about breakout that I’d like to address, based on conversations with many customers are below.

Can I connect a 1G device to a 10G breakout port?
No, the device must be 10G, but a 10G FEX can be connected to a breakout port and provide 1G connectivity.

Can I connect LX/LH/ER/ZR/DWDM oo other optics to a 10G breakout port? 
No, the optical QSFPs that support breakout (QSFP-40G-SR, QSFP-40G-CSR, FET-40G) require you to use the same type of optic on both ends. For example, FET to FET, CSR to CSR, SR to SR, etc.

Is breakout the same as BiDi optics?
No, BiDi solve the challenge by reducing the cabling required for 40G to 40G connections, while breakout focuses on 10G to 40G connectivity. BiDi optics are also supported on the Nexus 7000 and 7700 on both the F3 and M2 series modules.

Can I run breakout through my structured wiring system?
Certainly! It might look like the equipment below.

 

 

 

 

Hopefully you see the benefit of the breakout feature with the capability to run both 10G and 40G connectivity on the same module on a port by port basis with the F3 or M2 module. Routers, firewalls, switches, NAS appliances or servers can be connected to breakout ports without any issues.  Breakout is just one of many great capabilities of the F3 module.

 

 

Ron Fuller, CCIE No. 5851 (Routing and Switching/Storage Networking), is a technical marketing engineer (TME) on the Nexus 7000 team for Cisco. He has 21 years of experience in the industry and has held certifications from Novell, HP, Microsoft, ISC2, SNIA, and Cisco. Ron's focus is working with customers worldwide to address their challenges with comprehensive end-to-end data center architectures and how they can best use Cisco technology to their advantage.

Ron has had the opportunity to speak at Cisco Live on VDCs, NX-OS Multicast, and general design. He lives in Ohio with his wife and four wonderful children and enjoys travel and auto racing. He can be found on Twitter: @ccie5851.

 

NX-OS and Cisco Nexus Switching: Next-Generation Data Center Architectures, 2nd Edition

By Ron Fuller, David Jansen, Matthew McPherson.

Series: Networking Technology

Published: March 15, 2013

SBN-10: 0-13-288356-2

ISBN-13: 978-0-13-288356-6

Published by Cisco Press.

 

DISCOUNT: 35% off ciscopress.com for Cisco Technical Newsletter customers; use code CISCOTECH at checkout.

 

This article is featured in the April 2014 issue of the Cisco TS Newsletter. Are you subscribed?

 

 

Comments
mnagaboi
Cisco Employee
Cisco Employee

Thank you so much for sharing the information in a clear way. Really useful..

fly
Level 2
Level 2

does CRS-1 support 40G to 10G breakout, i see crs-1 support QSFP-40G-SR4

but i only find 100G support breakout on CRS-1, haven't find 40G support breakout

thank you!

Tom

marco.merlo
Level 1
Level 1

Thank you for this amazing post!

Indeed cisco docs on Nexus 7k reports

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/interfaces/configuration/guide/b-Cisco-Nexus-7000-Series-NX-OS-Interfaces-Configuration-Guide/conf-int-breakout-mode.html

Breakout mode is supported on following devices:

  • On Nexus 7000 - N7K-F312FQ-25, N7K-M206FQ-23L (NX-OS 6.2(6))

  • On Nexus 7700 - N77-F324FQ-25 (NX-OS 6.2.(8))

and

"When the breakout mode feature is configured, the module is reloaded and the configuration for the interface is removed"

but the same DOC on Nexus 9k says that module reload is not required when breakout is configured on a single port.

Now I need to breakout a port on a N77-F324FQ-25 on a VDC in a Nexus7700 running 6.2(6b).

Yout post says it's supported, Cisco doc says it's not but cisco CLI on the same SWITCH seems to support

 interface breakout module x port y map

command

So my questions are

1)Is actually breakout supported on N7700 running 6.2(6b)?

2)Is actually a moduel reload required?

3)Are there some restrictions when the card is shared among VDCs?

Regards

MM

Thank you for the good post.

can you advise me the correct QSFP model for N7K.

We are looking for single mode QSFP for N7K

NX-OS - 6.2(6)

Line Card: N7K-F312FQ-25

Hari

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Review Cisco Networking for a $25 gift card