ASK THE EXPERTS - NEXUS 5000 AND 2000 SERIES

[ciscomoderator]

Welcome to the Cisco Networking  Professionals Ask the Expert conversation. This is an opportunity to learn how to troubleshoot Nexus 5000 & 2000 series with Lucien Avramov.  Lucien Avramov is a Customer Support Engineer at the Cisco Technical  Assistance Center. He currently works in the data center switching team  supporting customers on the Cisco Nexus 5000 and 2000. He was previously  a technical leader within the network management team. Lucien holds a  bachelor's degree in general engineering and a master's degree in  computer science from Ecole des Mines d'Ales. He also holds the  following certifications: CCIE #19945 in Routing and Switching, CCDP,  DCNIS, and VCP #66183.

Remember to use the rating system to let Lucien know if you have received an adequate response.

Lucien might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered   questions in other discussion forums shortly after the event. This  event  lasts through August 30, 2010. Visit this forum often to view  responses  to your questions and the questions of other community  members.

[balsheikh]

Hi Lucien,

I have multiple concerns and appreciate your comment in this regard.

how is Nexus 5k along with extend fabric 2k can approve STP in a data center ?

is there  any plan down in the road map to have Nexus 2K supporting FCoE and Nexus 5k to be L3 switch ?

Thanks

Belal

[Lucien Avramov]

1.I'm not sure I understand what you mean by approve Spanning Tree Protocol.

There are two spanning tree features you can configure on the fabric extender: spanning-tree port type edge and spanning-tree port type edge trunk (if you have a trunk to an ESX server for example).

As far as how the 5k talks to the 2k for packet forwarding, it's via the VNTag. VNTag is a Network Interface Virtualization (NIV) technology. It allows the fabric extender to act as a data path of the Nexus 5000 for all policy and forwarding. It's added to the packet between the fabric extender and the Nexus 5k. Finally, it's stripped before the packet is sent to hosts.

2. You can have vpc and FCoE on the 2232. Also, there is a plan to have the Nexus 5k to have an L3 feature. Please consult with your Cisco system engineer / account team for the details.

More information on FCoE on the 2232, section Fibre Channel over Ethernet Support:

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/layer2/421_n2_1/Cisco_n5k_layer2_config_gd_rel_421_N2_1_chapter16.html

The datasheets are a good reference for the features supported:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/data_sheet_c78-461802.html

[tenaro.gusatu.novici]

Hi Lucien,

could you please make quick/short comparison between Nexus 5k with 2000.

Thanks,

Tenaro

[Lucien Avramov]

I'm glad you asked.

The Nexus 2000 also called the fabric extender is an extension of the Nexus 5000. It's configured from the Nexus 5000 directly.

Often times it's perceived as a lower end switch for data center due to it's nomination. It's actually not a switch, and it needs a Nexus 5000 to function.

The goal is to provide access to the server farm. As it's name fabric extender, you can see it as a NIC card extender between your server NICS to the Nexus 5000. It has up to 48 ports and there are 3 different kind of Nexus 2000: supporting 1GB with 2148, 100mg/1gb with 2248 and 10 GB to the NIC with the 2232.

The Nexus 2000 provides you more physical ports to the servers  avoiding to use the 10 GB dedicated ports on the Nexus 5000 for 1gb servers for instance. Also it simplifies the cabling, you can have server racks with a Nexus 2000 on top going to Nexus 5000s.

Product page (look at the video):

http://www.cisco.com/en/US/products/ps10110/index.html

Here are the datasheets for each:

http://www.cisco.com/en/US/products/ps10110/prod_models_home.html

White paper:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/white_paper_c11-594477.html

[tenaro.gusatu.novici]

Thanks for the great (short and precise) answer!

Just to complete this story, could you please confirm (or deny) my assumption: customer will be able to get all kind of ports (100/1000/10000 Mbps) thanks to different SFPs and later, when more ports will be needed customer can add Nexus 2000, right?

Regards,

Tenaro

[Lucien Avramov]

You are getting it, but let me be more precise, as it's not exactly what you said:

-The Nexus 5000 is a cut-trough switch with dedicated ports. Each port supports 10 GB speed. On the 5020, the first 16 ports you can set the speed to 1GB. On the 5010, the first 8 ports can be set to 1GB. You can connect 1GB servers to those ports.

-If you have 100 megs  servers, you need to use another device than the Nexus 5000. The best would be Nexus 2248 but another switch such as a Catalyst 3750-GE will work too (1GB capable so it can connect to the Nexus 5000).

-The Nexus 2148 and 2248 have 48 RJ-45 connections for the servers and 4 10 GE ports to the Nexus 5000.

Also, note that you have a great oversubcription rate on the Nexus 2000: for example the 2148/2248, have 48 1GB ports a total of 48 GBps. It also has 4 10 GB ports uplink to the Nexus 5000, totalling a 40GBps. This means that, economically it's better to use a Nexus 2000 connected to a Nexus 5000: this will use only 4 10 GB ports and you will be able to connect up to 48 servers instead of 8 for 5010 / 16 for 5020.

You may now wonder what SFPs are supported for the Nexus switches, so here is the compatibility matrix, look at data-center section:

http://www.cisco.com/en/US/docs/interfaces_modules/transceiver_modules/compatibility/matrix/OL_6974.html#wp55411

[tenaro.gusatu.novici]

Me again,

if it is not out of the scope, could you please compare Nexus 5000 vs Fiber Interconnect 6100? Can we say that one is a subset of another or each device has its own unique features?

For example, I know 5k can accept 1Gbps or 10Gbps connections from servers and also FC connections from storages; seems to me same can be achived on 6120XP, with appropriate SFPs, of course.

Regards,

Tenaro

[Lucien Avramov]

The architecture of the 6120 and the Nexus 5000 is similar.

However the use of them is different:

The 6120 / 6140 are dedicated devices for UCS-B chassis. They are configured trough the UCS manager that is a software residing on the 6100.

You can not connect other devices to it, whereas the Nexus 5000 is a switch and you can connect to it servers, switches 1GB and 10 GB or 6100s.

[tenaro.gusatu.novici]

This is very important for me and it will be great if you can confirm: storage (using FC interface) can't be connected directly to 6120XP, i.e. Nexus must be used to establish communication between the UCS and the storage?

[Lucien Avramov]

You can configure the 6100 in switch mode and connect it to storage. For some specifics such as NPIV you will need a Nexus 5000 or an MDS.

FCoE configuration:

http://www.cisco.com/en/US/products/ps10281/products_configuration_example09186a0080afd130.shtml

White paper:

http://www.cisco.com/en/US/prod/collateral/ps10265/ps10276/white_paper_c11-555663_ps10279_Products_White_Paper.html

[huangedmc]

Q1

When will the Nexus 2000's support spanning-tree?

If we have a blade enclosure w/ Cisco CGESM or 3100's where 10Gbps isn't required, we'd like the flexibility to uplink to Nexus 2K's.

However, we were told we'd need to filter BPDU's on the blade switches as a workaround; otherwise Nexus 2K's would err-disable the trunk ports.

That's a bit dangerous to run switches w/o STP.

Q2

If two servers in the same VLAN need to communicate to each other on the same Nexus 2K FEX, would packets be locally switched between the two ports on the FEX, or would they need to go to the Nexus 5K, and then back down to the FEX?

Q3

How come Cisco dropped the support for "write mem" & PAgP ether-channel on Nexus?

It's quite an inconvenience to have to do "copy run start", and since "write" is the command to erase NVRAM, we couldn't create a CLI alias that starts anything w/ "write".

One of the recommendations to avoid split-brain scenario w/ VSS is to use enhanced PAgP on Cat 6K's.

As a result, we'd have inconsistant configurations in the datacenter where PAgP is used on Cat 6K's, and LAcP is used on Nexus...

Not a big deal, but we'd like to keep things consistant.

Q4

As of today, you can only mesh between servers & FEX's, or between FEX's & 5K's, but not both.

Will there be support to do both in the future?

Q5

When FCoE is implemented, how does Nexus handle the priority queue for data traffic when there's congestion?

Obviously if the network is sized & configured properly, this wouldn't be an issue, but this is more a hypothetical question, and also to satisfy our storage team, to make sure we have a true lossless network to support their SAN.

Between the SAN traffic & priority queue, who gets forwarded first?

[mohsin.khan]

Hi Can you please compare the use of flexlinks as opposed to using STP?
What is the roadmap for using dual administration point for FEX, i.e 1 2k connected with active and redundant N5k, is it available today, or is it not even on the roadmap?

We have aggregated all our 10/100 links on 3560 and connected it to N5k using twingig convertor.. is it the right decision? Don't want to use core (6500 in our case) for seperate leg...

Do we have all COS features available on 2k/5k?

What importance does 5k have in the presence of N7k?

Lastly, kindly brief abt 1k (VM arch) and trasparent migration of services over VM stacks (don't know if i used the right terms).

regards,

Mohsin

[Lucien Avramov]

1.Flexlink is a fast convergence feature and allows one of the switchport interface to backup another switchport interface. It permits faster STPconvergence. Here are more details about it:

http://www.cisco.com/en/US/docs/switches/lan/catalyst3550/software/release/12.2_25_see/configuration/guide/swflink.html

2. You could configure active / passive fex by shutting down on of the uplinks to the fex on the Nexus 5000. If you have 2200s then you can do VPC to the FEX and VPC to the host, which is far a better option than active / passive, since with active / passive you loose uplink bandwidth.

3. Yes this is a good choice, except that if you go with Nexus 2248, then you can connect 100mg/1GB servers to it.

4. Regarding COS, it is supported and here is the configuration guide:

http://www.cisco.com/en/US/partner/docs/switches/datacenter/nexus5000/sw/qos/421_n2_1/Cisco_Nexus_5000_Series_NX-OS_Quality_of_Service_Configuration_Guide_Rel_421_N2_1_chapter3.html#con_1120879

5. The N7k can provide you with the distribution layer. You can have a fully VPC redundant HA design. You could if you like use a pair of catalyst 6k in VSS upstream, however the Nexus 7000 will provide you much better fabric performance and oversubcription rate.

6. The Cisco 1000v supports VN-Link which provides:

-Policy-based virtual machine connectivity

-Mobile virtual machine security and network policy

-Non-disruptive operational model for your server virtualization and networking teams

Here are a few good documents:

Data Sheets:

http://www.cisco.com/en/US/products/ps9902/products_data_sheets_list.html

White papers:

http://www.cisco.com/en/US/products/ps9902/prod_white_papers_list.html

Q&A:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9902/qa_c67-556624.html

[Lucien Avramov]

Q1:

Nexus 2000 is not a switch, it's a fabric extender. Picture it as a hardware extension of your server NICs. For this reason it's not running spanning-tree.

You can however configure the spanning tree type as edge port trunk and connect a server with a trunk to the FEX.

Q2:

The packets will always go from the Nexus 2000 to the Nexus 5000 and back.

Q3:

Nexus runs a different operating system, called NX-OS. It's a data center focused feature set for mission critical environments, 24x7 continuous operations, high density / performance ethernet and data center specific link-layer types.

It is different, so it does have advantages, such as being able to run any command from anywhere in configuration mode, and it does miss certain commands such as the write mem.

However, as I long time IOS user, I can tell you that even write mem was too long to type, and using ''wr'' was faster.

Good news is that you can actually create an alias for wr on NX-OS:

SJ-SV-N5K-3(config)# cli alias name wr copy r s

SJ-SV-N5K-3(config)# wr

[########################################] 100%

Regarding PaGP as you said it's not supported at this point and LACP is the recommended. I don't have more information as why it was decided to proceed this way. My guessing is that it was what was first demanded by the industry, or more largely deployed in the data centers.

Q4:

Yes it's in the works. I would suggest you to contact your system engineer / account team for the details and the timelines.

Q5:

FCoE traffic is marked on Nexus 5000 with a COS of 3. It's allocated a separate priority-queue and buffers for this traffic. It's considered as high priority traffic for the switch and preceeds the rest of the traffic when a congestion occurs. F