Re: Ask the Expert: Cisco Nexus 2000, 5000, and 6000 Series Swit - Page 3

ciscomoderator · ‎06-28-2013

with Cisco Expert Vinayak Sudame

Welcome to the Cisco Support Community Ask the Expert conversation. This is an opportunity to learn and ask questions how to configure and troubleshoot the Cisco Nexus 2000, 5000 and 6000 Series Switches with Cisco subject matter expert Vinayak Sudame. You can ask any question on configuration, troubleshooting, features, design and Fiber Channel over Ethernet (FCoE).

Vinayak Sudame is a Technical Lead in Data Center Switching Support Team within Cisco's Technical Services in RTP, North Carolina. His current responsibilities include but are not limited to Troubleshooting Technical support problems and Escalations in the areas of Nexus 5000, Nexus 2000, FCoE. Vinayak is also involved in developing technical content for Cisco Internal as well as external. eg, Nexus 5000 Troubleshooting Guide (CCO), Nexus 5000 portal (partners), etc. This involves cross team collaboration and working with multiple different teams within Cisco. Vinayak has also contributed to training account teams and partners in CAE (Customer Assurance Engineering) bootcamp dealing with Nexus 5000 technologies. In the past, Vinayak's responsibilities included supporting MDS platform (Fiber Channel Technologies) and work with EMC support on Escalated MDS cases. Vinayak was the Subject Matter Expert for Santap Technologies before moving to Nexus 5000 support. Vinayak holds a Masters in Electrical Engineering with Specialization in Networking from Wichita State University, Kansas. He also holds Cisco Certification CCIE (#20672) in Routing and Switching.

Remember to use the rating system to let Vinayak know if you have received an adequate response.

Vinayak might not be able to answer each question due to the volume expected during this event. Remember that you can continue the conversation on the Data Center sub-community, Other Data Center Topics discussion forum shortly after the event.

This event last through Friday July 12, 2013. Visit the community often to view responses to youe questions of other community members.

crtide · ‎07-10-2013

Hello Steve,

I am not referring to vPC on the host ports, I am actually trying to bundle ports on same 2k to form a channel-group.

The 5k is a Nexus5548 running version 5.2(1)N1(1) and the 2k is a C2248TP-E-1GE.

Please see the error message I get when I try to add a second interface to the channel-group below

channel-group xxx mode active command failed: port not compatible [max members on FEX exceeded]

vdsudame · ‎07-10-2013

Hi Bashir

How many ports are you trying to bundle in the port channel on the fex ? Also can you paste in the configuration of existing fex port channel and one member interface which is already in the port channel and the interface which you are trying to bundle into the existing port channel ?

Thanks, Vinayak

crtide · ‎07-10-2013

Hello Vinayak,

Although I am trying to bundle up to 8x1Gb ports when I try to add more than one port on same 2k I get the error message. Also I gave the wrong switch spec above cos I logged on to the wrong switch to take my specifications. The switch with the issue is a Nexus 5020 running version: 5.2(1)N1(2a).

the config of the port channel is

interface port-channel210

switchport mode trunk

switchport trunk native vlan 204

switchport trunk allowed vlan 202

no logging event port link-status

no snmp trap link-status

vpc 210

configuration of a member port is below

interface Ethernet105/1/9
description TDC2SOLC002 OB NET 0
switchport mode trunk
switchport trunk native vlan 204
switchport trunk allowed vlan 202
no snmp trap link-status
no logging event port link-status
channel-group 210 mode active

PO summary info below

        --------------------------------------------------------------------------------
Group Port-       Type     Protocol Member Ports
      Channel
--------------------------------------------------------------------------------
210   Po210(SD)   Eth      LACP      Eth105/1/9(I)

Could it be that with a vPC, you can only have one channel member per 2k ?

vdsudame · ‎07-10-2013

ok.. if this is a 5020, what model of fex it is ? Is it still a 2248 or 2148 ? We dont support more than one member in a port channel on 2148 fex. that was our first series of fexes which did not support more than one port in a port channel. please check if its a 2148 fex.

Thanks, Vinayak

crtide · ‎07-11-2013

Hello Vinayak,

Thanks, the 2k is a N2K-C2148T-1GE so that confirms it can only do one member per 2k which supports the error message displayed. Are you saying the limitation will not exist if the 2k were to be 2248? I thought the 2k being a remote linecard will have the capabilities given to it by the parent 5k

Thanks

Bash

vdsudame · ‎07-11-2013

Bashir

This is a hardware limitation is only with 2148 FEX, independent of the parent 5k switch. This is documented in our guides for quite a while now.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/mkt_ops_guides/513_n1_1/n5k_enhanced_vpc.html#wp1175308

The Cisco Nexus 2148 device does not support port channels. With the Cisco Nexus 2148 device, the host vPC can have up to two ports with one from each Cisco Nexus 2148.

The Cisco Nexus 2248, Cisco Nexus 2224, Cisco Nexus 2232 and Cisco Nexus 2248TP-E devices support hardware port channels and up to 16 ports in a host vPC with up to 8 ports from each FEX.

You can check our Configuration Limits guide on CCO for number of fex port channels supported on 2248.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/configuration_limits/limits_521/nexus_5000_config_limits_521.html

Thanks, Vinayak.

huangedmc · ‎07-10-2013

hi Vinayak,

Questions regarding ISSU:

Just to confirm...when a N5K performs ISSU upgrade, neither the 5K, nor the 2K/FEX will reboot?

Will they just start running w/ the new version when ISSU is completed, or is a reboot required ?

===

We've had Nexus 7K/5K/2K's for several years, but were never able to get the 5K & 2K's to perform ISSU.

It appears if a port is in non-edge, designated forwarding SPT state, the code upgrade needs to be disruptive.

There are two reasons why we have such ports on our N5K's:

1. Usually the N7K's are the root bridge for most of the VLAN's, but sometimes a 5K is the bridge for a VLAN because that particular VLAN only needs to be present on that 5K.

This results in one of the uplinks to 7K's in non-edge, designated forwarding state.

2. We have several blade chassis switches connected to the 5K's.

One of the ports facing those switches are also in non-edge, designated forwarding state. (not edge P2P because they're not host ports)

How do we get around this, and achieve ISSU?

N7K-----N7K

| |

N5K-----N5K

| |

Blade------Blade

thx

Kevin

vdsudame · ‎07-10-2013

Hi Kevin

Your understanding is correct.

If the switch supports ISSU, then both 5k and 2k will be upgraded non-disruptively without any reboots. The control plane of 5k and 2k gets upgraded, its called hitless upgrade.

For ISSU to Proceed, Check the Following Criteria :

1. No Topology change must be active in any STP instance

2. Bridge assurance(BA) should not be active on any port (except MCT)

3. There should not be any Non Edge Designated Forwarding port (except MCT)

4. ISSU criteria must be met on the VPC Peer Switch as well

If any of the above criteria is not met, ISSU will not be successful, unfortunately.

This is as per the guidelines in cisco documentation:

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/upgrade/503_N1_1/n5k_upgrade_downgrade_503.html#wp640746

which you might be already be aware of. Based on the conditions you pointed out in your post, in that topology ISSU will not be possible, unfortunately.

You will need to redesign your topology in such a way that the criteria above are met and then only you will be able to perform ISSU. Unfortunately this is how it is.

Thanks, Vinayak

huangedmc · ‎07-10-2013

Thanks for the reply, but could you please tell me how other customers work around this issue?

We're not the only customer who connect blade switches to Nexus 5K, are we?

Do they simply make the ports between 5K & blade switches edge ports, and hope no loop will occur?

vdsudame · ‎07-10-2013

correct .. there are lot of customers with blade switches most of them run with VPCs.

Yes, you can configure the 5k base ports going to blade server switches as edge trunk ports but we are exposed to loops as well. So certain customers have requirements of running spanning tree between nexus and blade switches so they cant really move them to edge ports.

For the 1st criteria you mentioned, if its a FCoE vlan spanning tree should already be disabled for that vlan and it should not interrupt ISSU process. But if its not FCoE Vlan and it is defined only on one switch then only way i know of is to disable spanning tree for that vlan before doing ISSU. We dont suggest or prefer doing that in our best practice documentation but that is the only way where we can fulfill the criteria for ISSU to succeed.

It might be best to have a vpc environment in your setup so switches can have full redundancy ie double sided vpc between 7ks and 5ks and 5ks and blade switches by cross connecting them and thereby eventhough you perform disruptive upgrades, we have another path for traffic to pass through.

Thanks, Vinayak

muca · ‎07-10-2013

Hi Vinayak

Is there a switchport protected equivalent command on the Nexus switches?

Thanks

Murilo

vdsudame · ‎07-10-2013

Murilo,

We dont have an exact equivalent command to "switchport protected" but if we configure the switch with private vlans and configure ports in isolated mode it would server similar purpose.

http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/layer2/521_n1_1/b_Nexus_5000_Layer2_Config_521N11_chapter_0100.html

Thanks, Vinayak

Ask the Expert: Cisco Nexus 2000, 5000, and 6000 Series Switches