topic Re: Nexus 5010 - not allowing more than 1 fex interface into a p in Server Networking

Nexus 5010 - not allowing more than 1 fex interface into a port-channel

dave.chadwick — Sat, 23 Jan 2010 18:13:21 GMT

Hi - I am trying to create a port-channel that contains 2 fex interfaces to provide SLB to a UNiX server, each time I try to add more that 1 interface into the Channel-group I get the following message:-"port channel cannot have more than 1 members". I have configured the fex interfaces up as trunk interfaces and have set the lacp to active mode.

interface port-channel62
switchport mode trunk

interface Ethernet107/1/37
description NAS DATAMOVER L cge2
switchport mode trunk
channel-group 62 mode active

NEXUS-2-5010-B(config-if)# int e107/1/38
NEXUS-2-5010-B(config-if)# channel-group 62 mode act
port channel cannot have more than 1 members

Please note that I am trying to configure multiple interfaces on the FEX not on the 5010...

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

Lucien Avramov — Sat, 23 Jan 2010 21:10:33 GMT

Here is your answer:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/configuration_guide_c07-543563.html

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

dave.chadwick — Sun, 24 Jan 2010 09:49:47 GMT

I'm not concerned about the vPC connections, I already have my vPC's configured from the 2148T FEX to the 5010's.. I'm talking about setting up an etherchannel connection to a server, which when configuring up the GEC the Nexus says "no more that 1 connection within the port-channel" .

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

steve_mils — Thu, 28 Jan 2010 11:00:54 GMT

Hi David,

I see exactly the same problem - did you manage to find an answer?

Thanks, Steve

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

rawlinsm — Mon, 01 Feb 2010 19:21:33 GMT

Same problem here.

I have 4 ports on a server divided evenly between 2 2048's

I would like all 4 ports to be seen to the server as an aggregated 4Gig connection that will use LACP.

If I run 1 connection from each 2048 and put them in the same VPC, No Problem.

When I try to add the second port on a 2048 to the same port-channel I get the same error.

Something more like this:

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

andy.steere — Mon, 01 Feb 2010 20:33:22 GMT

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

andy.steere — Mon, 01 Feb 2010 20:36:30 GMT

What happens when you create a new port-channel (different channel#) and add it to the same vPC? Like this:

N5k-1(config-if)# int ethernet 100/1/1

N5k-1(config-if)# channel-group 10

N5k-1(config-if)# int po10

N5k-1(config-if)# vpc 10

N5k-1(config-if)# int ethernet 100/1/2

N5k-1(config-if)# channel-group 11

N5k-1(config-if)# int po11

N5k-1(config-if)# vpc 10

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

rawlinsm — Mon, 01 Feb 2010 21:11:51 GMT

a VPC can only belong to a single Port-Channel

You get an error:

LVC-5020-1(config)# int po 50
LVC-5020-1(config-if)# vpc 50
LVC-5020-1(config-if)# int po 51
LVC-5020-1(config-if)# vpc 50
ERROR: Operation failed: [vPC already exists]

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

Lucien Avramov — Mon, 01 Feb 2010 21:19:50 GMT

Correst, it's one VPC per port-channel

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

rawlinsm — Mon, 01 Feb 2010 21:33:32 GMT

So, , , How do you bind multiple ports together to a server in the nexus environment?

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

alytle111 — Mon, 01 Feb 2010 21:58:04 GMT

Reported problem to TAC. They stated the current Nexus-OS does not support more than one interface per port-channel attached to a single 2k...if you want more than 2 ports per port-channel then use 5k ports.

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

Lucien Avramov — Thu, 04 Feb 2010 06:25:38 GMT

TAC should have also tell you that the product N2248T is the one that will support this feature.

I guess this post can be marked as resolved

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

slandeira — Wed, 22 Dec 2010 16:14:45 GMT

I have the same problem. Finally, how you solved it?

Thanks.

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

DAVIDE RENDINIELLO — Tue, 22 Feb 2011 10:20:08 GMT

Where I can find cisco's official documentation on this problem?

Is possible that NX-OS 5 has fixed it?

BR,

Dave

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

tiwang — Tue, 22 Feb 2011 10:34:14 GMT

I had the same problem - the only fix is to buy 2248 instead of 2148 - is some broken design in the asic's as far as I know

best regards /ti

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

vdsudame — Tue, 22 Feb 2011 16:10:10 GMT

There is nothing broken or any kind of design flaw here. This is a hardware limitation with the Nexus 2148 platform. This was our first FEX and at the time there was no support in n2k hardware to support more than one link in a portchannel from the fex to the host. Since this is a hardware limitation there wont be any software fix for it.

The current documentation is here:

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps10110/data_sheet_c78-507093.html

Table 9 - Layer 2 Features

PortChannel on server ports on Cisco Nexus 2200 platforms

Note: The platform says Nexus 2200. It does not account Nexus 2148.

Re: Nexus 5010 - not allowing more than 1 fex interface into a p

habadr — Sun, 27 Feb 2011 16:09:30 GMT

Hi David

You have to use 2 separate vPCs, each made of "2" interfaces and splitting vlans, or use server's TLB (Transmit Load balancing) as described in following design guide

http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9670/C07-572829-01_Design_N5K_N2K_vPC_DG.pdf

Page 37 and figure 29

Thanks

Hatim Badr

Nexus 5010 - not allowing more than 1 fex interface into a port-

Manas Dutta — Thu, 14 Mar 2013 17:01:59 GMT

Hello Lucein,

I am facing the same issue but while trying to connect my 2248 ports to NetApp FAS3200 Dual Controller.

The 10Gig connection works fine with vpc 13 and vpc 14 as follows (member ports are 5548 ports):

interface port-channel13

description 10G connection to NetApp_Controller1

switchport mode trunk

switchport trunk allowed vlan 161

spanning-tree port type edge trunk

speed 10000

duplex full

vpc 13

interface port-channel14

description 10G connection to NetApp_Controller2

switchport mode trunk

switchport trunk allowed vlan 161

spanning-tree port type edge trunk

speed 10000

vpc 14

But when I use for 2248 ports which are connected a dual-homed (EvPC) to both 5Ks, I get the error '

operation failed vpc already exists' when I try to use vpc 17 for e102/1/17 & e103/1/17 and vpc 18 for e102/1/18 & e103/1/18.

--------------------------------------------------------------------------------

Group Port- Type Protocol Member Ports

Channel

--------------------------------------------------------------------------------

17 Po17(SU) Eth LACP Eth102/1/17(P) Eth103/1/17(P)

18 Po18(SU) Eth LACP Eth102/1/18(P) Eth103/1/18(P)

However, they are up in PO right now but I can't add them to vpc. No matter what vpc number I use. It gives the same error.

Please help.

Nexus 5010 - not allowing more than 1 fex interface into a port-

Nicholas Sheridan — Tue, 09 Apr 2013 16:31:49 GMT

Hatim,

Good reference, many thanks.

Nik

Hi,I had the same problem,

Andrew Sparkes — Mon, 30 Jun 2014 11:47:19 GMT

Hi,

I had the same problem, but with slightly different models.

My setup used a Nexus 5548 and 2248 FEX.

The fix to the problem, shut down the uplink ports on the 5548 and disassociate the physical ports with the port channel and delete the port channel.

Next, reassign the two uplink ports to the FEX and recreate the port channel. This allowed me to apply the necessary configuration without error messages. I have provided an example below of the steps carried out.

show FEX - confirmation FEX is online and ports associated.

FEX: 104 Description: FEX104-Po104   state: Online
FEX version: 7.0(2)N1(1) [Switch version: 7.0(2)N1(1)]
FEX Interim version: 7.0(2)N1(1)
Switch Interim version: 7.0(2)N1(1)
E
Fabric interface state:
    Po104 - Interface Up. State: Active
    Eth1/18 - Interface Up. State: Active
    Eth1/26 - Interface Up. State: Active
Fex Port        State Fabric Port

show port-channel - confirmation port-channel has been built, but only one port assigned

104 Po104(SD) Eth NONE Eth1/18(D)

initial configuration applied and error message when trying to add second port to PO.

interface Ethernet1/18
description fex 104 port 1
switchport mode fex-fabric
fex associate 104
channel-group 104

interface Ethernet1/26
description fex 104 port 2
switchport mode fex-fabric
fex associate 104

conf t

int e1/26

channel-group 104

command failed: port not compatible [port dynamic logical-interface scale disable]

interface port-channel104
description fex 104
switchport mode fex-fabric
fex associate 104

The above shows the configuration applied and the error message recieved when trying to add second port to PO.

The next set of commands resolved the issue.

conf t

int e1/18
shut
switchport mode access
int e1/26
shut
switchport mode access
exit

no int po 104

Reapply configuration, but using port range command.

int e1/18 , e1/26
switchport mode fex-fabric
fex associate 104
channel-group 104

show commands confirm FEX returned to service and both ports in PO

sh fex

104 FEX104-Po104 Online N2K-C2248TP-E-1GE FOX1744G5TC

show port-channel summary
104 Po104(SU) Eth NONE Eth1/18(P) Eth1/26(P)

Hopefully the above will assist and is easy to follow.