FEX with Nexus 7000 connection

[clark white]

Dears,

Please find the attached state of a fex

Nexus:version 7.3(1)D1(1)

we have a FE of 2232 model connecting to N7k cross connection means 1 no's of N2k  uplink to N7k1 and another uplink to N7K2. i have configured the below configuration for the fex, Please confirm m i missing anything becz the fex stucks in connected state.

Anybody can confirm to me the cross connection from FE to N7K are supported becz, if i shutdown the link from N7K1 the N7K2 link shows me online, if both links are up then only on one N7k it show online. as per the below link it is supported.

http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html

interface Ethernet1/48
switchport
switchport mode fex-fabric
fex associate 112
channel-group 112

interface port-channel112
switchport
switchport mode fex-fabric
fex associate 112

thanks

[B.kablawi90]

Hello,

First of all please send the log of show fex details

As I understood, you have single fex connected up to two N7K.

Are your trying to make vPC ? I don't think that dual home vPC active/active is supported in 7K

You mentioned only one port (ether1/48) configured in the port channel (po12)

Have added the port of the other 7K in the port channel ?

[clark white]

Dears

attached are the logs when i configure the boxes by the below configuration, when i connect a laptop on FE it shows me interface not connected

interface Ethernet1/48
switchport
switchport mode fex-fabric
fex associate 112
channel-group 112
no shutdown

interface port-channel112
switchport
switchport mode fex-fabric
fex associate 112
vpc 112

The show fex details doesn't show anything related to the fex which i m working becz the fex is in connected state and not online.

I m trying to achieve the below Dual Homed Host (Active/Standby) and Active-Active FEX (VPC) Design according to the link provided. http://www.cisco.com/c/en/us/support/docs/switches/nexus-2000-series-fabric-extenders/200363-Nexus-2000-Fabric-Extenders-Supported-Un.html#anc6

yes i have added the port in the another Nexus also.

Can u write for the me the configuration for the active active vpc design as per the diagram shown in the link above.

thanks

[B.kablawi90]

I have limited resource as i'm out of the office now, but currently the log is referring to peerlink issue which suspend the fex.

Another think, have you forced the speed or it's auto negotiation ?

interface Ethernet104/1/1 are being suspended. (Reason: Peer-link is not UP)

NX7KCSW-1-CORE %ETHPORT-5-SPEED: Interface Ethernet104/1/1, operational speed changed to 1 Gbps

Will get you back.

[clark white]

Dear 

thanks for the reply,

Another think, have you forced the speed or it's auto negotiation ?

hardcoded the speed but no success

(Reason: Peer-link is not UP) ?? which peer link it is showing as not up ?? as i can see the vpc brief everything is seen as a success.

can you write for me the correct configuration for dual home FE

thanks

[B.kablawi90]

One of the common reason for FEX connect state is the speed misconfiguration, to verify, check below example

ex:

N5K1(Config)# feature fex

N5K1(Config)#  int e1/4 - 5

N5K1(Config-int-range)# switchport mode fex

N5K1(Config-int-range)# fex associate 101

N5K1(Config-int-range)#  no shut

N5K1(Config-int-range)#end

N5K1# sho fex (See Attached Nu1)

FEX STATUS CONNECTED

N5K1(Config-if)# sho int e101/1/1  status (See Attached Nu2)

YOU PROBALLY WILL SEE  sfpInvali IN THE STATUS

sfpIvali means a problem with the SFP it self  or sped problem we can verify it by the following command

N5K1# show int e101/1/1 transceiver (See Attached Nu2)

IF YOU GOT THE SAME JUST ADJUST THE SPEED FORCE IT TO BE 1G OR 10G

THEN

sho int e101/1/1 status (See Attached Nu3)

WILL GET THE STATUS OF THE INTERFACE CHANGED TO CONNECTED AND THE FEX STATE IMAGE DOWNLOAD (See Attached Nu4)

vPC Active/Active, I have tried on 5K but as your 7K version 7.3 it must be supported

-Remember Fex in 7K always must be in port channel

-The configuration must be the same in both boxes and the interface numbers too

First of all make sure the fexs are working without the vPC, then shutdown the interfaces toward the fex from on of the boxes during the configuration

I draw simple topology (See Attached 5)

N7K1

VPC KEEPALIVE LINK

vpc domain 7

vpc peer-keepalive destination (THE IP OF THE N7K2 I USED THE MNGT INT IP)

end

show vpc

VPC PEERLINK CONFIG

int e1/1 - 3

shutdown

channel-group 70 mode active

int po 70

vpc peer-link

switchport mode trunk

panning-tree port type network

speed 1000

no shut

FEX CONFIG

THE OTHER VPC MEMBER IN THE N5K2 SHOULD BE SHUTDOWN WHILE ADDING THE FEXs TO THE FIRST SWITCH

int e1/4-5

shut

channel-group 701 mode on  DOWN TO FABRIC MUST BE ON IT DOSEN'T SUPPORT LCAP

int  pot 701

vpc 501

switchport mode fex-fabric

fex associate 101

int e1/4-5

no shutdown

end

N7K2

VPC KEEPALIVE LINK

vpc domain 7

vpc peer-keepalive destination (THE IP OF THE N7K1 I USED THE MNGT INT IP)

end

show vpc

VPC PEERLINK CONFIG

int e1/1 - 3

shutdown

channel-group 70 mode active

int po 70

vpc peer-link

switchport mode trunk

panning-tree port type network

speed 1000

no shut

FEX CONFIG

int e1/4-5

shut

channel-group 701 mode on 

int  pot 701

vpc 501

switchport mode fex-fabric

fex associate 101

int e1/4-5

no shutdown

end

I hope that I could help you and provide some support to your issue, if not i'm happy to participate in this community, anyway keep me posted with your final solution

Wish you all the best

[clark white]

Dear 

thanks for the reply i have configured the same way as you have mentioned the FEX configuration, my another FEX is coming up and when i connnect my PC it works fine but the one which i was discussing above is still the same, it comes up online but when i connect any pc no effect  ( link not connected) .

i have 2 doubts for the logs which i get on console when configuring FEX.

1. Configuring vPC will loss all FEX configuration. Continue (yes/no)? [no] yes

the above message is received on the console when i enter the vpc command in the port-channel interface

2.CORE(config-if)# interface eth2/34

CORE(config-if)# channel-group 40

command failed: port not compatible [Members should have same fex config]

please correct me if i m wrong the above error is becz of port group?? if i am assigning port e1/4-5 on F3 in channel-group 2 then the port 1/1-2 should also be a part of the same port channel., it cannot be configured with different port channels??

thanks

[B.kablawi90]

You are welcome any time :)

Now the problem became more clear.

The main prerequisite of vPC Active/Active FEX is to have the configuration fully synchronized in both parent switches, as they are two separate control planes and both of these control planes managing the same downstream fex that means the config of N7K-1 must be exactly a copy of N7K-2 configuration everything must be the same 

Each change happen in one of the parent switch must configured equally in the another switch

Even the ports numbers of the member ports must be the same in both boxes and same port group number

So that you got this message, command failed: port not compatible [Members should have same fex config]

Using config sync feature help to to auto sync the config between the switches via CFSoIP 

Answering your question, No, it means if you have N7k-1 port e1/4-5 in channel-group 5, then the N7k-2 must has e1/4-5 in channel-group 5
and N7k-1 e1/1-2 in channel-group 10, then the N7k-2 must has port e1/1-2 in channel-group 10 

Please consider that port toward your laptop must be access port type edge, and consider using vpc peer-gateway if you will use FHRP

below links will help you

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/design_guide_c07-625857.html#_Toc271759476

check vPC with Fabric Extender Active-Active Design section

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/configuration_guide_c07-543563.html

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus2000/sw/configuration/guide/b_Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_7_x/b_Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_rel_7_x_chapter_011.html

http://www.cisco.com/c/dam/en/us/td/docs/switches/datacenter/sw/design/vpc_design/vpc_best_practices_design_guide.pdf


 

[clark white]

Dear 

thank for the insight information but there is small confusion.

Even the ports numbers of the member ports must be the same in both boxes and same port group number

so you are confirming me that if in N7k1 if we are using port e1/1-2 for Po11 then on N7k2 also it should be e1/1-2 and po11,

F3 cards are having a port group of 4 ports (1,2,3,4) so they cannot be split to two port channels ??

Virtual port channels (vPC) topologies require identical configurations on peer switches. As a result, you must repeat configurations on both peer switches. This process, which can cause errors due to misconfigurations or omissions, can result in additional service disruptions because of mismatched configurations. Configuration synchronization eliminates these problems by allowing you to configure one switch and automatically synchronize the configuration on the peer switch.

In a vPC topology, each Cisco Nexus 7000 Series switch must have some matching parameters. You can use a vPC consistency check to verify that both Cisco Nexus 7000 Series switches have the same configuration (Type 1 or Type 2). If they do not match, depending on whether it is a global (for example, spanning-tree port mode), a port-level (for example, speed, duplex, or channel-group type), or even a port-channel interface, the vPC can go into a suspended state or a VLAN can go into a blocking state on both peer switches. As a result, you must ensure that the configuration from one switch is copied identically to the peer switch.

Configuration synchronization allows you to synchronize the configuration between a pair of switches in a network. Configuration synchronization and vPCs are two independent features and configuration synchronization does not eliminate vPC consistency checks. The checks will continue. If there is a configuration mismatch, the vPC can still go into a suspended state

by enabling config-sync it will sync switch 1 config to switch 2 and vice versa so for example if i have a layer 3 connectivity on port e1/1 of switch 1 to an external router so it will replicate to the switch 2 on port e1/1 though i have only one link to the router so this means i will lose one port on switch 2 if config sync is enabled,

OR

config sync is only applicable for layer 2 ports and vPC and global commands on switch.

Thanks

[B.kablawi90 · Accepted solution]

so you are confirming me that if in N7k1 if we are using port e1/1-2 for Po11 then on N7k2 also it should be e1/1-2 and po11

Yes

"F3 cards are having a port group of 4 ports (1,2,3,4) so they cannot be split to two port channels ??"

I did not get your question very well, draw me your topology if possible

You need portchannel between the chassis to run vpc features and for data plane traffic, and another portchannel down to the FEXs

"by enabling config-sync it will sync switch 1 config to switch 2 and vice versa so for example if i have a layer 3 connectivity on port e1/1 of switch 1 to an external router so it will replicate to the switch 2 on port e1/1 though i have only one link to the router so this means i will lose one port on switch 2 if config sync is enabled"

If you have layer 3 connectivity, you will need to use FHRP and vpc advanced command such as peer-gateway in vpc domain configuration 

Check out this article (7K Active/Active vpc with HSRP enabled) 

https://supportforums.cisco.com/document/98811/peer-gateway-feature-nexus-7000

"config sync is only applicable for layer 2 ports and vPC and global commands on switch"

Config sync is not tied to a single protocol, its for the whole configuration designed for active/active vpc scenario to prevent split bran when running two control planes

[clark white]

Dear

thanks for your replies,

as it is confirmed by you that if i m using e1/1 and e1/2 in N7K1 for Po1 then the same should be used on N7K2 for Po1 but remaining ports e1/3 and e1/4 they can be used as Po2 on N7k1 and N7k2.

why i am asking for remaining ports becz they belong to same port-group as what i know that in F2 cards port-groups cannot be split in two port-channels but in F3 cards we can split in 2 port-channels ???

thanks

[B.kablawi90]

The port-group refers to the ability of utilizing the capacity which means not every port in the line card has 40G. the 40G divided on 4 ports within the port-group which must be in configured in shared mode. if you want to use 40G for single port you must configure it it as a dedicated #rate-mode dedicated (only one port with 40G will be active and the rest three ports in the port-gourp will be disabled)

So, I don't see any problem in doing two port-channel in a single port-group. If it did not work simply choose another ports belong to another port-group in the line card.

in M1 cards the ports arranged in port-group like this
Port-gourp:1 (1,3,5,7)
Port-gourp:2 (2,4,6,8)

in F1 cards the ports arranged in port-group like this
Port-group:1 (1,2)
Port-gourp:2 (3,4)

in F2 cards the ports arranged in port-group like this
Port-group:1 (1,2,3,4)
Port-group:2 (3,4,5,6)

in F3 cards the ports arranged in port-group like this
Port-group:1 (1,2,3,4,5,6,7,8)
Port-group:2 (9,10,...........16)

In case if you are using vDC all the ports within port-group must be in the same vDC

 
I haven't heard any limitation for the port-channels within the port group, try to split the port-group in two port-channels and check the error-message if any

Good luck.