FEX configuration on Nexus 7010

s-durando · ‎11-22-2010

Hi,

I have a Nexus 7K running 5.1.(1a) with N2248T as Fabric Extender.

I had two ports on N7K-M132XP-12 module connected to N2248T.

In order to activated fex feature I issued the following commands :

install feature-set fex
feature-set fex

When I tried to configure the 10G ports as fabric-extender on N7K I had the following error:

Switch(config)# interface Ethernet10/27
Switch(config-if)# switchport
Switch(config-if)# switchport mode fex-fabric
ERROR: Ethernet10/27: Configuration does not match the port capability.

What is missing in my configuration?

Regards

Stefano

Jerry Ye · ‎11-22-2010

Since the configuration restriction said all port in the same port group will need to be in fex-fabric mode. What does the config for other ports in this port group looks like?

I am interested on port 25, 27, 29 and 31.

Regards,

jerry

s-durando · ‎11-23-2010

Ports in the same port-group have different configurations so I'm going to change the config in order to use 25,27,29,31 as fabric extender ports.

Thanks a lot

Stefano

s-durando · ‎12-02-2010

Jerry,

I tried to configure all port 25, 27, 29 and 31 in fex-fabric mode but I have always the same message:

'Configuration does not match the port capability'

I also followed this documentation but it doesn't work.

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/interfaces/Configuring_the_Cisco_Nexus_2000_Series_Fabric_Extender_chapter1.html

Any suggestion?

Thanks

Stefano

Jerry Ye · ‎12-02-2010

Can you post the config for all these 4 ports?

Regards,

jerry

s-durando · ‎12-02-2010

Jerry,

port config is very simple:

interface Ethernet10/25
switchport

interface Ethernet10/27
switchport

interface Ethernet10/29
switchport

interface Ethernet10/31
switchport

Regards

Stefano

Jerry Ye · ‎12-02-2010

Try this:

interface e10/25 , e10/27 , e10/29 , e10/31

switchport mode fex-fabric

Regards,

jerry

s-durando · ‎12-02-2010

I tried but I had always the same message:

Switch(config)# interface e10/25 , e10/27 , e10/29 , e10/31
Switch(config-if-range)# switchport mode fex-fabric
ERROR: Ethernet10/25, Ethernet10/27, Ethernet10/29, Ethernet10/31: Configuration does not match the port capability.

Switch(config-if-range)#

I tried also to configure port-channel before and then assign physical interfaces to port-channel

interface port-channel10
switchport
switchport mode fex-fabric
fex associate 101

Switch(config-if)# interface e10/25 , e10/27 , e10/29 , e10/31
Switch(config-if-range)# channel-group 10
command failed: port not compatible [Ethernet Layer]

** You can use force option to override the port's parameters
** (e.g. "channel-group X force")
** Use "show port-channel compatibility-parameters" to get more information on failure

Switch(config-if-range)#

Regards

Stefano

Jerry Ye · ‎12-02-2010

Can you post the following command output:

show run | i feature

show run int e10/25

show run int e10/27

show run int e10/29

show run int e10/31

show int e10/25 cap

show int e10/27 cap

show int e10/29 cap

show int e10/31 cap

show ver

show fex

Regards,

jerry

s-durando · ‎12-03-2010

Jerry,

you find below the info you requested.

Thanks

Stefano

Switch# sh running-config | i feature
install feature-set fex
allow feature-set fex
feature-set fex
feature telnet
feature udld
feature interface-vlan
feature hsrp
feature lacp
feature vpc
feature vtp

Switch# sh running-config interface ethernet 10/25

!Command: show running-config interface Ethernet10/25
!Time: Fri Dec 3 19:04:56 2010

version 5.1(1a)

interface Ethernet10/25
switchport

Switch# sh running-config interface ethernet 10/27

!Command: show running-config interface Ethernet10/27
!Time: Fri Dec 3 19:05:15 2010

version 5.1(1a)

interface Ethernet10/27
switchport

Switch# sh running-config interface ethernet 10/29

!Command: show running-config interface Ethernet10/29
!Time: Fri Dec 3 19:05:19 2010

version 5.1(1a)

interface Ethernet10/29
switchport

Switch# sh running-config interface ethernet 10/31

!Command: show running-config interface Ethernet10/31
!Time: Fri Dec 3 19:05:23 2010

version 5.1(1a)

interface Ethernet10/31
switchport

Switch# sh interface ethernet 10/25 capabilities
Ethernet10/25
Model:                 N7K-M132XP-12
Type (SFP capable):    10Gbase-SR
Speed:                 10000
Duplex:                full
Trunk encap. type:     802.1Q
Channel:               yes
Broadcast suppression: percentage(0-100)
Flowcontrol:           rx-(off/on),tx-(off/on)
Rate mode:             dedicated/shared
QOS scheduling:        rx-(8q2t),tx-(1p7q4t)
CoS rewrite:           yes
ToS rewrite:           yes
SPAN:                  yes
UDLD:                  yes
Link Debounce:         yes
Link Debounce Time:    yes
MDIX:                  no
Pvlan Trunk capable:   no
Port Group Members:    25,27,29,31
TDR capable:           no
FabricPath capable:    no
Port mode:             Routed,Switched
FEX Fabric:            no
dot1Q-tunnel mode:     yes

Switch# sh interface ethernet 10/27 capabilities
Ethernet10/27
Model:                 N7K-M132XP-12
Type (SFP capable):    10Gbase-SR
Speed:                 10000
Duplex:                full
Trunk encap. type:     802.1Q
Channel:               yes
Broadcast suppression: percentage(0-100)
Flowcontrol:           rx-(off/on),tx-(off/on)
Rate mode:             shared
QOS scheduling:        rx-(8q2t),tx-(1p7q4t)
CoS rewrite:           yes
ToS rewrite:           yes
SPAN:                  yes
UDLD:                  yes
Link Debounce:         yes
Link Debounce Time:    yes
MDIX:                  no
Pvlan Trunk capable:   no
Port Group Members:    25,27,29,31
TDR capable:           no
FabricPath capable:    no
Port mode:             Routed,Switched
FEX Fabric:            no
dot1Q-tunnel mode:     yes

Switch# sh interface ethernet 10/29 capabilities
Ethernet10/29
Model:                 N7K-M132XP-12
Type (SFP capable):    10Gbase-LR
Speed:                 10000
Duplex:                full
Trunk encap. type:     802.1Q
Channel:               yes
Broadcast suppression: percentage(0-100)
Flowcontrol:           rx-(off/on),tx-(off/on)
Rate mode:             shared
QOS scheduling:        rx-(8q2t),tx-(1p7q4t)
CoS rewrite:           yes
ToS rewrite:           yes
SPAN:                  yes
UDLD:                  yes
Link Debounce:         yes
Link Debounce Time:    yes
MDIX:                  no
Pvlan Trunk capable:   no
Port Group Members:    25,27,29,31
TDR capable:           no
FabricPath capable:    no
Port mode:             Routed,Switched
FEX Fabric:            no
dot1Q-tunnel mode:     yes

Switch# sh interface ethernet 10/31 capabilities
Ethernet10/31
Model:                 N7K-M132XP-12
Type (SFP capable):    10Gbase-(unknown)
Speed:                 10000
Duplex:                full
Trunk encap. type:     802.1Q
Channel:               yes
Broadcast suppression: percentage(0-100)
Flowcontrol:           rx-(off/on),tx-(off/on)
Rate mode:             shared
QOS scheduling:        rx-(8q2t),tx-(1p7q4t)
CoS rewrite:           yes
ToS rewrite:           yes
SPAN:                  yes
UDLD:                  yes
Link Debounce:         yes
Link Debounce Time:    yes
MDIX:                  no
Pvlan Trunk capable:   no
Port Group Members:    25,27,29,31
TDR capable:           no
FabricPath capable:    no
Port mode:             Routed,Switched
FEX Fabric:            no
dot1Q-tunnel mode:     yes

Switch# sh version
Cisco Nexus Operating System (NX-OS) Software
TAC support: http://www.cisco.com/tac
Documents: http://www.cisco.com/en/US/products/ps9372/tsd_products_support_series_home.html
Copyright (c) 2002-2010, Cisco Systems, Inc. All rights reserved.
The copyrights to certain works contained in this software are
owned by other third parties and used and distributed under
license. Certain components of this software are licensed under
the GNU General Public License (GPL) version 2.0 or the GNU
Lesser General Public License (LGPL) Version 2.1. A copy of each
such license is available at
http://www.opensource.org/licenses/gpl-2.0.php and
http://www.opensource.org/licenses/lgpl-2.1.php
BIOS:      version 3.22.0
kickstart: version 5.1(1a)
system:    version 5.1(1a)
BIOS compile time:       02/20/10
kickstart image file is: bootflash:///n7000-s1-kickstart.5.1.1a.bin
kickstart compile time: 12/25/2020 12:00:00 [11/04/2010 23:14:37]
system image file is:    bootflash:///n7000-s1-dk9.5.1.1a.bin
system compile time:     11/4/2010 11:00:00 [11/05/2010 00:25:15]

Software

Hardware
cisco Nexus7000 C7010 (10 Slot) Chassis ("Supervisor module-1X")
Intel(R) Xeon(R) CPU with 4115812 kB of memory.
Processor Board ID JAF1436ANLK

Device name: DEA2
bootflash: 2029608 kB
slot0: 2074214 kB (expansion flash)

Kernel uptime is 1 day(s), 3 hour(s), 30 minute(s), 9 second(s)

Last reset at 148502 usecs after Thu Dec 2 15:35:31 2010

Reason: Reset Requested by CLI command reload
System version: 5.1(1a)
Service:

plugin
Core Plugin, Ethernet Plugin

CMP (Module 5) ok
CMP Software
CMP BIOS version:        02.01.05
CMP Image version:       5.1(1) [build 5.0(0.66)]
CMP BIOS compile time:   8/ 4/2008 19:39:40
CMP Image compile time: 9/2/2010 19:00:00

CMP (Module 6) ok
CMP Software
CMP BIOS version:        02.01.05
CMP Image version:       5.1(1) [build 5.0(0.66)]
CMP BIOS compile time:   8/ 4/2008 19:39:40
CMP Image compile time: 9/2/2010 19:00:00

Switch# sh fex
FEX         FEX           FEX                       FEX
Number    Description      State            Model            Serial
------------------------------------------------------------------------
---       --------               Offline     N2K-C2248TP-1GE   JAF1436CCAH

Jerry Ye · ‎12-03-2010

The part highlighted in RED is the problem:

Switch# sh interface ethernet 10/25 capabilities
Ethernet10/25
Model:                 N7K-M132XP-12
Type (SFP capable):    10Gbase-SR
Speed:                 10000
Duplex:                full
Trunk encap. type:     802.1Q
Channel:               yes
Broadcast suppression: percentage(0-100)
Flowcontrol:           rx-(off/on),tx-(off/on)
Rate mode:             dedicated/shared
QOS scheduling:        rx-(8q2t),tx-(1p7q4t)
CoS rewrite:           yes
ToS rewrite:           yes
SPAN:                  yes
UDLD:                  yes
Link Debounce:         yes
Link Debounce Time:    yes
MDIX:                  no
Pvlan Trunk capable:   no
Port Group Members:    25,27,29,31
TDR capable:           no
FabricPath capable:    no
Port mode:             Routed,Switched
FEX Fabric:            no
dot1Q-tunnel mode:     yes

My next questions are how old was this N7K and did you upgrade from a previous version to 5.1.1a? When you upgrade, did you upgrade the EPDL image also?

Regards,

jerry

s-durando · ‎12-03-2010

This N7K was shipped with NX-OS 5.0 and I upgraded to 5.1(1a) without upgrading EPDL image

Stefano

Jerry Ye · ‎12-03-2010

You should upgrade the EPLD image also. As you can see from the following link, the EPLD image is different from the software.

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/epld/epld_rn_5.1.html

Regards,

jerry

Network Pro · ‎10-07-2011

hi,

i have the same issue and just followed what you have said.

when i try to upgrade the epld on a 10gb module i get the msg

EPLD upgrade failed. Exiting..
Failure reason: Chain connectivity is broken while upgrading the FPGA

i have tried updating this on another module (1gb) and thats seems to be fine.

any thoughts on this pls ?

thanks