Nexus 5548UP N2K-B22HP-P status offline!?

rbko_ciscocco · ‎09-30-2015

Dear all,

We have two Nexus 5548UP (5.2(1)N1(9)) vPC connected in Dual-Homed Fabric-Extender B22HP (4 SFP cable from B22HP, two in each N5k switch) . In first N5k vPC switch everything works fine, but in secondd N5k switch the FEX status is Offline

FEX FEX FEX FEX
Number Description State Model Serial
------------------------------------------------------------------------
103 FEX0103 Offline N2K-B22HP-P FOC1906R0GM

104 FEX0104 Offline N2K-B22HP-P FOC1906R0K6

Below you can find the logs of port-channel 103 and FEX 103:

2015 Sep 30 10:51:48 nexsw-02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel103: first operational port changed from Ethernet1/10 to Ethernet1/9
2015 Sep 30 10:51:48 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 3 of Fex 103 that is connected with Ethernet1/10 changed its status from Active to Disconnected
2015 Sep 30 10:51:48 nexsw-02 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel103: Ethernet1/10 is down
2015 Sep 30 10:51:48 nexsw-02 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/10 is down (Link failure)
2015 Sep 30 10:51:48 nexsw-02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel103: first operational port changed from Ethernet1/9 to none
2015 Sep 30 10:51:48 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 4 of Fex 103 that is connected with Ethernet1/9 changed its status from Active to Disconnected
2015 Sep 30 10:51:48 nexsw-02 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel103: Ethernet1/9 is down
2015 Sep 30 10:51:48 nexsw-02 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel103: port-channel103 is down
2015 Sep 30 10:51:48 nexsw-02 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel103 is down (No operational members)
2015 Sep 30 10:51:48 nexsw-02 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/9 is down (Link failure)
2015 Sep 30 10:51:48 nexsw-02 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel103 is down (No operational members)
2015 Sep 30 10:51:49 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 3 of Fex 103 that is connected with Ethernet1/10 changed its status from Disconnected to Configured
2015 Sep 30 10:51:49 nexsw-02 %ETHPORT-3-IF_NON_CISCO_TRANSCEIVER: Non-Cisco transceiver on interface Ethernet1/10 is detected
2015 Sep 30 10:51:49 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 4 of Fex 103 that is connected with Ethernet1/9 changed its status from Disconnected to Configured
2015 Sep 30 10:51:49 nexsw-02 %ETHPORT-3-IF_NON_CISCO_TRANSCEIVER: Non-Cisco transceiver on interface Ethernet1/9 is detected
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-SPEED: Interface Ethernet1/10, operational speed changed to 10 Gbps
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/10, operational duplex mode changed to Full
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/10, operational Receive Flow Control state changed to off
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/10, operational Transmit Flow Control state changed to off
2015 Sep 30 10:51:52 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 0 of Fex 103 that is connected with Ethernet1/10 changed its status from Configured to Fabric Up
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-SPEED: Interface Ethernet1/9, operational speed changed to 10 Gbps
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/9, operational duplex mode changed to Full
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/9, operational Receive Flow Control state changed to off
2015 Sep 30 10:51:52 nexsw-02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/9, operational Transmit Flow Control state changed to off
2015 Sep 30 10:51:52 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 0 of Fex 103 that is connected with Ethernet1/9 changed its status from Configured to Fabric Up
2015 Sep 30 10:51:53 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 3 of Fex 103 that is connected with Ethernet1/10 changed its status from Fabric Up to Connecting
2015 Sep 30 10:51:53 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 3 of Fex 103 that is connected with Ethernet1/10 changed its status from Connecting to Active
2015 Sep 30 10:51:53 nexsw-02 %ETHPORT-5-SPEED: Interface port-channel103, operational speed changed to 10 Gbps
2015 Sep 30 10:51:53 nexsw-02 %ETHPORT-5-IF_DUPLEX: Interface port-channel103, operational duplex mode changed to Full
2015 Sep 30 10:51:53 nexsw-02 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel103, operational Receive Flow Control state changed to off
2015 Sep 30 10:51:53 nexsw-02 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel103, operational Transmit Flow Control state changed to off
2015 Sep 30 10:51:53 nexsw-02 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel103: Ethernet1/10 is up
2015 Sep 30 10:51:53 nexsw-02 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel103: first operational port changed from none to Ethernet1/10
2015 Sep 30 10:51:53 nexsw-02 %ETHPORT-5-IF_UP: Interface port-channel103 is up in mode Fex Fabric
2015 Sep 30 10:51:53 nexsw-02 %ETHPORT-5-IF_UP: Interface Ethernet1/10 is up in mode Fex Fabric
2015 Sep 30 10:51:54 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 4 of Fex 103 that is connected with Ethernet1/9 changed its status from Fabric Up to Connecting
2015 Sep 30 10:51:54 nexsw-02 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 4 of Fex 103 that is connected with Ethernet1/9 changed its status from Connecting to Active
2015 Sep 30 10:51:54 nexsw-02 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel103: Ethernet1/9 is up
2015 Sep 30 10:51:54 nexsw-02 %ETHPORT-5-IF_UP: Interface Ethernet1/9 is up in mode Fex Fabric

Do anyone have any idea what could be the problem?

BR

Enis

Mark Malone · ‎09-30-2015

Hi can you post the show fex 103 detail from both 5ks please

This was working fine before yes , no recent changes to config or physical changes to network?

rbko_ciscocco · ‎09-30-2015

Hi Mark,

Thanks for your response,

This is new installation.

Output of show fex from first N5k:

nexsw-01# show fex 103 detail
FEX: 103 Description: FEX0103 state: Online
FEX version: 5.2(1)N1(9) [Switch version: 5.2(1)N1(9)]
FEX Interim version: 5.2(1)N1(9)
Switch Interim version: 5.2(1)N1(9)
Extender Serial: FOC1906R0K6
Extender Model: N2K-B22HP-P, Part No: 73-13780-05
Bay: 1
Rack: HP_HO_RACK1
Enclosure: OA_ENC02_HO
Enclosure Serial: CZ35304W9M
Rack Id: Default RUID
Card Id: 145, Mac Addr: 84:b8:02:af:ef:c2, Num Macs: 64
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/7
FCoE Admin: true
FCoE Oper: true
FCoE FEX AA Configured: true
Fabric interface state:
Po103 - Interface Up. State: Active
Eth1/7 - Interface Up. State: Active
Eth1/8 - Interface Up. State: Active
Fex Port State Fabric Port
Eth103/1/1 Up Po103
Eth103/1/2 Up Po103
Eth103/1/3 Up Po103
Eth103/1/4 Up Po103
Eth103/1/5 Up Po103
Eth103/1/6 Down Po103
Eth103/1/7 Down Po103
Eth103/1/8 Down Po103
Eth103/1/9 Down Po103
Eth103/1/10 Down Po103
Eth103/1/11 Down Po103
Eth103/1/12 Down Po103
Eth103/1/13 Down Po103
Eth103/1/14 Down Po103
Eth103/1/15 Down Po103
Eth103/1/16 Down Po103
Logs:
09/28/2015 14:41:33.581603: Module register received
09/28/2015 14:41:33.582705: Registration response sent
09/28/2015 14:41:33.694533: Deleting route to FEX
09/28/2015 14:41:33.702564: Module disconnected
09/28/2015 14:41:33.703267: Module Offline
09/28/2015 14:41:33.705038: Deleting route to FEX
09/28/2015 14:41:33.711875: Module disconnected
09/28/2015 14:41:33.712894: Offlining Module
09/28/2015 14:41:33.726173: Deleting route to FEX
09/28/2015 14:41:33.733293: Module disconnected
09/28/2015 14:41:33.734281: Offlining Module
09/28/2015 14:42:03.592864: Module timed out
09/28/2015 14:42:06.920398: Module register received
09/28/2015 14:42:06.921495: Registration response sent
09/28/2015 14:42:07.398694: Module Online Sequence
09/28/2015 14:42:09.461537: Module Online

and output from second N5k:

nexsw-02# show fex 103 detail
FEX: 103 Description: FEX0103 state: Offline
FEX version: 5.2(1)N1(9) [Switch version: 5.2(1)N1(9)]
FEX Interim version: 5.2(1)N1(9)
Switch Interim version: 5.2(1)N1(9)
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/10
FCoE Admin: false
FCoE Oper: false
FCoE FEX AA Configured: true
Fabric interface state:
Po103 - Interface Up. State: Active
Eth1/9 - Interface Up. State: Active
Eth1/10 - Interface Up. State: Active
Logs:
09/30/2015 13:53:29.191914: Deleting route to FEX
09/30/2015 13:53:29.200751: Module disconnected
09/30/2015 13:53:29.202236: Offlining Module
09/30/2015 13:53:58.930998: Module timed out
09/30/2015 13:54:04.355255: Module register received
09/30/2015 13:54:04.363827: Registration response sent
09/30/2015 13:54:04.595202: Deleting route to FEX
09/30/2015 13:54:04.603638: Module disconnected
09/30/2015 13:54:04.604187: Module Offline
09/30/2015 13:54:04.605885: Deleting route to FEX
09/30/2015 13:54:04.613195: Module disconnected
09/30/2015 13:54:04.623703: Offlining Module
09/30/2015 13:54:04.631917: Deleting route to FEX
09/30/2015 13:54:04.639416: Module disconnected
09/30/2015 13:54:04.640586: Offlining Module
09/30/2015 13:54:34.373513: Module timed out
09/30/2015 13:54:37.841750: Module register received
09/30/2015 13:54:37.849741: Registration response sent
09/30/2015 13:54:38.97158: Deleting route to FEX
09/30/2015 13:54:38.103967: Module disconnected
09/30/2015 13:54:38.105033: Module Offline
09/30/2015 13:54:38.106970: Deleting route to FEX
09/30/2015 13:54:38.114146: Module disconnected
09/30/2015 13:54:38.116220: Offlining Module
09/30/2015 13:54:38.127120: Deleting route to FEX
09/30/2015 13:54:38.134133: Module disconnected
09/30/2015 13:54:38.138355: Offlining Module
09/30/2015 13:55:07.853492: Module timed out
09/30/2015 13:55:11.137147: Module register received
09/30/2015 13:55:11.145320: Registration response sent
09/30/2015 13:55:11.375026: Deleting route to FEX
09/30/2015 13:55:11.384406: Module disconnected
09/30/2015 13:55:11.385040: Module Offline
09/30/2015 13:55:11.386487: Deleting route to FEX
09/30/2015 13:55:11.393316: Module disconnected
09/30/2015 13:55:11.398037: Offlining Module
09/30/2015 13:55:11.416694: Deleting route to FEX
09/30/2015 13:55:11.423753: Module disconnected
09/30/2015 13:55:11.427449: Offlining Module
09/30/2015 13:55:41.151033: Module timed out
09/30/2015 13:59:44.904091: Module register received
09/30/2015 13:59:44.912227: Registration response sent
09/30/2015 13:59:45.147259: Deleting route to FEX
09/30/2015 13:59:45.154482: Module disconnected
09/30/2015 13:59:45.155467: Module Offline
09/30/2015 13:59:45.157340: Deleting route to FEX
09/30/2015 13:59:45.164498: Module disconnected
09/30/2015 13:59:45.165953: Offlining Module
09/30/2015 13:59:45.179396: Deleting route to FEX
09/30/2015 13:59:45.186903: Module disconnected
09/30/2015 13:59:45.188582: Offlining Module
09/30/2015 14:00:14.923504: Module timed out
09/30/2015 14:00:49.281975: Module register received
09/30/2015 14:00:49.290062: Registration response sent
09/30/2015 14:00:49.524784: Deleting route to FEX
09/30/2015 14:00:49.532200: Module disconnected
09/30/2015 14:00:49.532764: Module Offline
09/30/2015 14:00:49.535344: Deleting route to FEX
09/30/2015 14:00:49.542202: Module disconnected
09/30/2015 14:00:49.543198: Offlining Module
09/30/2015 14:00:49.562885: Deleting route to FEX
09/30/2015 14:00:49.569866: Module disconnected
09/30/2015 14:00:49.571517: Offlining Module
09/30/2015 14:01:19.293787: Module timed out

Mark Malone · ‎09-30-2015

Ok if its new need a bit more make sure config is ok first , I have similar device like one of these running in my DC and they need to be set as vpc , here's an example config of mine that's worked for me

This needs to be identical on each 5k in terms of config, if your config is already correct I would look at the physical connections try shut the fex 103 down on the 2nd 5k and bring it back up just in case something went wrong while it was communicating with the 5k

interface Ethernet2/3
description FEX140
switchport mode fex-fabric
fex associate 140
logging event port link-status
channel-group 140

5k1# sh run int po140

interface port-channel140
description FEX1140
switchport mode fex-fabric
fex associate 140
vpc 140

5k1# sh fex | i 140
140 FEX0140 Online N2K-B22DELL-P FOC1543R412

rbko_ciscocco · ‎09-30-2015

Hi Mark, I have almost similar config in my devices. Down you can find configs of both N5k's.

Config on first N5k (working fex):

nexsw-01# sh run int eth 1/7 - 8

!Command: show running-config interface Ethernet1/7-8
!Time: Wed Sep 30 17:17:07 2015

version 5.2(1)N1(9)

interface Ethernet1/7
switchport mode fex-fabric
fex associate 103
channel-group 103

interface Ethernet1/8
switchport mode fex-fabric
fex associate 103
channel-group 103

nexsw-01# sh run int port-channel 103

!Command: show running-config interface port-channel103
!Time: Wed Sep 30 17:17:24 2015

version 5.2(1)N1(9)

interface port-channel103
switchport mode fex-fabric
fex associate 103
vpc 103

nexsw-01# sh fex | inc 103
103 FEX0103 Online N2K-B22HP-P FOC1906R0K6

Config on second N5k (Offline FEX-es):

nexsw-02# sh run interface ethernet 1/9 - 10

!Command: show running-config interface Ethernet1/9-10
!Time: Wed Sep 30 17:16:25 2015

version 5.2(1)N1(9)

interface Ethernet1/9
switchport mode fex-fabric
fex associate 103
channel-group 103

interface Ethernet1/10
switchport mode fex-fabric
fex associate 103
channel-group 103

nexsw-02# sh run int port-channel 103

!Command: show running-config interface port-channel103
!Time: Wed Sep 30 17:16:33 2015

version 5.2(1)N1(9)

interface port-channel103
switchport mode fex-fabric
fex associate 103
vpc 103

nexsw-02# sh fex | inc 103
103 FEX0103 Offline N2K-B22HP-P FOC1906R0GM

Mark Malone · ‎10-01-2015

config looks fine , has the server been checked to make sure its definitely set for nic teaming on its end

can you post show vpc consistency-parameters vpc 103 / show port-channel summary
under the fex 103 you can also try add the type ----type NB22HP

when you disconnect it from 5k2 and then reconnect does it show anything useful in the logs relating to it

are you using the same cabling/fex connectors on each side , is it going by patch panel or is it direct to 5k2

http://www.cisco.com/c/en/us/products/collateral/switches/nexus-b22-blade-fabric-extender/guide_c07-686089.html

rbko_ciscocco · ‎10-26-2015

Hello,

Just to inform that the issue has been solved by changing the cables. This article was very helpful to me also: http://mycciedatacenter.blogspot.com/2013/06/fex-identity-mismatch-identity-mismatch.html,

prho-nexsw-01# show interface fex-fabric
Fabric Fabric Fex FEX
Fex Port Port State Uplink Model Serial
---------------------------------------------------------------
103 Eth1/7 Active 1 N2K-B22HP-P FOC1906R0K6
103 Eth1/8 Active 2 N2K-B22HP-P FOC1906R0K6
104 Eth1/9 Active 4 N2K-B22HP-P FOC1906R0GM
104 Eth1/10 Identity-Mismatch 3 N2K-B22HP-P FOC1906R0K6

Swapping cables with second switch solved my issue.

Thanks for your help,