FEX link redundancy...WTF...

[Andreas Wittemann]

Hi community,

a customer of us just removed one of his two ports used to link a FEX to it Nexus5K. Because he wanted to change the cabling a bit.

His idea was, since he has two "uplinks" to the bigger switch, all should be fine.

For sure, a FEX is no switch, and the uplink was no PortChannel, like he has this in his other switches configured on his campus.

But what made me wonder too...the FEX completely went offline, and sort of rebooted.

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet1/29 is down (Link failure)

2015 Jul  1 14:27:17 NEXUS1 last message repeated 1 time

2015 Jul  1 14:27:17 NEXUS1 %FEX-5-PINNING_CHANGED: Fex 101 pinning information is changed

2015 Jul  1 14:27:17 NEXUS1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 7 of Fex 101 that is connected with Ethernet1/29 changed its status from Active to Disconnected

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/9 is down (None)

2015 Jul  1 14:27:17 NEXUS1 last message repeated 1 time

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/10 is down (None)

2015 Jul  1 14:27:17 NEXUS1 last message repeated 1 time

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/1 is down (None)

2015 Jul  1 14:27:17 NEXUS1 last message repeated 1 time

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/2 is down (None)

2015 Jul  1 14:27:17 NEXUS1 last message repeated 1 time

2015 Jul  1 14:27:17 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel16: Ethernet101/1/11 is down

2015 Jul  1 14:27:17 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel16: port-channel16 is down

2015 Jul  1 14:27:17 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel17: Ethernet101/1/12 is down

2015 Jul  1 14:27:17 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel17: port-channel17 is down

2015 Jul  1 14:27:17 NEXUS1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel16: first operational port changed from Ethernet101/1/11 to none

2015 Jul  1 14:27:17 NEXUS1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel17: first operational port changed from Ethernet101/1/12 to none

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel16 is down (No operational members)

2015 Jul  1 14:27:17 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel17 is down (No operational members)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/11 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel16 is down (No operational members)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/12 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel17 is down (No operational members)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/32 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/31 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/30 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/29 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/28 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/27 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/26 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/25 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/24 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/23 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/22 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/21 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/20 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/19 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/18 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/17 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/9 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/10 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/1 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/2 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/3 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/4 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/5 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/6 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/7 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/8 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/16 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/15 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/14 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/13 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/11 is down (None)

2015 Jul  1 14:27:18 NEXUS1 %ETHPORT-5-IF_DOWN_NONE: Interface Ethernet101/1/12 is down (None)

2015 Jul  1 14:27:19 NEXUS1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 7 of Fex 101 that is connected with Ethernet1/29 changed its status from Disconnected to Configured

2015 Jul  1 14:27:51 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet1/29, operational speed changed to 10 Gbps

2015 Jul  1 14:27:51 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/29, operational duplex mode changed to Full

2015 Jul  1 14:27:51 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/29, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:51 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/29, operational Transmit Flow Control state changed to off

2015 Jul  1 14:27:51 NEXUS1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 0 of Fex 101 that is connected with Ethernet1/29 changed its status from Configured to Fabric Up

2015 Jul  1 14:27:51 NEXUS1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 7 of Fex 101 that is connected with Ethernet1/29 changed its status from Fabric Up to Connecting

2015 Jul  1 14:27:51 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet1/29 is up in mode Fex Fabric

2015 Jul  1 14:27:51 NEXUS1 %FEX-5-PINNING_CHANGED: Fex 101 pinning information is changed

2015 Jul  1 14:27:51 NEXUS1 %FEX-5-FEX_PORT_STATUS_NOTI: Uplink-ID 7 of Fex 101 that is connected with Ethernet1/29 changed its status from Connecting to Active

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/2, operational speed changed to 1 Gbps

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/2, operational duplex mode changed to Full

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/2, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/2, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/2 is up in mode access

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/12, operational speed changed to 10 Gbps

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/12, operational duplex mode changed to Full

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/12, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/12, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-SPEED: Interface port-channel17, operational speed changed to 10 Gbps

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface port-channel17, operational duplex mode changed to Full

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel17, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel17, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/11, operational speed changed to 10 Gbps

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/11, operational duplex mode changed to Full

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/11, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/11, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-SPEED: Interface port-channel16, operational speed changed to 10 Gbps

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface port-channel16, operational duplex mode changed to Full

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel16, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:58 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel16, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/10, operational speed changed to 10 Gbps

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/10, operational duplex mode changed to Full

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/10, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/10, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/10 is up in mode access

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_DOWN_LINK_FAILURE: Interface Ethernet101/1/10 is down (Link failure)

2015 Jul  1 14:27:59 NEXUS1 last message repeated 1 time

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/9, operational speed changed to 10 Gbps

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/9, operational duplex mode changed to Full

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/9, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/9, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/9 is up in mode access

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/1, operational speed changed to 1 Gbps

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/1, operational duplex mode changed to Full

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/1, operational Receive Flow Control state changed to off

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/1, operational Transmit Flow Control state changed to on

2015 Jul  1 14:27:59 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/1 is up in mode access

2015 Jul  1 14:28:02 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel16: Ethernet101/1/11 is up

2015 Jul  1 14:28:02 NEXUS1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel16: first operational port changed from none to Ethernet101/1/11

2015 Jul  1 14:28:02 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/11 is up in mode trunk

2015 Jul  1 14:28:02 NEXUS1 %ETHPORT-5-IF_UP: Interface port-channel16 is up in mode trunk

2015 Jul  1 14:28:03 NEXUS1 %ETH_PORT_CHANNEL-5-PORT_UP: port-channel17: Ethernet101/1/12 is up

2015 Jul  1 14:28:03 NEXUS1 %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel17: first operational port changed from none to Ethernet101/1/12

2015 Jul  1 14:28:03 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/12 is up in mode trunk

2015 Jul  1 14:28:03 NEXUS1 %ETHPORT-5-IF_UP: Interface port-channel17 is up in mode trunk

2015 Jul  1 14:28:04 NEXUS1 %ETHPORT-5-SPEED: Interface Ethernet101/1/10, operational speed changed to 10 Gbps

2015 Jul  1 14:28:04 NEXUS1 %ETHPORT-5-IF_DUPLEX: Interface Ethernet101/1/10, operational duplex mode changed to Full

2015 Jul  1 14:28:04 NEXUS1 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet101/1/10, operational Receive Flow Control state changed to off

2015 Jul  1 14:28:04 NEXUS1 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet101/1/10, operational Transmit Flow Control state changed to on

2015 Jul  1 14:28:04 NEXUS1 %ETHPORT-5-IF_UP: Interface Ethernet101/1/10 is up in mode access

# # # # # # # # # # # # # # #

Configuration of the N5K:

interface Ethernet1/29

  description UPLINK 1  FEX 101

  switchport mode fex-fabric

  fex associate 101

interface Ethernet1/30

  description UPLINK 2  FEX 101

  switchport mode fex-fabric

  fex associate 101

NEXUS1# sh fex 101

FEX: 101 Description: FEX0101   state: Online

  FEX version: 7.0(5)N1(1) [Switch version: 7.0(5)N1(1)]

  Extender Serial: xxxxxxxxxxxx

  Extender Model: N2K-C2232TM-E-10GE,  Part No: 73-14645-04

  Pinning-mode: static    Max-links: 1

  Fabric port for control traffic: Eth1/30

  FCoE Admin: false

  FCoE Oper: true

  FCoE FEX AA Configured: false

  Fabric interface state:

    Eth1/29 - Interface Up. State: Active

    Eth1/30 - Interface Up. State: Active

NEXUS1# sh fex

  FEX         FEX           FEX              FEX              Fex

Number    Description      State            Model            Serial

------------------------------------------------------------------------

101        FEX0101                Online  N2K-C2232TM-E-10GE   xxxxxxxxxxx

Any ideas how i could achieve redundancy between the two uplinks to the FEX so that such things can happen without this issue?

Many thanks in advance,

Andreas