Nexus 6K not seeing FEX 2248T

[CiscoMedMed]

I have a pair of Nexus 6001 switches and I believe both are connected to a pair of FEX2248. Port Eth 2/3 and 2/4 on each show configuration for FEX 100 and 101 respectively. However if I run a show fex 100 or 101 detail on N6K-1 nothing comes up but -2 I do see the fex. show interface status shows port 100/x/x and 101/x/x on N6k-2. There are a few ports in 100/x/x in UP status yet no mac addresses are visible in show mac address-table for those ports. Questions..

-Is there a way to vet physical connections to the Fex from each switch? 2/3 and 2/4 on N6K-1 show up/connected sw show int status and vlan 1. Show cdp nei appears to not apply to FEX. Anyhow that's the first question as the data center's 100 miles away. How can I remotely vet the cabling connections?
-Are FEX meant to be connected to a pair of switches in a redundancy cross-hatch A-C, B-C, A-D, B-D?
Or would they normally just extend one individual switch? 
-Should normal NXOS commands like sho mac address-table work for devices connected to the FEX?

These were setup before my arrival and I'm just getting around to investigating these oddities.

[balaji.bandi]

check show fex 100 or show fex 101 give you the port connected in the switch

example :

# show fex 153

Pinning-mode: static Max-links: 1
Fabric port for control traffic: Eth1/14
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Po153 - Interface Up. State: Active
Eth1/6 - Interface Up. State: Active
Eth1/14 - Interface Up. State: Active

[CiscoMedMed]

Thank you. The show commands on 6k-1 show no fex found. It looks like these never got cabled or configured correctly. The ports on the 6k-1 are configured for fex and show in up status. So I wonder how they're cabled. Something must be putting the ports in an up state. Mysteries of the previous regime!

NX6K-1# sho fex 100 detail
FEX 100 not found
NX6K-1# sho fex 101 detail
FEX 101 not found

NX6K2#sho fex 100 detail
FEX: 100 Description: 2348_10G_FEX1 state: Online
FEX version: 7.0(4)N1(1) [Switch version: 7.0(4)N1(1)]
FEX Interim version: 7.0(4)N1(1)
Switch Interim version: 7.0(4)N1(1)
Extender Serial: FOC2043R2SJ
Extender Model: N2K-C2348UPQ-10GE, Part No: 73-15489-04
Card Id: 265, Mac Addr: 00:eb:d5:05:a9:c2, Num Macs: 96
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 4
Fabric port for control traffic: Eth2/3
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Eth2/3 - Interface Up. State: Active

NX6K-2# sho fex 101 detail
FEX: 101 Description: 2348_10G_FEX2 state: Online
FEX version: 7.0(4)N1(1) [Switch version: 7.0(4)N1(1)]
FEX Interim version: 7.0(4)N1(1)
Switch Interim version: 7.0(4)N1(1)
Extender Serial: FOC2043R2S3
Extender Model: N2K-C2348UPQ-10GE, Part No: 73-15489-04
Card Id: 265, Mac Addr: 00:59:dc:52:d1:02, Num Macs: 96
Module Sw Gen: 21 [Switch Sw Gen: 21]
post level: complete
Pinning-mode: static Max-links: 4
Fabric port for control traffic: Eth2/4
FCoE Admin: false
FCoE Oper: true
FCoE FEX AA Configured: false
Fabric interface state:
Eth2/4 - Interface Up. State: Active

[balaji.bandi]

Can you post the config to help along with show modules to look, if you lookingfor help

if the nexus 6K VPC send both the device information.

[CiscoMedMed]

Below are the version, show inventory and port configs for the VPC peering connectivity between the NX6K's and the ports ostensibly configured for the FEX. 

system: version 7.0(4)N1(1)

01# show inventory
NAME: "Chassis", DESCR: "Nexus 6001 Chassis"
PID: N6K-C6001-64P , VID: V01 , SN: FOC1804R02X

NAME: "Module 1", DESCR: "Nexus 64 Supervisor"
PID: N6K-C6001-64P , VID: V01 , SN: FOC1803930B

NAME: "Module 2", DESCR: "Nexus 4xQSFP Ethernet Module"
PID: N6K-C6001-M4Q , VID: V01 , SN: FOC1803930B

02: sho inventory
NAME: "Chassis", DESCR: "Nexus 6001 Chassis"
PID: N6K-C6001-64P , VID: V01 , SN: FOC1804R02F

NAME: "Module 1", DESCR: "Nexus 64 Supervisor"
PID: N6K-C6001-64P , VID: V01 , SN: FOC180392WW

NAME: "Module 2", DESCR: "Nexus 4xQSFP Ethernet Module"
PID: N6K-C6001-M4Q , VID: V01 , SN: FOC180392WW

NAME: "FEX 100 CHASSIS", DESCR: "N2K-C2348UPQ-10GE CHASSIS"
PID: N2K-C2348UPQ-10GE , VID: V02 , SN: FOC2043R2SJ

NAME: "FEX 100 Module 1", DESCR: "Fabric Extender Module: 48x10GE, 24x10GE/6x40G QSFP Supervisor"
PID: N2K-C2348UPQ-10GE , VID: V02 , SN: FOC20433S9F

NAME: "FEX 101 CHASSIS", DESCR: "N2K-C2348UPQ-10GE CHASSIS"
PID: N2K-C2348UPQ-10GE , VID: V02 , SN: FOC2043R2S3

NAME: "FEX 101 Module 1", DESCR: "Fabric Extender Module: 48x10GE, 24x10GE/6x40G QSFP Supervisor"
PID: N2K-C2348UPQ-10GE , VID: V02 , SN: FOC20433S1S

NX6K-1:

interface Ethernet2/1
description VPC Peer Link
switchport mode trunk
channel-group 1 mode active

interface Ethernet2/2
description VPC Peer Link
switchport mode trunk
channel-group 1 mode active

interface Ethernet2/3
switchport mode fex-fabric
fex associate 100

interface Ethernet2/4
switchport mode fex-fabric
fex associate 101

NX6K-2

interface Ethernet2/1
description VPC Peer Link
switchport mode trunk
channel-group 1 mode active

interface Ethernet2/2
description VPC Peer Link
switchport mode trunk
channel-group 1 mode active

interface Ethernet2/3
switchport mode fex-fabric
fex associate 100

interface Ethernet2/4
switchport mode fex-fabric
fex associate 101

[Christopher Hart · Accepted solution]

Hello!

Based on the output provided so far, it sounds like you have the following topology. Note that I've purposefully only included FEX-100 in this topology for simplicity - it sounds like the topology is similar for FEX-101 as well.

+----------------+    Po1 - VPC Peer-Link    +----------------+
|     NX6K-1     +---------------------------+     NX6K-2     |
|                +---------------------------+                |
|                |         Eth2/1-2          |                |
+----------------+                           +----------------+
        |Eth2/3                                 Eth2/3|
        |                                             |
        |              +----------------+             |
        |      Uplink 1|    FEX-100     |Uplink 2     |
        +--------------+                +-------------+
                       |                |
                       +----------------+

Typically with this sort of topology, the FEX is dual-homed to both parent Nexus switches. This requires the FEX fabric interfaces on the parent switches (in your case, Eth2/3) to be bundled into a port-channel (let's call it port-channel100), which is then defined as a vPC (e.g. by configuring vpc 100 under port-channel100) on both vPC peers.

With your setup, the physical topology looks like a dual-homed setup - however, the configuration itself reflects a single-homed setup, as the parent switch's FEX fabric interfaces are physical interfaces instead of vPCs. As a result, it may be worth spending the time to convert this configuration into a full dual-homed setup - you'll gain additional redundancy in the event of a FEX fabric interface failure, and you'll decrease the amount of confusion this topology will create for other network engineers!

With respect to your original questions - it is expected behavior for a FEX's uplink interfaces to bring links up/up when connected to a parent switch with unconfigured or misconfigured physical interfaces. Unfortunately, until the parent switch's interface is properly configured as a FEX fabric interface, there's no easy way to determine that the remote partner on the other end of the link is indeed a FEX.

I hope this helps - please let us know if you have any other questions. Thank you!

-Christopher

[CiscoMedMed]

Exceptionally helpful. Thank you!

[balaji.bandi · Accepted solution]

follow below example : both the side for 100 and 101.

interface Ethernet2/3
switchport mode fex-fabric
fex associate 100
channel-group 100

interface port-channel100
switchport mode fex-fabric
fex associate 100
vpc 100