Nexus 5624Q and 2248TP-E, FEX interfaces not showing up

[Rollen]

This is driving me crazy and I hope some one can answer this. I'm working with a Nexus 5624Q and a 2248TP-E FEX. I'm trying to get the 5k to communicate with the 2k and show the interfaces. Below I included my configuration. I've done the break out command, I've done the port channel, I've connected all 4 10G connections to the uplinks and they are all showing green but then drop to link not connected after a few minutes. I've read every documentation and tried max links 4 with out port channel and even attempted to try VPC but still cannot get the fex interfaces to show up... what am I missing?

switch(config-if)# sh int po100 fex-intf 
Fabric           FEX
Interface        Interfaces
---------------------------------------------------
 Po100           
switch(config-if)# sh run

!Command: show running-config
!Time: Sat Jan 27 21:33:14 2001

version 7.1(0)N1(1b)
interface breakout slot 1 port 1-12 map 10g-4x

hostname switch

no feature telnet
cfs eth distribute
feature lldp
feature fex

username admin password 5 $1$lA3Ufc0l$8gqZmFwXyYZKyjQ7Gt5Ha.  role network-admin
no password strength-check
ip domain-lookup
fex 100
  pinning max-links 1
  description "FEX 100"
snmp-server user admin network-admin auth md5 0xff5615dc29622bf9811e35075902a51d
 priv 0xff5615dc29622bf9811e35075902a51d localizedkey

vlan 1
vrf context management

interface port-channel100
  switchport mode fex-fabric
  fex associate 100

interface Ethernet1/1/1

interface Ethernet1/1/2

interface Ethernet1/1/3

interface Ethernet1/1/4

interface Ethernet1/2/1

interface Ethernet1/2/2

interface Ethernet1/2/3

interface Ethernet1/2/4

interface Ethernet1/3/1

interface Ethernet1/3/2

interface Ethernet1/3/3

interface Ethernet1/3/4

interface Ethernet1/4/1

interface Ethernet1/4/2

interface Ethernet1/4/3

interface Ethernet1/4/4

interface Ethernet1/5/1

interface Ethernet1/5/2

interface Ethernet1/5/3

interface Ethernet1/5/4

interface Ethernet1/6/1

interface Ethernet1/6/2

interface Ethernet1/6/3

interface Ethernet1/6/4

interface Ethernet1/7/1

interface Ethernet1/7/2

interface Ethernet1/7/3

interface Ethernet1/7/4

interface Ethernet1/8/1

interface Ethernet1/8/2

interface Ethernet1/8/3

interface Ethernet1/8/4

interface Ethernet1/9/1

interface Ethernet1/9/2

interface Ethernet1/9/3

interface Ethernet1/9/4

interface Ethernet1/10/1

interface Ethernet1/10/2

interface Ethernet1/10/3

interface Ethernet1/10/4

interface Ethernet1/11/1
  switchport mode fex-fabric
  fex associate 100
  channel-group 100

interface Ethernet1/11/2
  switchport mode fex-fabric
  fex associate 100
  channel-group 100

interface Ethernet1/11/3
  switchport mode fex-fabric
  fex associate 100
  channel-group 100

interface Ethernet1/11/4
  switchport mode fex-fabric
  fex associate 100
  channel-group 100

interface Ethernet1/12/1

interface Ethernet1/12/2

interface Ethernet1/12/3

interface Ethernet1/12/4

interface Ethernet2/1

interface Ethernet2/2

interface Ethernet2/3

interface Ethernet2/4

interface Ethernet2/5

interface Ethernet2/6

interface Ethernet2/7

interface Ethernet2/8

interface Ethernet2/9

interface Ethernet2/10

interface Ethernet2/11

interface Ethernet2/12

interface mgmt0
  vrf member management
  ip address 192.168.70.250/24
line console
line vty
boot kickstart bootflash:/n6000-uk9-kickstart.7.1.0.N1.1b.bin
boot system bootflash:/n6000-uk9.7.1.0.N1.1b.bin
poap transit

switch(config-if)#     
switch(config-if)# sh int br

--------------------------------------------------------------------------------
Ethernet      VLAN    Type Mode   Status  Reason                   Speed     Por
t
Interface                                                                    Ch 
#
--------------------------------------------------------------------------------
Eth1/1/1      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/1/2      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/1/3      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/1/4      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/2/1      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/2/2      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/2/3      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/2/4      1       eth  access down    Transceiver validation fa   10G(D) --
Eth1/3/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/3/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/3/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/3/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/4/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/4/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/4/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/4/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/5/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/5/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/5/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/5/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/6/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/6/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/6/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/6/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/7/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/7/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/7/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/7/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/8/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/8/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/8/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/8/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/9/1      1       eth  access down    SFP not inserted            10G(D) --
Eth1/9/2      1       eth  access down    SFP not inserted            10G(D) --
Eth1/9/3      1       eth  access down    SFP not inserted            10G(D) --
Eth1/9/4      1       eth  access down    SFP not inserted            10G(D) --
Eth1/10/1     1       eth  access down    SFP not inserted            10G(D) --
Eth1/10/2     1       eth  access down    SFP not inserted            10G(D) --
Eth1/10/3     1       eth  access down    SFP not inserted            10G(D) --
Eth1/10/4     1       eth  access down    SFP not inserted            10G(D) --
Eth1/11/1     1       eth  fabric down    Link not connected          10G(D) 100
Eth1/11/2     1       eth  fabric down    Link not connected          10G(D) 100
Eth1/11/3     1       eth  fabric down    Link not connected          10G(D) 100
Eth1/11/4     1       eth  fabric down    Link not connected          10G(D) 100
Eth1/12/1     1       eth  access down    SFP not inserted            10G(D) --
Eth1/12/2     1       eth  access down    SFP not inserted            10G(D) --
Eth1/12/3     1       eth  access down    SFP not inserted            10G(D) --
Eth1/12/4     1       eth  access down    SFP not inserted            10G(D) --
Eth2/1        1       eth  access down    Transceiver validation fa   40G(D) --
Eth2/2        1       eth  access down    Transceiver validation fa   40G(D) --
Eth2/3        1       eth  access down    SFP not inserted            40G(D) --
Eth2/4        1       eth  access down    SFP not inserted            40G(D) --
Eth2/5        1       eth  access down    SFP not inserted            40G(D) --
Eth2/6        1       eth  access down    SFP not inserted            40G(D) --
Eth2/7        1       eth  access down    SFP not inserted            40G(D) --
Eth2/8        1       eth  access down    SFP not inserted            40G(D) --
Eth2/9        1       eth  access down    SFP not inserted            40G(D) --
Eth2/10       1       eth  access down    SFP not inserted            40G(D) --
Eth2/11       1       eth  access down    SFP not inserted            40G(D) --
Eth2/12       1       eth  access down    SFP not inserted            40G(D) --

--------------------------------------------------------------------------------
Port-channel VLAN    Type Mode   Status  Reason                    Speed   Proto
col
Interface                                                                  
--------------------------------------------------------------------------------
Po100        1       eth  fabric down    No operational members      auto(D)  no
ne

--------------------------------------------------------------------------------
Port   VRF          Status IP Address                              Speed    MTU
--------------------------------------------------------------------------------
mgmt0  --           down   192.168.70.250                          --       1500
switch(config-if)#

[Rollen]

Using sh fex and sh fex details I found the FEX communicating with my 5k switch. The FEX is stuck in a Image Download state and will not go past it. It keeps reboot and I can't get it to stop. The Switch and FEX version are exactly the same.

[Syed Aman Hussain]

Dear Rollen,

After physical wiring done from Parent switch eg. N7k or N5k to Fex , just issue the command Feature Fex , and wait for one minute then check sh fex , fex should be in discover mode , if not in discover then there is a problem in physical cabling

BR

SYED

[Rollen]

switch(config)# sh fex detail
FEX: 100 Description: FEX0100   state: Image Download
  FEX version: 7.1(0)N1(1b) [Switch version: 7.1(0)N1(1b)]
  FEX Interim version: 7.1(0)N1(1b)
  Switch Interim version: 7.1(0)N1(1b)
  Module Sw Gen: 12594  [Switch Sw Gen: 21]
  post level: complete
 pinning-mode: static    Max-links: 1
  Fabric port for control traffic: Eth1/11/2
  FCoE Admin: false
  FCoE Oper: true
  FCoE FEX AA Configured: false
  Fabric interface state:
    Po100 - Interface Up. State: Active
    Eth1/11/1 - Interface Down. State: Configured
    Eth1/11/2 - Interface Up. State: Active
    Eth1/11/3 - Interface Up. State: Active
    Eth1/11/4 - Interface Up. State: Active
Logs:
01/26/2001 16:06:24.307467: Image preload successful.
01/26/2001 16:06:25.715170: Deleting route to FEX
01/26/2001 16:06:25.724964: Module disconnected
01/26/2001 16:06:25.726443: Module Offline
01/26/2001 16:06:26.610114: Deleting route to FEX
01/26/2001 16:06:26.617711: Module disconnected
01/26/2001 16:06:26.619794: Offlining Module
01/26/2001 16:06:26.665467: Deleting route to FEX
01/26/2001 16:06:26.673062: Module disconnected
01/26/2001 16:06:26.675214: Offlining Module
01/26/2001 16:06:26.713752: Deleting route to FEX
01/26/2001 16:06:26.721332: Module disconnected
01/26/2001 16:06:26.723529: Offlining Module
01/26/2001 16:06:26.776957: Deleting route to FEX
01/26/2001 16:06:26.784563: Module disconnected
01/26/2001 16:06:26.786788: Offlining Module
01/26/2001 16:07:55.776947: Module register received
01/26/2001 16:07:55.778535: Image Version Mismatch
01/26/2001 16:07:55.779616: Registration response sent
01/26/2001 16:07:55.779920: Requesting satellite to download image
01/26/2001 16:08:26.011337: Deleting route to FEX
01/26/2001 16:08:26.018956: Module disconnected
01/26/2001 16:08:26.021390: Module Offline
01/26/2001 16:08:26.891585: Deleting route to FEX
01/26/2001 16:08:26.899150: Module disconnected
01/26/2001 16:08:26.901298: Offlining Module
01/26/2001 16:08:29.983343: Module register received
01/26/2001 16:08:29.984711: Image Version Mismatch
01/26/2001 16:08:29.985789: Registration response sent
01/26/2001 16:08:29.987794: Requesting satellite to download image
01/26/2001 16:09:03.994177: Module register received
01/26/2001 16:09:03.995535: Image Version Mismatch
01/26/2001 16:09:03.996641: Registration response sent
01/26/2001 16:09:03.998609: Requesting satellite to download image
01/26/2001 16:09:39.764487: Module register received
01/26/2001 16:09:39.765868: Image Version Mismatch
01/26/2001 16:09:39.766934: Registration response sent
01/26/2001 16:09:39.767218: Requesting satellite to download image
01/26/2001 16:10:09.534641: Deleting route to FEX
01/26/2001 16:10:09.542205: Module disconnected
01/26/2001 16:10:09.544163: Module Offline
01/26/2001 16:10:10.660592: Deleting route to FEX
01/26/2001 16:10:10.668215: Module disconnected
01/26/2001 16:10:10.670401: Offlining Module
01/26/2001 16:10:14.488537: Module register received
01/26/2001 16:10:14.489932: Image Version Mismatch
01/26/2001 16:10:14.491027: Registration response sent
01/26/2001 16:10:14.491312: Requesting satellite to download image
01/26/2001 16:10:48.525360: Module register received
01/26/2001 16:10:48.526735: Image Version Mismatch
01/26/2001 16:10:48.527805: Registration response sent
01/26/2001 16:10:48.528087: Requesting satellite to download image
01/26/2001 16:11:22.537272: Module register received
01/26/2001 16:11:22.538666: Image Version Mismatch
01/26/2001 16:11:22.539734: Registration response sent
01/26/2001 16:11:22.540028: Requesting satellite to download image
01/26/2001 16:11:56.567723: Module register received
01/26/2001 16:11:56.569095: Image Version Mismatch
01/26/2001 16:11:56.570168: Registration response sent
01/26/2001 16:11:56.570450: Requesting satellite to download image
01/26/2001 16:12:30.596332: Module register received
01/26/2001 16:12:30.597721: Image Version Mismatch
01/26/2001 16:12:30.598785: Registration response sent
01/26/2001 16:12:30.599070: Requesting satellite to download image

[Rollen]

Thank Syed Aman Hussain, I tried this and I can see the FEX switch but it is stuck in a Image Download state. I tried two different cables and get the same results.

Also I am using QSFP-4SFP10G-CU5M to connect between the Nexus 5624Q and the 2248TP-E. I don't see this cable on the compatibility matrix for the 2248TP-E but that shouldn't make a difference correct?

[derrick.ray1]

Rollen, 

Thank you for this post! It just saved my butt. I couldn't figure out what I was doing wrong!!

[Rollen]

Any one?...

[Rollen · Accepted solution]

7.1(0)N1(1b) has a bug that prevented me from associating the fex switch. Downgraded to version 7.0(8)N1(1) and now I can associate... I can't select my post as a correct answer but hopefully this helps any one else...

[Rollen]

Found the solution and want to resolve this thread. The problem was with the IOS and not the switch. We downgraded and had no problem after that.