Nexus 5624Q with FEX 2348UPQ, image download failed

[Geir Sand-Strand]

We experience problem with the FEX failing to download the image. It seems to go in loop trying to download the image.
We see the following messages in the log:
%DAEMON-3-SYSTEM_MSG: connect: Network is unreachable - in.tftpd
And show fex detail:
FEX: 102 Description: FEX0102　　 state: Offline
　 FEX version: Unknown [Switch version: 7.2(1)N1(1)]
　 FEX Interim version: Unknown
　 Switch Interim version: 7.2(1)N1(1)
　 Module Sw Gen: 12594　 [Switch Sw Gen: 21]
　 post level: complete
pinning-mode: static　　　 Max-links: 1
　 Fabric port for control traffic:
　　FCoE Admin: false
　 FCoE Oper: true
　 FCoE FEX AA Configured: false
　 Fabric interface state:
　　　 Po102 - Interface Down. State: Configured
　　　 Eth1/4 - Interface Down. State: Configured

Logs:
02/17/2016 08:57:05.870786: Module register received
02/17/2016 08:57:05.871180: Image Version Mismatch
02/17/2016 08:57:05.871426: Registration response sent
02/17/2016 08:57:05.871585: Requesting satellite to download image
02/17/2016 09:00:13.079145: Image preload failed.
02/17/2016 09:00:19.655089: Deleting route to FEX
02/17/2016 09:00:19.662366: Module disconnected
02/17/2016 09:00:19.663209: Module Offline
02/17/2016 09:00:19.685370: Deleting route to FEX
02/17/2016 09:00:19.692630: Module disconnected
02/17/2016 09:00:19.693599: Offlining Module
02/17/2016 09:03:25.668649: Module register received
02/17/2016 09:03:25.669047: Image Version Mismatch
02/17/2016 09:03:25.669300: Registration response sent
02/17/2016 09:03:25.669460: Requesting satellite to download image
02/17/2016 09:06:17.671649: Image preload failed.
02/17/2016 09:06:26.757317: Deleting route to FEX
02/17/2016 09:06:26.766860: Module disconnected
02/17/2016 09:06:26.767721: Module Offline
02/17/2016 09:06:26.787504: Deleting route to FEX
02/17/2016 09:06:26.794927: Module disconnected
02/17/2016 09:06:26.795967: Offlining Module
02/17/2016 09:09:33.846822: Module register received
02/17/2016 09:09:33.847218: Image Version Mismatch
02/17/2016 09:09:33.847471: Registration response sent
02/17/2016 09:09:33.847637: Requesting satellite to download image
02/17/2016 09:12:31.931056: Image preload failed.
02/17/2016 09:12:39.647541: Deleting route to FEX
02/17/2016 09:12:39.659849: Module disconnected
02/17/2016 09:12:39.661711: Module Offline
02/17/2016 09:12:39.677424: Deleting route to FEX
02/17/2016 09:12:39.684684: Module disconnected
02/17/2016 09:12:39.685648: Offlining Module
02/17/2016 09:15:45.853994: Module register received
02/17/2016 09:15:45.854395: Image Version Mismatch
02/17/2016 09:15:45.854643: Registration response sent
02/17/2016 09:15:45.854803: Requesting satellite to download image
02/17/2016 09:18:34.860097: Image preload failed.
02/17/2016 09:18:44.647638: Deleting route to FEX
02/17/2016 09:18:44.660746: Module disconnected
02/17/2016 09:18:44.661446: Module Offline
02/17/2016 09:18:44.678481: Deleting route to FEX
02/17/2016 09:18:44.685699: Module disconnected
02/17/2016 09:18:44.686668: Offlining Module
02/17/2016 09:21:50.652390: Module register received
02/17/2016 09:21:50.652796: Image Version Mismatch
02/17/2016 09:21:50.653037: Registration response sent
02/17/2016 09:21:50.653198: Requesting satellite to download image
02/17/2016 09:24:56.791825: Image preload failed.
02/17/2016 09:25:04.330652: Deleting route to FEX
02/17/2016 09:25:04.338057: Module disconnected
02/17/2016 09:25:04.338726: Module Offline
02/17/2016 09:25:04.367833: Deleting route to FEX
02/17/2016 09:25:04.375250: Module disconnected
02/17/2016 09:25:04.376287: Offlining Module

We have testet with several images, and they all give the same problem for the fex downloading the image.

Cisco bug CSCuw50341 is about the problem we encounter, and there is no workaround. We have not tested NXOS 7.3(0)N1(0.150) that the bug is know to affect.  It's also not in the download software library.

We have testet with the folowing NXOS versions:
7.2.1.N1.1
7.1.3.N1.2
7.1.0.N1.1b

Anyone know something about this issue?

BR

Geir

[rgeorgiev]

If you have n7k boxes see CSCuv91507 workaround.

[Geir Sand-Strand]

Have seen the CSCu91507, but 6.2.10 does not support F3-card in the 7010.  Therefore it will not take up the F3 card so the image can be downloaded to the FEX.

Tried using 6.2.14, but same result.

[Geir Sand-Strand]

Opened a TAC and got to try this:

　 1. 　Remove the NIF/HIF (ports connecting FEX and N5K) connections of 2348UPQ from 5624Q Parent switch.
　 2. 　Connect the FEX back again to N7k and make it online.
　 3. 　Load the dplug image matching the N7k image (load bootflash:<dplug filename>)
　 4. 　Now rlogin to the FEX as root. (rlogin -l root 127.1.1.<fexid>)
　 5. 　Do rm -rf /mnt/sysimage/*
　 6. 　Remove Physical NIF connections and HIF port connections if any (fex will shut off, thats fine)
　 7. 　Connect the FEX back up to the N5k/N6k. Now the FEX should come up

That seems to work for one of the 2348.  The other one seemed to look ok, but after image download successful, it never came back on.  Now I've got the STAT led solid amber.  Maybe a RMA for this one...?

[Geir Sand-Strand]

Well, had to do a RMA on the last N2K device.

Geir