Nexus 5548 Upgrade, FEX stuck in Blinking Green, uplinks not connected

[Net Admins]

Did the upgrade of a pair of 5548UP to 7.3(7)N1(1)

The pair was upgraded but the FEXes don't show up.

Physically they are are blinking green, all ports amber expect the uplinks that are turned off.

From the 5548UP the downlinks are not connected and show fex details shows nothing.

Has anyone been through such situation and was able to solve it?

[Sergiu.Daniluk]

Hi @Net Admins 

Do you still have the FEX configuration present? What FEXs model do you have?

Stay safe,

Sergiu

[Net Admins]

it turns out we hit this bug 
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvt58479

so watch out folks, be ready to downgrade quickly if you experience this too.

[paul driver]

excellent 
thanks for the feedback

[jlopezbo]

Hello Net Admin

If you are hitting CSCvt58479 then probably you may have some problems with the fex after the downgrade ( the fex may go to a booting loop trying to download the new version)

There is a work around to break out the booting loop

To recover the fex you need to upgrade or downgrade to new version ( version that is not affected by the bug)

Then we will upload dplug image to login to  a linux kernel of the switch and attach to the fex . Then we will manually delete the  7.3(7)N1(1 version and the fex will boot the new image 

7.x.x === New version

Action plan

Copy the dplug image for the 7.x.x  into the Nexus  bootflash:.

2. Load the dplug image matching the N5k image for the 7.x.x using the command "load bootflash:nuova-or-dplug-mzg.7.x.x.N1.1.bin".

3. Now, rlogin to the FEX as root. (rlogin -l root 127.15.1.<fexid> ). For example, if the FEX to recover is the FEX x, you can do rlogin -l root 127.15.1.(fex_ID).

4. Do rm -rf /mnt/sysimage/* >>> This command is supposed to delete the current image from the FEX's bootflash.

After some minutes the fex boot up

To follow the recover steps you will require to open a Cisco TAC  and they will help you