Solved: after NCS 5501-SE upgrade from 7.0.1 to 7.0.2 - UPGD FAIL status

from88 · ‎07-20-2020

Hello,

I upgraded new NCS5501-SE from 7.0.1 to 7.0.2 upgrade went smooth, but after doing last checks i see UPG-FAIL message is this something bad ?

RP/0/RP0/CPU0:ITXNCS01#show  hw-module fpd 
Mon Jul 20 11:32:26.100 UTC
                                                               FPD Versions
                                                               =================
Location   Card type             HWver FPD device       ATR Status   Running Programd
-----------------------------------------------------------------------------------
0/RP0      NCS-5501-SE           1.1   MB-MIFPGA            CURRENT    1.02    1.02  
0/RP0      NCS-5501-SE           1.1   Bootloader           CURRENT    1.22    1.22  
0/RP0      NCS-5501-SE           1.1   CPU-IOFPGA           UPGD FAIL  1.25    1.25  
0/RP0      NCS-5501-SE           1.1   MB-IOFPGA            CURRENT    1.11    1.11

How to fix it ? Thanks.

As I rememeber in 7.0.1 there was no error like that.

When tryingto di it manualy: upgrade hw-module location all fpd all

Getting this error:

0/RP0/ADMIN0:Jul 20 12:41:55.420 UTC: card_mgr[3087]: %DRIVER-CARD_MGR-1-UPGD_SKIP : FPD upgrade skipped for CPU-IOFPGA @location 0/RP0 .The Bootloader primary or golden version on the box is not compatible with the new packaged CPU-IOFPGA. Kindly 'force' upgrade the Bootloader prior to manually upgrading CPU-IOFPGA to avoid bricking the board

is it safe to do it this upgrade hw-module location all fpd all force command, as log suggests ?

from88 · ‎07-30-2020

thanks, as the support suggested, it could be due to reload command usage. They said, that on NCS devices we need to use reload location all.

View solution in original post

from88 · ‎07-20-2020

actualy i managed to upgrade it first forcing to upgrade Bootloader then CPU-IOFPGA.

but now it's hanging in RLOAD REQ phase, and even the box reload doesnt help.

RP/0/RP0/CPU0:ITXNCS01#show hw-module fpd
Mon Jul 20 15:40:41.120 UTC
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/RP0 NCS-5501-SE 1.1 MB-MIFPGA CURRENT 1.02 1.02
0/RP0 NCS-5501-SE 1.1 Bootloader CURRENT 1.22 1.22
0/RP0 NCS-5501-SE 1.1 CPU-IOFPGA RLOAD REQ 1.25 1.27
0/RP0 NCS-5501-SE 1.1 MB-IOFPGA CURRENT 1.11 1.11

any ideas ?

Thanks

from88 · ‎07-20-2020

After hardreboot (plug all the power off, then turn on) it started to show OK.

RP/0/RP0/CPU0:ITXNCS01#show hw-module fpd
Mon Jul 20 20:25:44.172 UTC
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/RP0 NCS-5501-SE 1.1 MB-MIFPGA CURRENT 1.02 1.02
0/RP0 NCS-5501-SE 1.1 Bootloader CURRENT 1.22 1.22
0/RP0 NCS-5501-SE 1.1 CPU-IOFPGA CURRENT 1.27 1.27
0/RP0 NCS-5501-SE 1.1 MB-IOFPGA CURRENT 1.11 1.11
RP/0/RP0/CPU0:ITXNCS01#

i hope it will continue to work that way:)

from88 · ‎07-30-2020

thanks, as the support suggested, it could be due to reload command usage. They said, that on NCS devices we need to use reload location all.

hostasaurus · ‎09-19-2022

@from88 can you let me know what you ran to accomplish this part: "actualy i managed to upgrade it first forcing to upgrade Bootloader then CPU-IOFPGA." I just went to 702 and ended up with the same "UPGD FAIL"; trying to figure out next steps. I did try reload location all but same issue afterward.

0/RP0/ADMIN0:Sep 18 23:00:20.748 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP0/ADMIN0:Sep 18 23:00:20.748 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force
0/RP0/ADMIN0:Sep 18 23:00:20.748 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP0/ADMIN0:Sep 18 23:00:20.751 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              CPU-IOFPGA           FALSE
0/RP0/ADMIN0:Sep 18 23:00:21.121 UTC: card_mgr[3079]: %DRIVER-CARD_MGR-1-UPGD_SKIP : FPD upgrade skipped for CPU-IOFPGA @location 0/RP0 .The Bootloader primary or golden version on the box is not compatible with the new packaged CPU-IOFPGA. Kindly 'force' upgrade the Bootloader prior to manually upgrading CPU-IOFPGA to avoid bricking the board.
0/RP0/ADMIN0:Sep 18 23:00:21.121 UTC: card_mgr[3079]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD CPU-IOFPGA@0/RP0 image programming completed with UPGD FAIL state Info: []

smilstea · ‎09-19-2022

sysadmin-vm:0_RP0# upgrade hw-module loc 0/RP0 fpd IOFPGA ?
Possible completions:
alternate
force
| Output modifiers
<cr>
sysadmin-vm:0_RP0# upgrade hw-module loc 0/RP0 fpd IOFPGA force ?
Possible completions:
alternate
| Output modifiers
<cr>
sysadmin-vm:0_RP0# upgrade hw-module loc 0/RP0 fpd IOFPGA force

Sam

hostasaurus · ‎09-19-2022

Unfortunately same failure:

sysadmin-vm:0_RP0# upgrade hw-module loc 0/RP0 fpd CPU-IOFPGA force
Mon Sep  19 15:20:22.711 UTC+00:00


0/RP0/ADMIN0:Sep 19 15:20:34.713 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP0/ADMIN0:Sep 19 15:20:34.713 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force
0/RP0/ADMIN0:Sep 19 15:20:34.713 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP0/ADMIN0:Sep 19 15:20:34.713 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              CPU-IOFPGA           TRUE
0/RP0/ADMIN0:Sep 19 15:20:35.074 UTC: card_mgr[3079]: %DRIVER-CARD_MGR-1-UPGD_SKIP : FPD upgrade skipped for CPU-IOFPGA @location 0/RP0 .The Bootloader primary or golden version on the box is not compatible with the new packaged CPU-IOFPGA. Kindly 'force' upgrade the Bootloader prior to manually upgrading CPU-IOFPGA to avoid bricking the board.
0/RP0/ADMIN0:Sep 19 15:20:35.074 UTC: card_mgr[3079]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD CPU-IOFPGA@0/RP0 image programming completed with UPGD FAIL state Info: []
0/RP0/ADMIN0:Sep 19 15:20:44.714 UTC: fpdserv[4252]: %INFRA-FPD_Manager-1-UPGRADE_ALERT :  FPD Upgrade Completed (use "show hw-module fpd" to check upgrade status)


sysadmin-vm:0_RP0# show hw-module fpd
Mon Sep  19 15:20:58.938 UTC+00:00
                                                                   FPD Versions
                                                                ===============
Location  Card type         HWver FPD device       ATR Status   Run    Programd
-------------------------------------------------------------------------------
0/RP0     NCS-5501-SE       1.1   Bootloader           CURRENT    1.22    1.22
0/RP0     NCS-5501-SE       1.1   CPU-IOFPGA           UPGD FAIL  1.25    1.25
0/RP0     NCS-5501-SE       1.1   MB-IOFPGA            CURRENT    1.11    1.11
0/RP0     NCS-5501-SE       1.1   SATA                 CURRENT    5.00    5.00

smilstea · ‎09-19-2022

the comment and error mentions 'actualy i managed to upgrade it first forcing to upgrade Bootloader then CPU-IOFPGA.'

Can you try force on bootloader first and then iofpga?

Sam

hostasaurus · ‎09-20-2022

Thanks! I was able to force Bootloader first, then force CPU-IOFPGA. I reloaded, but it didn't change from a new status of "RLOAD REQ". I worked with TAC at that point and they had me do an admin->"hw-module location all reload" and then everything went current! Was able to move on to 7.5.2 after that with zero issues.

smilstea · ‎09-20-2022

If you type reload it only reloads the VM not the hardware, the proper way to power-cycle/reload the hardware is with hw-module reload command.

Sam

from88 · ‎09-20-2022

hello, all.

As i digged in my notes, i was using these two commands:

4)upgrade hw-module location all fpd Bootloader force
5)upgrade hw-module location all fpd CPU-IOFPGA

its interesting why it so cumberstone upgrading NCS devices. Maybe uupgrades in other versions doesnt happens this way ?

please, inform how it goes.

hostasaurus · ‎09-20-2022

@from88 wrote:
its interesting why it so cumberstone upgrading NCS devices. Maybe uupgrades in other versions doesnt happens this way ?

I'm certainly in agreement with you on this process being a real headache, but if I had to guess why it seems so difficult from the perspective of the 5501, it is probably because much of this is to allow the chassis-based systems with redundant modules to be upgraded without outage, where you have to be able to update and reload portions of the device independently. But yeah, for a 5501 I would love a 1) copy file, 2) reload, 3) oh, i'm done.

from88 · ‎09-26-2022

yeah,
do i understand correctly that from 7.0.2 to 7.5.2 you can upgrade directly, and without any fails ?

hostasaurus · ‎09-26-2022

Yeah that was the path I took; just make sure all the "show hw-module fpd" outputs are clean before proceeding. Also, if you're used to just uploading the tar file, 7.5.2 grew too large to do that, so you'll have to extract it yourself and upload the mini iso and all the module rpms, install add source on them all, then prepare them as a combined command before you can activate.

EdvinasKairys11982 · ‎10-24-2022

Thank you, will keep in mind. Also - maybe you know if i can also - at the same upgrade iteration add one of the SMUs ? To the same .tarball or `to the `install add source harddisk` command)