cisco ASR9010 RSP440SE FPD update fail

[Yang Li]

location Specify a location
RP/0/RSP0/CPU0:ios(admin)#upgrade hw-module fpd all force location ?
0/1/CPU0 Fully qualified location specification
0/PS0/M2/SP Fully qualified location specification
0/PS0/M3/SP Fully qualified location specification
0/RSP0/CPU0 Fully qualified location specification
0/RSP1/CPU0 Fully qualified location specification
WORD Fully qualified location specification
all All locations
RP/0/RSP0/CPU0:ios(admin)#upgrade hw-module fpd all force location all ?
<cr>
RP/0/RSP0/CPU0:ios(admin)#upgrade hw-module fpd all force location all
Fri Jan 5 10:35:12.669 UTC

***** UPGRADE WARNING MESSAGE: *****
* This upgrade operation has a maximum timout of 90 minutes. *
* If you are executing the cmd for one specific location and *
* card in that location reloads or goes down for some reason *
* you can press CTRL-C to get back the RP's prompt. *
* If you are executing the cmd for _all_ locations and a node *
* reloads or is down please allow other nodes to finish the *
* upgrade process before pressing CTRL-C. *

% RELOAD REMINDER:
- The upgrade operation of the target module will not interrupt its normal
operation. However, for the changes to take effect, the target module
will need to be manually reloaded after the upgrade operation. This can
be accomplished with the use of "hw-module <target> reload" command.
- If automatic reload operation is desired after the upgrade, please use
the "reload" option at the end of the upgrade command.
- The output of "show hw-module fpd location" command will not display
correct version information after the upgrade if the target module is
not reloaded.
NOTE: Chassis CLI will not be accessible while upgrade is in progress.
Continue? [confirm]
This can take some time for a full chassis.
Ensure that system is not power cycled during the upgrades.
Please consult the documentation for more information.
Continue ? [no]: yes

FPD upgrade in progress on some hardware, reload/configuration change
on those is not recommended as it might cause HW programming failure
and result in RMA of the hardware.

Starting the upgrade/download of following FPDs:
=========== ==== ======= ======= =========== =========
Current Upg/Dng
Location Type Subtype Upg/Dng Version Version
=========== ==== ======= ======= =========== =========
0/RSP0/CPU0 lc cbc upg 16.116 16.116
0/RSP0/CPU0 lc fpga upg 0.11 0.11
0/RSP0/CPU0 lc fpga3 upg 4.09 4.09
0/RSP0/CPU0 lc fpga2 upg 1.09 1.09
0/RSP0/CPU0 lc rommon upg 0.75 0.75
0/FT0/SP ft cbc upg 5.03 5.03
0/FT0/SP ft cbc upg 5.03 5.03
0/FT1/SP ft cbc upg 5.03 5.03
0/FT1/SP ft cbc upg 5.03 5.03
0/BPID0/SP bp cbc upg 7.104 7.104
------------------------------------------------------
Starting the upgrade/download of following FPDs:
=========== ==== ======= ======= =========== =========
Current Upg/Dng
Location Type Subtype Upg/Dng Version Version
=========== ==== ======= ======= =========== =========
0/RSP1/CPU0 lc cbc upg 16.116 16.116
0/RSP1/CPU0 lc fpga upg 0.10 0.11
0/RSP1/CPU0 lc fpga3 upg 4.09 4.09
0/RSP1/CPU0 lc fpga2 upg 1.06 1.09
0/RSP1/CPU0 lc rommon upg 0.73 0.75
------------------------------------------------------
Starting the upgrade/download of following FPDs:
=========== ==== ======= ======= =========== =========
Current Upg/Dng
Location Type Subtype Upg/Dng Version Version
=========== ==== ======= ======= =========== =========
0/1/CPU0 lc cbc upg 20.118 20.118
0/1/CPU0 lc fpga2 upg 1.04 1.04
0/1/CPU0 lc fpga4 upg 1.05 1.05
0/1/CPU0 lc rommon upg 3.03 3.03
0/1/0 spa fpga8 upg 1.07 1.07
0/1/1 spa fpga8 upg 1.07 1.07
------------------------------------------------------
FPD upgrade in progress. Max timeout remaining 89 min.
FPD upgrade in progress. Max timeout remaining 88 min.
FPD upgrade in progress. Max timeout remaining 87 min.
FPD upgrade in progress. Max timeout remaining 86 min.
FPD upgrade in progress. Max timeout remaining 85 min.
FPD upgrade in progress. Max timeout remaining 84 min.
Successfully upgraded cbc for A9K-RSP440-SE on location 0/RSP1/CPU0 from 16.116 to 16.116
Successfully upgraded fpga for A9K-RSP440-SE on location 0/RSP1/CPU0 from 0.10 to 0.11
Successfully upgraded fpga3 for A9K-RSP440-SE on location 0/RSP1/CPU0 from 4.09 to 4.09
Successfully upgraded fpga2 for A9K-RSP440-SE on location 0/RSP1/CPU0 from 1.06 to 1.09
Successfully upgraded rommon for A9K-RSP440-SE on location 0/RSP1/CPU0 from 0.73 to 0.75
FPD upgrade in progress. Max timeout remaining 83 min.
FPD upgrade in progress. Max timeout remaining 82 min.
FPD upgrade in progress. Max timeout remaining 81 min.
FPD upgrade in progress. Max timeout remaining 80 min.
FPD upgrade in progress. Max timeout remaining 79 min.
FPD upgrade in progress. Max timeout remaining 78 min.
FPD upgrade in progress. Max timeout remaining 77 min.
FPD upgrade in progress. Max timeout remaining 76 min.
FPD upgrade in progress. Max timeout remaining 75 min.
FPD upgrade in progress. Max timeout remaining 74 min.
LC/0/1/CPU0:Jan 5 10:51:27.968 : dumper[56]: %OS-DUMPER-4-SIGSEGV : Thread 20 received SIGSEGV - Segmentation Fault
LC/0/1/CPU0:Jan 5 10:51:27.968 : dumper[56]: %OS-DUMPER-4-SIGSEGV_INFO : Accessed BadAddr 0x400f at PC 0x4f4eaffc. Signal code 1 - SEGV_MAPPER. Address
not mapped.
LC/0/1/CPU0:Jan 5 10:51:27.968 : dumper[56]: %OS-DUMPER-4-CRASH_INFO : Crashed pid = 45088 (pkg/bin/lda_server)
Successfully upgraded cbc for A9K-MOD160-SE on location 0/1/CPU0 from 20.118 to 20.118
Successfully upgraded fpga2 for A9K-MOD160-SE on location 0/1/CPU0 from 1.04 to 1.04
Successfully upgraded fpga4 for A9K-MOD160-SE on location 0/1/CPU0 from 1.05 to 1.05
Successfully upgraded rommon for A9K-MOD160-SE on location 0/1/CPU0 from 3.03 to 3.03
Successfully upgraded fpga8 for A9K-MPA-8X10GE on location 0/1/0 from 1.07 to 1.07
FAILED to upgrade fpga8 for A9K-MPA-8X10GE on location 0/1/1 from 1.07 to 1.07
LC/0/1/CPU0:Jan 5 10:51:35.231 : lc_fpd_upgrade[263]: %PLATFORM-UPGRADE_FPD-3-GENERIC_FPD_ERROR : ERROR occured while FPD upgrade: Failed to send Progra
m message: No such process
FPD upgrade in progress. Max timeout remaining 73 min.
FPD upgrade in progress. Max timeout remaining 72 min.
FPD upgrade in progress. Max timeout remaining 71 min.
FPD upgrade in progress. Max timeout remaining 70 min.
FPD upgrade in progress. Max timeout remaining 69 min.
FPD upgrade in progress. Max timeout remaining 68 min.
FPD upgrade in progress. Max timeout remaining 67 min.
FPD upgrade in progress. Max timeout remaining 66 min.
FPD upgrade in progress. Max timeout remaining 65 min.
FPD upgrade in progress. Max timeout remaining 64 min.
FPD upgrade in progress. Max timeout remaining 63 min.
FPD upgrade in progress. Max timeout remaining 62 min.
FPD upgrade in progress. Max timeout remaining 61 min.
FPD upgrade in progress. Max timeout remaining 60 min.
FPD upgrade in progress. Max timeout remaining 59 min.
FPD upgrade in progress. Max timeout remaining 58 min.
FPD upgrade in progress. Max timeout remaining 57 min.
FPD upgrade in progress. Max timeout remaining 56 min.
FPD upgrade in progress. Max timeout remaining 55 min.
FPD upgrade in progress. Max timeout remaining 54 min.
FPD upgrade in progress. Max timeout remaining 53 min.
FPD upgrade in progress. Max timeout remaining 52 min.
FPD upgrade in progress. Max timeout remaining 51 min.
FPD upgrade in progress. Max timeout remaining 50 min.
FPD upgrade in progress. Max timeout remaining 49 min.
FPD upgrade in progress. Max timeout remaining 48 min.
FPD upgrade in progress. Max timeout remaining 47 min.
FPD upgrade in progress. Max timeout remaining 46 min.
FPD upgrade in progress. Max timeout remaining 45 min.
FPD upgrade in progress. Max timeout remaining 44 min.
FPD upgrade in progress. Max timeout remaining 43 min.
FPD upgrade in progress. Max timeout remaining 42 min.
FPD upgrade in progress. Max timeout remaining 41 min.
FPD upgrade in progress. Max timeout remaining 40 min.
FPD upgrade in progress. Max timeout remaining 39 min.
FPD upgrade in progress. Max timeout remaining 38 min.
FPD upgrade in progress. Max timeout remaining 37 min.
FPD upgrade in progress. Max timeout remaining 36 min.
FPD upgrade in progress. Max timeout remaining 35 min.
FPD upgrade in progress. Max timeout remaining 34 min.
FPD upgrade in progress. Max timeout remaining 33 min.
FPD upgrade in progress. Max timeout remaining 32 min.
FPD upgrade in progress. Max timeout remaining 31 min.
FPD upgrade in progress. Max timeout remaining 30 min.
FPD upgrade in progress. Max timeout remaining 29 min.
FPD upgrade in progress. Max timeout remaining 28 min.
FPD upgrade in progress. Max timeout remaining 27 min.
FPD upgrade in progress. Max timeout remaining 26 min.
FPD upgrade in progress. Max timeout remaining 25 min.
FPD upgrade in progress. Max timeout remaining 24 min.
FPD upgrade in progress. Max timeout remaining 23 min.
FPD upgrade in progress. Max timeout remaining 22 min.
FPD upgrade in progress. Max timeout remaining 21 min.
FPD upgrade in progress. Max timeout remaining 20 min.
FPD upgrade in progress. Max timeout remaining 19 min.
FPD upgrade in progress. Max timeout remaining 18 min.
FPD upgrade in progress. Max timeout remaining 17 min.
FPD upgrade in progress. Max timeout remaining 16 min.
FPD upgrade in progress. Max timeout remaining 15 min.
FPD upgrade in progress. Max timeout remaining 14 min.
FPD upgrade in progress. Max timeout remaining 13 min.
FPD upgrade in progress. Max timeout remaining 12 min.
FPD upgrade in progress. Max timeout remaining 11 min.

[Yang Li]

RP/0/RSP0/CPU0:Jan 5 12:05:13.442 : upgrade_fpd_cli[65787]: %PLATFORM-UPGRADE_FPD-4-WARNING : A Warning message while doing fpd_upgrade: FPD upgrade has exceeded the maximum time window, the process will terminate now. Please check the status of the hardware and reissue the upgrade command if required.
FPD ERROR:Error while waiting for FPD upgrade complete messages from the nodes:FPD upgrade has exceeded the maximum time window

FPD upgrade has ended.

FPD upgrade in progress, aborting now is not recommended
as it might cause HW programming failure and result in RMA
of the hardware.
Continue ? [no]:
Continuing with FPD upgrade

[Yang Li]

@Harold Ritter bro can you help me check this Problem?

[Ruben Cocheno]

@Yang Li 

can you can the following output?

show hw-module fpd location all
show run

Despite some components being upgraded "successfully", It seems that you got a segmentation fault, and not sure if is just cosmetic so i would suggest the TAC route at his point.

LC/0/1/CPU0:Jan 5 10:51:27.968 : dumper[56]: %OS-DUMPER-4-SIGSEGV : Thread 20 received SIGSEGV - Segmentation Fault
LC/0/1/CPU0:Jan 5 10:51:27.968 : dumper[56]: %OS-DUMPER-4-SIGSEGV_INFO : Accessed BadAddr 0x400f at PC 0x4f4eaffc. Signal code 1 - SEGV_MAPPER. Address
not mapped.
LC/0/1/CPU0:Jan 5 10:51:27.968 : dumper[56]: %OS-DUMPER-4-CRASH_INFO : Crashed pid = 45088 (pkg/bin/lda_server)

[Yang Li]

Dear @Ruben Cocheno 

RP/0/RSP0/CPU0:A9K#show hw-module fpd location all
Fri Jan 5 15:38:33.161 UTC

===================================== ==========================================
Existing Field Programmable Devices
==========================================
HW Current SW Upg/
Location Card Type Version Type Subtype Inst Version Dng?
============ ======================== ======= ==== ======= ==== =========== ====
0/RSP0/CPU0 A9K-RSP440-SE 1.0 lc cbc 0 16.116 No
lc fpga1 0 0.11 No
lc fpga3 0 4.09 No
lc fpga2 0 1.09 No
lc rommon 0 0.75 No
--------------------------------------------------------------------------------
0/FT0/SP ASR-9006-FAN-V2 1.0 ft cbc 7 5.03 No
--------------------------------------------------------------------------------
0/FT1/SP ASR-9006-FAN-V2 1.0 ft cbc 8 5.03 No
--------------------------------------------------------------------------------
0/BPID0/SP ASR-9006-BPID2 1.0 bp cbc 11 7.104 No
--------------------------------------------------------------------------------
0/PS0/M2/SP PWR-3KW-AC-V2 1.0 pm fpga11 15 6.04^ No
pm fpga12 15 6.02^ No
pm fpga13 15 6.02^ No
--------------------------------------------------------------------------------
0/PS0/M3/SP PWR-3KW-AC-V2 1.0 pm fpga11 16 6.04^ No
pm fpga12 16 6.02^ No
pm fpga13 16 6.02^ No
--------------------------------------------------------------------------------
0/RSP1/CPU0 A9K-RSP440-SE 1.0 lc cbc 0 16.116 No
lc fpga2 0 1.09 No
lc fpga3 0 4.09 No
lc fpga1 0 0.11 No
lc rommon 0 0.75 No
--------------------------------------------------------------------------------
0/1/CPU0 A9K-MOD160-SE 1.0 lc cbc 0 20.118 No
lc fpga2 0 1.04 No
lc fpga4 0 1.05 No
lc rommon 0 3.03 No
--------------------------------------------------------------------------------
0/1/0 A9K-MPA-8X10GE 1.102 spa fpga8 0 1.07 No
--------------------------------------------------------------------------------
0/1/1 A9K-MPA-8X10GE 1.102 spa fpga8 1 1.07 No
--------------------------------------------------------------------------------
NOTES:
1. ^ One or more FPD will be intentionally skipped from upgrade using CLI with option "all" or during "Auto fpd".
It can be upgraded only using the "admin> upgrade hw-module fpd <fpd> location <loc>" CLI with exact location.
RP/0/RSP0/CPU0:A9K#

RP/0/RSP0/CPU0:A9K#show running-config
Fri Jan 5 15:38:54.929 UTC
Building configuration...
!! IOS XR Configuration 5.3.4
!! Last configuration change at Fri Jan 5 15:17:51 2024 by admin
!
hostname A9K
interface MgmtEth0/RSP0/CPU0/0
shutdown
!
interface MgmtEth0/RSP0/CPU0/1
shutdown
!
interface MgmtEth0/RSP1/CPU0/0
shutdown
!
interface MgmtEth0/RSP1/CPU0/1
shutdown
!
interface TenGigE0/1/0/0
!
interface TenGigE0/1/0/1
!
interface TenGigE0/1/0/2
!
interface TenGigE0/1/0/3
!
interface TenGigE0/1/0/4
!
interface TenGigE0/1/0/5
!
interface TenGigE0/1/0/6
!
interface TenGigE0/1/0/7
!
interface TenGigE0/1/1/0
!
interface TenGigE0/1/1/1
!
interface TenGigE0/1/1/2
!
interface TenGigE0/1/1/3
!
interface TenGigE0/1/1/4
!
interface TenGigE0/1/1/5
!
interface TenGigE0/1/1/6
!
interface TenGigE0/1/1/7
!

[Yang Li]

@Ruben Cocheno  

Can you help me check if the FPD upgrade is normal?

best wishes

[anpetit · Accepted solution]

Hello @Yang Li ,
I would try to restart this process: process restart lc_fpd_upgrade

Let me know if it helps.

Have a nice day. 
Kind Regards,

Antoine