FPD Upgrade MOP

dpacific · ‎02-08-2021

The Need

In some situation NCS560 RP become unresponsive after reload or powercycle.

In many NCS560 deployments are in remote location, deployment might be large and human intervention should be kept at minimum

Engineering team have been working on a strategy to have functionalities to assess/troubleshoot and recovery such situation remotely

Depending on the release it is important to fulfill the mandatory SMU installation.
All the fixes are available stating from 7.2.2 onward. This guide applies also to more recent version by using the bundled FPD package unless explicitly advised from some Cisco representative

SMU Summary Table

IOS XR Version	SMU	Link	Notes
6.6.3	CSCvt69762 (sysadmin) CSCvt26907 (sysadmin) CSCvv88955 (sysadmin) CSCvw42910 (XR) CSCvw52338 (XR)	link
7.0.1	CSCvw42910 (XR) CSCvw52338 (XR) CSCvt69762 (sysadmin)	link	It is mandatory to move to the EMR release of the same minor release 7.0.2 No solution for TAM area corruption available in that release.
7.0.2	CSCvt69762 (sysadmin) CSCvt26907 (sysadmin) CSCvv48056 (sysadmin) CSCvw42910 (XR) CSCvw52338 (XR)	link
7.1.1	CSCvv41516 (sysadmin) CSCvt69762 (sysadmin) CSCvt26907 (sysadmin) CSCvw42910 (XR) CSCvw52338 (XR)	link	It is advisable to move to the EMR release of the same minor release 7.1.2
7.1.2	CSCvv41516 (sysadmin) CSCvt69762 (sysadmin) CSCvt26907 (sysadmin) CSCvw42910 (XR) CSCvw52338 (XR)	link
7.2.1	CSCvv41516 (sysadmin) CSCvt69762 (sysadmin) CSCvt26907 (sysadmin) CSCvw42910 (XR) CSCvw52338 (XR)	link

FPD Upgrade MOP

The idea is to reach a situation that FPD version are at or above minimum requirement

ADM	1.06
IOFPGA	0.64
BIOS	0.16

Assumptions

RP0 is the Active RP

RP1 is the Standby RP

below the assumed prompt for various location so that all command to navigate the various location are omitted

XR - RP0

RP/0/RP0/CPU0:ios#

XR - RP1

RP/0/RP1/CPU0:ios#

sysadmin - RP0

sysadmin-vm:0_RP0#

sysadmin - RP1

sysadmin-vm:0_RP1#

sysadmin - RP0 shell

[sysadmin-vm:0_RP0:~]$

sysadmin - RP1 shell

[sysadmin-vm:0_RP1:~]$

Sample System running 7.1.2 and necessary SMUs activated

below the SMUs that are expected to be active and committed on the box prior to start this process

XR VM

RP/0/RP0/CPU0:ios#sh install active summary 
    Active Packages: 10
        ncs560-xr-7.1.2 version=7.1.2 [Boot image]
        ncs560-mcast-2.0.0.0-r712
        ncs560-mgbl-2.0.0.0-r712
        ncs560-mpls-1.0.0.0-r712
        ncs560-mpls-te-rsvp-2.0.0.0-r712
        ncs560-ospf-2.0.0.0-r712
        ncs560-k9sec-2.1.0.0-r712
        ncs560-iosxr-fwding-2.1.0.1-r712.CSCvp67225 
        ncs560-os-support-2.1.0.1-r712.CSCvv41516 
        cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .xr

sysadmin

sysadmin-vm:0_RP0# show install active summ
     Active Packages: 4
        ncs560-sysadmin-7.1.2 version=7.1.2 [Boot image]
        ncs560-sysadmin-ncs560-7.1.2.1-r712.CSCvv41516      
        cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .admin
        cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .host

Verify FPD Versions on the RPs

RP/0/RP0/CPU0:ios#sh hw-module fpd
                                                               FPD Versions
                                                               =================
Location   Card type             HWver FPD device       ATR Status   Running Programd
-----------------------------------------------------------------------------------
0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02
0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95
0/RP0      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04
0/RP0      N560-RSP4-E           0.0   IOFPGA               NEED UPGD  0.53    0.53
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15
0/RP0      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30
0/RP1      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04
0/RP1      N560-RSP4-E           0.0   IOFPGA               NEED UPGD  0.53    0.53
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15
0/RP1      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30
0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Check TAM area on both RPs (evaluate the removal of the section)

It was important not to have the TAM area corrupted otherwise a subsequent reload would lead to an RMA

With this MOP, since we are installing FPD version 0.64, this IOFPGA FPD upgrade and RP reload will lead to TAM corruption recovery as per the note reported below

RP0

sysadmin-vm:0_RP0# show control-driver info location 0/RP0
<snip>
Card view from local RP:
        TAM Memory is not corrupted
        Can proceed with IOFPGA upgrade if required
<snip>

RP1

sysadmin-vm:0_RP0# show control-driver info location 0/RP1
<snip>
Card view from local RP:
        TAM Memory is not corrupted
        Can proceed with IOFPGA upgrade if required
<snip>

Note: in case of corrupted TAM Memory this does not represent a big problem as long as we are upgrading at least to 0.64 IOFPGA version

Sample output of corrupted TAM Memory

sysadmin-vm:0_RP1# show control-driver info location 0/RP0
<snip>
Card view from local RP:    
        TAM Memory is corrupted       
        Do not proceed with IOFPGA upgrade
<snip>

so if all prerequisites reported in this document are fulfilled it is possible to proceed with the upgrade.

The reason why the output says "Do not proceed with IOFPGA upgrade" is because the long term fix have been made available only via IOFPGA 0.64 while the command to verify have been made available earlier

Perform IOFPGA upgrade on both RPs

RP/0/RP0/CPU0:ios#upgrade hw-module location all fpd IOFPGA
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP0/CPU0:RONALDO-RF2a#0/RP0/ADMIN0:Oct 16 15:19:26.609 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              IOFPGA               FALSE
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              IOFPGA               FALSE

Verify

RP/0/RP0/CPU0:ios#sh hw-module fpd
Fri Oct 16 15:22:11.256 CEST
                                                               FPD Versions
                                                               =================
Location   Card type             HWver FPD device       ATR Status   Running Programd
-----------------------------------------------------------------------------------
0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02
0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95
0/RP0      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04
0/RP0      N560-RSP4-E           0.0   IOFPGA               RLOAD REQ  0.53    0.64
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15
0/RP0      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30
0/RP1      N560-RSP4-E           0.0   ADM                  NEED UPGD  1.04    1.04
0/RP1      N560-RSP4-E           0.0   IOFPGA               RLOAD REQ  0.53    0.64
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         NEED UPGD  0.15    0.15
0/RP1      N560-RSP4-E           0.0   SATA                 NEED UPGD  1.30    1.30
0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

RP reload one by one for IOFPGA upgrade to complete

Standby RP first

sysadmin-vm:0_RP0# hw-module location 0/RP1 reload
Reload hardware module ? [no,yes] yes

and once the redundancy is restored we can reload RP0 (it will trigger a failover)

sysadmin-vm:0_RP0# hw-module location 0/RP0 reload
Reload hardware module ? [no,yes] yes

Verify ADM version

on new active RP1

[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.2
Bootloader Version : 0.0.9
Config Version : 1.4

then on standby RP0

[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.2
Bootloader Version : 0.0.9
Config Version : 1.4

Perform remaining FPD upgrade on both RPs

ADM

RP/0/RP1/CPU0:ios#upgrade hw-module location all FPd ADM
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 15:57:14.332 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              ADM                  FALSE
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              ADM                  FALSE

BIOS

RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD PRIMARY-BIOS
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              PRIMARY-BIOS         FALSE
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              PRIMARY-BIOS         FALSE

SATA

RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD SATA
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location           FPD name             Force
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1              SATA                 FALSE
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0              SATA                 FALSE

Verify

RP/0/RP1/CPU0:ios#sh hw-module fpd
                                                               FPD Versions
                                                               =================
Location   Card type             HWver FPD device       ATR Status   Running Programd
-----------------------------------------------------------------------------------
0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02
0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95
0/RP0      N560-RSP4-E           0.0   ADM                  RLOAD REQ  1.04    1.04
0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ  0.15    0.16
0/RP0      N560-RSP4-E           0.0   SATA                 RLOAD REQ  1.30    2.10
0/RP1      N560-RSP4-E           0.0   ADM                  RLOAD REQ  1.04    1.04
0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         RLOAD REQ  0.15    0.16
0/RP1      N560-RSP4-E           0.0   SATA                 RLOAD REQ  1.30    2.10
0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Power Cycle procedure to complete remaining FPD upgrade

Note: only if ADM firmware version is 1.14.2
It is important to sync all filesystems on all VMs/host prior to use the reload script

[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.4 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.4 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 chvrf 0 ssh 10.0.2.2 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 chvrf 0 ssh 10.0.2.2 sync

no output is expected for these commands

then reload the chassis using the special command below

[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/reboot_system.sh
Initiating power-cycle of 0/RP0
Initiating power-cycle of 0/RP1

Note: only if ADM firmware version is 1.14.3 and some FPD on RP are requiring reload

Standby RP first

sysadmin-vm:0_RP1# hw-module location 0/RP0 reload
Reload hardware module ? [no,yes] yes

and once the redundancy is restored we can reload RP1 (it will trigger a failover)

sysadmin-vm:0_RP1# hw-module location 0/RP1 reload
Reload hardware module ? [no,yes] yes

Verify FPD versions

RP/0/RP0/CPU0:RONALDO-RF2a#sh hw-module fpd
Fri Oct 16 16:24:18.849 CEST
                                                               FPD Versions
                                                               =================
Location   Card type             HWver FPD device       ATR Status   Running Programd
-----------------------------------------------------------------------------------
0/5        A900-IMA8Z            0.0   IMFPGA               NEED UPGD 17.02   17.02
0/14       A900-IMA8CS1Z-M       0.0   IMFPGA               CURRENT    1.95    1.95
0/RP0      N560-RSP4-E           0.0   ADM                  CURRENT    1.06    1.06
0/RP0      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64
0/RP0      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT    0.16    0.16
0/RP0      N560-RSP4-E           0.0   SATA                 CURRENT    2.10    2.10
0/RP1      N560-RSP4-E           0.0   ADM                  CURRENT    1.06    1.06
0/RP1      N560-RSP4-E           0.0   IOFPGA               CURRENT    0.64    0.64
0/RP1      N560-RSP4-E           0.0   PRIMARY-BIOS         CURRENT    0.16    0.16
0/RP1      N560-RSP4-E           0.0   SATA                 CURRENT    2.10    2.10
0/FT0      A907-FAN-E            1.0   PSOC                 CURRENT    1.65    1.65

Verify ADM version

[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.3
Bootloader Version : 0.0.9
Config Version : 1.6

[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.3
Bootloader Version : 0.0.9
Config Version : 1.6

Upgrade other FPDs on other locations (if any)

RP/0/RP0/CPU0:ios#upgrade hw-module location 0/5 FPD IMFPGA
Fri Oct 16 16:28:15.006 CEST
upgrade command issued (use "show hw-module fpd" to check upgrade status)

How to make sure to run latest FPD on NCS560

The Need

SMU Summary Table

FPD Upgrade MOP

Assumptions

Sample System running 7.1.2 and necessary SMUs activated

XR VM

sysadmin

Verify FPD Versions on the RPs

Check TAM area on both RPs (evaluate the removal of the section)

RP0

RP1

Sample output of corrupted TAM Memory

Perform IOFPGA upgrade on both RPs

Verify

RP reload one by one for IOFPGA upgrade to complete

Verify ADM version

Perform remaining FPD upgrade on both RPs

ADM

BIOS

SATA

Verify

Power Cycle procedure to complete remaining FPD upgrade

Note: only if ADM firmware version is 1.14.2It is important to sync all filesystems on all VMs/host prior to use the reload script

Note: only if ADM firmware version is 1.14.3 and some FPD on RP are requiring reload

Verify FPD versions

Verify ADM version

Upgrade other FPDs on other locations (if any)

Note: only if ADM firmware version is 1.14.2
It is important to sync all filesystems on all VMs/host prior to use the reload script