02-08-2021 10:20 AM - edited 10-01-2021 05:19 AM
In some situation NCS560 RP become unresponsive after reload or powercycle.
In many NCS560 deployments are in remote location, deployment might be large and human intervention should be kept at minimum
Engineering team have been working on a strategy to have functionalities to assess/troubleshoot and recovery such situation remotely
Depending on the release it is important to fulfill the mandatory SMU installation.
All the fixes are available stating from 7.2.2 onward. This guide applies also to more recent version by using the bundled FPD package unless explicitly advised from some Cisco representative
IOS XR Version |
SMU |
Link |
Notes |
6.6.3 |
CSCvt69762 (sysadmin) |
|
|
7.0.1 |
CSCvw42910 (XR) |
link |
It is mandatory to move to the EMR release of the same minor release 7.0.2 |
7.0.2 |
CSCvt69762 (sysadmin) |
|
|
7.1.1 |
CSCvv41516 (sysadmin) |
It is advisable to move to the EMR release of the same minor release 7.1.2 |
|
7.1.2 |
CSCvv41516 (sysadmin) |
|
|
7.2.1 |
CSCvv41516 (sysadmin) |
|
The idea is to reach a situation that FPD version are at or above minimum requirement
ADM |
1.06 |
IOFPGA |
0.64 |
BIOS |
0.16 |
RP0 is the Active RP
RP1 is the Standby RP
below the assumed prompt for various location so that all command to navigate the various location are omitted
XR - RP0
RP/0/RP0/CPU0:ios#
XR - RP1
RP/0/RP1/CPU0:ios#
sysadmin - RP0
sysadmin-vm:0_RP0#
sysadmin - RP1
sysadmin-vm:0_RP1#
sysadmin - RP0 shell
[sysadmin-vm:0_RP0:~]$
sysadmin - RP1 shell
[sysadmin-vm:0_RP1:~]$
below the SMUs that are expected to be active and committed on the box prior to start this process
RP/0/RP0/CPU0:ios#sh install active summary
Active Packages: 10
ncs560-xr-7.1.2 version=7.1.2 [Boot image]
ncs560-mcast-2.0.0.0-r712
ncs560-mgbl-2.0.0.0-r712
ncs560-mpls-1.0.0.0-r712
ncs560-mpls-te-rsvp-2.0.0.0-r712
ncs560-ospf-2.0.0.0-r712
ncs560-k9sec-2.1.0.0-r712
ncs560-iosxr-fwding-2.1.0.1-r712.CSCvp67225
ncs560-os-support-2.1.0.1-r712.CSCvv41516
cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .xr
sysadmin-vm:0_RP0# show install active summ
Active Packages: 4
ncs560-sysadmin-7.1.2 version=7.1.2 [Boot image]
ncs560-sysadmin-ncs560-7.1.2.1-r712.CSCvv41516
cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .admin
cisco-klm-rsp4-0.1.p1-r0.0.r712.CSCvt26907 .host
RP/0/RP0/CPU0:ios#sh hw-module fpd
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP0 N560-RSP4-E 0.0 IOFPGA NEED UPGD 0.53 0.53
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP0 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/RP1 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP1 N560-RSP4-E 0.0 IOFPGA NEED UPGD 0.53 0.53
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP1 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
It was important not to have the TAM area corrupted otherwise a subsequent reload would lead to an RMA
With this MOP, since we are installing FPD version 0.64, this IOFPGA FPD upgrade and RP reload will lead to TAM corruption recovery as per the note reported below
sysadmin-vm:0_RP0# show control-driver info location 0/RP0
<snip>
Card view from local RP:
TAM Memory is not corrupted
Can proceed with IOFPGA upgrade if required
<snip>
sysadmin-vm:0_RP0# show control-driver info location 0/RP1
<snip>
Card view from local RP:
TAM Memory is not corrupted
Can proceed with IOFPGA upgrade if required
<snip>
Note: in case of corrupted TAM Memory this does not represent a big problem as long as we are upgrading at least to 0.64 IOFPGA version
sysadmin-vm:0_RP1# show control-driver info location 0/RP0
<snip>
Card view from local RP:
TAM Memory is corrupted
Do not proceed with IOFPGA upgrade
<snip>
so if all prerequisites reported in this document are fulfilled it is possible to proceed with the upgrade.
The reason why the output says "Do not proceed with IOFPGA upgrade" is because the long term fix have been made available only via IOFPGA 0.64 while the command to verify have been made available earlier
RP/0/RP0/CPU0:ios#upgrade hw-module location all fpd IOFPGA
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP0/CPU0:RONALDO-RF2a#0/RP0/ADMIN0:Oct 16 15:19:26.609 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 IOFPGA FALSE
0/RP0/ADMIN0:Oct 16 15:19:26.610 CEST: fpdserv[4356]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 IOFPGA FALSE
RP/0/RP0/CPU0:ios#sh hw-module fpd
Fri Oct 16 15:22:11.256 CEST
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP0 N560-RSP4-E 0.0 IOFPGA RLOAD REQ 0.53 0.64
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP0 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/RP1 N560-RSP4-E 0.0 ADM NEED UPGD 1.04 1.04
0/RP1 N560-RSP4-E 0.0 IOFPGA RLOAD REQ 0.53 0.64
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS NEED UPGD 0.15 0.15
0/RP1 N560-RSP4-E 0.0 SATA NEED UPGD 1.30 1.30
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
Standby RP first
sysadmin-vm:0_RP0# hw-module location 0/RP1 reload
Reload hardware module ? [no,yes] yes
and once the redundancy is restored we can reload RP0 (it will trigger a failover)
sysadmin-vm:0_RP0# hw-module location 0/RP0 reload
Reload hardware module ? [no,yes] yes
on new active RP1
[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.2
Bootloader Version : 0.0.9
Config Version : 1.4
then on standby RP0
[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.2
Bootloader Version : 0.0.9
Config Version : 1.4
RP/0/RP1/CPU0:ios#upgrade hw-module location all FPd ADM
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 15:57:14.332 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 ADM FALSE
0/RP1/ADMIN0:Oct 16 15:57:14.333 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 ADM FALSE
RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD PRIMARY-BIOS
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 PRIMARY-BIOS FALSE
0/RP1/ADMIN0:Oct 16 16:00:29.257 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 PRIMARY-BIOS FALSE
RP/0/RP1/CPU0:ios#upgrade hw-module location all FPD SATA
upgrade command issued (use "show hw-module fpd" to check upgrade status)
RP/0/RP1/CPU0:RONALDO-RF2a#0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Upgrade for the following FPDs has been committed:
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : Location FPD name Force
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : ==================================================
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP1 SATA FALSE
0/RP1/ADMIN0:Oct 16 16:02:06.472 CEST: fpdserv[4503]: %INFRA-FPD_Manager-1-UPGRADE_ALERT : 0/RP0 SATA FALSE
RP/0/RP1/CPU0:ios#sh hw-module fpd
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM RLOAD REQ 1.04 1.04
0/RP0 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS RLOAD REQ 0.15 0.16
0/RP0 N560-RSP4-E 0.0 SATA RLOAD REQ 1.30 2.10
0/RP1 N560-RSP4-E 0.0 ADM RLOAD REQ 1.04 1.04
0/RP1 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS RLOAD REQ 0.15 0.16
0/RP1 N560-RSP4-E 0.0 SATA RLOAD REQ 1.30 2.10
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.4 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.4 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.4.1 chvrf 0 ssh 10.0.2.2 sync
[sysadmin-vm:0_RP1:~]$chvrf 0 ssh 192.0.0.1 chvrf 0 ssh 10.0.2.2 sync
no output is expected for these commands
then reload the chassis using the special command below
[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/reboot_system.sh
Initiating power-cycle of 0/RP0
Initiating power-cycle of 0/RP1
Standby RP first
sysadmin-vm:0_RP1# hw-module location 0/RP0 reload
Reload hardware module ? [no,yes] yes
and once the redundancy is restored we can reload RP1 (it will trigger a failover)
sysadmin-vm:0_RP1# hw-module location 0/RP1 reload
Reload hardware module ? [no,yes] yes
RP/0/RP0/CPU0:RONALDO-RF2a#sh hw-module fpd
Fri Oct 16 16:24:18.849 CEST
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/5 A900-IMA8Z 0.0 IMFPGA NEED UPGD 17.02 17.02
0/14 A900-IMA8CS1Z-M 0.0 IMFPGA CURRENT 1.95 1.95
0/RP0 N560-RSP4-E 0.0 ADM CURRENT 1.06 1.06
0/RP0 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP0 N560-RSP4-E 0.0 PRIMARY-BIOS CURRENT 0.16 0.16
0/RP0 N560-RSP4-E 0.0 SATA CURRENT 2.10 2.10
0/RP1 N560-RSP4-E 0.0 ADM CURRENT 1.06 1.06
0/RP1 N560-RSP4-E 0.0 IOFPGA CURRENT 0.64 0.64
0/RP1 N560-RSP4-E 0.0 PRIMARY-BIOS CURRENT 0.16 0.16
0/RP1 N560-RSP4-E 0.0 SATA CURRENT 2.10 2.10
0/FT0 A907-FAN-E 1.0 PSOC CURRENT 1.65 1.65
[sysadmin-vm:0_RP0:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.3
Bootloader Version : 0.0.9
Config Version : 1.6
[sysadmin-vm:0_RP1:~]$/opt/cisco/calvados/sbin/show_adm_version.sh
ADM 1266 Version info
Firmware Version : 1.14.3
Bootloader Version : 0.0.9
Config Version : 1.6
RP/0/RP0/CPU0:ios#upgrade hw-module location 0/5 FPD IMFPGA
Fri Oct 16 16:28:15.006 CEST
upgrade command issued (use "show hw-module fpd" to check upgrade status)
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: