Buy or Renew
Buy or Renew
Cisco + Splunk: It’s a new day for your data. Learn more.
 
cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
cancel
Start a conversation
1691
Views
0
Helpful
5
Replies

NCS540 upgrade process from 7.3.2 to 7.5.2 success, fails after reload

decode.chr13
Level 1
Level 1

‎10-20-2022 01:12 AM

Hello,

I'm trying to upgrade N540-ACC-SYS with AC power supplies from 7.3.2 to 7.5.2, but I encounter a major issue related to power supplies not being recognized.

I also tried to usb boot directly to 7.5.2 and the same thing happends.

rtr# show platform
Node Type State Config state
--------------------------------------------------------------------------------
0/PM0 N540-PWR400-D UNKNOWN

I'll paste the whole process below.

=========== currently running 7.3.2 (pre-checks before upgrade) =============================
rtr# show ver
Cisco IOS XR Software, Version 7.3.2
Copyright (c) 2013-2021 by Cisco Systems, Inc.

Build Information:
Built By : ingunawa
Built On : Wed Oct 13 20:17:42 PDT 2021
Built Host : iox-ucs-021
Workspace : /auto/srcarchive17/prod/7.3.2/ncs540/ws
Version : 7.3.2
Location : /opt/cisco/XR/packages/
Label : 7.3.2

cisco NCS-540 () processor
System uptime is 15 hours 10 minutes


rtr# show platform
Node Type State Config state
--------------------------------------------------------------------------------
0/RP0/CPU0 N540-ACC-SYS(Active) IOS XR RUN NSHUT
0/RP0/NPU0 Slice UP
0/FT0 N540-FAN OPERATIONAL NSHUT
0/FT1 N540-FAN OPERATIONAL NSHUT
0/FT2 N540-FAN OPERATIONAL NSHUT
0/FT3 N540-FAN OPERATIONAL NSHUT

rtr# show hw-module fpd
Auto-upgrade:Enabled
FPD Versions
==============
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------------
0/RP0 N540-ACC-SYS 1.0 MB-MIFPGA CURRENT 0.05 0.05
0/RP0 N540-ACC-SYS 1.0 Bootloader CURRENT 1.14 1.14
0/RP0 N540-ACC-SYS 1.0 CPU-IOFPGA CURRENT 0.07 0.07
0/RP0 N540-ACC-SYS 1.0 MB-IOFPGA CURRENT 0.23 0.23
0/RP0 N540-ACC-SYS 1.0 SATA-M500IT-MU-B CURRENT 4.00 4.00

rtr# show inventory
NAME: "0/RP0", DESCR: "Cisco NCS 540 16G Series Fixed Router 24x10G,8x25G,2x100G"
PID: N540-ACC-SYS , VID: V01, SN: XXXXXX

NAME: "Rack 0", DESCR: "NCS 540 Series 16G 24x1/10GE, 8x10/25GE, 2x100GE Fixed Chassis"
PID: N540-ACC-SYS , VID: V01, SN: XXXXXX

NAME: "0/FT0", DESCR: "NCS 540 Fan"
PID: N540-FAN , VID: V01, SN: N/A

NAME: "0/FT1", DESCR: "NCS 540 Fan"
PID: N540-FAN , VID: V01, SN: N/A

NAME: "0/FT2", DESCR: "NCS 540 Fan"
PID: N540-FAN , VID: V01, SN: N/A

NAME: "0/FT3", DESCR: "NCS 540 Fan"
PID: N540-FAN , VID: V01, SN: N/A

Upgrade process:

=========== 7.5.2 =========================================
rtr# install add source http://10.120.0.37:8000/ NCS540-iosxr-k9-7.5.2.tar
RP/0/RP0/CPU0:2022 Oct 20 07:30:29.310 UTC: sdr_instmgr[1212]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 6 finished successfully

rtr# install activate id 6 noprompt
2022-10-20 07:32:19 Install operation 7 started by admin:
2022-10-20 07:32:19 install activate id 6
2022-10-20 07:32:19 No install operation in progress at this moment
2022-10-20 07:32:19 Checking system is ready for install operation
2022-10-20 07:32:21 'install activate' in progress
2022-10-20 07:32:21 ISO ncs540-mini-x-7.5.2 in input package list. Going to upgrade the system to version 7.5.2.
...
2022-10-20 07:38:31 Action 1: install prepare action completed successfully
2022-10-20 07:38:34 Prepare operation completed. Trigger activate.
This may take a while...
2022-10-20 07:38:34 Prepare completed. Operation ID 7 will be taken up for activate operation
2022-10-20 07:38:34 Activate operation ID is: 7 for 'install activate' ID:7
2022-10-20 07:38:36 Install operation 7 started by admin:
install activate noprompt synchronous
2022-10-20 07:38:38 Action 1: install activate action started
2022-10-20 07:38:39 The software will be activated with reload upgrade
2022-10-20 07:38:41 Following nodes are available for System Upgrade activate:
2022-10-20 07:38:42 0/RP0
2022-10-20 07:41:26 Action 1: install activate action completed successfully
2022-10-20 07:41:37 Install operation 7 finished successfully
RP/0/RP0/CPU0:2022 Oct 20 07:41:37.409 UTC: sdr_instmgr[1212]: %INSTALL-INSTMGR-2-OPERATION_SUCCESS : Install operation 7 finished successfully
2022-10-20 07:41:37 Ending operation 7
RP/0/RP0/CPU0:2022 Oct 20 07:41:37.759 UTC: sdr_instmgr[1212]: %INSTALL-INSTMGR-2-SYSTEM_RELOAD_INFO : The whole system will be reloaded to complete install operation 7

======== Post upgrade, after reboot ========

0/RP0/ADMIN0:Oct 20 07:46:45.898 UTC: shelf_mgr[3168]: %INFRA-SHELF_MGR-3-CARD_HW_FAILED : Card: 0/PM0 hardware state going to FAILED
0/RP0/ADMIN0:Oct 20 07:46:46.035 UTC: card_mgr[3122]: %LIBRARY-PSU-3-INVALID_DATA_IN_IDPROM : Power Module 1 IDPROM has wrong FRU type 0xab010207 or PID N540-PWR400-A.
0/RP0/ADMIN0:Oct 20 07:46:46.060 UTC: pm[3019]: %INFRA-Process_Manager-3-PROCESS_RESTART : Process envmon (IID: 0) restarted
0/RP0/ADMIN0:Oct 20 07:46:53.782 UTC: card_mgr[3122]: %LIBRARY-PSU-3-INVALID_DATA_IN_IDPROM : Power Module 0 IDPROM has wrong FRU type 0xab010207 or PID N540-PWR400-A.
0/RP0/ADMIN0:Oct 20 07:47:01.018 UTC: card_mgr[3122]: %LIBRARY-PSU-3-INVALID_DATA_IN_IDPROM : Power Module 1 IDPROM has wrong FRU type 0xab010207 or PID N540-PWR400-A.
0/RP0/ADMIN0:Oct 20 07:47:01.110 UTC: pm[3019]: %INFRA-Process_Manager-3-PROCESS_RESTART : Process envmon (IID: 0) restarted
RP/0/RP0/CPU0:2022 Oct 20 07:48:05.647 UTC: ct_li_feature_ctl[1297]: %SECURITY-CT_LICTRL-3-ERR : Failed to set default value for LI flag error: 'Consent Token' detected the 'fatal' condition 'Failed to read from TAM'
....
0/RP0/ADMIN0:2022 Oct 20 07:49:47.179 UTC: inst_agent[2614]: %INFRA-INSTAGENT-2-SWP_REVERTED : Reverted committed software profile to active software profile as node has booted from committed partition
0/RP0/ADMIN0:2022 Oct 20 07:49:51.804 UTC: card_mgr[2607]: %LIBRARY-PSU-3-INVALID_DATA_IN_IDPROM : Power Module 0 IDPROM has wrong FRU type 0xab010207 or PID N540-PWR400-A.
0/RP0/ADMIN0:2022 Oct 20 07:49:51.855 UTC: card_mgr[2607]: %LIBRARY-PSU-3-INVALID_DATA_IN_IDPROM : Power Module 1 IDPROM has wrong FRU type 0xab010207 or PID N540-PWR400-A.
0/RP0/ADMIN0:2022 Oct 20 07:49:56.776 UTC: inst_agent[2614]: %INFRA-INSTAGENT-4-SELF_TEST_HOST_ADMIN_MISMATCH_MSG : Sysadmin and host software versions are not same on this node. Trying to recover now
0/RP0/ADMIN0:2022 Oct 20 07:50:10.382 UTC: vm_manager[2669]: %INFRA-VM_MANAGER-4-INFO : Info: vm_manager brought down VM default-sdr--2

==== Post boot checks, after login ======

rtr# show ver
Cisco IOS XR Software, Version 7.5.2
Copyright (c) 2013-2022 by Cisco Systems, Inc.

Build Information:
Built By : ingunawa
Built On : Tue Apr 26 18:53:00 PDT 2022
Built Host : iox-lnx-010
Workspace : /auto/srcarchive14/prod/7.5.2/ncs540/ws
Version : 7.5.2
Location : /opt/cisco/XR/packages/
Label : 7.5.2

cisco NCS-540 () processor
System uptime is 1 minute

rtr# show platform
Node Type State Config state
--------------------------------------------------------------------------------
0/RP0/CPU0 N540-ACC-SYS(Active) IOS XR RUN NSHUT
0/RP0/NPU0 Slice UP
0/FT0 N540-FAN OPERATIONAL NSHUT
0/FT1 N540-FAN OPERATIONAL NSHUT
0/FT2 N540-FAN OPERATIONAL NSHUT
0/FT3 N540-FAN OPERATIONAL NSHUT
0/PM0 N540-PWR400-D UNKNOWN !(This line was not present in 7.3.2)

rtr# show hw-module fpd !(initial at boot)
Auto-upgrade:Enabled
FPD Versions
==============
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------------
0/RP0 N540-ACC-SYS MB-MIFPGA NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_0 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_1 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_10 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_11 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_12 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_13 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_14 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_15 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_16 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_17 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_18 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_19 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_2 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_20 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_21 NOT READY
0/RP0 N540-ACC-SYS SSFP_E1F_22 NOT READY

rtr# show hw-module fpd !(after couple of minutes)
Auto-upgrade:Enabled
FPD Versions
==============
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------------
0/RP0 N540-ACC-SYS 1.0 MB-MIFPGA CURRENT 0.05 0.05
---just this line ---

rtr# show inventory
--no output --

As mentioned, the node has AC power supplies.

Is there anything that I can do to upgrade? Did I miss any required SMU?

Thanks

 

Labels:
0 Helpful
5 Replies 5

anpetit
Cisco Employee
Cisco Employee

‎10-20-2022 08:42 AM

Hi @decode.chr13 , 

It looks like fpd upgrade issue for 0/PM0 and 0/PM1. 

Do you have output from "admin show logging" ?

Otherwise, would it be possible to force these FPD upgrades ? 

PM0
admin
upgrade hw-module location 0/PM0 fpd SDG-PriMCU-DCFW
upgrade hw-module location 0/PM0 fpd SDG-SecMCU-DCFW
hw-module location 0/PM0 reload

PM1
admin
upgrade hw-module location 0/PM1 fpd all
hw-module location 0/PM1 reload

I've shared version with fpd name, and fpd-all. Both should worked. 

Kind Regards,

Antoine

 

0 Helpful

decode.chr13
Level 1
Level 1
In response to anpetit

‎10-20-2022 09:45 AM

Hello Antoine,

Thank you for response.

I attached the admin show logging output.

I tried to force fpd upgrades for PM, but I receive a strange output:

When I'm in admin or XR VM and issue: 
upgrade hw-module location pm-all fpd all force

I get following output

0/RP0/ADMIN0:Oct 20 16:25:00.442 : fpdserv[3827]: %INFRA-FPD_Manager-3-UPGRADE_ERROR : Upgrade command has not be committed yet, please wait ...
0/RP0/ADMIN0:Oct 20 16:25:01.441 : fpdserv[3827]: %INFRA-FPD_Manager-3-UPGRADE_ERROR : Invalid Upgrade command, No FPDs are ready for upgrade

I tried your commands, but I get a syntax error:

sysadmin-vm:0_RP0# upgrade hw-module location 0/PM0 fpd SDG-PriMCU-DCFW
----------------------------------------------^
syntax error: unknown argument

Possible options are:

sysadmin-vm:0_RP0# upgrade hw-module location ?
Possible completions:
0/RP0 all pm-all

... so on this path I can only do: upgrade hw-module location pm-all fpd all (which is giving that output)

if I try in xr vm:

rtr# upgrade hw-module location ?
all All locations
help-loc display All locations available
pm-all Power tray/module specific locations
0/FT0 Fully qualified location specification
0/FT1 Fully qualified location specification
0/FT2 Fully qualified location specification
0/FT3 Fully qualified location specification
0/RP0 Fully qualified location specification

#upgrade hw-module location pm-all fpd ?
WORD input the regexp pattern of the fpd name
all All FPD
help-fpd display all fpds in selected locations

I tried: upgrade hw-module location pm-all fpd SDG-PriMCU-DCFW
Invalid fpd name 

So the only option is upgrade hw-module location pm-all fpd all, which gives that output...

Any more ideas would be great.

Thank you

 

 

 

 

Preview file
5 KB
0 Helpful

decode.chr13
Level 1
Level 1
In response to decode.chr13

‎10-20-2022 09:46 AM

tried to reload the PM0, but got this error:

sysadmin-vm:0_RP0# hw-module location 0/PM0 reload
Reload hardware module ? [no,yes] yes
result Card graceful reload request on 0/PM0 failed.

0 Helpful

anpetit
Cisco Employee
Cisco Employee
In response to decode.chr13

‎10-21-2022 12:47 AM

Hello @decode.chr13 , 

Thank you for the feedback.

In this case, I recommend to open a TAC case if device is under contract, to be able to troubleshoot properly this issue as more output will be needed. 

Have a nice day. 

Kind Regards,

Antoine

0 Helpful

decode.chr13
Level 1
Level 1
In response to anpetit

‎10-24-2022 02:28 AM

Hi Antoine,

We'll look for the service contract to open a TAC.

Thank you for support!

0 Helpful
Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Customers Also Viewed These Support Documents