NCS 5501-SE reboot after upgrade from 6.6.3 to 7.1.2

FabriceLaman · ‎12-14-2021

Hi all,

I upgraded NCS5501-SE from 6.6.3 to 7.1.2, upgrade went smooth, but after upgrading fpd, i this message telling to reload.

RP/0/RP0/CPU0:RT#sh hw-module fpd
Tue Dec 14 14:15:22.672 UTC
FPD Versions
=================
Location Card type HWver FPD device ATR Status Running Programd
-----------------------------------------------------------------------------------
0/RP0 NCS-5501-SE 1.1 MB-MIFPGA CURRENT 1.02 1.02
0/RP0 NCS-5501-SE 1.1 Bootloader RLOAD REQ 1.15 1.22
0/RP0 NCS-5501-SE 1.1 CPU-IOFPGA UPGD FAIL 1.14 1.14
0/RP0 NCS-5501-SE 1.1 MB-IOFPGA CURRENT 1.11 1.11
RP/0/RP0/CPU0:RT#

after the reload, i cannot access CLI and the router reboot continuously.

How to fix it ? Thanks

tkarnani · ‎12-14-2021

Can you provide the console logs so we can see why the router is in a reboot loop?

we may have to try a USB boot if the router does not stabilize, it will wipe the disk and try to bring it up with a fresh install.

you can download the usb boot file from here, extract it to a usb and try to boot from it. esc or del to get into the bios and select usb boot

https://software.cisco.com/download/home/286291132/type/280805694/release/7.3.2

FabriceLaman · ‎12-21-2021

Hi @ tkarnani,

cannot access the CLI, and the USB boot don't work.

Thank.

decode.chr13 · ‎12-14-2021

Hi,

You need first to upgrade all succesfully. Use:

upgrade hw-module location all fpd all force

Then your status will have only CURRENT and RLOAD REQ. Use this command (will reload the entire router!):

admin hw-module location all reload

or

reboot the router by unpluging from power.

HTH

fracjackmac · ‎12-19-2021

Hello @FabriceLaman,

I'm not sure if you're still dealing w/ the booting issue, but I noticed an item in the output provided that you didn't mention: the CPU-IOFPGA firmware upgrade failure.

0/RP0   NCS-5501-SE       1.1   Bootloader    RLOAD REQ   1.15  1.22
0/RP0   NCS-5501-SE       1.1   CPU-IOFPGA    UPGD FAIL   1.14  1.14

The post here (Upgrade from 7.0.1 to 7.0.2) provides information about the failed CPU-IOFPGA upgrade with a warning about "bricking the board".

If you're still stuck in a boot loop, please try removing power from the unit as the post and @decode.chr13 suggest.

If this doesn't solve the boot loop issue, you can try the USB Boot that @tkarnani suggests, but I suspect you'll need to open a ticket w/ TAC to determine if the firmware upgrade failure is preventing the router from booting to a stable state.

Good luck!!!

fjm

@ittybittypacket

pau.nadeu · ‎04-08-2022

Hello Fabrice,

We are experiencing same problem on NCS-5501-SE. Downgrade from 6.x to 7.2.2 was ok, then we decided to upgrade 7.2.2 to 7.4.2 having same behaviour, boot loop. We tried to connect console cable (we was using to upgrade), but we don't get any output (we sometimes see some "?" signs on it, baudrate it's set on 1152200, 8n1)

Here it's the log of what's happen before the problem:

RP/0/RP0/CPU0:cr01.bcneq01.es#install prepare id 26
Thu Apr  7 20:34:39.662 UTC
Apr 07 20:34:41 Install operation 28 started by root:
  install prepare id 26
Apr 07 20:34:41 Package list:
Apr 07 20:34:41  ncs5500-ospf-3.0.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-mpls-te-rsvp-3.1.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-eigrp-1.0.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-k9sec-3.1.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-mcast-3.0.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-mpls-2.1.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-li-1.0.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-isis-2.1.0.0-r742.x86_64
Apr 07 20:34:41  ncs5500-mini-x-7.4.2
Apr 07 20:34:41  ncs5500-mgbl-3.0.0.0-r742.x86_64
Apr 07 20:34:41 Install operation will continue in the background

RP/0/RP0/CPU0:cr01.bcneq01.es#show install request
Thu Apr  7 20:38:02.604 UTC
User nearip, Op Id 28
install prepare
ncs5500-isis-2.1.0.0-r742.x86_64
ncs5500-eigrp-1.0.0.0-r742.x86_64
ncs5500-mpls-2.1.0.0-r742.x86_64
ncs5500-mgbl-3.0.0.0-r742.x86_64
ncs5500-li-1.0.0.0-r742.x86_64
ncs5500-mpls-te-rsvp-3.1.0.0-r742.x86_64
ncs5500-mini-x-7.4.2
ncs5500-mcast-3.0.0.0-r742.x86_64
ncs5500-ospf-3.0.0.0-r742.x86_64
ncs5500-k9sec-3.1.0.0-r742.x86_64
install operation 28 is in progress
Triggered prepare operation in sysadmin.

Updates from sysadmin for this operation.
Install prepare operation 28 is in progress
Initial config parsed. Stage one orchestration in progress.
Stage one orchestration completed.
Stage two orchestration started.
Monitor progress for prepare operation.
Prepare operation completed.
Node(s)                   State                     Action
0/RP0                     Completed                 Partition preparation completed


0/RP0/ADMIN0:Apr  7 20:39:40.893 UTC: card_mgr[3109]: %DRIVER-CARD_MGR-1-UPGD_SKIP : FPD upgrade skipped for CPU-IOFPGA @location 0/RP0 .The Bootloader primary or golden version on the box is not compatible with the new packaged CPU-IOFPGA. Kindly 'force' upgrade the Bootloader prior to manually upgrading CPU-IOFPGA to avoid bricking the board.
0/RP0/ADMIN0:Apr  7 20:39:40.893 UTC: card_mgr[3109]: %INFRA-FPD_Driver-1-UPGRADE_ALERT : FPD CPU-IOFPGA@0/RP0 image programming completed with UPGD FAIL state Info: []
0/RP0/ADMIN0:Apr  7 20:39:40.947 UTC: card_mgr[3109]: %DRIVER-CARD_MGR-1-GOLDEN_UPGD : FPD upgrade in progress on backup Bootloader (Backup) @location 0/RP0, power-cycle/reload/configuration change on the node is not recommended as it might cause HW programming failure and result in RMA of the hardware.

# upgrade hw-module location 0/RP0 fpd CPU-IOFPGA force
# show hw-module fpd

After this, the current status of router it's like you can see at bottom video.

(view in My Videos)

FabriceLaman · ‎04-10-2022

Hi,

I see that you use baudrate set to *1152200*... use *115200 *to connect to
the console.

Best regards,

Fabrice LAMAN

pau.nadeu · ‎04-10-2022

Sorry, was a typo error. Using 115200 it’s giving blank output or some “?” sometimes. How you fixed bootloader / CPUIOFPGA?

FabriceLaman · ‎04-11-2022

Hi Pau.Nadeu

For the issue with *CPUIOFPGA*, I had no solution, even with the Cisco TAC
Support, which asked me to make an RMA.

Also when you upgrade or downgrade on IOS XR, you have to give the router
time to upgrade or downgrade the line cards as it does it in the
background, so if you run another upgrade or configuration it could crash
the line cards or boot loader, like the current case.

Best regards,

pau.nadeu · ‎04-11-2022

Already done successfully on other NE-5501-SE. It was strange. Supplier will do RMA.

jmartinez20 · ‎01-10-2023

I'm having a similar issue, I'm not able to boot, I tried to recover from BOOT image via usb and now it's just stuck in with message:

Booting CISCO Route Processor Module

Any help or advice is appreciated. image used: ncs5500-usb_boot-7.8.1

smilstea · ‎01-11-2023

This thread is about FPDs not upgrading properly. Is your issue with FPDs or the router not booting at all?

Are you on the right console? You can use ctrl + o as in orange to switch between the 3 consoles.

How long did you wait for the RP to boot?

Do you have any session logs that can be shared to show the bootloader, booting, reimage, etc?

Sam

jmartinez20 · ‎01-11-2023

I was able to press CRTL + O to boot up from USB.I waited for about 1hr after using the Boot image, and still nothing. The unit keeps rebooting and only sees the message below: Booting CISCO Route Processor Module

Thank you for your reply Sam,

smilstea · ‎01-12-2023

How did you decompress the tar file to the USB?

These are the steps:

Procedure

Step 1

Connect the USB drive to your local machine and format it with FAT32 or MS-DOS file system using the Windows Operating System or Apple MAC Disk Utility.

Step 2

Copy the compressed boot file to the USB drive.

Step 3

Verify that the copy operation is successful. To verify, compare the file size at source and destination. Additionally, verify the MD5 checksum value.

Step 4

Extract the content of the compressed boot file by unzipping it inside the USB drive. This converts the USB drive to a bootable drive.

Note

The content of the zipped file ("EFI" and "boot" directories) should be extracted directly into root of the USB drive. If the unzipping application places the extracted files in a new folder, move the "EFI" and "boot" directories to root of the USB drive.

Step 5

Eject the USB drive from your local machine.

Also what size is the USB drive and is it formatted FAT32?

IF we have verified these steps are correct then we should open a TAC case as we would need a webex to do some deeper troubleshooting.

Sam

jmartinez20 · ‎01-12-2023

shouldn't the unit at least give me the option to go into maintenance mode? Where I can at least run the USB. It's not letting me do it anymore. Thank you for your insight. I think we need to RMA.