NC55-RP embraced the definition of insanity after upgrade to 7.7.21

ThomasD86 · ‎02-29-2024

Hi,

as per the title, after we upgrade a NCS5508 to 7.7.21 from 7.3.2, the standby RP can no longer be utilised, it goes in a boot loop.
What I've observed is the following:

The RP can bake the XR image successfully however at the end I see this in the logs:

0/RP1/ADMIN0:2024 Feb 29 13:19:47.195 : inst_agent[2029]: %INFRA-INSTAGENT-4-XR_PART_PREP_RESP : SDR/XR partition preparation completed successfully  
RP/0/RP1/CPU0:2024 Feb 29 13:20:09.586 : tmgctrl[327]: %PLATFORM-CLKCTRL_MAIN-6-NOT_SUPPORTED : Frequency and Time Synchronization is not supported on this board 
RP/0/RP1/CPU0:2024 Feb 29 13:20:11.822 : show_enc_status_edm[159]: sysdb edm register failed: ['sysdb' detected the 'warning' condition 'No role obtained from Process Manager it may not be ready. Please retry again']
RP/0/RP1/CPU0:2024 Feb 29 13:20:12.377 : syslogd[190]: %SECURITY-PKI-6-LOG_INFO_DETAIL : FIPS mode function notification registration failed, retrying with ECM -  'sysdb' detected the 'try again' condition 'The SysDB server detected a timeout from the system.Operation may need to be retried again.' 
RP/0/RP1/CPU0:2024 Feb 29 13:20:12.719 : packet[304]: %PKT_INFRA-PAKSRV-7-SIGNAL : signal: packet cfg init not applied. 
RP/0/RP1/CPU0:2024 Feb 29 13:20:13.824 : show_enc_status_edm[159]: sysdb edm register failed: ['sysdb' detected the 'warning' condition 'No role obtained from Process Manager it may not be ready. Please retry again']
RP/0/RP1/CPU0:2024 Feb 29 13:20:15.825 : show_enc_status_edm[159]: sysdb edm register failed: ['sysdb' detected the 'warning' condition 'No role obtained from Process Manager it may not be ready. Please retry again']
RP/0/RP1/CPU0:2024 Feb 29 13:20:09.586 : tmgctrl[327]: %PLATFORM-CLKCTRL_MAIN-6-NOT_SUPPORTED : Frequency and Time Synchronization is not supported on this board 
RP/0/RP1/CPU0:2024 Feb 29 13:20:11.822 : show_enc_status_edm[159]: sysdb edm register failed: ['sysdb' detected the 'warning' condition 'No role obtained from Process Manager it may not be ready. Please retry again']
RP/0/RP1/CPU0:2024 Feb 29 13:20:12.377 : syslogd[190]: %SECURITY-PKI-6-LOG_INFO_DETAIL : FIPS mode function notification registration failed, retrying with ECM -  'sysdb' detected the 'try again' condition 'The SysDB server detected a timeout from the system.Operation may need to be retried again.' 
RP/0/RP1/CPU0:2024 Feb 29 13:20:18.459 : rmf_svr[429]: %HA-REDCON-6-GO_STANDBY : this card going standby, location RP/0/RP1/CPU0 
RP/0/RP1/CPU0:2024 Feb 29 13:20:22.129 : syslog_dev[119]: sdr_instagt[361] PID-3013: RL: waiting for completion of shutdown delay 30 secs, pending secs 29 
RP/0/RP1/CPU0:2024 Feb 29 13:20:22.129 : sdr_instagt[361]: %INFRA-REBOOT_LIB-6-SHUTDOWN_DELAY : Waiting for completion of shutdown delay 30 secs, pending 29 secs  
RP/0/RP1/CPU0:2024 Feb 29 13:20:51.129 : syslog_dev[119]: sdr_instagt[361] PID-3013: RL: Reboot initiated with code 36, cause Reboot triggered by failed install operation reboot_timeout 30 shutdown delay 30 
RP/0/RP1/CPU0:2024 Feb 29 13:20:51.129 : sdr_instagt[361]: %INFRA-REBOOT_LIB-5-REBOOT_INITIATED : Reboot initiated with code:36 cause:'Reboot triggered by failed install operation' reboot timeout:30 shutdown delay: 30

At this point the RP reboots and then starts once again to bake the XR image only to fail at the end over and over (and over). While I appreciate the tenacity of the RP, is there a way to stop it?

Thanks

anpetit · ‎02-29-2024

Hello @ThomasD86 ,
Hope you are doing well.

In admin mode you can shut down the RP.
You can use: admin show controller card-mgr inventory summary to check which RP is the master/slave one.

Then you can proceed with:

RP/0/RP0/CPU0:NCS5500#admin
sysadmin-vm:0_RP0# hw-module location 0/RP0 or 1 shutdown

Once this is done, your passive RP should stop to be stuck in boot loop.

I would then tried to re-image with the following CLI below. Note, this CLI can take 20 up to 50minutes.

RP/0/RP0/CPU0:NCS5500#admin hw-module location 0/RP0 or 1 bootmedia network reload

Let me know if it helps you, otherwise a TAC case will be needed to troubleshoot and RCA the issue properly.

Have a nice day.
Kind Regards,
Antoine

ThomasD86 · ‎03-02-2024

Hi Antoine,

I am doing fine thank you, hope the same goes for you.
I've tried to reimagine the RP1 with the cli provided but nothing changes. The RP bakes the XR images successfully, for a short while after that (roughly 15-20 second) the RP role changes to "(Standby)" and the status to "BOOTING" but then it reloads.

We even tried to reload the chassis to see if that would help but it actually made things worse as not only the RP1 is still not working but now even the LC equipped on the box started to display the same behaviour as the RP. I ended up raising a SR to TAC to help me figure this out.

Wish you a good day,
Regards

anpetit · ‎03-13-2024

Hello @ThomasD86 ,

Thanks for your reply and sorry for my late reply.
Hope you have now a solution to your issue. We can close this thread if you want.

Have a nice day.
Antoine

ThomasD86 · ‎03-13-2024

Hi Antoine,

ultimately we fixed the issue by re-imaging the router with an usb boot drive.

Regards