ios xr upgrade software from 7.2.2 to 7.5.5

username11 · ‎09-15-2022

this error which I can not resolve

install package replace /harddisk:/ncs540l-x64-7.5.2.iso
Thu Sep 15 08:27:40.385 UTC
Install replace operation 6.1.1 has started
Install operation will continue in the background
RP/0/RP0/CPU0:ios#RP/0/RP0/CPU0:Sep 15 08:27:50.214 UTC: instorch[263]: %INFRA-INSTALL-3-ACTION_FAILED : Packaging operation 6.1.1 failed - Internal error on 1 node. First error: 0/RP0/CPU
RP/0/RP0/CPU0:Sep 15 08:27:50.369 UTC: instorch[263]: %INFRA-INSTALL-3-ERROR_OUTSIDE_TRANSACTION : Install action 'Replace' encountered an error outside a transaction: Internal error on 1

smilstea · ‎09-15-2022

Can you share the show install log detail, specifically for operation 6?

Sam

username11 · ‎09-16-2022

RP/0/RP0/CPU0:ios#show install log detail
Fri Sep 16 08:12:43.190 UTC
2022-09-16 08:12:04 UTC Transaction 9 started
2022-09-16 08:12:04 UTC Atomic change 9.1 started
2022-09-16 08:12:04 UTC Packaging operation 9.1.1 started
2022-09-16 08:12:04 UTC Replace
2022-09-16 08:12:04 UTC /harddisk:/ncs540l-x64-7.5.2.iso
2022-09-16 08:12:10 UTC Packaging operation 9.1.1 failed: Internal error on 1 node. First error: 0/RP0/CPU0: [Errno 2] No such file or directory: '/boot/bzImage-4.8.28-WR9.0.0.20_cgl' -> '/boot/bzImage-4.8.28-WR9.0.0.20_cgl-install-backup-1663315929.bak_3570_140176538415168'RP/0/RP0/CPU0:ios#

smilstea · ‎09-16-2022

As I mentioned in my second comment this upgrade is unsupported, you will need to go to the break release of 7.3.1 first or ipxe boot the router with 7.5.2.

Sam

smilstea · ‎09-15-2022

Upgrade from 7.2.2 to 7.5.2 is not supported. Unfortunately the upgrade guide for 7.5.2 is not available yet.

This is from the 7.4.2 upgrade guide but the same applies to 7.5.2:

7.3.1 is considered a break release for direct upgrade, due to a known issue with kernel cache corruption that was fixed in releases 7.2.1 and 7.3.1. Please refer to CSCvr79738 for more information. This issue applies to new NCS540L PIDs shipped with 7.0.1, but it is possible that certain affected units that were subsequently upgraded to other production releases might fail the upgrade to release 7.3.1 from any previous releases. With this bug, operations on very large files, for example, an iso image, may fail because of cache corruption. In case of failure, clear the cache and retry the install operation.
Hence software upgrade/downgrade using "install replace" or "install package replace" CLI from 7.0.x/7.1.x/7.2.x to 7.4.2 is NOT supported and will need to be done via re-image using iPXE/USB boot.

https://www.cisco.com/c/en/us/td/docs/iosxr/ncs5xx/system-setup/72x/b-system-setup-cg-72x-ncs540/medium-xr-routers-overview.html#id_114188

Can you try ipxe to reimage router or upgrade to 7.3.1 first?

Sam

marx82 · ‎09-23-2022

Hi @smilstea

I have case 694308713 open but perhaps other people might be interested.

I have a test node where I managed to upgrade from 7.1.2 to 7.3.1.
It would be nice if you could share how to identify the devices where the kernel is corrupted .

I have 4000 node waiting to be upgraded and this issue could become very expensive.

RP/0/RP0/CPU0:004#show install active summary
Fri Sep 23 09:24:53.909 CEST
Active Packages: XR: 151 All: 1198
Label: 7.3.1
Software Hash: 883356826270dff0d2727c2212cfa8fb

Optional Packages Version
---------------------------------------------------- ---------------------------
xr-bgp 7.3.1v1.0.0-1
xr-ipsla 7.3.1v1.0.0-1
xr-is-is 7.3.1v1.0.0-1
xr-lldp 7.3.1v1.0.0-1
xr-mcast 7.3.1v1.0.0-1
xr-mpls-oam 7.3.1v1.0.0-1
xr-ncs540l-mcast 7.3.1v1.0.0-1
xr-ncs540l-netflow 7.3.1v1.0.0-1
xr-netflow 7.3.1v1.0.0-1
xr-ospf 7.3.1v1.0.0-1
xr-perfmgmt 7.3.1v1.0.0-1
xr-track 7.3.1v1.0.0-1

smilstea · ‎09-23-2022

I don't see a corrupt kernel from these outputs, how did you determine the kernel was corrupt? Did you follow the upgrade guide and install all prerequisite SMUs to prevent corruption?

Looking at your ticket its the same thing I have said in this forum post, please check the upgrade guide before upgrading, it lists the supported upgrades for a reason. Upgrading directly to 7.4.2 or 7.5.x is not supported and will lead to problems, the upgrade guide says to upgrade to 7.3.1 first, this will prevent the corruption. Or you can iPXE/USB boot directly to avoid 2-step upgrade.

Sam

marx82 · ‎09-26-2022

Hi,

I didn't verify that the kernel was corrupted. Reading CSCvr79738 , it seems that the issue is triggered by a (kernel) cache issue.
Perhaps the output I copied was confusing in that context, sorry for that.

As I managed to upgrade to 7.3.1 from 7.1.2 , it seems I did not hit that issue so perhaps in some cases the upgrade is possible to 7.3 (and next releases) without using a USB drive as indicated in the upgrade guide. If I would be able to check how many devices are affected by issue CSCvr79738, perhaps that could simplify the upgrade procedure.

EDIT: actually, after reading CSCvr79738 once again , the trigger of the kernel cache corruptions seems to be the file size of the ISO. So if I knew what was the size that does not cause corruption of the kernel cache.....we could probably upgrade the nodes without problems...I suppose.

EDIT2: attaching 7.3.1 installation guide section stating that 7.3.1 required USB boot (which does not seem to be the case?)