09-15-2022 01:43 AM
this error which I can not resolve
install package replace /harddisk:/ncs540l-x64-7.5.2.iso
Thu Sep 15 08:27:40.385 UTC
Install replace operation 6.1.1 has started
Install operation will continue in the background
RP/0/RP0/CPU0:ios#RP/0/RP0/CPU0:Sep 15 08:27:50.214 UTC: instorch[263]: %INFRA-INSTALL-3-ACTION_FAILED : Packaging operation 6.1.1 failed - Internal error on 1 node. First error: 0/RP0/CPU
RP/0/RP0/CPU0:Sep 15 08:27:50.369 UTC: instorch[263]: %INFRA-INSTALL-3-ERROR_OUTSIDE_TRANSACTION : Install action 'Replace' encountered an error outside a transaction: Internal error on 1
09-15-2022 08:29 AM
Can you share the show install log detail, specifically for operation 6?
Sam
09-16-2022 01:10 AM
RP/0/RP0/CPU0:ios#show install log detail
Fri Sep 16 08:12:43.190 UTC
2022-09-16 08:12:04 UTC Transaction 9 started
2022-09-16 08:12:04 UTC Atomic change 9.1 started
2022-09-16 08:12:04 UTC Packaging operation 9.1.1 started
2022-09-16 08:12:04 UTC Replace
2022-09-16 08:12:04 UTC /harddisk:/ncs540l-x64-7.5.2.iso
2022-09-16 08:12:10 UTC Packaging operation 9.1.1 failed: Internal error on 1 node. First error: 0/RP0/CPU0: [Errno 2] No such file or directory: '/boot/bzImage-4.8.28-WR9.0.0.20_cgl' -> '/boot/bzImage-4.8.28-WR9.0.0.20_cgl-install-backup-1663315929.bak_3570_140176538415168'RP/0/RP0/CPU0:ios#
09-16-2022 01:27 AM
09-15-2022 02:20 PM
Upgrade from 7.2.2 to 7.5.2 is not supported. Unfortunately the upgrade guide for 7.5.2 is not available yet.
This is from the 7.4.2 upgrade guide but the same applies to 7.5.2:
7.3.1 is considered a break release for direct upgrade, due to a known issue with kernel cache corruption that was fixed in releases 7.2.1 and 7.3.1. Please refer to CSCvr79738 for more information. This issue applies to new NCS540L PIDs shipped with 7.0.1, but it is possible that certain affected units that were subsequently upgraded to other production releases might fail the upgrade to release 7.3.1 from any previous releases. With this bug, operations on very large files, for example, an iso image, may fail because of cache corruption. In case of failure, clear the cache and retry the install operation.
Hence software upgrade/downgrade using "install replace" or "install package replace" CLI from 7.0.x/7.1.x/7.2.x to 7.4.2 is NOT supported and will need to be done via re-image using iPXE/USB boot.
Can you try ipxe to reimage router or upgrade to 7.3.1 first?
Sam
09-23-2022 01:28 AM - edited 09-23-2022 01:29 AM
Hi @smilstea
I have case 694308713 open but perhaps other people might be interested.
I have a test node where I managed to upgrade from 7.1.2 to 7.3.1.
It would be nice if you could share how to identify the devices where the kernel is corrupted .
I have 4000 node waiting to be upgraded and this issue could become very expensive.
RP/0/RP0/CPU0:004#show install active summary
Fri Sep 23 09:24:53.909 CEST
Active Packages: XR: 151 All: 1198
Label: 7.3.1
Software Hash: 883356826270dff0d2727c2212cfa8fb
Optional Packages Version
---------------------------------------------------- ---------------------------
xr-bgp 7.3.1v1.0.0-1
xr-ipsla 7.3.1v1.0.0-1
xr-is-is 7.3.1v1.0.0-1
xr-lldp 7.3.1v1.0.0-1
xr-mcast 7.3.1v1.0.0-1
xr-mpls-oam 7.3.1v1.0.0-1
xr-ncs540l-mcast 7.3.1v1.0.0-1
xr-ncs540l-netflow 7.3.1v1.0.0-1
xr-netflow 7.3.1v1.0.0-1
xr-ospf 7.3.1v1.0.0-1
xr-perfmgmt 7.3.1v1.0.0-1
xr-track 7.3.1v1.0.0-1
09-23-2022 08:47 AM
I don't see a corrupt kernel from these outputs, how did you determine the kernel was corrupt? Did you follow the upgrade guide and install all prerequisite SMUs to prevent corruption?
Looking at your ticket its the same thing I have said in this forum post, please check the upgrade guide before upgrading, it lists the supported upgrades for a reason. Upgrading directly to 7.4.2 or 7.5.x is not supported and will lead to problems, the upgrade guide says to upgrade to 7.3.1 first, this will prevent the corruption. Or you can iPXE/USB boot directly to avoid 2-step upgrade.
Sam
09-26-2022 02:18 AM - edited 09-26-2022 02:42 AM
Hi,
I didn't verify that the kernel was corrupted. Reading CSCvr79738 , it seems that the issue is triggered by a (kernel) cache issue.
Perhaps the output I copied was confusing in that context, sorry for that.
As I managed to upgrade to 7.3.1 from 7.1.2 , it seems I did not hit that issue so perhaps in some cases the upgrade is possible to 7.3 (and next releases) without using a USB drive as indicated in the upgrade guide. If I would be able to check how many devices are affected by issue CSCvr79738, perhaps that could simplify the upgrade procedure.
EDIT: actually, after reading CSCvr79738 once again , the trigger of the kernel cache corruptions seems to be the file size of the ISO. So if I knew what was the size that does not cause corruption of the kernel cache.....we could probably upgrade the nodes without problems...I suppose.
EDIT2: attaching 7.3.1 installation guide section stating that 7.3.1 required USB boot (which does not seem to be the case?)
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide