XR upgrade 5.3.1 -> 6.3.3/6.4.2 fails on "install add" claiming corrupt PIE files

[drumfrodo]

We have tried to upgrade a customers network of ASR9001 routers running XR 5.3.1 to 6.3.3, and also 6.4.2, but both of them fails when attempting the "install add"-operation. Image download and files has been equisitely verified with md5, and the files are not corrupted. However, the ASR9001 seems to think so, for some reason: 

Install operation 79 '(admin) install add tar

    /sftp://********:*************@yyy.yyy.yyy.yyy/ASR9K-iosxr-px-k9-6.3.3.tar'

    started by user 'admin' via CLI at 06:44:38 UTC Mon Sep 17 2018.

    Info:     The following files were extracted from the tar file

    Info:     '/tmp/instdir_sftp_files/temp_sftp_file_08257603' and will be

    Info:     added to the entire router:

    Info:

    Info:         README-ASR9K-iosxr-px-k9-6.3.3.txt (skipped - not a pie)

    Info:         gisobuild.py (skipped - not a pie)

    Info:         asr9k-video-px.pie-6.3.3

    Info:         asr9k-services-px.pie-6.3.3

    Info:         asr9k-services-infra-px.pie-6.3.3

    Info:         asr9k-optic-px.pie-6.3.3

    Info:         asr9k-mpls-px.pie-6.3.3

    Info:         asr9k-mini-px.pie-6.3.3

    Info:         asr9k-mgbl-px.pie-6.3.3

    Info:         asr9k-mcast-px.pie-6.3.3

    Info:         asr9k-li-px.pie-6.3.3

    Info:         asr9k-k9sec-px.pie-6.3.3

    Info:         asr9k-fpd-px.pie-6.3.3

    Info:         asr9k-doc-px.pie-6.3.3

    Info:         asr9k-bng-px.pie-6.3.3

    Info:         asr9k-asr9000v-nV-px.pie-6.3.3

    Info:

    Error:    Cannot proceed with the add operation because the pie file

    Error:    '/tmp/install/tar/instdir/8918452_223000000/asr9k-video-px.pie-6.

    Error:    3.3' is corrupt.

    Error:

    Error:    The following pies were not added due to an error:

    Error:        asr9k-video-px.pie-6.3.3

    Error:        asr9k-services-px.pie-6.3.3

    Error:        asr9k-services-infra-px.pie-6.3.3

    Error:        asr9k-optic-px.pie-6.3.3

    Error:        asr9k-mpls-px.pie-6.3.3

    Error:        asr9k-mini-px.pie-6.3.3

    Error:        asr9k-mgbl-px.pie-6.3.3

    Error:        asr9k-mcast-px.pie-6.3.3

    Error:        asr9k-li-px.pie-6.3.3

    Error:        asr9k-k9sec-px.pie-6.3.3

    Error:        asr9k-fpd-px.pie-6.3.3

    Error:        asr9k-doc-px.pie-6.3.3

    Error:        asr9k-bng-px.pie-6.3.3

    Error:        asr9k-asr9000v-nV-px.pie-6.3.3

    Install operation 79 failed at 06:59:10 UTC Mon Sep 17 2018.

It has also been attempted to extract the .tar on the FTP server, verified the individual PIE-files with the correct md5, and running the install add-operation directly on the PIE-files. However, the result is the same. And this is now the case for two different software releases, and on two different boxes..!

I would suspect that some bridge SMUs or something similar was missing, but there isn't any to add for release 5.3.1, which seems legit as the official docs claims bridge SMUs are not used after Release 4.3. So, has anyone experienced something similar? What could be the problem here?

[smilstea]

What is the MD5 of the video file so I can verify internally? Not the entire tar, just the video pie file.

Also if you copy the video pie file to the router via SFTP and try install add does it work? What is its MD5 (show md5 file will tell us). I want to verify its not an SFTP issue.

Thanks,

Sam

[drumfrodo]

We have verified MD5 on both the tar and individual PIE files. This happens to all files, not just one PIE. And it happens even if the files are manually copied to the router, and md5 checked using show file md5, before doing install add. Adding 5.3.4-files however, seems to work. So we currently suspect undocumented incompatibility/bug in upgrading directly from 5.3.1 to 6.x..

[smilstea]

The error message only complains about the video pie file, however the install add is an all or nothing event so that is why they all fail. Are you saying that other times you attempt the upgrade other pie files are the ones showing as corrupt in the error message?

I still need the MD5 values to verify internally on my end if its an issue with the tar you download / is on cisco.com on something else as I will attempt the install add myself.

Sam

[smilstea]

I have verified both the k9 and regular tar file in my lab.

RP/0/RSP1/CPU0:ASR9006-M#copy ftp://xxx:xxxxx@172.18.104.30/asr9k-video-px.pie-6$
Tue Sep 18 05:26:14.451 EDT
Destination filename [/harddisk:/asr9k-video-px.pie-6.3.3]?
Accessing ftp://xxx:*@172.18.104.30;calo-mgmt/asr9k-video-px.pie-6.3.3
CC
1536980 bytes copied in      0 sec
RP/0/RSP1/CPU0:ASR9006-M#admin
Tue Sep 18 05:26:18.942 EDT
RP/0/RSP1/CPU0:ASR9006-M(admin)#install add harddisk:/asr9k-video-px.pie-6.3.3
Tue Sep 18 05:26:25.760 EDT
Install operation 213 '(admin) install add /harddisk:/asr9k-video-px.pie-6.3.3'
started by user 'smilstea' via CLI at 05:26:25 EDT Tue Sep 18 2018.
The install operation will continue asynchronously.
RP/0/RSP1/CPU0:ASR9006-M(admin)#Info:     The following package is now available to be activated:
Info:
Info:         disk0:asr9k-video-px-6.3.3
Info:
Info:     The package can be activated across the entire router.
Info:
Install operation 213 completed successfully at 05:26:36 EDT Tue Sep 18 2018.

RP/0/RSP1/CPU0:ASR9006-M(admin)#exit
RP/0/RSP1/CPU0:ASR9006-M#show md5 file /harddisk:/asr9k-video-px.pie-6.3.3
Tue Sep 18 05:26:54.841 EDT
21be4aeaca0176468585accfacb210e1
RP/0/RSP1/CPU0:ASR9006-M#

-bash-4.2$ ls -l | grep 6.3.3 | grep video
-rw-r--r--   1 lab  lab      1536980 Aug 28 18:11 asr9k-video-px.pie-6.3.3
-bash-4.2$ md5sum asr9k-video-px.pie-6.3.3
21be4aeaca0176468585accfacb210e1  asr9k-video-px.pie-6.3.3
-bash-4.2$

Thanks,

Sam

[drumfrodo]

What version is your box running? It seems 5.3.1 does not recognize the PIE-files for Release 6.x, and claims they are corrupt. Upgrade to 5.3.4 succeeds however, and after that the 6.3.3-files also succeeds. So there seems to be something with 5.3.1..

[smilstea]

This sounds like the root certificate issue, hard to know without logs. The reason the files would show as corrupt is that they use the new certificate to sign them so they aren't valid with the old certificate.

https://www.cisco.com/c/en/us/td/docs/routers/technotes/MOP-CSS-to-Abraxas.html

Sam

[smilstea]

There appears to be an issue with earlier versions of code, I can reproduce this in my lab on 5.1.3 and we are taking a look.

Thanks,

Sam

[Santosh Sharma]

There can be issue with file transfer to the router. Lets try copying the pie in the router and check md5. Lets do install add incase md5 is proper

[drumfrodo]

We have already done this, and the md5 is verified after the files have been transferred to the router. But when on version 5.3.1 the router still complains that the PIE-files are corrupt (we have tried many of them, not just one or two). However, upgrading to 5.3.4 solves the issue, and we are then able to upgrade to 6.3.3.

[tkarnani]

 Hi All,

from the 6.3.3 upgrade doc, it seems only from 5.3.4 and above is supported.

maybe some file system change was done which makes this seem like a file corruption issue

i have attached the doc

[mivens]

That doc needs adding at

https://www.cisco.com/web/Cisco_IOS_XR_Software/index.html#ASR9000

The latest MOP there is for 6.1.3 (and other platforms not just ASR 9000 also need updating).

[drumfrodo]

Where did you find this document?? I have been searching all over Cisco.com and went through all available Upgrade and Software Management guides for ASR9000 / IOS-XR. Also, Release Notes for 6.3.3 does not mention any prerequisite version for a normal install.

[tkarnani]

it would be nice if it was posted.

however if you go to the download software section, there is a doc.tar

that is where i found it

[nkarpysh · Accepted solution]

Hi All,

Sam has recently posted an update for this problem. Hope it will help to work this around in future:
https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xr-software/213745-ios-xr-upgrade-changes-certificate-expi.html

Plus a not from Xander:

https://community.cisco.com/t5/service-providers-blogs/upgrading-to-6-3-2-plus-releases-coming-from-43x-51x-52x-need-to/ba-p/3714718#M901