ASR-9001 with IOS XR 5.1.3 Turboboot to 6.5.3 Issues

[PK99]

We got a ASR-9001 and it was loaded with 5.1.3 things seemed normal.  We then proceeded to turboboot the 9001 via USB to 6.5.3.  After issuing the commands in ROMMON it was loading the mini file and loaded 6.5.3 and then got this error

......................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
program load complete, entry point: 0x2037e0, size: 0x2850d7b1

Boot Status 17
Config = SMP, Running = SMP
Board type: 0x00100401
Card Capability = 0x00000000
BSP: Board type : ASR9K-RSP-I
tracelogger: starting tracing in background ring mode
tracelogger running with args: -startring -F 1 -F 2
czip:326:mount=/dev/shmem, image=0, remove=0x1
czip: strings=ffff23f8, as=ffff28a0, num=0x340
czip: 356:start_addr=60185000, in_size=0x24dcf1f1

Restricted Rights Legend

Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.

cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706

Cisco IOS XR Software for the Cisco XR ASR9K, Version 6.5.3
Copyright (c) 2019 by Cisco Systems, Inc.
/pkg/lib//install_debug_remote: No such file or directory
Feb 13 11:55:43.438: Install Setup: Booting with committed software
Initializing Punt-switch ...
Initializing Punt-switch ...
Set RP CPU Cluster port in DSA mode
Init STP group for Cluster ports forwarding state control done
Cluster: Boot Request Filter configured
Cluster: Control Link monitor proto filter configured
Successfully initialized punt-switch(26 ports).
SPP has connected to PAKMAN....
SPP registered CPU port 4 with DPAA
SPP Port Idx 0 MAC addr --> 00:70:72:00:00:08
RP/0/RSP0/CPU0:Feb 13 11:55:13.241 UTC: imio[64]: %PLATFORM-IMIO-3-OPERATION_NOTIFY : imio_main: cold start

ios con0/RSP0/CPU0 is now available

Press RETURN to get started.

SYSTEM CONFIGURATION IN PROCESS

The startup configuration for this device is presently loading.
This may take a few minutes. You will be notified upon completion.

Please do not attempt to reconfigure the device until this process is complete.
RP/0/RSP0/CPU0:Feb 13 12:00:14.102 UTC: envmon[211]: %PLATFORM-ENVMON-4-FANTRAY_RPM : Fan tray RPM warning on slot 0/FT0/SP
Feb 13 12:00:20.077 : RP/0/RSP0/CPU0:Feb 13 12:00:20.074 UTC: easybake3[190]: %INSTALL-TURBOBOOT-3-INVALID_DEVICE : The requested install device disk0: is either corrupt or missing. Turboboot cannot be performed. The turboboot settings will be cleared, and the node will reset. Please repair or insert the device and retry the boot sequence.
TURBOBOOT: Failed to obtain turboboot parameters: 'Turboboot' detected the 'fatal' condition 'No devices that qualifies for boot device are found in the system'
Feb 13 12:00:20.105 : TURBOBOOT: Resetting TURBOBOOT rommon variable to (TURBOBOOT=).

So I can see that Turboboot cannot see disk0, so I rebooted and went back to ROMMON.

When I issue dir disk0: it is now the USB DRIVE, along with disk1:

There is a disk0a: with the following files

rommon B10 > dir disk0a:
FAILED
rwxrwxr-x 4096 .
rwxrwxr-x 4096 ..
r--r--r-- 244736 .bitmap
r--r--r-- 8192 .inodes
rw------- 0 .boot
rw------- 0 .altboot
rwxr-xr-x 4096 LOST.DIR
rwxr-xr-x 4096 LOST.DIR
r--r--r-- 0 .longfilenames
rwxr-xr-x 4096 usr
rwxr-xr-x 4096 var
rwx------ 14312 PNEB-RPE-GATE-1
rwx------ 14312 PNEB-RPE-GATE-1
rommon B11 > dir
Incorrect Usage
rommon B12 > dir help
Device not found help
rommon B13 > dir?

monitor: command "dir?" not found
rommon B14 > dir ?
Device not found ?
rommon B15 > dir:

monitor: command "dir:" not found

issuing dir ? now gives me incomplete warning

I originally issued

turboboot=on,disk0,format

but should I actually point it disk0a: ?

turboboot=on,disk0a,format

or do I need to leave that alone?  

Any help is appreciated as this 9001 is just "dead" state right now.

[PK99]

So I was able to get 5.3.4 mini vm file and was able to get router into IOS XR.  I then formatted disk0: (it sees all the disks in IOS XR).  I copied 5.3.4.tar file to disk0:

Performed - admin install add tar disk0:file name.tar

then I noticed all the packages were expanded to mem: instead of disk0:

so I activated mem:*5.4.3*

then Install commit

I rebooted, then it went to ROMMON.  I could not fid the files anywhere ROMMON, so I had to boot off mini vm again from USB.

Not sure what's going on at this point.

Don't know why it keeps putting the packages in mem:  (I can't find the actual location)

See below

RP/0/RSP0/CPU0:ios#sh install log 2 detail
Sat Feb 13 17:16:31.435 UTC

Install operation 2 started by user 'root' via CLI at 17:09:15 UTC Sat Feb 13
2021.
(admin) install add tar /disk0:ASR9K-iosxr-px-k9-5.3.4.tar
Install operation 2 completed successfully at 17:16:16 UTC Sat Feb 13 2021.

Install logs:
Install operation 2 '(admin) install add tar
/disk0:ASR9K-iosxr-px-k9-5.3.4.tar' started by user 'root' via CLI at
17:09:15 UTC Sat Feb 13 2021.
Info: The following files were extracted from the tar file
Info: '/disk0:ASR9K-iosxr-px-k9-5.3.4.tar' and will be added to the
Info: entire router:
Info:
Info: asr9k-video-px.pie-5.3.4
Info: asr9k-optic-px.pie-5.3.4
Info: asr9k-mgbl-px.pie-5.3.4
Info: asr9k-k9sec-px.pie-5.3.4
Info: asr9k-bng-px.pie-5.3.4
Info: asr9k-services-px.pie-5.3.4
Info: asr9k-mpls-px.pie-5.3.4
Info: asr9k-mcast-px.pie-5.3.4
Info: asr9k-fpd-px.pie-5.3.4
Info: asr9k-asr901-nV-px.pie-5.3.4
Info: README-asr9k-k9sec-px-5.3.4.txt (skipped - not a pie)
Info: asr9k-services-infra-px.pie-5.3.4
Info: asr9k-mini-px.pie-5.3.4
Info: asr9k-li-px.pie-5.3.4
Info: asr9k-doc-px.pie-5.3.4
Info: asr9k-asr9000v-nV-px.pie-5.3.4
Info:
Warning: Skipped adding the following package as it was already present:
Warning: mem:asr9k-mini-px-5.3.4
Warning: Please check:
Warning: - the set of active packages using '(admin) show install
Warning: active'.
Warning: - the set of inactive packages using '(admin) show install
Warning: inactive'.
Info: The following packages are now available to be activated:
Info:
Info: mem:asr9k-video-px-5.3.4
Info: mem:asr9k-optic-px-5.3.4
Info: mem:asr9k-mgbl-px-5.3.4
Info: mem:asr9k-k9sec-px-5.3.4
Info: mem:asr9k-bng-px-5.3.4
Info: mem:asr9k-services-px-5.3.4
Info: mem:asr9k-mpls-px-5.3.4
Info: mem:asr9k-mcast-px-5.3.4
Info: mem:asr9k-fpd-px-5.3.4
Info: mem:asr9k-asr901-nV-px-5.3.4
Info: mem:asr9k-services-infra-5.3.4
Info: mem:asr9k-li-px-5.3.4
Info: mem:asr9k-doc-px-5.3.4
Info: mem:asr9k-9000v-nV-px-5.3.4
Info:
Info: The packages can be activated across the entire router.
Info:
Install operation 2 completed successfully at 17:16:16 UTC Sat Feb 13 2021.

--------------------------------------------------------------------------------

Any ideas?

[smilstea · Accepted solution]

So you did turboboot properly, the issue is during the turboboot process the system couldn't read disk0 so instead of baking the image to disk0 it just put the image in memory. We use membooting internally since we change images a lot and dont want to wear out the disks, but for production environments you don't want that.

My suspicion is that the disk has gone bad based on what you have said and the 9001 will need to be replaced.

Sam

[PK99]

When I turbobooted it would get to add a password section then it would fail and reboot with a message about'Turboboot' detected the 'fatal' condition 'No devices that qualifies for boot device are found in the system'

Then I manually pointed ROMMON to boot off disk1: (USB) and told it to boot min-vm.  I would successfully get to IOS, and ever see disk0:  I formatted disk0:, added the entire tar file to disk0 successfully but when I performed the install add tar, it would not write to disk0 but rather mem:

Strange thing is, it clearly sees disk0: as I success fully copied different tar files to it.  Just won't extract the pie files to disk0 for some reason.

I'm tempted to open the case and take out the 6GB flash and put a good one from a RSP440 and format it and see what happens.  

We've had this unit as a test unit forever I just learned and was just sitting on a rack.  

If the unit is bad, I probably will need to RMA it but I' guessing our Smartnet on this unit is probably long expired.   Will Cisco allow us to just purchase another new contract for this?

I found like 4 different parts - 

CON-SNT-ASR9001

CON-SNT-ASR90012

CON-SNT-ASR90014

I can't find any info on the differences.  We don't need anything but NBD for our lab stuff since it isn't critical.

Couple things Sam, could you shed some light on the disk portion above?  Strange thing is the 9901 was functioning perfectly fine when I booted it up and let it run for over an hour with MPA-20X1GE.  Then I shut it down and did the first Turboboot on it to 6.5.3 (I later learned I accidently copied x64 version to the USB and I TURBOBOOTED the x64 mini vm image by accident and all hell broke loose from there.  

2nd - could you shed some light on the differences of the CON-SNT?

Thanks!

[PK99]

Tried TURBO BOOT again today here is what I get

Try to load image as per local BOOT variable
Unable to parse local BOOT variable

Boot Status 15
rommon B7 > TURBOBOOT=on,disk0,format
rommon B8 > sync
rommon B9 > boot disk1:/asr9k-mini-px.vm-5.3.4

Boot Status 16
FAILED
Located asr9k-mini-px.vm-5.3.4, start cluster is 199
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

Under Boot Status 16 I notice it says FAILED.  I reckon that's the part that it cannot see disk0

[PK99]

UPDATE -

After many tries, Turboboot FINALLY worked....

in ROMMON i booted into disk1:mini-vm file

Once in IOSXR (although lite version since all the packages needed to be installed), I formatted disk0:

reloaded router, back to ROMMON.  I proceeded to TURBOBOOT normally but this time for some reason it seemed be baking properly. 

Thanks again Sam!