One possible explanation is

Aaziz AMHAOUCH · ‎10-13-2015

Hi all,

We need to upgrade an ASR9010 from 4.3.2 to 5.3.0 in order to install VSM cards. We did an upgrade in a test environment and we had the same problem as in this Cisco Bug: CSCuv20072 - Software upgrade causes reboot loop: https://tools.cisco.com/quickview/bug/CSCuv20072

In order to fix this issue, we used Turboboot and we accessed CLI but when we restart the chassis it goes to reboot loop again !!

Is there any solution for this solution ?

Many thanks!

Best Regards,

Aaziz

xthuijs · ‎10-13-2015

Hi Aaziz, did you use the turboboot with the format option?

One thing would be the clean the bootflash and disk from all unused packages.

xander

Aaziz AMHAOUCH · ‎10-13-2015

Hi Alexander,

We used turboboot with format option and it comes up but when we restart the node the problem appears again. (router goes in reboot loop).

Thanks!

Aaziz

xthuijs · ‎10-13-2015

can you verify if there is a (wrong) image on the bootflash?

and also the boot loop, could you paste a cycle of that to see what it is going through, just to see if this is the same or different issue.

cheers

xander

xthuijs · ‎10-13-2015

oh btw, are you willing/able to run 532? 530 is the first release of 53 which then has received 2 rebuilds already to 532, it may be best use of time and investigation to be on the right release if that is possible.

xander

Aaziz AMHAOUCH · ‎10-13-2015

Hi Xander,

We tried 5.3.2 and it gives the same problem: reboot loop

BR,

Aaziz

xthuijs · ‎10-13-2015

Hi Aaziz,

I would need some logging and details to see what is going on and help identify the root cause.

xander

Aaziz AMHAOUCH · ‎10-13-2015

Hi Xander,

Thank you for your help. You can find attached logging from upgrading to 5.3.2

BR,

Aaziz

xthuijs · ‎10-13-2015

I dont think the format option worked, it seems that there are 2 boot vars and images available.

it seems to load bootloader 53 and xr image 43 that indeed will result in bad stuff.

try this first:

from rommon unset boot:

BOOT=disk0:asr9k-os-mbi-4.3.2/0x100305/mbiasr9k-rsp3.vm,1;disk0:asr9k-os-mbi-5.3.2/0x100305/mbiasr9k-rsp3.vm,2;

and reset it to BOOT=disk0:asr9k-os-mbi-5.3.2/0x100305/mbiasr9k-rsp3.vm

if no dice.

unset BOOT

TURBOBOOT=on,disk0,format

sync

and boot the vm then. that should fix your issue.

cheers

xander

Aaziz AMHAOUCH · ‎10-13-2015

Thanks Alexander, I will do it tomorrow and will get back to you.

BR,

Aaziz

Aaziz AMHAOUCH · ‎10-14-2015

Hi Alexander,

We have proceed with the steps thats you have decribes and failed to brings up the system. Kindly find bellow logs :

rommon 2 > unset BOOT
unset: "BOOT" does not exist
rommon 3 > TURBOBOOT=on,disk0,format
rommon 4 > sync
rommon 5 > mediaboot usb:\asr9k-mini-px.vm-5.3.2
boot_disk2 - Launching image.

#################################################################
Program load complete, Entry point: 0x40d9d8, size: 0x1e8d781d
ًَض{£I¾t…+ڑ
éaژمَCPU reset reason = 8 (CPU_RESET_WDOG_POR)
########## Environment Variables ##########
PS1=rommon ! >
?=0
IOX_ADMIN_CONFIG_FILE=
ACTIVE_FCD=1
CLUSTER_RACK_ID=0
CLUSTER_NO_BOOT=
BOOT_DEV_SEQ_CONF=
BOOT_DEV_SEQ_OPER=
BSI=0
TURBOBOOT=on,disk0,format
###########################################
Total DRAM Memory: 6144 MB
MAC Address from cookie: d8:67:d9:66:d1:30
Board type: 0x00100306
Chassis type: 0x00ef02fe
Slot number: 05
Chassis serial: FOX1544GTKV

##########################################################
System Bootstrap, Version 0.71 [ASR9K x86 ROMMON],
Copyright (c) 1994-2012 by Cisco Systems, Inc.
Compiled on Tue 06/18/2013 22:50:54.15 by myekkar

Rommon : 0.71
Ibex Peak : 6
Jasper Forest: 1.0
Zen-JF : 0.7.92
CBC0 : Part 1=16.115, Part 2=16.115, Act Part=2
Laguna : 0.10.0 (primary)
Dao : 1.6.0 (primary)
UTI : 4.9 (primary)
Timex : 0.1.1 (primary)
Board : 4
==========================================================
Sending Boot Image validation request.
HIT CTRL-C to abort
----------- [CTRL-C] Boot Image validation: Premature termination
Boot Image validation aborted.
rommon 1 > priv
You now have access to the full set of monitor commands.
Warning: some commands will allow you to destroy your
configuration and/or system images and could render
the machine unbootable.
rommon 2 > diswd
Watchdog Disabled
rommon 3 >
rommon 3 >
rommon 3 >
rommon 3 > mediaboot usb:\asr9k-mini-px.vm-5.3.2
boot_disk2 - Launching image.

#################################################################
Program load complete, Entry point: 0x40d9d8, size: 0x1e8d781d
ënˆçC[د

Many thanks for your support

Best Regards

xthuijs · ‎10-14-2015

It looks like it doesn't want to turboboot.

in rommon use the command "dev" to see the mounted disk devices.

depending on where your usb is it may be:

disk1, disk2, or usb

we need to use either:

mediaboot usb:/xxx

boot usb:/xxx

boot disk2:/xxx

apologies for this hassle, it should be easier, but we'll get through it once this turboboot succeeds.

worse case, we'll do a tftp boot if the USB can't be read/used.

cheers

xander

Aaziz AMHAOUCH · ‎10-14-2015

Many Thanks for your quick feedback, usb device is recognized. Kindly refer to below logs:

rommon 1 > dir usb:
usb:\Directory of: usb:\

10/13/15 05:11p10/13/15 05:11p <DIR> 4,096 ASR9K-iosxr-px-k9-5.3.0
10/13/15 05:11p10/13/15 05:11p <DIR> 4,096 ASR9K-iosxr-px-5.3.2-turboboot
10/13/15 05:14p10/13/15 05:14p 259,481,600 asr9k-vsm-cgv6-5.3.2.00.ova
10/12/15 02:02p10/12/15 02:02p 259,420,160 asr9k-vsm-cgv6-5.3.0.00.ova
10/13/15 05:16p10/13/15 05:16p <DIR> 4,096 ASR9K-iosxr-px-5.3.0-turboboot
10/13/15 01:03a10/13/15 01:03a <DIR> 4,096 LOST.DIR
09/01/15 10:29a09/01/15 10:29a 538,177,949 asr9k-mini-px.vm-5.3.2
07/27/15 09:44a07/27/15 09:44a <DIR> 20,480 OBS DOC
05/31/15 11:13p05/31/15 11:13p <DIR> 4,096 Visio 2003
06/04/15 10:04p06/04/15 10:04p <DIR> 4,096 Next Wave Systems tools
10/07/15 11:54a10/07/15 11:54a 0 PS_Toolkits_V100R001C02SPC563.zip
10/13/15 04:58p10/13/15 04:58p <DIR> 4,096 5.3.2
4 File(s) 1,057,079,709 bytes
8 Dir(s)

Best Regrads

Aaziz AMHAOUCH · ‎10-16-2015

Hi Alexander,

We used Turboboot 4.3.2 instead of 5.3.2 and it works: The node comes UP with 4.3.2 version and after that we tried again upgrade to 5.3.2 and this time it works !!

- Is there a problem on Turboboot of 5.3.0 and 5.3.2 ?

- Is it recommended to upgrade from 4.3.2 to 5.X.X directly ?

Also we will upgrade a CRS3 from 4.3.2 to 5.1.3, what is the best practice to proceed this upgrade ? the cards that exist in CRS are:

Node Type PLIM State Config State
------------- ----------------- ------------------ --------------- ---------------
0/0/CPU0 FP-140G 14-10GbE IOS XR RUN PWR,NSHUT,MON
0/1/CPU0 FP40 42-1GbE IOS XR RUN PWR,NSHUT,MON
0/1/0 FP40(SPA) 8X1GE OK PWR,NSHUT,MON
0/1/1 FP40(SPA) 5X1GE OK PWR,NSHUT,MON
0/1/2 FP40(SPA) 8X1GE OK PWR,NSHUT,MON
0/1/3 FP40(SPA) 8X1GE OK PWR,NSHUT,MON
0/1/4 FP40(SPA) 5X1GE OK PWR,NSHUT,MON
0/1/5 FP40(SPA) 8X1GE OK PWR,NSHUT,MON
0/2/CPU0 FP40 4-10GbE IOS XR RUN PWR,NSHUT,MON
0/6/CPU0 FP40 4-10GbE IOS XR RUN PWR,NSHUT,MON
0/7/CPU0 FP-140G 14-10GbE IOS XR RUN PWR,NSHUT,MON
0/RP0/CPU0 RP(Standby) N/A IOS XR RUN PWR,NSHUT,MON
0/RP1/CPU0 RP(Active) N/A IOS XR RUN PWR,NSHUT,MON

Best Regards,

Aaziz

xthuijs · ‎10-16-2015

One possible explanation is this. Your current rommon ver is pretty old in terms of revision. long time ago we identified an issue with tftp and all that in rommon that were related to file sizes. it could be the filesize difference between 43 and 53 that was acting up on us.

so would be good to uprev your rommon right away to the fpd levels from 53.

the easiest upgrade path is always to use the mini-pie to apply and activate that on the existing release. single reboot, that's it.

there a very few reasons to do a turboboot, which is like taking the windows install CD and upgrading from XP to 7 as instead of running XP and using the upgrade installation to get to the new ver. So turboboot is kind of like fresh installing, which sometimes is needed, but we try to prevent that as much as possible.

The upgrade guides for the platform explains which from releases we validated this new rls to and reports any gotcha's, if any.

cheers!

xander

Upgrade ASR9010 from 4.3.2 to 5.3.0