ASR9000/XR Understanding Turboboot and initial System bring up

[xthuijs]

• Introduction
• File System overview
• Summary steps for using turboboot
• Steps to Turboboot
  • Clear the ROM Monitor environmental variables on all RSPs
  • Clear disk mirroring variables
  • Disable the CPU watchdog
  • Define the network and IP settings on the mgmt interface
  • Set TFTP environment variables
  • Set the Turboboot variable on the RSP
  • ASR9K/CRS-PRP Additional Information
  • Boot the remote mini.vm file
  • Restore disk mirroring
• How to boot from the external USB port
• How to update the FPD's
  • Disk Space occupied for each image
• Related Information

---

Introduction

This document provides an understanding of what Turboboot is and how to bring up a system running IOS-XR from scratch

There are two ways to upgrade the system:

• One is to install add and activate the new XR software version. At a minimum this would require that mini.pie file
• The second way is by performing a turboboot, fresh install, by booting the mini.vm file from rommon

This executable mini.vm file needs to be transferred via TFTP (on the RSP2) or can be loaded from the external USB port or TFTP (on the RSP440 and CRS-PRP). On the 9001 the USB ability is added in rommon 2.03 (5.1.1 release version).

No other media or protocols are possible to be used for a turboboot other then the ones specified above. Ex FTP is not allowed

File System overview

XR devices have multiple medias for storage and they all have their individual purpose.

Volume	RSP2	RSP440	Trident	Typhoon
disk0:	Embedded USB	SSD (SATA)
disk0a:	Embedded USB	SSD (SATA)
disk1:	Embedded USB	SSD (SATA)
disk1a:	Embedded USB	SSD (SATA)
harddisk:	Harddisk (SAS)	Embedded USB
harddiska:	Harddisk (SAS)	Embedded USB
harddiskb:	Harddisk (SAS)	Embedded USB
compactflash:	Compactflash1	External USB1
lcdisk0:			Embedded USB	Embedded USB
lcdisk0a:			Embedded USB	Embedded USB
bootflash:	NOR Flash		NOR Flash	NOR Flash
configflash:	NOR Flash
nvram:	NVSRAM	NVSRAM
Kernel dump	Harddisk (SAS)	SSD (SATA)	TFTP	bootflash:
	1. Removable
Access (Mount) Points (in /dev)
Volume	RSP2	RSP440	Trident	Typhoon
disk0:	disk00t77	hd0t77	qsm to active rsp	qsm to active rsp
disk0a:	disk00t78	hd0t78	qsm to active rsp	qsm to active rsp
disk1:	disk10t77	hd1t77	qsm to active rsp	qsm to active rsp
disk1a:	disk10t78	hd1t78	qsm to active rsp	qsm to active rsp
harddisk:	hd0t79	usb00t77	qsm to active rsp	qsm to active rsp
harddiska:	hd0t77	usb00t78	qsm to active rsp	qsm to active rsp
harddiskb:	hd0t78	usb00t11
compactflash:	disk20t6,11,121	usb10t6,11,121
lcdisk0:			lcdisk00t77	lcdisk00t77
lcdisk0a:			lcdisk00t78	lcdisk00t78
bootflash:	fs0p1		fs0p1	fs0p1
configflash:	fs1p1
nvram:	nvram	nvram
Kernel dump	hd0t80	hd0t80 or hd1t802		fs0p2
	1. Any one	2. Either one
Usage
Volume	RSP2	RSP440	Trident	Typhoon
disk0:	IOS-XR Packages, Configs	IOS-XR Packages, Configs
disk0a:	sysmgr_debug	sysmgr_debug
disk1:	IOS-XR Packages (if Mirrored)	IOS-XR Packages (if Mirrored)
disk1a:	wdsysmon_debug	wdsysmon_debug
harddisk:	Crash files, logs	Crash files, logs
harddiska:	NP logs, crash files	NP logs, crash files
harddiskb:
compactflash:	File Copy	File Copy
lcdisk0:			Kernel dump files	Kernel dump files
lcdisk0a:
bootflash:	MBI Images
configflash:	OBFL
nvram:	Configs	Configs
Kernel dump	Raw kernel dumps	Raw kernel dumps		Raw kernel dumps
Filesystems
Volume	RSP2	RSP440	Trident	Typhoon
disk0:	QNX4	QNX4
disk0a:	QNX4	QNX4
disk1:	QNX4	QNX4
disk1a:	QNX4	QNX4
harddisk:	QNX4	QNX4
harddiska:	QNX4	QNX4
harddiskb:	QNX4	FAT
compactflash:	FAT1	FAT1,2
lcdisk0:			QNX4	QNX4
lcdisk0a:			QNX4	QNX4
bootflash:	FFSv3		FFSv3	FFSv3
configflash:	FFSv3
nvram:	Cisco	Cisco
Kernel dump	Raw	Raw	File	Raw
	1. FAT F/S only	2. Flash Media only
Approximate Parition Size (minimum)
Volume	RSP2	RSP440	Trident	Typhoon
disk0:	1.6GB	11.0GB
disk0a:	0.4GB	2.2GB
disk1:	1.6GB	11.0GB
disk1a:	0.4GB	2.2GB
harddisk:	35GB	3.1/6.2GB
harddiska:	8GB	0.4/0.8GB
harddiskb:	8GB	0.4/0.8GB
compactflash:	1GB	1-32GB
lcdisk0:			1.6GB	1.6GB
lcdisk0a:			0.4GB	0.4GB
bootflash:	44MB		56MB	56MB
configflash:	28MB
nvram:	220K	500K
Kernel dump	21GB	500MB x 2		24MB

Note that unlike many IOS devices, nvram is NOT used for the configuration storage. Configurations are stored in a database on the boot disk (often disk0). Typically only rommon variables and license info are stored in nvram.

Summary steps for using turboboot

Because a  turboboot can erase configuration, SSH keys, and other items such as  licenses the following should be done to check and backup any files

1. Run a cfs check in admin & non-admin mode

2. Copy active licenses and SNMP files to tftp server

3. Copy running config to a tftp-server or laptop

4. Capture "show ipv4 int brief" output to a text file

5. Capture "show ipv6 int brief | i Up/Up" output to a text file

6.  Offline. Edit the saved RSP config - add "no shutdown" for all physical  interfaces that are up/up from the above IPv4 & IPv6 interface  captures and save cfg changes. Note that it is not necessary to “no  shut” sub-interfaces, only the main physical interface.

7. Connect a laptop console cable to the RSP in RSP0 slot and enable a log file to monitor and capture the RSP bootup logs.

8 . Turn the power supplies on to power up the asr9k system. (approx. 7-12 minutes)

9.  After the LED's indicate IOS-XR on the LC's, and ACTV or STBY on the  RSP’s, log in via the console of the RSP that is ACTV and run some  preliminary checks to check system stability.

NOTE: The default root-system username and password on the RSP440 are root/root

(if root/root does not work also try cisco/cisco, or admin/admin or viking/viking)

10. Verify the ASR9K IOS XR version

11. Run a cfs check in admin & non-admin mode

12. (Optional) Install add & commit any missing SW packages (pies) or required SMU’s

13. Upgrade FPD in admin mode

14. Reload any nodes that had FPD upgrades

15.  Configure the Mgmt ethernet interface with an IP address to reach tftp  server & load and commit the saved RSP config from tftp server or  laptop

a) or log into the console and cut & paste a saved cfg from laptop

b) or copy saved cfg from laptop to usb, then insert usb into RSP440 and copy and commit cfg

c) copy licenses and snmp files back to the RSP’s

16. (Optional) create and generate new crypto keys if required.

Steps to Turboboot

As mentioned Turbobooting means that you load the "VM" (virtual machine) XR base OS image.

Turboboot is started from Rommon and is essentially the same as putting a disk with the desired OS in your laptop, reboot the machine to boot from CDROM, and installing the base OS.

Before the Turboboot process starts, you can instruct the system to wipe all files from the system and start clean or install the image to be turbobooted along side with any existing releases currently found on the disk. (see Set the Turboboot variables on the RSP)

Turbobooting may be required if you want to sweep clean your system, or we also had some issues in XR4.2.0 with the RSP2 whereby the upgrade pie could not be loaded. A turboboot was required in that case also.

Some or all of these procedures below are needed.

The command "set" gives you an overview of all the rommon environment variables currently set to their values.

Clear the ROM Monitor environmental variables on all RSPs

unset BOOT
unset TFTP_FILE
sync

the command *unset* clears the variable value from rommon.

the command *sync *saves or writes the newly set and unset variables to persistent memory so they are saved cross reloads and power cycles.

Clear disk mirroring variables

unset BOOT_DEV_SEQ_OPER
unset MIRROR_ENABLE
sync

By default, the two internal USB partitions (disk0 and disk1) are mirrored to each other, if you break the mirror, turboboot will only affect the disk

that you are turbobooting target to and not the other one (nice if you want to fall back).

Disable the CPU watchdog

priv
diswd <- Disable the CPU watchdog

If you omit this step and the TFTP download for the turboboot mini-vm image takes longer than 30 minutes due to network delays etc, then the RSP might reset and you'll have to start over. Disabling this watchdog makes sure the system is not going to reload during the transfer of the image in rommon.

Define the network and IP settings on the mgmt interface

IP_ADDRESS=ip_address
IP_SUBNET_MASK=mask
DEFAULT_GATEWAY=ip_address

Set TFTP environment variables

TFTP_RETRY_COUNT=4

sets the number of retries to contact the tftp server

TFTP_TIMEOUT=6000

sets the TFTP timeout for the transfer, you may need to set this larger to prevent abort during xfer if there are network delays

TFTP_CHECKSUM=1

whether checksum on the transfer is needed, this is adviceable in case the image gets corrupted during transfer.

TFTP_SERVER=server_ip_addr

the server address can also be specified in the boot statement, or fixed in the rommon variable.

TFTP_MGMT_INTF=0

which of the 2 mgmt interfaces you want to use, either 0 or 1 with 0 being the default.

TFTP_BLKSIZE=1400

Setting a larger TFTP block size is recommended to pack larger packets and transfer the VM image quicker. Note that for CRS this variable is TFTP_BLOCK_SIZE.

Set the Turboboot variable on the RSP

TURBOBOOT=on, {boot-device},[format | clean],[nodisablebreak]

on tells us to install add and install activate the packages when we boot from the VM image.

boot-device is which device we want to use to install the OS, typically disk0

format tells us to replace the OS completely except for the admin configuration

clean tells us to replace the OS completely, but other files such as the admin or exec configuration are saved

nodisablebreak allows us to terminate the turboboot via a break signal. The default is to ignore breaks

Example:

TURBOBOOT=on,disk0,format
sync

This will instruct the system to do a turboboot with disk0 as the selected boot device and to use the format option. The format key is optional.

Currently today we only support targeted install to disk0 but this will change likely in XR4.3.1 whereby you can use disk1 as install target.

NOTE: a recent tac case showed that the command for turboboot failed on the ASR9001.

Supposedly this was made to work by omitting the colon after disk0:

Suggesting to try the disk0 (without colon) if the command with colon fails.

ASR9K/CRS-PRP Additional Information

In CRS the format option works with FAT16 but not FAT32 or QNX4 so a new variable must also be used.

In ASR9K the format and clean options do work but in order to erase the exec configuration, admin configuration, and every other file this additional variable must be used.

For these scenarios the following must be set.

TURBOBOOT=on,disk0

MEDIA_FORMAT=disk0:,QNX4

Note: If the format or clean options are set in turboboot or confreg 0x2142 is set when also having the MEDIA_FORMAT variable set then when prompted for a new username/password we will be unable to write this to the disk. To fix this go back to rommon and properly set the variables.

Boot the remote mini.vm file

(Works only with the VM image, not the TAR file or mini.pie)

rommon> boot tftp://server/directory/filename

During the boot process the image is copied first on to the memory(RAM) and is installed from memory(RAM). Once it is insalled from memory, it will copy the image back on to disk0: and reload the device. Wait till you get the message "SYSTEM CONFIGURATION COMPLETED"

Output of show install active when in memory,

RP/0/RSP0/CPU0:ios#sh install active
<SNIP>
Active Packages:
mem:asr9k-mini-p-4.2.0

Output of show install active after image copied on to disk0:,

RP/0/RSP0/CPU0:ios#sh install active
<SNIP>
Active Packages:
disk0:asr9k-mini-p-4.2.0

The system will also self unset the TURBOBOOT rommon variable.

Restore disk mirroring

To restore disk mirroring, use the mirror command in the global configuration mode. For more information on the mirror command, see the "Boot Commands on Cisco IOS XR Software" module in Cisco ASR 9000 Series Aggregation Services Router System Management Command Reference.

How to boot from the external USB port

The RSP-440 (and 9001 with rommon 2.03) can boot from the USB front panel port. Instead of using "boot tftp:// or boot disk0:/" you need to use a different command, mediaboot.

The command is:

rommon> mediaboot usb:\release_mini.vm

In later revisions of the rommon, the mediaboot has been superseded to boot usb:/<file>

so make sure you try them both.

NOTE:

Some newer rommon versions on the 9001 want to use the boot usb:/ directive. (see Q&A/comment section below this article).

It is also seen in rommon versions post 2.04 that the usb is referred to as disk1 in which case you can use: boot disk1:/...

To find out the mapping of the usb disk use the rommon "dev" command to see all filesystem devices.

On the CRS-PRP use boot disk2:hfr-mini-px.vm<image>

CRS does not use the mediaboot command.

How to update the FPD's

FPD upgrade for all ASR9K devices using FPD.

a) Enter admin mode via the admin command, and capture the output of the current firmware versions using CLI show hw-module fpd location all. save this output to a text file. Notice any LC that has a “yes” in the Upg/Dng?  column. This indicates the FPD should be upgraded or downgraded to match the current FPD version.

b) From admin mode upgrade FPD using the CLI: upgrade hw-module fpd location r/s/m

or if all locations require FPD upgrade (suggested) use CLI:* upgrade hw-module fpd location all *

Disk Space occupied for each image

Simplest way is to use the ksh df utility.

Install a release and packages and run df:

# df /disk0:

/dev/disk00t77           3813344    733477   3079867      20%  /dev/disk0:/   

Divide the highlighted number by 2000. That gives the approximate size in MB. 366MB in this case.

Repeat for any other releases we should be interested in.

If you do an upgrade, gather the df output before and after upgrade and compute the difference in df output.

Related Information

Xander Thuijs, CCIE #6775

Principal Engineer ASR9000

Sam Milstead,

Customer Support Engineer TAC XR

[Nitin Pabbi]

Hi Martine,

1) No you have made correct settings now. When we perform turboboot then system boot directly from XR image and with default config. Hence system require to make changes at default confreg.

Default confreg setting is 0x2102. You can verify the same through command :-

sh variables boot

2) Since you have Active and Committed Packages are :-
     disk0:asr9k-mini-px-5.2.4

    disk0:asr9k-services-infra-5.2.4

    disk0:asr9k-services-px-5.2.4

    disk0:asr9k-fpd-px-5.2.4

And Inactivated packages are :-
 Inactive Packages:

    disk0:asr9k-asr901-nV-px-5.2.4

    disk0:asr9k-bng-px-5.2.4

    disk0:asr9k-li-px-5.2.4

    disk0:asr9k-doc-px-5.2.4

    disk0:asr9k-mpls-px-5.2.4

    disk0:asr9k-optic-px-5.2.4

    disk0:asr9k-9000v-nV-px-5.2.4

    disk0:asr9k-video-px-5.2.4

    disk0:asr9k-mcast-px-5.2.4

    disk0:asr9k-mgbl-px-5.2.4

    disk0:asr9k-asr903-nV-px-5.2.4

    disk0:asr9k-k9sec-px-5.2.4

Assuming inactive packages are not required for your router/ network then you can safely remove these packages from disk to make it free.

Command is :- admin install remove inactive  ===> This will remove all inactive packages through single command.

Or if you think some packages you may need in future then below is command to remove selected.

Command is :- admin install remove  disk0:asr9k-video-px-5.2.4  ===> You can opt any other package. This is for an example.

Another way is through using ID process:-

First check :- show install log. Then select respective ID for inactive package and use knob

Command is :- admin install remove  ID 10  ===> This ID refer the specific package.

I would suggest you to use second option suggested above.

3) Well I checked that this error message is harmless. you can ignore it.

BTW, command output would be like :-

RP/0/RSP0/CPU0:ios#run cat /pkg/startup/route_proxy.startup
Mon Aug 13 05:49:41.915 UTC
name:route_proxy
item:/cfg/gl/onep/
ignore_on_sc:ON
respawn: ON
standby_capable:ON

Thanks

Nitin Pabbi

[jmartinez20]

Hello Nitin,

Thanks a lot for all your help, Everything you mention was clear and precise.

Best Regards,

Martine

[djherteen]

Greetings,

I've recently purchased a 9904 with two RSP-440-LT's for our lab.

The arrived with 5.3.2 but I'm trying to get them to 5.1.3.

Long story short, I can't seem to turboboot these RSP's to anything other that 5.3.2 or 5.3.1.

All attempts to downgrade seem to fail. They always hang at the same spot:

#################################################################
Program load complete, Entry point: 0x40d878, size: 0x16d6d8c
Startup-code initializing, running on Storm (00100313)
BSP date: May 7 2014 01:40:11
Debugging enabled; debug_flag = 1, vendor_debug_flag = 0
Page Address Extension mode (PAE) enabled with 36bit physical addressing.
Running on a ASR9K x86 RSP
Found 8192MB of main memory
Reading back MTRR config from boot processor.
Found LP at APIC ID 6
IPL reports 2 logical processors
Family = 00000006, Model = 0000001e
2 CPU cores available
Initializing logical processor 0 (APIC ID 0).......OK (BSP)
Initializing logical processor 1 (APIC ID 6).......OK
Chassis Type: ASR9904 (00ef02f6)
Region shmwin1: 70000000:b0000000
Region pakman: b0000000:b8000000
###############################################################################################################################

System page at phys:0001b000 user:fed1d000 kern:fed20000
Starting next program at vfe443a54

Anybody have any thoughts on this ?

Thanks,

Darin

[smilstea]

Hi Darin,

The -LT varieties of RSP-440 were introduced in 5.3.0, prior to 5.3.0 the software has no knowledge of this hardware and will not let the hardware boot.

See the "Hardware Features Introduced in Cisco IOS XR Software Release 5.3.0 " section of the below

http://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k_r5-3/general/release/notes/reln_530a9k.html

Thanks,

Sam

[djherteen]

Sam,

Thank you very much for this quick reply as I am working in the lab as I type!

I will head on back to 5.3.x

Thanks again!!

[debrbake]

I am trying to install an image 440 RSP ASR 9006 from an external USB.

rommon 9 > set
PS1=rommon ! >
?=0
IOX_ADMIN_CONFIG_FILE=
ACTIVE_FCD=1
BSI=0
CLUSTER_NO_BOOT=
BOOT_DEV_SEQ_CONF=
TURBOBOOT=on,


rommon 11 > mediaboot usb:\asr9k-mini-px.vm-5.3.3
boot_disk2 - Launching image.


#################################################################
Program load complete, Entry point: 0x40d9d8, size: 0x1ede1c88
f{CPU reset reason = 2 (CPU_RESET_OIR_POR)
########## Environment Variables ##########
PS1=rommon ! >
?=0
IOX_ADMIN_CONFIG_FILE=
ACTIVE_FCD=1
BSI=0
CLUSTER_NO_BOOT=
BOOT_DEV_SEQ_CONF=
TURBOBOOT=on,
###########################################
Total DRAM Memory: 6144 MB
MAC Address from cookie: d4:6d:50:3a:28:a8
Board type: 0x00100306
Chassis type: 0x00ef02fb
Slot number: 00
Chassis serial: FOX1842GRZ8

##########################################################
System Bootstrap, Version 0.71 [ASR9K x86 ROMMON],
Copyright (c) 1994-2012 by Cisco Systems, Inc.
Compiled on Tue 06/18/2013 22:50:54.15 by myekkar

Rommon : 0.71
Ibex Peak : 6
Jasper Forest: 1.0
Zen-JF : 0.7.92
CBC0 : Part 1=16.115, Part 2=16.115, Act Part=2
Laguna : 0.10.0 (primary)
Dao : 1.6.0 (primary)
UTI : 4.9 (primary)
Timex : 0.2.1 (primary)
Board : 5
==========================================================
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
Interface link changed state to UP.
Interface link state up.
................Boot Image validation: Boot response timeout
Sending Boot Image validation request.
HIT CTRL-C to abort
Interface link changed state to UP.
Interface link state up.
................Boot Image validation: Boot response timeout
Sending Boot Image validation request.
HIT CTRL-C to abort
-----------
Interface link did not come up. Timed out.
Boot Image validation: Link failure
Sending Boot Image validation request.
HIT CTRL-C to abort
Interface link changed state to UP.
Interface link state up.

I thought I installed the image but keep getting this boot image verification link error.  Can I not get to the command line without assigning IP address and verifying the image?  When I tried installing the image again, I get an error reading the external USB drive.  

rommon 19 > mediaboot usb:\asr9k-k9sec-px.pie-5.3.3
copy_image_from_media_to_ram: could not open file usb:asr9k-k9sec-px.pie-5.3.3.

rommon 22 > set
PS1=rommon ! >
IOX_ADMIN_CONFIG_FILE=
ACTIVE_FCD=1
BSI=0
CLUSTER_NO_BOOT=
BOOT_DEV_SEQ_CONF=
TURBOBOOT=on
?=0 

----------------------------------

I just wanted everyone to know I got this to work with the following commands.  Also just remember to copy the mini vm image (usb:asr9k-mini-px.vm-5.1.3) over to the hard drive once you have it up and running and set the rommon to boot from disk, so you don't lose what you just did on reload.

rommon 1 > set

PS1=rommon ! >

?=0

IOX_ADMIN_CONFIG_FILE=

ACTIVE_FCD=1

BSI=0

CLUSTER_NO_BOOT=

BOOT_DEV_SEQ_CONF=

BOOT_DEV_SEQ_OPER=

TURBOBOOT=on,

rommon 2 > unset BOOT_DEV_SEQ_OPER

rommon 3 > unset MIRROR_ENABLE

unset: "MIRROR_ENABLE" does not exist

rommon 4 > sync

rommon 5 > priv

You now have access to the full set of monitor commands.

Warning: some commands will allow you to destroy your

configuration and/or system images and could render

the machine unbootable.

rommon 6 > unset TFTP_FILE

unset: "TFTP_FILE" does not exist

rommon 7 > dlswd

monitor: command "dlswd" not found

rommon 8 > diswd

Watchdog Disabled

rommon 9 > set

PS1=rommon ! >

?=0

IOX_ADMIN_CONFIG_FILE=

ACTIVE_FCD=1

BSI=0

CLUSTER_NO_BOOT=

BOOT_DEV_SEQ_CONF=

TURBOBOOT=on,

rommon 1 > boot usb:asr9k-mini-px.vm-5.1.3  

Beginning Media boot:

boot_disk2 - Launching image.

#################################################################

Program load complete, Entry point: 0x40d878, size: 0x1761cbc3

   Startup-code initializing, running on Storm (00100307)

[Aleksandar Vidakovic]

Please try with 'boot' instead of 'mediaboot'.

[mauricio.andrade81]

Hi,

 Any SMU for 4.2.3 and RSP440-LT ??

regards,

[djherteen]

My apologies as I thought I had replied to you.

You were spot on, thanks again for they reply!

[smilstea]

Hi Mauricio,

As I mentioned to djherteen the RSP440-LT cards were not available before 5.3.0 so there are no SMUs available in 4.2.3 for that card as it will not boot.

As well SMUs are built per platform and architecture (right now the 9K supports 1 architecture) so any SMU on 9K can be loaded on any box of the same version of code.

Thanks,

Sam

[Bradley Marshall]

Wanted to share:

IOS XR 5.3.3 Turboboot (from 4.3.4 SP5).

Turbobooting from the mini-px, I had to use the following command to get the vm to load, if not, the RSP would stay in a LOAD or MBI status (LCD screen) for 30 minutes before the CPU Watchdog would reset the RSP back to ROMMON. Reseating the card caused us to receive a Power Sequencer failure and the RSP had to be RMA'd (PSEQ—CBC detected power sequencer failure on card LCD, no console access).

The command that worked, would probably work with harddisk: as well:

"boot usb:\asr9k-mini-px.vm-5.3.3"

RSP 440, FYI, Codes below:

INIT—Card is inserted and microcontroller is initialized

BOOT—Board is powered on and CPU is booting

IMEM—Start initializing memory

IGEN—Start initializing the board

ICBC—Start initializing communication with the microcontroller

SCPI—Board is not plugged in properly

RSP-440/RSP 440-Lite

• STID—CBC was unable to read slot ID pins correctly
• PSEQ—CBC detected power sequencer failure
• DBPO—CBC detected an issue during board power up
• KPWR—CBC detected an issue during board power up
• LGNP—CBC detected an issue during board power up
• LGNI—CBC detected an issue during board power up

RMN—All tests are finished and ROMMON is ready for commands

LOAD—Downloading MBI image to CPU

RRST—ROMMON is performing a soft reset after 5 consecutive MBI validation requests timed out

MVB—ROMMON trying MBI validation boot

MBI—Starting execution of MBI

IOXR—Cisco IOS XR software is starting execution

LDG—The RSP is loading (MBI started and card preparing for activity)

INCP—The software or configuration is incompatible with the RSP

OOSM—The RSP is in Out of Service, Maintenance mode

ACTV—RSP role is determined to be active RSP

STBY—RSP role is determined to be standby RSP

PREP—Preparing disk boot

[Aleksandar Vidakovic]

hi Bradley,

thanks for sharing the experience.

I;m very sorry to hear that the RSP ran into the power sequencer failure. We have just posted yesterday the 5.3.3 SMU for CSCux24553. This will recover the line cards that report power sequencer failures during bootup. I hope this will avoid similar experiences in upcoming upgrades.

We don't recommend turboboot, it should be used as a last option. Normal upgrade from 4.3.4 to 5.3.3 is supported and tested. Was there a particular reason for using the turboboot method?

Regards,

Aleksandar

[Bradley Marshall]

Aleksandar, 

Thanks know you for the reply. We were able to perform the turboboot twice now successfully. I will post information about my upgrade details to IOS XR 5.3.3 with a turboboot so others can benefit from the information.

We turboboot upgrade our ASR 9Ks annually and we like using the turboboot feature as we have to isolate the chassis interfaces from the network either way if a reboot is involved. The turboboot ensures that we have a procedure that never changes with every upgrade except for the names of the files used. It also allows us to ensure we do a full reset (multiple resets with a turboboot) on all the elements of the chassis (cards,  CPUs,  power equipment). This helps us do fail over testing of the hardware and can identify hardware issues in a controlled environment with tier 3 and onsite engineers engaged. I find the turboboot to be any and repeatable,  the process hardly changes year over year.

It will also make sure we do not run into any issues with leftover software loads and memory space issues on the RSPs.

Why do you not recommend a turboboot? Is this harder on the equipment? There is no risk to the network as the chassis is already manually isolated from the environment and we gracefully turn up the interfaces to return the units to service. 

[Aleksandar Vidakovic]

hi Bradley,

thanks for the detailed explanation. We don't recommend turboboot simply because there is no rollback. If for whatever reason you decide to downgrade, the only option is again a turboboot.

Will you be deploying individual SMUs or Service Packs on top of 5.3.3? If it's the latter, we should have the 5.3.3 SP1 available within a week's time, likely even faster.

Regards,

Aleksandar

[Bradley Marshall]

Aleksandar,

Thank you for your reply. Please let me show you a guaranteed rollback option that works flawlessly if you are using dual RSPs (redundancy) in a single chassis. This rollback has been tested successfully in our lab environment multiple times and has now been used in production with no issues.

Our upgrade process has us manually turn down the interfaces on the cards in a controlled manner. We isolate the chassis as we have redundant chassis connected with EtherBundles, so there is another ASR9K up and running with all traffic. The running-config and admin running-config are now saved. We then save a copy of the running-config and admin running-config to the harddisk: of the ASR9K so it can be loaded once the turboboot is complete.

Now that the chassis is isolated and the configs are saved (across both RSPs), we physically unseat the STBY RSP from the chassis backplane so that it powers down and the show redundancy command reports the RSP gone. Now with one RSP in place (ACTV), we will turboboot the RSP /chassis leaving the secondary RSP out the entire time.

Once the upgrade is complete and traffic returned to the box, ACTV RSP firmware upgraded with everything functioning as normal, the STBY RSP can be returned to the chassis backplane (this removes the rollback option). We can then manually FPD the STBY RSP to upgrade the firmware to match the newly upgraded via turboboot RSP.

If there is an issue with the software upgrade during turboboot, we unseat the ACTV card (so both RSPs are disconnected from the backplane), re-seat the old STBY RSP into the chassis, and the chassis reboots on the previous RSP with all existing software and config (with interfaces still manually turned down). This lets us rollback in about 10 minutes at any stage of the turboboot and have the ability to gracefully return traffic to the box as the interfaces are still manually down.

Thank you for the news on the service pack! This is good to know and will work well with my rollout plans. We will definitely deploy the service pack.