cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
Announcements
Webcast SD-WAN

ASR9000/XR Understanding Turboboot and initial System bring up

40644
Views
50
Helpful
153
Comments



Introduction

This document provides an understanding of what Turboboot is and how to bring up a system running IOS-XR from scratch

There are two ways to upgrade the system:

  • One is to install add and activate the new XR software version. At a minimum this would require that mini.pie file
  • The second way is by performing a turboboot, fresh install, by booting the mini.vm file from rommon

This executable mini.vm file needs to be transferred via TFTP (on the RSP2) or can be loaded from the external USB port or TFTP (on the RSP440 and CRS-PRP). On the 9001 the USB ability is added in rommon 2.03 (5.1.1 release version).

No other media or protocols are possible to be used for a turboboot other then the ones specified above. Ex FTP is not allowed

File System overview

XR devices have multiple medias for storage and they all have their individual purpose.

 
VolumeRSP2RSP440TridentTyphoon
disk0:Embedded USBSSD (SATA)  
disk0a:Embedded USBSSD (SATA)  
disk1:Embedded USBSSD (SATA)  
disk1a:Embedded USBSSD (SATA)  
harddisk:Harddisk (SAS)Embedded USB  
harddiska:Harddisk (SAS)Embedded USB  
harddiskb:Harddisk (SAS)Embedded USB  
compactflash:Compactflash1External USB1  
lcdisk0:  Embedded USBEmbedded USB
lcdisk0a:  Embedded USBEmbedded USB
bootflash:NOR Flash NOR FlashNOR Flash
configflash:NOR Flash   
nvram:NVSRAMNVSRAM  
Kernel dumpHarddisk (SAS)SSD (SATA)TFTPbootflash:
 1. Removable   
     
Access (Mount) Points (in /dev)
VolumeRSP2RSP440TridentTyphoon
disk0:disk00t77hd0t77qsm to active rspqsm to active rsp
disk0a:disk00t78hd0t78qsm to active rspqsm to active rsp
disk1:disk10t77hd1t77qsm to active rspqsm to active rsp
disk1a:disk10t78hd1t78qsm to active rspqsm to active rsp
harddisk:hd0t79usb00t77qsm to active rspqsm to active rsp
harddiska:hd0t77usb00t78qsm to active rspqsm to active rsp
harddiskb:hd0t78usb00t11  
compactflash:disk20t6,11,121usb10t6,11,121  
lcdisk0:  lcdisk00t77lcdisk00t77
lcdisk0a:  lcdisk00t78lcdisk00t78
bootflash:fs0p1 fs0p1fs0p1
configflash:fs1p1   
nvram:nvramnvram  
Kernel dumphd0t80hd0t80 or hd1t802 fs0p2
 1. Any one2. Either one  
     
Usage
VolumeRSP2RSP440TridentTyphoon
disk0:IOS-XR Packages, ConfigsIOS-XR Packages, Configs  
disk0a:sysmgr_debugsysmgr_debug  
disk1:IOS-XR Packages (if Mirrored)IOS-XR Packages (if Mirrored)  
disk1a:wdsysmon_debugwdsysmon_debug  
harddisk:Crash files, logsCrash files, logs  
harddiska:NP logs, crash filesNP logs, crash files  
harddiskb:    
compactflash:File CopyFile Copy  
lcdisk0:  Kernel dump filesKernel dump files
lcdisk0a:    
bootflash:MBI Images   
configflash:OBFL   
nvram:ConfigsConfigs  
Kernel dumpRaw kernel dumpsRaw kernel dumps Raw kernel dumps
     
Filesystems
VolumeRSP2RSP440TridentTyphoon
disk0:QNX4QNX4  
disk0a:QNX4QNX4  
disk1:QNX4QNX4  
disk1a:QNX4QNX4  
harddisk:QNX4QNX4  
harddiska:QNX4QNX4  
harddiskb:QNX4FAT  
compactflash:FAT1FAT1,2  
lcdisk0:  QNX4QNX4
lcdisk0a:  QNX4QNX4
bootflash:FFSv3 FFSv3FFSv3
configflash:FFSv3   
nvram:CiscoCisco  
Kernel dumpRawRawFileRaw
 1. FAT F/S only2. Flash Media only  
     
Approximate Parition Size (minimum)
VolumeRSP2RSP440TridentTyphoon
disk0:1.6GB11.0GB  
disk0a:0.4GB2.2GB  
disk1:1.6GB11.0GB  
disk1a:0.4GB2.2GB  
harddisk:35GB3.1/6.2GB  
harddiska:8GB0.4/0.8GB  
harddiskb:8GB0.4/0.8GB  
compactflash:1GB1-32GB  
lcdisk0:  1.6GB1.6GB
lcdisk0a:  0.4GB0.4GB
bootflash:44MB 56MB56MB
configflash:28MB   
nvram:220K500K  
Kernel dump21GB500MB x 2 24MB
 

Note that unlike many IOS devices, nvram is NOT used for the configuration storage. Configurations are stored in a database on the boot disk (often disk0). Typically only rommon variables and license info are stored in nvram.

Summary steps for using turboboot

Because a  turboboot can erase configuration, SSH keys, and other items such as  licenses the following should be done to check and backup any files

1. Run a cfs check in admin & non-admin mode

2. Copy active licenses and SNMP files to tftp server

3. Copy running config to a tftp-server or laptop

4. Capture "show ipv4 int brief" output to a text file

5. Capture "show ipv6 int brief | i Up/Up" output to a text file

6.  Offline. Edit the saved RSP config - add "no shutdown" for all physical  interfaces that are up/up from the above IPv4 & IPv6 interface  captures and save cfg changes. Note that it is not necessary to “no  shut” sub-interfaces, only the main physical interface.

7. Connect a laptop console cable to the RSP in RSP0 slot and enable a log file to monitor and capture the RSP bootup logs.

8 . Turn the power supplies on to power up the asr9k system. (approx. 7-12 minutes)

9.  After the LED's indicate IOS-XR on the LC's, and ACTV or STBY on the  RSP’s, log in via the console of the RSP that is ACTV and run some  preliminary checks to check system stability.

NOTE: The default root-system username and password on the RSP440 are root/root

(if root/root does not work also try cisco/cisco, or admin/admin or viking/viking)

 

10. Verify the ASR9K IOS XR version

11. Run a cfs check in admin & non-admin mode

12. (Optional) Install add & commit any missing SW packages (pies) or required SMU’s

13. Upgrade FPD in admin mode

14. Reload any nodes that had FPD upgrades

15.  Configure the Mgmt ethernet interface with an IP address to reach tftp  server & load and commit the saved RSP config from tftp server or  laptop

a) or log into the console and cut & paste a saved cfg from laptop

b) or copy saved cfg from laptop to usb, then insert usb into RSP440 and copy and commit cfg

c) copy licenses and snmp files back to the RSP’s

16. (Optional) create and generate new crypto keys if required.

Steps to Turboboot

 

As mentioned Turbobooting means that you load the "VM" (virtual machine) XR base OS image.

Turboboot is started from Rommon and is essentially the same as putting a disk with the desired OS in your laptop, reboot the machine to boot from CDROM, and installing the base OS.

Before the Turboboot process starts, you can instruct the system to wipe all files from the system and start clean or install the image to be turbobooted along side with any existing releases currently found on the disk. (see Set the Turboboot variables on the RSP)

Turbobooting may be required if you want to sweep clean your system, or we also had some issues in XR4.2.0 with the RSP2 whereby the upgrade pie could not be loaded. A turboboot was required in that case also.

Some or all of these procedures below are needed.

The command "set" gives you an overview of all the rommon environment variables currently set to their values.

Clear the ROM Monitor environmental variables on all RSPs

unset BOOT
unset TFTP_FILE
sync

the command *unset* clears the variable value from rommon.

the command *sync *saves or writes the newly set and unset variables to persistent memory so they are saved cross reloads and power cycles.

Clear disk mirroring variables

unset BOOT_DEV_SEQ_OPER
unset MIRROR_ENABLE
sync

By default, the two internal USB partitions (disk0 and disk1) are mirrored to each other, if you break the mirror, turboboot will only affect the disk

that you are turbobooting target to and not the other one (nice if you want to fall back).

Disable the CPU watchdog

priv
diswd <- Disable the CPU watchdog

If you omit this step and the TFTP download for the turboboot mini-vm image takes longer than 30 minutes due to network delays etc, then the RSP might reset and you'll have to start over. Disabling this watchdog makes sure the system is not going to reload during the transfer of the image in rommon.

Define the network and IP settings on the mgmt interface

IP_ADDRESS=ip_address
IP_SUBNET_MASK=mask
DEFAULT_GATEWAY=ip_address

Set TFTP environment variables

TFTP_RETRY_COUNT=4

sets the number of retries to contact the tftp server

TFTP_TIMEOUT=6000

sets the TFTP timeout for the transfer, you may need to set this larger to prevent abort during xfer if there are network delays

TFTP_CHECKSUM=1

whether checksum on the transfer is needed, this is adviceable in case the image gets corrupted during transfer.

TFTP_SERVER=server_ip_addr

the server address can also be specified in the boot statement, or fixed in the rommon variable.

TFTP_MGMT_INTF=0

which of the 2 mgmt interfaces you want to use, either 0 or 1 with 0 being the default.

TFTP_BLKSIZE=1400

Setting a larger TFTP block size is recommended to pack larger packets and transfer the VM image quicker. Note that for CRS this variable is TFTP_BLOCK_SIZE.

Set the Turboboot variable on the RSP

TURBOBOOT=on, {boot-device},[format | clean],[nodisablebreak]

on tells us to install add and install activate the packages when we boot from the VM image.

boot-device is which device we want to use to install the OS, typically disk0

format tells us to replace the OS completely except for the admin configuration

clean tells us to replace the OS completely, but other files such as the admin or exec configuration are saved

nodisablebreak allows us to terminate the turboboot via a break signal. The default is to ignore breaks

Example:

TURBOBOOT=on,disk0,format
sync

This will instruct the system to do a turboboot with disk0 as the selected boot device and to use the format option. The format key is optional.

Currently today we only support targeted install to disk0 but this will change likely in XR4.3.1 whereby you can use disk1 as install target.

NOTE: a recent tac case showed that the command for turboboot failed on the ASR9001.

Supposedly this was made to work by omitting the colon after disk0:

Suggesting to try the disk0 (without colon) if the command with colon fails.

 

ASR9K/CRS-PRP Additional Information

 

In CRS the format option works with FAT16 but not FAT32 or QNX4 so a new variable must also be used.

In ASR9K the format and clean options do work but in order to erase the exec configuration, admin configuration, and every other file this additional variable must be used.

 

For these scenarios the following must be set.

TURBOBOOT=on,disk0

MEDIA_FORMAT=disk0:,QNX4

 

Note: If the format or clean options are set in turboboot or confreg 0x2142 is set when also having the MEDIA_FORMAT variable set then when prompted for a new username/password we will be unable to write this to the disk. To fix this go back to rommon and properly set the variables.

 

Boot the remote mini.vm file

(Works only with the VM image, not the TAR file or mini.pie)

rommon> boot tftp://server/directory/filename

During the boot process the image is copied first on to the memory(RAM) and is installed from memory(RAM). Once it is insalled from memory, it will copy the image back on to disk0: and reload the device. Wait till you get the message "SYSTEM CONFIGURATION COMPLETED"

Output of show install active when in memory,

RP/0/RSP0/CPU0:ios#sh install active
<SNIP>
Active Packages:
mem:asr9k-mini-p-4.2.0

Output of show install active after image copied on to disk0:,

RP/0/RSP0/CPU0:ios#sh install active
<SNIP>
Active Packages:
disk0:asr9k-mini-p-4.2.0

The system will also self unset the TURBOBOOT rommon variable.

Restore disk mirroring

To restore disk mirroring, use the mirror command in the global configuration mode. For more information on the mirror command, see the "Boot Commands on Cisco IOS XR Software" module in Cisco ASR 9000 Series Aggregation Services Router System Management Command Reference.

How to boot from the external USB port

The RSP-440 (and 9001 with rommon 2.03) can boot from the USB front panel port. Instead of using "boot tftp:// or boot disk0:/" you need to use a different command, mediaboot.

The command is:

rommon> mediaboot usb:\release_mini.vm

In later revisions of the rommon, the mediaboot has been superseded to boot usb:/<file>

so make sure you try them both.

 

NOTE:

Some newer rommon versions on the 9001 want to use the boot usb:/ directive. (see Q&A/comment section below this article).

It is also seen in rommon versions post 2.04 that the usb is referred to as disk1 in which case you can use: boot disk1:/...

To find out the mapping of the usb disk use the rommon "dev" command to see all filesystem devices.

 

On the CRS-PRP use boot disk2:hfr-mini-px.vm<image>

CRS does not use the mediaboot command.

How to update the FPD's

FPD upgrade for all ASR9K devices using FPD.

a) Enter admin mode via the admin command, and capture the output of the current firmware versions using CLI show hw-module fpd location all. save this output to a text file. Notice any LC that has a “yes” in the Upg/Dng?  column. This indicates the FPD should be upgraded or downgraded to match the current FPD version.

b) From admin mode upgrade FPD using the CLI: upgrade hw-module fpd location r/s/m

or if all locations require FPD upgrade (suggested) use CLI:* upgrade hw-module fpd location all *

Disk Space occupied for each image

Simplest way is to use the ksh df utility.

Install a release and packages and run df:

# df /disk0:

/dev/disk00t77           3813344    733477   3079867      20%  /dev/disk0:/   

Divide the highlighted number by 2000. That gives the approximate size in MB. 366MB in this case.

Repeat for any other releases we should be interested in.

If you do an upgrade, gather the df output before and after upgrade and compute the difference in df output.

Related Information

Xander Thuijs, CCIE #6775

Principal Engineer ASR9000

Sam Milstead,

Customer Support Engineer TAC XR

Comments
Beginner

I think the key issue with the ASR9001 is that it didnt work with a TFTP_BLKSIZE of 1400. Once I changed it to 1024 (which is a setting I happened to choose by chance after trying every other conceivable combination of 5 different TFTP servers and an Ubuntu VM running atftpd) it worked.

Hopefully this will help others with the same problem

Community Member

Hi Xander,

Does the turboboot work for 9006 as well? The error mssage I get is:

TFTP attempt 1 of 1

Kawela link up, waiting for ARP reply ...

Receiving ASR9K432/asr9k-mini-px.vm-4.3.2 from 192.168.1.248 !!!!!!!!!!!!!!!!!!!

. . .

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

File reception completed.

loadprog: error - not an executable ELF file

Thanks.

Cisco Employee

there are 2 possible reasons for this:

in releases prior to 43 the P and PX were split images for RSP2 and RSP440 respectively

(combined in 43)

which leaves one potential possibility for 43 which is a corrupted image or booting a non vm file.

depending on teh speed of xfer, you may need to set the appropriate larger block size of 1450 and disable the watchdog timeout.

effectively this message means that you are loading an incomplete non executable VM image.

regards

xander

Community Member

"install add tar" command worked for my previous upgrading to 4.2.1, but it fails at upgrading to 4.3.2, only turboboot works for upgrading to 4.3.2

The image itself should be fine, since the same image works for 9001, 9006 is upgraded with the same image. The BLKSIZE is increased to 1450, but still get the same error .

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

File reception completed.

loadprog: error - not an executable ELF file

rommon 1 > unset BOOT

rommon 2 > unset TFTP_FILE

rommon 3 > unset BOOT_DEV_SEQ_OPER

rommon 4 > unset MIRROR_ENABLE

unset: "MIRROR_ENABLE" does not exist

rommon 5 > priv

You now have access to the full set of monitor commands.

Warning: some commands will allow you to destroy your

configuration and/or system images and could render

the machine unbootable.

rommon 6 > diswd

Watchdog Disabled

rommon 7 > IP_ADDRESS=192.168.1.22

rommon 8 > IP_SUBNET_MASK=255.255.255.0

rommon 9 > DEFAULT_GATEWAY=192.168.1.251

rommon 10 > TFTP_RETRY_COUNT=4

rommon 11 > TFTP_TIMEOUT=6000

rommon 12 > TFTP_CHECKSUM=1

rommon 13 > TFTP_SERVER=192.168.1.248

rommon 16 > TFTP_MGMT_INTF=0

rommon 17 > TFTP_BLKSIZE=1450

rommon 18 > sync

rommon 19 > TURBOBOOT=on,disk0:

rommon 20 > sync

rommon 21 > boot tftp://192.168.1.248/ASR9K432/asr9k-mini-px.vm-4.3.2

Beginning TFTP boot:

TFTP server specified is:

TFTP filename specified is: ASR9K432/asr9k-mini-px.vm-4.3.2

I monitor the TFTP server, no error message.

Cisco Employee

ah I know what you are likely running into, your rommon version may not be up to par for the RSP you have...

you need min version 0.61

http://www.cisco.com/web/Cisco_IOS_XR_Software/pdf/ASR9000_Upgrade_Procedure_431.pdf

xander

Community Member

BINGO, I got rommon 0.51

Thank you very much!!!!!

Cisco Employee

awesome! thanks for confirming.

yeah there are sneaky tricks available to boot a rommon image from rommon with a boot statement that updates the fpd, but that is tricky, you need to get the rommon image from one router that has the right version and copy it from its fpd dir to the tftp server and then boot tftp://server/rommon-A.fpd or whatever that image is. there are 2 parts A and B.

easier is probably, if there are limited devices/lab, to load 423 with the fpd pie, upgrade the rommon and then either use the mini-pie to upgrade or the turboboot with the vm...

reason for the rommon need is that the rommon needs to understand that p and px are combined, that is the underlying technical rationale behind it...

cheers

xander

Community Member

Hello all,

I need to downgrade an ASR9K from 4.3.2 to 4.3.1. Should I use turboboot procedure?

Thanks

Yannis

Cisco Employee

yannis, nope you can just do an install admin activate on the 431 mini-pie (and all other optional pie's you're running on your current release).

regards

xander

Community Member

You are right Alexander, I tried it and it works.

Thanks!

Yannis

Beginner

After a upgrade to 4.3.0 and some patches the usage of the bootflash is about 90%

sh filesystem
Thu Jan  9 06:46:37.933 UTC
File Systems:

     Size(b)     Free(b)        Type  Flags  Prefixes
           -           -     network     rw  qsm/dev/fs/tftp:
           -           -     network     rw  qsm/dev/fs/rcp:
           -           -     network     rw  qsm/dev/fs/ftp:
  1006632960  1006349312  dumper-lnk     rw  qsm/dumper_disk0a:
  3019898880  2383787008  dumper-lnk     rw  qsm/dumper_harddisk:
      224256      195584  dumper-lnk     rw  qsm/dumper_nvram:
  4026515456  2631576576  dumper-lnk     rw  qsm/dumper_disk0:
    28966912    28943732  dumper-lnk     rw  qsm/dumper_configflash:
    44695552     4668992  dumper-lnk     rw  qsm/dumper_bootflash:
  4026515456  2631576576  flash-disk     rw  disk0:
  3019898880  2383787008    harddisk     rw  harddisk:
  1006632960  1006349312  flash-disk     rw  disk0a:
    44695552     4668992       flash     rw  bootflash:
    28966912    28943732       flash     rw  configflash:
      224256      195584       nvram     rw  nvram:
ASR90001#

when I look at all the files in the bootflash I am unable to locate the files which take up all this space. We have this on multiple devices. Any commands availble to show hidden files (if any) would a chkdsk fix this.

Cisco Employee

Hi,

Correct, the bootflash is used for the MBI and thats about it.

The disk usage can go up by copying and deleting files from bootflash, increasing the flash overhead.

We recommend using the harddisk for file storage.

To "reclaim" the space on a 9001 the router would have to be reloaded.

HTH,

Sam

Hello Alexander. Thank you very much for this post.

I was able to download new image from tftp server (tftp32), I used  TFTP_BLKSIZE=1024, it didn't work with any other BLKSIZE. I have ASR9001, when system was upgraded to IOS XR 5.1.0 vm file, I was able to install all other packets. It works properly now, thank you.

Vsevolod.

Beginner

Rather than dividing by 2000, probably easier to use "df -h" or "df -k" from ksh

Community Member

I am currently stuck in rommon after getting stuck in a continuous reload cycle, due to corrupt files while upgrading to 4.3.2 from 4.2.1. I am getting the loadprog error when trying to turboboot, this is likely due to the rommon version as I was unable to do my FPD upgrades yet. Is there a way to update the rommon from rommon?

Thanks

CreatePlease to create content
Content for Community-Ad
June's Community Spotlight Awards