on 01-14-2013 08:10 AM
This document provides an understanding of what Turboboot is and how to bring up a system running IOS-XR from scratch
There are two ways to upgrade the system:
This executable mini.vm file needs to be transferred via TFTP (on the RSP2) or can be loaded from the external USB port or TFTP (on the RSP440 and CRS-PRP). On the 9001 the USB ability is added in rommon 2.03 (5.1.1 release version).
No other media or protocols are possible to be used for a turboboot other then the ones specified above. Ex FTP is not allowed
XR devices have multiple medias for storage and they all have their individual purpose.
Volume | RSP2 | RSP440 | Trident | Typhoon |
disk0: | Embedded USB | SSD (SATA) | ||
disk0a: | Embedded USB | SSD (SATA) | ||
disk1: | Embedded USB | SSD (SATA) | ||
disk1a: | Embedded USB | SSD (SATA) | ||
harddisk: | Harddisk (SAS) | Embedded USB | ||
harddiska: | Harddisk (SAS) | Embedded USB | ||
harddiskb: | Harddisk (SAS) | Embedded USB | ||
compactflash: | Compactflash1 | External USB1 | ||
lcdisk0: | Embedded USB | Embedded USB | ||
lcdisk0a: | Embedded USB | Embedded USB | ||
bootflash: | NOR Flash | NOR Flash | NOR Flash | |
configflash: | NOR Flash | |||
nvram: | NVSRAM | NVSRAM | ||
Kernel dump | Harddisk (SAS) | SSD (SATA) | TFTP | bootflash: |
1. Removable | ||||
Access (Mount) Points (in /dev) | ||||
Volume | RSP2 | RSP440 | Trident | Typhoon |
disk0: | disk00t77 | hd0t77 | qsm to active rsp | qsm to active rsp |
disk0a: | disk00t78 | hd0t78 | qsm to active rsp | qsm to active rsp |
disk1: | disk10t77 | hd1t77 | qsm to active rsp | qsm to active rsp |
disk1a: | disk10t78 | hd1t78 | qsm to active rsp | qsm to active rsp |
harddisk: | hd0t79 | usb00t77 | qsm to active rsp | qsm to active rsp |
harddiska: | hd0t77 | usb00t78 | qsm to active rsp | qsm to active rsp |
harddiskb: | hd0t78 | usb00t11 | ||
compactflash: | disk20t6,11,121 | usb10t6,11,121 | ||
lcdisk0: | lcdisk00t77 | lcdisk00t77 | ||
lcdisk0a: | lcdisk00t78 | lcdisk00t78 | ||
bootflash: | fs0p1 | fs0p1 | fs0p1 | |
configflash: | fs1p1 | |||
nvram: | nvram | nvram | ||
Kernel dump | hd0t80 | hd0t80 or hd1t802 | fs0p2 | |
1. Any one | 2. Either one | |||
Usage | ||||
Volume | RSP2 | RSP440 | Trident | Typhoon |
disk0: | IOS-XR Packages, Configs | IOS-XR Packages, Configs | ||
disk0a: | sysmgr_debug | sysmgr_debug | ||
disk1: | IOS-XR Packages (if Mirrored) | IOS-XR Packages (if Mirrored) | ||
disk1a: | wdsysmon_debug | wdsysmon_debug | ||
harddisk: | Crash files, logs | Crash files, logs | ||
harddiska: | NP logs, crash files | NP logs, crash files | ||
harddiskb: | ||||
compactflash: | File Copy | File Copy | ||
lcdisk0: | Kernel dump files | Kernel dump files | ||
lcdisk0a: | ||||
bootflash: | MBI Images | |||
configflash: | OBFL | |||
nvram: | Configs | Configs | ||
Kernel dump | Raw kernel dumps | Raw kernel dumps | Raw kernel dumps | |
Filesystems | ||||
Volume | RSP2 | RSP440 | Trident | Typhoon |
disk0: | QNX4 | QNX4 | ||
disk0a: | QNX4 | QNX4 | ||
disk1: | QNX4 | QNX4 | ||
disk1a: | QNX4 | QNX4 | ||
harddisk: | QNX4 | QNX4 | ||
harddiska: | QNX4 | QNX4 | ||
harddiskb: | QNX4 | FAT | ||
compactflash: | FAT1 | FAT1,2 | ||
lcdisk0: | QNX4 | QNX4 | ||
lcdisk0a: | QNX4 | QNX4 | ||
bootflash: | FFSv3 | FFSv3 | FFSv3 | |
configflash: | FFSv3 | |||
nvram: | Cisco | Cisco | ||
Kernel dump | Raw | Raw | File | Raw |
1. FAT F/S only | 2. Flash Media only | |||
Approximate Parition Size (minimum) | ||||
Volume | RSP2 | RSP440 | Trident | Typhoon |
disk0: | 1.6GB | 11.0GB | ||
disk0a: | 0.4GB | 2.2GB | ||
disk1: | 1.6GB | 11.0GB | ||
disk1a: | 0.4GB | 2.2GB | ||
harddisk: | 35GB | 3.1/6.2GB | ||
harddiska: | 8GB | 0.4/0.8GB | ||
harddiskb: | 8GB | 0.4/0.8GB | ||
compactflash: | 1GB | 1-32GB | ||
lcdisk0: | 1.6GB | 1.6GB | ||
lcdisk0a: | 0.4GB | 0.4GB | ||
bootflash: | 44MB | 56MB | 56MB | |
configflash: | 28MB | |||
nvram: | 220K | 500K | ||
Kernel dump | 21GB | 500MB x 2 | 24MB | |
Note that unlike many IOS devices, nvram is NOT used for the configuration storage. Configurations are stored in a database on the boot disk (often disk0). Typically only rommon variables and license info are stored in nvram.
Because a turboboot can erase configuration, SSH keys, and other items such as licenses the following should be done to check and backup any files
1. Run a cfs check in admin & non-admin mode
2. Copy active licenses and SNMP files to tftp server
3. Copy running config to a tftp-server or laptop
4. Capture "show ipv4 int brief" output to a text file
5. Capture "show ipv6 int brief | i Up/Up" output to a text file
6. Offline. Edit the saved RSP config - add "no shutdown" for all physical interfaces that are up/up from the above IPv4 & IPv6 interface captures and save cfg changes. Note that it is not necessary to “no shut” sub-interfaces, only the main physical interface.
7. Connect a laptop console cable to the RSP in RSP0 slot and enable a log file to monitor and capture the RSP bootup logs.
8 . Turn the power supplies on to power up the asr9k system. (approx. 7-12 minutes)
9. After the LED's indicate IOS-XR on the LC's, and ACTV or STBY on the RSP’s, log in via the console of the RSP that is ACTV and run some preliminary checks to check system stability.
NOTE: The default root-system username and password on the RSP440 are root/root
(if root/root does not work also try cisco/cisco, or admin/admin or viking/viking)
10. Verify the ASR9K IOS XR version
11. Run a cfs check in admin & non-admin mode
12. (Optional) Install add & commit any missing SW packages (pies) or required SMU’s
13. Upgrade FPD in admin mode
14. Reload any nodes that had FPD upgrades
15. Configure the Mgmt ethernet interface with an IP address to reach tftp server & load and commit the saved RSP config from tftp server or laptop
a) or log into the console and cut & paste a saved cfg from laptop
b) or copy saved cfg from laptop to usb, then insert usb into RSP440 and copy and commit cfg
c) copy licenses and snmp files back to the RSP’s
16. (Optional) create and generate new crypto keys if required.
As mentioned Turbobooting means that you load the "VM" (virtual machine) XR base OS image.
Turboboot is started from Rommon and is essentially the same as putting a disk with the desired OS in your laptop, reboot the machine to boot from CDROM, and installing the base OS.
Before the Turboboot process starts, you can instruct the system to wipe all files from the system and start clean or install the image to be turbobooted along side with any existing releases currently found on the disk. (see Set the Turboboot variables on the RSP)
Turbobooting may be required if you want to sweep clean your system, or we also had some issues in XR4.2.0 with the RSP2 whereby the upgrade pie could not be loaded. A turboboot was required in that case also.
Some or all of these procedures below are needed.
The command "set" gives you an overview of all the rommon environment variables currently set to their values.
unset BOOT
unset TFTP_FILE
sync
the command *unset* clears the variable value from rommon.
the command *sync *saves or writes the newly set and unset variables to persistent memory so they are saved cross reloads and power cycles.
unset BOOT_DEV_SEQ_OPER
unset MIRROR_ENABLE
sync
By default, the two internal USB partitions (disk0 and disk1) are mirrored to each other, if you break the mirror, turboboot will only affect the disk
that you are turbobooting target to and not the other one (nice if you want to fall back).
priv
diswd <- Disable the CPU watchdog
If you omit this step and the TFTP download for the turboboot mini-vm image takes longer than 30 minutes due to network delays etc, then the RSP might reset and you'll have to start over. Disabling this watchdog makes sure the system is not going to reload during the transfer of the image in rommon.
IP_ADDRESS=ip_address
IP_SUBNET_MASK=mask
DEFAULT_GATEWAY=ip_address
TFTP_RETRY_COUNT=4
sets the number of retries to contact the tftp server
TFTP_TIMEOUT=6000
sets the TFTP timeout for the transfer, you may need to set this larger to prevent abort during xfer if there are network delays
TFTP_CHECKSUM=1
whether checksum on the transfer is needed, this is adviceable in case the image gets corrupted during transfer.
TFTP_SERVER=server_ip_addr
the server address can also be specified in the boot statement, or fixed in the rommon variable.
TFTP_MGMT_INTF=0
which of the 2 mgmt interfaces you want to use, either 0 or 1 with 0 being the default.
TFTP_BLKSIZE=1400
Setting a larger TFTP block size is recommended to pack larger packets and transfer the VM image quicker. Note that for CRS this variable is TFTP_BLOCK_SIZE.
TURBOBOOT=on, {boot-device},[format | clean],[nodisablebreak]
on tells us to install add and install activate the packages when we boot from the VM image.
boot-device is which device we want to use to install the OS, typically disk0
format tells us to replace the OS completely except for the admin configuration
clean tells us to replace the OS completely, but other files such as the admin or exec configuration are saved
nodisablebreak allows us to terminate the turboboot via a break signal. The default is to ignore breaks
Example:
TURBOBOOT=on,disk0,format
sync
This will instruct the system to do a turboboot with disk0 as the selected boot device and to use the format option. The format key is optional.
Currently today we only support targeted install to disk0 but this will change likely in XR4.3.1 whereby you can use disk1 as install target.
NOTE: a recent tac case showed that the command for turboboot failed on the ASR9001.
Supposedly this was made to work by omitting the colon after disk0:
Suggesting to try the disk0 (without colon) if the command with colon fails.
In CRS the format option works with FAT16 but not FAT32 or QNX4 so a new variable must also be used.
In ASR9K the format and clean options do work but in order to erase the exec configuration, admin configuration, and every other file this additional variable must be used.
For these scenarios the following must be set.
TURBOBOOT=on,disk0
MEDIA_FORMAT=disk0:,QNX4
Note: If the format or clean options are set in turboboot or confreg 0x2142 is set when also having the MEDIA_FORMAT variable set then when prompted for a new username/password we will be unable to write this to the disk. To fix this go back to rommon and properly set the variables.
(Works only with the VM image, not the TAR file or mini.pie)
rommon> boot tftp://server/directory/filename
During the boot process the image is copied first on to the memory(RAM) and is installed from memory(RAM). Once it is insalled from memory, it will copy the image back on to disk0: and reload the device. Wait till you get the message "SYSTEM CONFIGURATION COMPLETED"
Output of show install active when in memory,
RP/0/RSP0/CPU0:ios#sh install active
<SNIP>
Active Packages:
mem:asr9k-mini-p-4.2.0
Output of show install active after image copied on to disk0:,
RP/0/RSP0/CPU0:ios#sh install active
<SNIP>
Active Packages:
disk0:asr9k-mini-p-4.2.0
The system will also self unset the TURBOBOOT rommon variable.
To restore disk mirroring, use the mirror command in the global configuration mode. For more information on the mirror command, see the "Boot Commands on Cisco IOS XR Software" module in Cisco ASR 9000 Series Aggregation Services Router System Management Command Reference.
The RSP-440 (and 9001 with rommon 2.03) can boot from the USB front panel port. Instead of using "boot tftp:// or boot disk0:/" you need to use a different command, mediaboot.
The command is:
rommon> mediaboot usb:\release_mini.vm
In later revisions of the rommon, the mediaboot has been superseded to boot usb:/<file>
so make sure you try them both.
NOTE:
Some newer rommon versions on the 9001 want to use the boot usb:/ directive. (see Q&A/comment section below this article).
It is also seen in rommon versions post 2.04 that the usb is referred to as disk1 in which case you can use: boot disk1:/...
To find out the mapping of the usb disk use the rommon "dev" command to see all filesystem devices.
On the CRS-PRP use boot disk2:hfr-mini-px.vm<image>
CRS does not use the mediaboot command.
FPD upgrade for all ASR9K devices using FPD.
a) Enter admin mode via the admin command, and capture the output of the current firmware versions using CLI show hw-module fpd location all. save this output to a text file. Notice any LC that has a “yes” in the Upg/Dng? column. This indicates the FPD should be upgraded or downgraded to match the current FPD version.
b) From admin mode upgrade FPD using the CLI: upgrade hw-module fpd location r/s/m
or if all locations require FPD upgrade (suggested) use CLI:* upgrade hw-module fpd location all *
Disk Space occupied for each image
Simplest way is to use the ksh df utility.
Install a release and packages and run df:
# df /disk0:
/dev/disk00t77 3813344 733477 3079867 20% /dev/disk0:/
Divide the highlighted number by 2000. That gives the approximate size in MB. 366MB in this case.
Repeat for any other releases we should be interested in.
If you do an upgrade, gather the df output before and after upgrade and compute the difference in df output.
Xander Thuijs, CCIE #6775
Principal Engineer ASR9000
Sam Milstead,
Customer Support Engineer TAC XR
I just had same issue Simon and changing TFTP block size to 1024 was only way I could get it to work. I've never had to set this for the ASR9001 Turboboot process before though. I typically use the default or 1400.
Hi,
If we use for eg. TURBOBOOT=on,disk0,clean, how are licenses and disk partitioning preserved? How about disk-mirroring: for eg. mirror location 0/ 0/cpu0 disk0: disk1:? Is it preserved?
Thank you,
Karlo.
Hi Karlo,
Disk mirroring needs to be turned off before turbobooting.
unset BOOT_DEV_SEQ_OPER
unset MIRROR_ENABLE
sync
Clean should preserve licensing. Essentially clean just deletes the package directories, but saves directories such as licensing and configuration.
Thanks,
Sam
So basically after the upgrade you have to re-enable disk mirroring: mirror location 0/ 0/cpu0 disk0: disk1: or is it done automatically as it loads the configuration?
Thanks, K.
truthfully, I see no purpose for disk mirroring at all. it is the same physical usb disk chip, so there is no redundancy benefit there on the asr9000. in fact breaking the mirror gives you 2 times the disk space for additional image storing, which is particularly useful on the RSP2 which is 2x2G.
ps as for licensing, if you do a turboboot, I would always recommend backing up config and license files.
regards
xander
After everything is written to disk and we reload to disk boot, on that boot up we apply the old configuration and that should include disk mirroring if it was previously in the running-config.
Thanks,
Sam
Hm, now I'm a bit confused here:
We have two RSPs the chassis. I thought that if we configure disk mirroring we would gain this: Disk mirroring replicates the critical data on the primary boot device onto another storage device on the same RSP, henceforth referred to as the secondary device. If the primary boot device fails, applications continue to be serviced transparently by the secondary device, thereby avoiding a switchover to the standby RSP. The failed primary storage device can be replaced or repaired without disruption of service. (Source: Configuration Guide, Release 4.2.x)
So for example if the disk0 on RSP0 goes corrupt, it switches over to disk1 on RSP1, then we can format disk0 and thereby fix it.
I can't really test this as I find it hard to corrupt disk0. ;-)
Thanks, Karlo.
Agreed, disk mirroring was a 3.8.x-ish implementation, but is not common anymore on CRS or at all on ASR9K.
Also agreed on backing up the files.
They are under disk0:/license
For the config do copy run <destination>.
Sam
We hit the thread limit for branches :)
That section you quoted brings up an interesting point, it looks like at least the end of the paragraph is CRS specific as ASR9K does not have removable disks, which without removable disks there really is no benefit to disk mirroring.
The idea in CRS was if disk0 failed you could keep using disk1 without having to failover the RP, and simply replace the 'bad' disk0.
With ASR9K the same basic concept applies, we mirror disk0 to disk1 on that RSP, but because we cannot field replace the disk there is little gain. By default RSP0 and RSP1 will sync their disk0 data so we still have redundancy in that sense.
I'm sure I could find a way to cause some disk corruption, but I would rather we not mess up your router :)
Thanks,
Sam
Xander, is USB mediaboot on the 9001 exclusive to 5.1.x and rommon 2.03? We went from 4.3.1 to 5.2.0 on ours which installed rommon 2.04 and mediaboot is not an option there.
The rommon support for the 9001 USB went into 511, so you should be good. I forgot to mention and let me update the doc on it also that there is an inconsistency between RSP models and 9001 to use the external USB. I apologize.
On 9001, it is called disk1, and you dont need the media boot, but regular boot command
Here are the details:
RP/0/RSP0/CPU0:ios#format disk1: Format will destroy all data on "disk1:". Continue? [confirm] Device partition disk1: is now formated and is available for use. RP/0/RSP0/CPU0:ios#copy tftp: disk1: Thu Oct 17 13:47:20.022 UTC Address or name of remote host []?1.1.1.1 Source filename [/tftp:]?asr9k-mini-px.vm_nebs1 Destination filename [/disk1:/asr9k-mini-px.vm_nebs1]? Accessing tftp://1.1.1.1/asr9k-mini-px.vm_nebs1 CCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCCC 405843128 bytes copied in 671 sec ( 604074)bytes/sec RP/0/RSP0/CPU0:ios#reload rommon B1 > rommon B1 > rommon B1 > rommon B1 > dir disk1: File System: FAT32 3 0 drw- LOST.DIR 4 405843128 -rw- asr9k-mini-px.vm_nebs1 rommon B2 > rommon B5 > b disk1:/asr9k-mini-px.vm_nebs1 Boot Status 16 Located asr9k-mini-px.vm_nebs1, start cluster is 4 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! <<<<<<remove usb>>>>> rommon B1 > dir disk1: USB device not found rommon B2 > rommon B1 > dir disk0a: rwxrwxr-x 4096 . rwxrwxr-x 4096 .. r--r--r-- 250880 .bitmap r--r--r-- 8192 .inodes rw------- 0 .boot rw------- 0 .altboot rwxr-xr-x 4096 LOST.DIR rwxr-xr-x 4096 LOST.DIR r--r--r-- 0 .longfilenames rwxrwxrwx 4096 sysmgr_debug rwxr-xr-x 4096 usr rwxr-xr-x 4096 var rwxr--r-- 405843128 asr9k-mini-px.vm_nebs1 rommon B2 > <<<<TURBOBOOT>>> rommon B3 > TURBOBOOT=on,disk0,format rommon B4 > rommon B4 > rommon B4 > sync rommon B5 > b disk1:/asr9k-mini-px.vm_nebs1 Boot Status 16 Located asr9k-mini-px.vm_nebs1, start cluster is 4 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
xander
Ah that did work great. Thanks!
Xander,
When mediaboot in used on IOX XR 5.1.3 doing turboboot i doesn't work. Beloew is the related output.
Error
rommon 1 > mediaboot usb:asr9k-mini-px.vm-5.1.3
boot_disk2 - Launching image.
#################################################################
Program load complete, Entry point: 0x40d878, size: 0x1761cbc3
òóÂãY$)k
œÊŸý)ºþ
No Error
rommon 1 > boot usb:asr9k-mini-px.vm-5.1.3
Beginning Media boot:
boot_disk2 - Launching image.
#################################################################
Program load complete, Entry point: 0x40d878, size: 0x1761cbc3
Startup-code initializing, running on Storm (00100307)
BSP date: May 7 2014 01:40:11
Debugging enabled; debug_flag = 1, vendor_debug_flag = 0
Page Address Extension mode (PAE) enabled with 36bit physical addressing.
Running on a ASR9K x86 RSP
Found 12288MB of main memory
Reading back MTRR config from boot processor.
Found LP at APIC ID 2
Found LP at APIC ID 4
Found LP at APIC ID 6
IPL reports 4 logical processors
Family = 00000006, Model = 0000001e
4 CPU cores available
Initializing logical processor 0 (APIC ID 0).......OK (BSP)
Initializing logical processor 1 (APIC ID 2).......OK
Initializing logical processor 2 (APIC ID 4).......OK
Initializing logical processor 3 (APIC ID 6).......OK
Chassis Type: ASR9010-V2 (00ef02fa)
Region shmwin1: 70000000:b0000000
Region pakman: b0000000:b8000000
Can you try using 'boot' instead of 'mediaboot'?
Also Xander answered a similar question a few weeks ago with outputs provided as well.
Thanks,
Sam
Yes and it works. Thats what im trying to point out because Xander in this document specifies this:
The RSP-440 (and 9001 with rommon 2.03) can boot from the USB front panel port. Instead of using "boot tftp:// or boot disk0:/" you need to use a different command, mediaboot.
The command is:
rommon> mediaboot usb:\release_mini.vm
And doesn't work.
Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: