1. I was able to turboboot directly from 4.3.4 SP5 to 5.3.3 with no SMUs. Same with 5.1.3. Version 4.3.4 SP5 to 5.1.3 required SMUs due to the Abraxis server change for validating files.
... View more
Thanks for the reply and the info about keeping the STBY RSP in ROMMON for safe keeping. There is a security for me with removing the RSP in any event where it could possibly sync before I know I do not want to rollback. If i have to turboboot rollback we are talking about almost 2 hours and 15 minutes of rollback if the 2nd RSP is compromised. If I can just re-seat that second RSP, 15 minutes rollback.
What you shared above is great and will work. Maybe I am too much of a nervous person to rely on it to not wipe my rollback RSP and throw a potential rollback outside of the maintenance window!
... View more
Turboboot timeline (4.3.4 SP5 to 5.3.3):
3:03am - Router fully isolated, “admin config-register boot-mode rom-monitor location all” then “admin reload location all” executed
3:06am – From ROMMON set turboboot, then “boot harddisk:\asr9k-mini-px.vm-5.3.3”
3:27am - Complete - All Cards are up and ready for next step in the upgrade (admin show platform)
3:28am – “install add tar harddisk:ASR9K-iosxr-px-K9-5.3.3.tar synchronous”
3:34am - Complete
3:35am – “install activate id 1 synchronous”
3:51am - Complete
3:52am – “admin upgrade hw-module fpd all location all”
4:06am - Complete
4:07am – “admin reload location all”
4:15am - Complete - All Cards are up and ready to restore ASR9K chassis traffic (admin show plat)
4:16am - Load saved config and start to bring ASR9K back on the network
Cleanup: After all OSS alarms verified/cleaned and all services back up, reinserted RSP1 (back-out RSP) and upgraded required FPD firmware (CBC, FPGA1, FPGA2, ROMMON).
... View more
Thank you for your reply. Please let me show you a guaranteed rollback option that works flawlessly if you are using dual RSPs (redundancy) in a single chassis. This rollback has been tested successfully in our lab environment multiple times and has now been used in production with no issues.
Our upgrade process has us manually turn down the interfaces on the cards in a controlled manner. We isolate the chassis as we have redundant chassis connected with EtherBundles, so there is another ASR9K up and running with all traffic. The running-config and admin running-config are now saved. We then save a copy of the running-config and admin running-config to the harddisk: of the ASR9K so it can be loaded once the turboboot is complete.
Now that the chassis is isolated and the configs are saved (across both RSPs), we physically unseat the STBY RSP from the chassis backplane so that it powers down and the show redundancy command reports the RSP gone. Now with one RSP in place (ACTV), we will turboboot the RSP /chassis leaving the secondary RSP out the entire time.
Once the upgrade is complete and traffic returned to the box, ACTV RSP firmware upgraded with everything functioning as normal, the STBY RSP can be returned to the chassis backplane (this removes the rollback option). We can then manually FPD the STBY RSP to upgrade the firmware to match the newly upgraded via turboboot RSP.
If there is an issue with the software upgrade during turboboot, we unseat the ACTV card (so both RSPs are disconnected from the backplane), re-seat the old STBY RSP into the chassis, and the chassis reboots on the previous RSP with all existing software and config (with interfaces still manually turned down). This lets us rollback in about 10 minutes at any stage of the turboboot and have the ability to gracefully return traffic to the box as the interfaces are still manually down.
Thank you for the news on the service pack! This is good to know and will work well with my rollout plans. We will definitely deploy the service pack.
... View more
Thanks know you for the reply. We were able to perform the turboboot twice now successfully. I will post information about my upgrade details to IOS XR 5.3.3 with a turboboot so others can benefit from the information.
We turboboot upgrade our ASR 9Ks annually and we like using the turboboot feature as we have to isolate the chassis interfaces from the network either way if a reboot is involved. The turboboot ensures that we have a procedure that never changes with every upgrade except for the names of the files used. It also allows us to ensure we do a full reset (multiple resets with a turboboot) on all the elements of the chassis (cards, CPUs, power equipment). This helps us do fail over testing of the hardware and can identify hardware issues in a controlled environment with tier 3 and onsite engineers engaged. I find the turboboot to be any and repeatable, the process hardly changes year over year.
It will also make sure we do not run into any issues with leftover software loads and memory space issues on the RSPs.
Why do you not recommend a turboboot? Is this harder on the equipment? There is no risk to the network as the chassis is already manually isolated from the environment and we gracefully turn up the interfaces to return the units to service.
... View more
Wanted to share:
IOS XR 5.3.3 Turboboot (from 4.3.4 SP5).
Turbobooting from the mini-px, I had to use the following command to get the vm to load, if not, the RSP would stay in a LOAD or MBI status (LCD screen) for 30 minutes before the CPU Watchdog would reset the RSP back to ROMMON. Reseating the card caused us to receive a Power Sequencer failure and the RSP had to be RMA'd (PSEQ—CBC detected power sequencer failure on card LCD, no console access).
The command that worked, would probably work with harddisk: as well:
RSP 440, FYI, Codes below:
INIT—Card is inserted and microcontroller is initialized
BOOT—Board is powered on and CPU is booting
IMEM—Start initializing memory
IGEN—Start initializing the board
ICBC—Start initializing communication with the microcontroller
SCPI—Board is not plugged in properly
STID—CBC was unable to read slot ID pins correctly
PSEQ—CBC detected power sequencer failure
DBPO—CBC detected an issue during board power up
KPWR—CBC detected an issue during board power up
LGNP—CBC detected an issue during board power up
LGNI—CBC detected an issue during board power up
RMN—All tests are finished and ROMMON is ready for commands
LOAD—Downloading MBI image to CPU
RRST—ROMMON is performing a soft reset after 5 consecutive MBI validation requests timed out
MVB—ROMMON trying MBI validation boot
MBI—Starting execution of MBI
IOXR—Cisco IOS XR software is starting execution
LDG—The RSP is loading (MBI started and card preparing for activity)
INCP—The software or configuration is incompatible with the RSP
OOSM—The RSP is in Out of Service, Maintenance mode
ACTV—RSP role is determined to be active RSP
STBY—RSP role is determined to be standby RSP
PREP—Preparing disk boot
... View more