topic Re: ASR 1001-x upgrade path in Routing and SD-WAN

ASR 1001-x upgrade path

netspazz — Tue, 09 Mar 2021 21:23:46 GMT

Looking at the process to upgrade an ASR 1001-x, currently 16.9.2. Doing some research this looks to be not the old way of uploading the bin, changing the boot var and performing a reload. I have a few questions.

1. the document shows a "subpackage" install. Is this they path that must be taken? uploading the bin then expanding all the packages? this seams to be a much more intense process then the old way.

2. currently my boot var shows nothing configured, and nothing for the standby boot var. is this normal? I do see the .bin file sitting on the bootflash.

3. I am running redundancy, sso, so I believe this is normal for an upgrade.

thanks for any help.

Re: ASR 1001-x upgrade path

netspazz — Tue, 09 Mar 2021 22:00:21 GMT

I have a couple additional comments.

1. How do you tell if you need to upgrade using subpackages? when I run "sh ver installed" it shows package, so I am assuming that i must upgrade using subpackages.

2. I noticed that the web GUI has an area to upload/install software. Would this actually be the best way, it seems behind the scene it is unpacked and installed by the GUI.

Re: ASR 1001-x upgrade path

Leo Laohoo — Tue, 09 Mar 2021 22:23:18 GMT

@netspazz wrote:

Doing some research this looks to be not the old way of uploading the bin, changing the boot var and performing a reload.

Welcome to IOS-XE.
There are (currently) two ways to upgrade in IOS-XE: Bundle Mode and Install Mode.

What you've just describe is "Bundle Mode". It is an "easy" way but the bootup will take longer and will chew up extra memory (unpacking the packages to memory).

For obvious reasons, Install Mode is highly recommended.

Next, for some demented reasons, Cisco now "hides" the documentation in about what ROMMON/Bootstrap version a router needs to be running, as a minimum, and what IOS-XE to run.

So you will need to find that documentation to make sure the IOS-XE version you run "matches" with the ROMMON version. Otherwise, upgrade to the ROMMON to the latest version will save you the hassle in the future.

Re: ASR 1001-x upgrade path

netspazz — Tue, 09 Mar 2021 23:11:26 GMT

thanks for the information. Its starting to make sense. Looking at the "install" command documentation, I see 2 types.

1. Super Package install—Applicable to Dual RP
2. Sub-package install—Applicable to Dual RP

Not sure what the difference is, or maybe depends on the hardware.

Re: ASR 1001-x upgrade path

Leo Laohoo — Sun, 11 Apr 2021 23:10:54 GMT

1. Copy the IOS-XE file into the bootflash:

2. Copy the ROMMON file into the bootflash:

3. Upgrade the ROMMON (DO NOT REBOOT)

upgrade rom-monitor filename bootflash:FILENAME all

4. Unpack the IOS-XE file

install add file bootflash:IOS-XE_FILENAME.bin activate commit

5. Change the boot variable string to point to the "packages.conf" file

conf t
 no boot system flash bootflash:<OLD IOS>
 boot system flash bootflash:packages.conf
 boot system flash bootflash:<OLD IOS>
end
wr

6. Examine the "packages.conf" (2nd page) to make sure the new packages are pointing to the new IOS version

more bootflash:packages.conf

7. Save the config

8. Reboot the appliance.

9. After the reboot, clean up the bootflash:

install remove inactive

Re: ASR 1001-x upgrade path

netspazz — Wed, 10 Mar 2021 00:56:43 GMT

Thanks for the step by step. Would your steps be the way to go? I will look into the ROMMON, don't see it mentioned in the document.

I looked at the "install" method and the steps it takes. Since I have redundancy configured I thought I would use the following for dual RP.

1. # install add file bootflash:<file.bin>

2. # install activate location standby

3. # redundancy force-switchover

4. # install abort-timer stop

5. # install activate location standby

6. # install commit

cisco is really making this harder than it should be, or I am missing something.

Re: ASR 1001-x upgrade path

Leo Laohoo — Wed, 10 Mar 2021 01:47:48 GMT

@netspazz wrote:

I will look into the ROMMON, don't see it mentioned in the document.

The only ROMMON matrix I know of is the ones for the ISR 4000 routers (Recommended ROMMON Release).

@netspazz wrote:

Would your steps be the way to go?

Your call. The steps I have listed are the steps I use. Regularly.

@netspazz wrote:

cisco is really making this harder than it should be, or I am missing something.

No, this is the new Cisco "standard".

Re: ASR 1001-x upgrade path

netspazz — Wed, 10 Mar 2021 17:01:04 GMT

Thanks, it makes more sense now. Do some more research basically .bin=bundle mode, .conf=Install mode.

Looking at my system it has the .bin so I am using Bundle mode, which is interesting since it came from cisco and I thought Install mode was the default mode.

I will test out the install mode on a spare router. My only question is since it takes a reload then the redundancy of the RP does not matter then?

thanks again.

Re: ASR 1001-x upgrade path

Leo Laohoo — Thu, 11 Mar 2021 04:22:10 GMT

The RP will reboot because the parent appliance will reboot.

Re: ASR 1001-x upgrade path

Simba_NetEngg — Fri, 09 Apr 2021 19:39:03 GMT

Hello Leo,

Thanks for describing the step by step process.

I have some additional queries regarding the ASR1001-X upgrade process. It would be very nice of you to comment on my below queries:

1. I could not find any document which mentions the upgrade path. Our routers are currently working on version 16.9.2 and we are planning to upgrade to 16.12.5. Can this be done in one single step or i need to perform an intermediate OS upgrade?

2. If i understand it correctly, the method you described here is Bundle mode upgrade, as i do not see any "install" commands? You mentioned with this method the device will take relatively longer time to boot up, please can you advise on how much downtime time should be expected if we use this method.

3. I have two routers which are in simplex mode(non-redundant) but running HSRP primary/secondary. Please can you advise it the method you described would be the best upgrade method to follow in this setup.

Re: ASR 1001-x upgrade path

Leo Laohoo — Sat, 10 Apr 2021 00:47:20 GMT

@Simba_NetEngg wrote:

Our routers are currently working on version 16.9.2 and we are planning to upgrade to 16.12.5. Can this be done in one single step or i need to perform an intermediate OS upgrade?

Upgrade from 16.9.X to 16.12.X is possible. Make sure the ROMMON is 16.12(2r). If it is not, upgrade the ROMMON and the IOS-XE.

Out of curiosity, are the routers configured with LACP?

@Simba_NetEngg wrote:

the method you described here is Bundle mode upgrade

The method I've described (above) as Install Mode and takes up to seven (7) minutes for the router to boot up.

NOTE: If it is possible, do a COLD boot (turn the router OFF for about 5 seconds before powering it back up).

@Simba_NetEngg wrote:

I have two routers which are in simplex mode(non-redundant) but running HSRP primary/secondary. Please can you advise it the method you described would be the best upgrade method to follow in this setup.

No different. Just reboot/cold boot one router before doing the next one.

Re: ASR 1001-x upgrade path

Simba_NetEngg — Sat, 10 Apr 2021 11:20:32 GMT

Hello Leo,

Thanks for your valuable response.

1. The current ROMMON version in our routers is 16.9(4r).
I was not planning to upgrade the ROMMON originally as i found below Cisco document which says that the minimum and recommended ROMMOM version for OS Version 16.12.X is 16.9(4r). Please can you share if there is any specific reason due to which i should upgrade ROMMON to 16.12(2r)?

https://www.cisco.com/c/en/us/td/docs/routers/asr1000/rommon/asr1000-rommon-upg-guide.html?referring_site=RE?pos=1&page=https://www.cisco.com/c/en/us/td/docs/routers/asr1000/release/notes/xe-16-12/asr1000-rel-notes-xe-16-12.html

2. Yes we are running LACP.

3. If this is the install method, should i be using any install commands like below too or only the ones described in your message are sufficient.

install add file bootflash:asr1001x-universalk9.16.12.05.SPA.bin
install activate file bootflash:asr1001x-universalk9.16.12.05.SPA.bin

4. Also one additional question, in your original thread, you mentioned in step 5 regarding boot variable that the boot order should have packages.conf as the primary OS and <OLD IOS> as secondary. In my case, old IOS is asr1001x-universalk9.16.09.02.SPA.bin, so will my command be "boot system flash bootflash:asr1001x-universalk9.16.09.02.SPA.bin" or do i need to expand the old IOS into packages and add the packages.conf file of the OLD IOS in boot variable, same as the new IOS.

5. I will be upgrading this remotely, so cold reboot will be difficult. Though i will try to arrange someone onsite if you are suggesting it.

Sorry for so many questions, i am very new to upgrading IOS XE. 🙂

Thanks in advance.

Re: ASR 1001-x upgrade path

Simba_NetEngg — Sat, 10 Apr 2021 16:04:24 GMT

Hello Leo,

Thanks for your valuable response.

2. Yes we are running LACP.

3. If this is the install method, should i be using any install commands like below too or only the ones described in your message are sufficient.

install add file bootflash:asr1001x-universalk9.16.12.05.SPA.bin
install activate file bootflash:asr1001x-universalk9.16.12.05.SPA.bin

5. I will be upgrading this remotely, so cold reboot will be difficult. Though i will try to arrange someone onsite if you are suggesting it.

Sorry for so many questions, i am very new to upgrading IOS XE.

Thanks in advance.

Re: ASR 1001-x upgrade path

Leo Laohoo — Sun, 11 Apr 2021 04:12:36 GMT

@Simba_NetEngg wrote:

2. Yes we are running LACP.

So far, CSCvw74609 has only been reported to affect ISR 4k so be careful.

@Simba_NetEngg wrote:

Please can you share if there is any specific reason due to which i should upgrade ROMMON to 16.12(2r)?

The router is going to get upgraded to a new IOS-XE version. Upgrade the ROMMON while you can. In the future, someone will/might "forget" this tiny tidbit (ROMMON version to IOS-XE version) and spend countless hours wondering why the router refuses to load the new firmware. Remember, when the router refuses to load the new firmware (because the ROMMON version is not "compatible), it means the router has booted into ROMMON. And the drive to the site to the site (to fix this oversight) is usually lonely and sad -- Which means no one can hear you scream.

IMPORTANT:
Make ROMMON upgrade a "habit" especially with IOS-XE upgrade. Why? Because Cisco has a funny way of hiding the ROMMON version matrix. In some platforms, the ROMMON version matrix is in the Release Notes of the IOS-XE, however, in some platforms, it is hidden or un-linked.

@Simba_NetEngg wrote:

install add file bootflash:asr1001x-universalk9.16.12.05.SPA.bin
install activate file bootflash:asr1001x-universalk9.16.12.05.SPA.bin

Cut it down to one line:

install add file bootflash:asr1001x-universalk9.16.12.05.SPA.bin activate commit

@Simba_NetEngg wrote:

Also one additional question, in your original thread, you mentioned in step 5 regarding boot variable that the boot order should have packages.conf as the primary OS and <OLD IOS> as secondary. In my case, old IOS is asr1001x-universalk9.16.09.02.SPA.bin, so will my command be "boot system flash bootflash:asr1001x-universalk9.16.09.02.SPA.bin" or do i need to expand the old IOS into packages and add the packages.conf file of the OLD IOS in boot variable, same as the new IOS.

Good point.

No need to "expand" 16.9.2.

So do this:

conf t
 no boot system flash bootflash:asr1001x-universalk9.16.09.02.SPA.bin
 boot system flash bootflash:packages.conf
 boot system flash bootflash:asr1001x-universalk9.16.09.02.SPA.bin
end
wr

Hope this helps.

TIP:

To anyone who is about to embark on IOS-XE & ROMMON upgrade for the first time, connect the console port and enable logging. This will make troubleshooting easier.

Re: ASR 1001-x upgrade path

Simba_NetEngg — Sun, 11 Apr 2021 14:31:14 GMT

Hello Leo,

Thanks for the wise words, will keep them in mind during this and future upgrades.

1. I do not see any option to download ROMMON version 16.12(2r) in Cisco downloads section for ASR 1001-X. The only 16.X ROMMON version currently available for ASR 1001-X is 16.9(4R). I believe i will have to keep the ROMMON as it is for now, or if you suggest otherwise.

2. To sum it up, i will be using below commands/steps during the upgrade using Install mode after copying the IOS image to flash. Please let me know if they look fine to you and should be sufficient to perform the task.

Step a. Expand OS:

request platform software package expand file bootflash:asr1001x-universalk9.16.12.05.SPA.bin over verb

Step b. Install OS and activate:

install add file bootflash:asr1001x-universalk9.16.12.05.SPA.bin activate commit

Step c. Configure Boot variables:(Currently there is no boot variable set, as this is the first time we are upgrading the devices after installation)

conf t
boot system flash bootflash:packages.conf
boot system flash bootflash:asr1001x-universalk9.16.09.02.SPA.bin
end
wr

Step d. Reboot the appliance.

Step e. Cleanup bootflash after upgrade goes successful:

request platform software package clean

------------

Thanks in advance.

Re: ASR 1001-x upgrade path

Leo Laohoo — Sun, 11 Apr 2021 23:14:26 GMT

@Simba_NetEngg wrote:

request platform software package expand file bootflash:asr1001x-universalk9.16.12.05.SPA.bin over verb

request platform software package clean

My sincerest apologies. This command is only compatible on the 9300. Any other appliances and this command is incompatible.
I have edited my response(s) to the correct commands.

16.9(4r) is fine.

Re: ASR 1001-x upgrade path

Simba_NetEngg — Tue, 13 Apr 2021 12:34:48 GMT

Hello Leo,

Noted and understood so far.

Just one more concern at this moment before i book the maintenance window.

Please can you confirm if the "packages.conf" should come in the boot variables in place of the IOS file itself - "asr1001x-universalk9.16.12.05.SPA.bin". Asking because i am not sure how packages.conf will be created if we are not running the expand command on asr1001x-universalk9.16.12.05.SPA.bin. Will it be generated as a result of the install and activate command?

Thanks in advance.

Re: ASR 1001-x upgrade path

Leo Laohoo — Tue, 13 Apr 2021 13:08:48 GMT

@Simba_NetEngg wrote:

Will it be generated as a result of the install and activate command?

Yes and no.
Yes, the "packages.conf" will be created every time the firmware gets unpacked.

I have seen where the script throws up an error (at the end of the process) and "refuses" to create the file. It will, however, create a new file called "asr1001x-universalk9.16.12.05.SPA.conf". If this is the case, then rename that file to "packages.conf" is the fix.

Re: ASR 1001-x upgrade path

Simba_NetEngg — Tue, 13 Apr 2021 14:39:39 GMT

Welp, then i guess we will know this during the upgrade itself.

Will let you know how it goes.

Thanks for all the help!

Re: ASR 1001-x upgrade path

TW80CJ5 — Fri, 19 Nov 2021 13:24:58 GMT

Hey Leo!

This post was very helpful...I do have a question for you I saw a previous post of yours: what's the difference between:

request platform software package expand file bootflash:asr1001x-universalk9.16.12.03.SPA.bin over verb

and

install add file bootflash:IOS-XE_FILENAME.bin activate commit

I used the latter to successfully upgrade our ASR 1001 HX router. So thanks a ton!

Also, where you list:

conf t
no boot system flash bootflash:<OLD IOS>
boot system flash bootflash:packages.conf
boot system flash bootflash:<OLD IOS>
end
wr

Can I point to the new IOS.bin file in case the packages.conf doesn't load correctly? My current boot variable is the following:

bootvariable = bootflash:/packages.conf,1; bootflash:/asr1000-universalk9.16.06.04.SPA.bin,1;

You were tremendously helpful with 16.12.05 to .06 upgrade. I successfully upgraded my 9300's to 16.12.06 and am running the "sh platform software status control-processor br"