ISR 4431 from version 16.9.5 to 17.9.5?

[nict]

I've read the documentation Cisco 4000 Series ISRs Software Configuration Guide, Cisco IOS XE 17 which says 

"When you upgrade from Cisco IOS XE 3.x to 16.x image, you should first upgrade the rommon release to the 16.7(5r) rommon release. After upgrading to the 16.7(5r) rommon release, based on the IOS XE 16.x image, the rommon release can be auto-upgraded to a later rommon release."

But this only mentions for 16.x image's - Can someone confirm if it is possible to upgrade from IOS XE 16.9.5, ROMMON version 16.7(5r), directly to 17.9.5a?

Thanks.

[Leo Laohoo]

---

@nict wrote:

"When you upgrade from Cisco IOS XE 3.x to 16.x image, you should first upgrade the rommon release to the 16.7(5r) rommon release. After upgrading to the 16.7(5r) rommon release, based on the IOS XE 16.x image, the rommon release can be auto-upgraded to a later rommon release."

---

For "consistency" purposes (or lack of), this is both true and false.  

It is true that in 17.X.X releases, future ROMMON upgrade is rolled into the firmware.  For example, ROMMON version 17.5(1r) is rolled into 17.6.X and 17.9.X, HOWEVER, it is not rolled into 17.12.X.  Another thing is Cisco has released ROMMON version 17.15(1r).  There are "inconsistencies" with what Cisco is saying and what Cisco is actually doing.  

Another thing, the "latest" ROMMON version is 16.12(2r) and, frankly speaking, I'd recommend upgrading to this version than 16.7(5r) because that recommendation was probably not updated to keep up with the times.

After upgrading the ROMMON, direct upgrade to 17.9.X is doable.  

But DO NOT, regardless what TAC says, conduct a direct upgrade, from 16.X.X to 17.12.X.  Failure to do so will cause the router to perform a silent boot-crash-loop.  Upgrade first to 17.6.X or 17.9.X before jumping to 17.12.X (and later).  This fact is not found in any public-facing Cisco documentation.  

[nict]

Thank you so much for your reply Leo. I am going to upgrade to 16.12(2r) before upgrading to 17.9.5a.

It is kinda funny you mention the boot-crash-loop. Last week, I upgrade an ISR 1100 from 16.9.x to 17.12.x and I experienced the boot-loop-crash. There was no way to break into Rommon or boot up on the previous image on the ISR - We had to make a RMA to get a new ISR. On the other ISR 1100's, I went with the solution to upgrade to 17.9.x directly instead, which works fine.

[Leo Laohoo]

---

@nict wrote:
Last week, I upgrade an ISR 1100 from 16.9.x to 17.12.x and I experienced the boot-loop-crash. There was no way to break into Rommon or boot up on the previous image on the ISR

---

I found out about this "undocumented feature" when we tried to upgrade a 1121 router from 17.3.X to 17.12.3 back in April 2024.

It took me about 2 hours to figure out what was happening.  I was very lucky the router was in front of me.  

Imagine if you were the person who "pushed the button" and instructed DNAC to upgrade routers.  

---

@nict wrote:
We had to make a RMA to get a new ISR.

---

Unfortunately, that was a "rookie mistake" from Cisco TAC.  

Any half-decent Cisco TAC engineer would know how to break into a boot-crash-loop like that.  Not only that, what caused the boot-crash-loop can be undone.  I know this fact because I fixed our router.  

[nict]

@Leo Laohoo wrote:

---

@nict wrote:
Last week, I upgrade an ISR 1100 from 16.9.x to 17.12.x and I experienced the boot-loop-crash. There was no way to break into Rommon or boot up on the previous image on the ISR

---

I found out about this "undocumented feature" when we tried to upgrade a 1121 router from 17.3.X to 17.12.3.It took me about 2 hours to figure out what was happening.  I was very lucky the router was in front of me.  

Imagine if you were the person who "pushed the button" and instructed DNAC to upgrade routers.  

I was not aware of the problem with upgrading for 16.9.x to 17.12.x directly, but I actually think Cisco has documented it here

"To upgrade to Cisco IOS XE Dublin 17.12.x, follow these steps:

1. If you are on a device that is running software version between Cisco IOS XE 16.x to Cisco IOS XE 17.4.x, upgrade to any IOS XE image between Cisco IOS XE 17.5.x to Cisco IOS XE 17.10.x.

2. After performing step a, upgrade to Cisco IOS XE 17.12.x.

3. For devices that are running on software version Cisco IOS XE 17.5.x or later, you can upgrade to Cisco IOS XE 17.12.x directly." 

But I will agree, that they don't mention what could happen and why you can't upgrade directly to 17.12.x.

---

---

@nict wrote:
We had to make a RMA to get a new ISR.

---

Unfortunately, that was a "rookie mistake" from Cisco TAC.  

Any half-decent Cisco TAC engineer would know how to break into a boot-crash-loop like that.  Not only that, what caused the boot-crash-loop can be undone.  I know this fact because I fixed our router.

---

How would you break it? I tried myself with SecureCRT on my Macbook. I did CTRL + C, Space, Enter, ALT + F4 and "Break" directly from the GUI. Nothing worked. I tried for a cople of hours. I also tried with my Windows Machine, with both SecureCRT and Putty. Nothing worked, the boot processed just continued. The extra thing was, that Password Recovery was also disabled. I don't know if that was the problem.

[Leo Laohoo]

---

@nict wrote:
How would you break it? I tried myself with SecureCRT on my Macbook.

---

I have SecureCRT but I treat it like a normal password-recovery procedure, however, I have a Ctrl+Break button to do just that when I do have to conduct password-recovery.  

---

@nict wrote:
why you can't upgrade directly to 17.12.x.

---

Because it will shed a bad light of Cisco.  

[shambhu.kumar]

First upgrade ROMMON 16.12(2r) regardless it is 3.x or  16.x image,once ROMMON upgraded to 16.12(2r) and then upgrade 17.9.5a directly.

ISR4321#sh version
Cisco IOS XE Software, Version 03.16.04b.S - Extended Support Release
Cisco IOS Software, ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 15.5(3)S4b, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport

ISR4321#sh platform
Chassis type: ISR4321/K9

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ISR4321/K9 ok 26w6d
0/0 ISR4321-2x1GE ok 26w6d
R0 ISR4321/K9 ok, active 26w6d
F0 ISR4321/K9 ok, active 26w6d
P0 PWR-4320-AC ok 26w6d
P2 ACS-4320-FANASSY ok 26w6d

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 15030325 16.2(2r)
R0 15030325 16.2(2r)
F0 15030325 16.2(2r)

================================================================

Copy ROMMON 16.12(2r) and recommended IOS-XE image

ISR4321#dir
Directory of bootflash:/

11 drwx 16384 Aug 28 2017 14:49:50 +03:00 lost+found
112449 drwx 4096 Aug 28 2017 15:11:53 +03:00 .prst_sync
12 -rw- 486645440 Aug 28 2017 15:02:49 +03:00 isr4300-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin
16065 drwx 4096 Aug 28 2017 14:50:54 +03:00 .installer
80321 drwx 4096 Aug 28 2017 15:12:28 +03:00 core
240961 drwx 4096 Aug 28 2017 15:11:01 +03:00 .rollback_timer
13 -rw- 0 Aug 28 2017 15:11:11 +03:00 tracelogs.622
144577 drwx 4096 Nov 10 2024 13:59:15 +03:00 tracelogs
14 -rw- 30 May 2 2024 14:28:04 +03:00 throughput_monitor_params
15 -rw- 15125 Apr 7 2024 15:04:23 +03:00 7April2024bkp
16 -rw- 5032908 Nov 11 2024 08:31:57 +03:00 isr4200_4300_rommon_1612_2r_SPA.pkg
17 -rw- 776466178 Nov 11 2024 08:48:21 +03:00 isr4300-universalk9.17.12.04.SPA.bin

3232477184 bytes total (1798492160 bytes free)

Upgrade ROMMON
ISR4321# upgrade rom-monitor filename bootflash:isr4200_4300_rommon_1612_2r_SPA.pkg R0
ROMMON upgrade complete.
To make the new ROMMON permanent, you must restart the RP.
<<<Please reload here>>>>>

ISR4321#sh platform
Chassis type: ISR4321/K9

Slot Type State Insert time (ago)
--------- ------------------- --------------------- -----------------
0 ISR4321/K9 ok 00:03:15
0/0 ISR4321-2x1GE ok 00:01:58
R0 ISR4321/K9 ok, active 00:03:15
F0 ISR4321/K9 ok, active 00:03:15
P0 PWR-4320-AC ok 00:02:58
P2 ACS-4320-FANASSY ok 00:02:58

Slot CPLD Version Firmware Version
--------- ------------------- ---------------------------------------
0 15030325 16.12(2r)
R0 15030325 16.12(2r)
F0 15030325 16.12(2r)

Change the boot variable for your recommended IOS-XE image
ISR4321(config)#boot system bootflash:isr4300-universalk9.17.12.04.SPA.bin
ISR4321(config)#boot system bootflash:isr4300-universalk9.03.16.04b.S.155-3.S4b-ext.SPA.bin
ISR4321#wr
ISR4321#reload

Cisco IOS XE Software, Version 17.12.04
Cisco IOS Software [Dublin], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Version 17.12.4, RELEASE SOFTWARE (fc3)

[nict]

Thank you for your response!

I have never performed a ROMmon upgrade before, so I have a couple of questions, if you don’t mind:

Is it possible to upgrade the ROMmon without an immediate reload, then later select the 17.9.x image and perform the reload? Or would it be better to reload right after upgrading the ROMmon, as you demonstrated, and then select the new image and reload again?

The reason I’m asking is that the customer wants the ROMmon upgrade done in advance, but they plan to reload the ISR at a specific time to minimize user impact.

[Leo Laohoo]

---

@nict wrote:
Is it possible to upgrade the ROMmon without an immediate reload

---

I perform a ROMMON upgrade and an IOS upgrade in one reload.   Please read this:  Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode)

---

@nict wrote:
The reason I’m asking is that the customer wants the ROMmon upgrade done in advance, but they plan to reload the ISR at a specific time to minimize user impact.

---

Please read this:  Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode).  Because all your questions are explained (with pictures!) in this document.

[shambhu.kumar · Accepted solution]

Hello,

You can perform this activity in single reload.

Change the boot variable,

ISR4321(config)#boot system bootflash:isr4300-universalk9.17.12.04.SPA.bin

and then Upgrade ROMMON

ISR4321# upgrade rom-monitor filename bootflash:isr4200_4300_rommon_1612_2r_SPA.pkg R0

Once it prompts "To make the new ROMMON permanent, you must restart the RP." then reload.

Hope this will help.

[Leo Laohoo]

---

@shambhu.kumar wrote:
ISR4321(config)#boot system bootflash:isr4300-universalk9.17.12.04.SPA.bin

---

That's Bundle Mode. 

Bundle Mode means the router, switch, WLC boots the BIN file.  Because the BIN file is the compact form of the firmware, the packages are extracted in the memory of the platform.  This means the boot-up time is longer and utilizes more CPU and memory resources.  It is NOT possible to install any SMU in Bundle Mode.  

Install Mode means all the packages files are fully extracted and the switch goes through the list to boot the different packages in order.  Because the package files are already extracted into the flash, the boot-up time is faster and uses less memory and CPU resources.  Because the package files are fully extracted, the stack master is capable (if the commands are present or enabled) to push the firmware down to any stack member joining the stack with incorrect firmware version.

For obvious reasons, Cisco TAC recommends Install Mode and only use Bundle Mode in "extreme cases" such as booting a platform in ROMMON.

I would never encourage nor recommend anyone to consider using Bundle Mode.  

[shambhu.kumar]

We do understand difference of Install and bundle mode. This is 4000 Series ISR router. There is no concept of stack master.

customer has very specific requirement about user impact

How many reload do you expect  to upgrade from 3.X to 17.9.5, including upgrade of ROMMON. through Install mode.

[Leo Laohoo]

---

@shambhu.kumar wrote:
How many reload do you expect  to upgrade from 3.X to 17.9.5, including upgrade of ROMMON. through Install mode.

---

Upgrade the ROMMON, do not reload, unpack the firmware Packages from the BIN file.  And then reload. 

One reload.  Install Mode. 

It can be done.  I do this all the time.  As a matter of fact, I even "wrote the book" about it:  Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode)

Be mindful that it is very easy to hit CSCvg37458 with Bundle Mode.  Why?  Because in Bundle Mode firmware upgrade, the boot variable string has to change.  This is open to mistakes.  And when the mistake occurs, the router may boot into ROMMON because the boot variable string is incorrect or the syntax is wrong.  And Cisco makes it easy because of inconsistencies:  Some firmware version will allow "boot system bootflash:" while some version is exclusively "boot system flash bootflash:".  Remember, this is not just platform specific but, rather, IOS version specific.  

But in Install Mode, the boot variable string is always pointed to the "packages.conf" file.  

[shambhu.kumar]

Bug CSCvg37458 is not causing due to bundle mode. if ROMMON version is older than 16.9.1r. then only it cause problem.

https://bst.cisco.com/bugsearch/bug/CSCvg37458?rfs=qvred.

flash" & "bootflash" statement logic fixed in ROMMON release 16.9(1r) onwards.

and of course your book needs update also.

[Leo Laohoo]

---

@shambhu.kumar wrote:

Bug CSCvg37458 is not causing due to bundle mode. if ROMMON version is older than 16.9.1r. then only it cause problem.

---

I think you misunderstood CSCvg37458. 

Do not take CSCvg37458 for granted.  It does not discriminate.  It does not matter if I am using Install Mode or Bundle Mode - As long as the boot variable string is incorrect, the router will boot into ROMMON.  

Let me repeat what I said previously:  In Bundle Mode, every firmware update the boot variable string must change (to point to the new firmware).  This is prone to mistakes.  Mistakes that will become CSCvg37458.  

---

@shambhu.kumar wrote:
if ROMMON version is older than 16.9.1r. then only it cause problem.

---

Try it.  Try it several times.  Make the router boot the correct firmware with an incorrect boot variable string.  See what happens.  

I would also like to take this opportunity to conclude the discussion between Install Mode vs Bundle Mode by saying this:  I've pointed out the main difference between the two so people reading this thread can make their own decision(s).