Router freeze for long time not boot - after IOS upgrade

[robert-gorajski]

Hello!

I have a C1111-8P router. Unfortunately I can't update the IOS image. For the time being I have temporarily set gi0/0/1 as a DHCP client. Then I downloaded the IOS image from my TFTP server and calculated the MD5 sum.

Directory of bootflash:/

   11  drwx            16384  Apr 14 2021 08:13:07 +00:00  lost+found
163201  drwx             4096  Jan 16 2025 17:57:21 +00:00  .prst_sync
   12  -rw-        457963732  Apr 14 2021 08:21:21 +00:00  c1100-universalk9_ias_npe.16.09.05.SPA.bin
114241  drwx             4096  Jan 16 2025 17:56:07 +00:00  .installer
89761  drwx             4096  Apr 14 2021 08:27:24 +00:00  core
65281  drwx             4096  Apr 14 2021 08:26:02 +00:00  .rollback_timer
73441  drwx             4096  Apr 14 2021 08:26:03 +00:00  gs_script
106081  drwx            73728  Jan 16 2025 18:41:25 +00:00  tracelogs
122401  drwx             4096  Jul 10 2021 13:27:30 +00:00  .dbpersist
24481  -rw-               30  Jan 16 2025 17:57:21 +00:00  throughput_monitor_params
81601  drwx             4096  Apr 14 2021 08:27:02 +00:00  onep
24482  -rw-               35  Jan 16 2025 12:44:44 +00:00  pnp-tech-time
24483  -rw-            66041  Jan 16 2025 12:44:54 +00:00  pnp-tech-discovery-summary
24484  -rw-            10220  Jan 16 2025 17:57:31 +00:00  vlan.dat
24485  -rw-              567  Jul 11 2021 06:26:35 +00:00  1625984510647sh_tech.txt
24486  -rw-              567  Jul 11 2021 06:27:22 +00:00  1625984554263sh_tech.txt
24487  -rw-        701111540  Jan 16 2025 18:34:27 +00:00  c1100-universalk9.17.12.04a.SPA.bin
130561  drwx             4096  Jan 16 2025 14:50:11 +00:00  SHARED-IOX
24488  -rwx                0  Jan 16 2025 14:50:11 +00:00  mode_event_log
138721  drwx             4096  Jan 16 2025 14:50:11 +00:00  pcap

2908606464 bytes total (1578356736 bytes free)

Configuration:

Router(config)#boot system flash c1100-universalk9.17.12.04a.SPA.bin

Router#show bootvar
BOOT variable = c1100-universalk9_ias_npe.16.09.05.SPA.bin,1;c1100-universalk9.17.12.04a.SPA.bin,1;

Router(config)#no boot system flash c1100-universalk9_ias_npe.16.09.05.SPA.bin


Router#show bootvar
BOOT variable = c1100-universalk9.17.12.04a.SPA.bin,1;
CONFIG_FILE variable does not exist
BOOTLDR variable does not exist
Configuration register is 0x2102

Standby not ready to show bootvar
Router(config)#do wr
Building configuration...

[OK]
Router(config)#
*Jan 16 18:50:31.493: %SYS-2-PRIVCFG_ENCRYPT: Successfully encrypted private config file


Router(config)#reload

Reload:

System Bootstrap, Version 16.9(1r), RELEASE SOFTWARE
Copyright (c) 1994-2018  by cisco Systems, Inc.


Current image running: Boot ROM0

Last reset cause: LocalSoft
C1111-8P platform with 4194304 Kbytes of main memory


........

Located c1100-universalk9.17.12.04a.SPA.bin
####################################################
.
.
.
Package header rev 3 structure detected
IsoSize = 646684672
Calculating SHA-1 hash...Validate package: SHA-1 hash:
        calculated C8FC40FC:3F5B9694:390E1398:ACD90582:1CB4E2E6
        expected   C8FC40FC:3F5B9694:390E1398:ACD90582:1CB4E2E6

RSA Signed RELEASE Image Signature Verification Successful
Image validated

Router freeze for long time, and show error:

System Bootstrap, Version 16.9(1r), RELEASE SOFTWARE
Copyright (c) 1994-2018  by cisco Systems, Inc.

Current image running: Boot ROM0

Last reset cause: Unrecoverable Error
C1111-8P platform with 4194304 Kbytes of main memory

After rollback to universalk9_ias_npe.16.09.05.SPA.bin with rommon. The router starts without any errors.
Can I ask for help in solving this problem?

[Leo Laohoo]

---

@robert-gorajski wrote: 

System Bootstrap, Version 16.9(1r), RELEASE SOFTWARE

---

I suspect this is the problem -- ROMMON version 16.9(1r). 

This is going to be a complicated answer.  

Refer to Cisco ISR1000 ROMmon Compatibility Matrix (Table 3) and it specifically states that the router needs to be on ROMMON version 17.5(1r) if upgrading to 17.5.X 17.6.X or 17.9.X. 

Finally, it is not possible to perform a direct upgrade, from 16.X.X, to 17.12.X (and later) without going through an intermediate version like 17.6.X or 17.9.X.  This information or fact is undocumented.  The reason why is because of ROMMON version 17.6(1r), which is not a downloadable file.  

ROMMON version 17.6(1r) is incorporated into 17.6.X and 17.9.X only.  It is not incorporated in 17.12.X nor 17.15.X and will not be incorporated in any later versions other than 17.6.X and 17.9.X.  And like before, this fact is not documented anywhere else -- Even Cisco TAC was kept in the dark by this information.  

To upgrade to Cisco IOS XE Dublin 17.12.x, follow these steps:

1. If you are on a device that is running software version between Cisco IOS XE 16.x to Cisco IOS XE 17.4.x, upgrade to any IOS XE image between Cisco IOS XE 17.5.x to Cisco IOS XE 17.10.x.

2. After performing step a, upgrade to Cisco IOS XE 17.12.x.

3. For devices that are running on software version Cisco IOS XE 17.5.x or later, you can upgrade to Cisco IOS XE 17.12.x directly.

---

So here is the plan: 

1.  Force the router to boot 16.9.5.  

2.  Read this:  Cisco ISR & ASR 1k Routers: IOS-XE/Firmware Upgrade (Install Mode)

3.  Upgrade the ROMMON to version 17.5(1r) but DO NOT REBOOT the router.

4.  Upgrade to router firmware to either 17.6.X or 17.9.X.  Reboot

5.  Upgrade to router firmware to 17.12.X & reboot. 

[liviu.gheorghe]

Is there a particular reason for using IOS 17.12.4a? If there is, I would suggest upgrading first the ROMMON version to 17.5(1r) as per Cisco recommendation: https://www.cisco.com/c/en/us/td/docs/routers/access/isr1100/software/configuration/xe-17/isr1100-sw-config-xe-17/Configuring_ROMMON.html

and then try again with IOS 17.12.4a

If there isn't a good reason for IOS 17.12.4a, can you try booting the router with IOS 17.9.5e? I'm using some C1111 with this image and don't have any issues.

HTH