9800-40 17.9.5 update to 17.9.6 rollback after 5 hrs

[David Ritter]

I picked a wonderful Sunday to update my 9800-40 ha stack at 0900 hrs

Update complete and all AP's reattached by 0945.  However the system reverted back to 17.9.5.

C-WLC1#sh ver

Cisco IOS XE Software, Version 17.09.05

Cisco IOS Software [Cupertino], C9800 Software (C9800_IOSXE-K9), Version 17.9.5, RELEASE SOFTWARE (fc1)

C-WLC1 uptime is 18 hours, 15 minutes‌

Uptime for this control processor is 18 hours, 17 minutes‌

System returned to ROM by Image Install Abort  at 19:39:23 PDT Tue Sep 10 2024‌

System restarted at 15:13:44 GMT Sun Dec 22 2024‌

System image file is "bootflash:packages.conf"

Last reload reason: Image Install Abort

the TIME is correct but the time zone was not PST and has since been corrected.

Anyone have any ideas?  I think I will hit up TAC next year!

the system is working on the 17.9.5, 335 ap's and 782 clients.

the whole exercise was related to FN74222 potential loss of config during SSO switch over.

[marce1000]

  - Did you commit the install in the final step of the upgrade procedure ('install commit...., or a related command , please check documentation).

 M.

[David Ritter]

indeed I poked the button..
I executed the Download& Install routine on 12/19.
I waited until 12/21 to do the global AP Image Predownload.
On 12/22
two more steps
I did the install/activate routine
and somewhere was the Save Configuration & Activate.
However I may have missed a gui step.

[marce1000 · Accepted solution]

    - Ref https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/222431-upgrade-the-catalyst-9800-wlc-quick-st.html#toc-hId-24522175
   >...

install commit

The commit make the changes permanent

 M.

[David Ritter]

lol I suspect there were too many buttons.steps.. as in the very last .

so the idea is to be able to test the result and if all is well log back in and commit.

I guess I did not revisit the last time..

[David Ritter]

during this event, all AP's jump'd ship joined the 5520 on 8.10.196.0.  It took 20 mins or so for them to rediscover the 9800, revert back and redownload the 17.9.6 code.   So I was quite happy when all the AP's were back home with a batch of clients that I forgot to revisit the Software Update page and commit to posterity the new code.

[Leo Laohoo]

FN74222 - Full or Partial Cisco 9800 Series Wireless Controller Configuration Loss after High-Availability Stateful Switchover Failover

[Leo Laohoo]

NSFW + One-Hit-Wonder

[Gaurav Kansal]

Hi,

After upgrade if you don't commit your WLC will roll back in some hours. Maybe you missed read the process because it clearly displayed with time remaining for rollback.

Regards
Gaurav Kansal

[liviu.gheorghe]

When upgrading through the GUI, I found it useful to check the "One-Shot Install Upgrade" checkbox in order to not forget committing.

[David Ritter]

OK team, Happy New Year. 

So now that the 9800 has rolled back to 17.9.5 and the previous 17.9.6  .bin file still exists in bootflash, how do I repeat the process without redoing the Download & Install? 

is it as simple as selecting transport type > device, File system > bootflash and pointing at the 17.9.6 file in file path?

[Leo Laohoo]

Post the complete output to the following command: dir flash:

[David Ritter]

here ye are:
Thu Jan 02 2025 23:10:18 GMT-0800 (Pacific Standard Time)
===================================================================================
#dir
Directory of bootflash:/
106497 drwx 4096 Jan 2 2025 21:01:33 -08:00 .dbpersist
67 -rw- 57428 Dec 27 2024 20:30:09 -08:00 DIN.DDO.AP5.5508_2c0be9b9baa0.crash
50 -rw- 30808 Dec 26 2024 22:06:41 -08:00 VIS.TPC.B.2.4538_2c0be9c17680.crash
30 -rw- 33314 Dec 23 2024 18:29:27 -08:00 SCDF.HU1.Y.33F8_501cb0592360.crash
28 -rw- 97224 Dec 22 2024 17:38:35 -08:00 SCDF.HU1.x.04B0_007e95c1fb20.crash
1245185 drwx 4096 Dec 22 2024 15:16:10 -08:00 .installer
811009 drwx 4096 Dec 22 2024 15:15:00 -08:00 certs
19 -rw- 616 Dec 22 2024 15:14:26 -08:00 vlan.dat
1146881 drwx 4096 Dec 22 2024 15:13:53 -08:00 .prst_sync
14 -rw- 137940 Dec 22 2024 15:13:49 -08:00 memleak.tcl
1114113 drwx 4096 Dec 22 2024 15:13:32 -08:00 dc_profile_dir
27 -rw- 90 Dec 22 2024 15:13:13 -08:00 act2_cleanup_2024_12_22_23_13_13.log
29 -rw- 5542 Dec 22 2024 15:09:46 -08:00 packages.conf
368641 drwx 4096 Dec 22 2024 15:08:35 -08:00 .rollback_timer
26 -rw- 90 Dec 22 2024 09:14:12 -08:00 act2_cleanup_2024_12_22_17_14_11.log
1024004 -rw- 37306641 Dec 19 2024 11:59:30 -08:00 C9800-rpboot.17.09.06.SPA.pkg
1024002 -rw- 5542 Dec 19 2024 11:59:30 -08:00 C9800-40-universalk9_wlc.17.09.06.SPA.conf
1024003 -rw- 1413284880 Dec 19 2024 11:58:41 -08:00 C9800-mono-universalk9_wlc.17.09.06.SPA.pkg
18 -rw- 1450333533 Dec 19 2024 11:55:52 -08:00 C9800-40-universalk9_wlc.17.09.06.SPA.bin
66 -rw- 56540 Oct 24 2024 12:41:48 -07:00 SCDF.HU2.2.33F0_501cb0592340.crash
827393 drwx 4096 Sep 12 2024 10:10:02 -07:00 .cloudm
61 -rw- 90 Sep 12 2024 10:06:47 -07:00 act2_cleanup_2024_9_12_17_6_46.log
58 -rw- 90 Sep 10 2024 19:43:57 -07:00 act2_cleanup_2024_9_11_2_43_57.log
65 -rw- 67 Sep 10 2024 19:32:47 -07:00 collated_file_err_log_20240910-193247
64 -rw- 90 Sep 6 2024 17:49:44 -07:00 act2_cleanup_2024_9_7_0_49_43.log
48 -rw- 42331 Jul 17 2024 08:19:55 -07:00 debugTrace_5c71.0d99.df48.txt
59 -rw- 8215 Jul 5 2024 12:17:46 -07:00 debugTrace_007e.95c2.04b0.txt
56 -rw- 133781 Jul 3 2024 22:51:44 -07:00 OE.TCiCT.MSaliakis.5D70_70ea1a6261a0.crash
1007620 -rw- 37292247 Jun 24 2024 21:22:12 -07:00 C9800-rpboot.17.09.05.SPA.pkg
1007619 -rw- 1414837264 Jun 24 2024 21:21:22 -07:00 C9800-mono-universalk9_wlc.17.09.05.SPA.pkg
53 -rw- 88463 Jun 2 2024 01:51:27 -07:00 PVL.AG.01.CED8_2cf89bfd8c60.crash
55 -rw- 51472 Apr 29 2024 16:42:37 -07:00 debugTrace_8a49.3d1b.9cde.txt
54 -rw- 50762 Apr 29 2024 16:11:00 -07:00 debugTrace_7c2a.314c.324a.txt
52 -rw- 7240863 Apr 29 2024 15:36:56 -07:00 debugTrace_cef7.6b43.d6b6.txt
51 -rw- 7097488 Apr 29 2024 14:51:24 -07:00 debugTrace_0ed6.246f.c03e.txt
40 -rw- 245974 Apr 23 2024 15:42:05 -07:00 debugTrace_a4c3.f0c2.8a85.txt
47 -rw- 79788 Apr 4 2024 07:13:14 -07:00 TUL.TDO.ap04.13E0_c4b9cde8b060.crash
46 -rw- 97833 Mar 28 2024 06:49:00 -07:00 OE.TCiCT.DRitter.DF48_5c710dbe00a0.crash
49 -rw- 10367 Feb 21 2024 16:56:54 -08:00 debugTrace_4ca6.4d85.daa0.txt
45 -rw- 51028 Nov 14 2023 10:35:02 -08:00 debugTrace_c4f7.d514.1c08.txt
43 -rw- 124633 Sep 18 2023 20:25:44 -07:00 OE.TCiCT.xx.C8B0_c4f7d5d17620.crash
42 -rw- 798 Aug 31 2023 14:32:55 -07:00 1693517341239sh_tech.txt
516097 dr-x 4096 Aug 3 2023 13:38:44 -07:00 .wl_persistent_db
41 -rw- 116658 Jun 9 2023 12:35:49 -07:00 APA4B4.392B.FC40_cc7f753cf660.crash
34 -rw- 2 May 10 2023 12:34:32 -07:00 debugTrace_172.16.222.34.txt
802817 drwx 4096 Apr 12 2023 11:46:36 -07:00 .PATCH
37 -rwx 0 Apr 9 2023 12:56:56 -07:00 mode_event_log
1179649 drwx 4096 Apr 9 2023 12:56:56 -07:00 SHARED-IOX
794625 drwx 4096 Dec 29 2022 09:00:35 -08:00 .PATCH-backup
32 -rw- 82736 Dec 27 2022 00:45:07 -08:00 AP007E.95C2.04B0_007e95c1fb20.crash
1327105 drwx 4096 Nov 16 2022 08:45:19 -08:00 webui_helper
671745 drwx 4096 Aug 16 2022 10:35:04 -07:00 EFI
12 -rw- 26515 Oct 15 2021 11:23:20 -07:00 debugTrace_00b0.e191.62a0.txt
25 -rw- 1869518 May 14 2021 09:19:44 -07:00 debugTrace_5c5f.6787.6950.txt
24 -rw- 5242880 May 12 2021 11:12:52 -07:00 ssd
20 -rw- 271 May 11 2021 14:19:33 -07:00 btdecode_err
23 -rw- 2 May 11 2021 14:08:17 -07:00 debugTrace_172.22.92.47.txt
22 -rw- 2 May 11 2021 14:01:46 -07:00 debugTrace_172.16.6.60.txt
21 -rw- 146 Jan 16 2021 20:51:00 -08:00 topnmemuse
17 -rw- 107999 May 9 2020 17:20:25 -07:00 pnp-tech-discovery-summary
16 -rw- 35 May 9 2020 17:20:15 -07:00 pnp-tech-time
1515521 drwx 4096 Apr 28 2020 01:54:00 -07:00 onep
15 -rw- 3 Apr 28 2020 01:53:59 -07:00 .wlc_air_lic
13 -rw- 130536449 Apr 27 2020 10:07:11 -07:00 mcpwlc-diag-ids-rel24.4-1904290014.bin
11 drwx 16384 Apr 27 2020 09:58:13 -07:00 lost+found
24811823104 bytes total (19036651520 bytes free)

[marce1000]

 - Examine the content of:  packages.conf

  M.

[Leo Laohoo]

---

@David Ritter wrote:

Dec 22 2024 15:09:46 -08:00 packages.conf
Dec 19 2024 11:59:30 -08:00 C9800-rpboot.17.09.06.SPA.pkg
Dec 19 2024 11:59:30 -08:00 C9800-40-universalk9_wlc.17.09.06.SPA.conf
Dec 19 2024 11:58:41 -08:00 C9800-mono-universalk9_wlc.17.09.06.SPA.pkg
Dec 19 2024 11:55:52 -08:00 C9800-40-universalk9_wlc.17.09.06.SPA.bin​

---

Thanks, @David Ritter. 

Now, can I see the "first page" of the output of the following: 

more bootflash:packages.conf | begin rp_boot

I want to see if the packages are pointing to 17.9.5 or 17.9.6.

If the file "packages.conf" is pointing to 17.9.6 then it is fine to simply reboot the WLC. 

If the file "packages.conf" is still pointing to 17.9.5, the fix is ridiculously simple: 

rename bootflash:packages.conf bootflash:packages.conf.bak
copy bootflash:C9800-mono-universalk9_wlc.17.09.06.SPA.pkg bootflash:packages.conf

And then reboot the controller.