9800-40 HA SSO - Software Update

[Marco111]

Hi, I currently have 2x 9800s in HA SSO and want to update from 17.6.4 to 17.9.4a, I cant seem to locate a guide for the process when setup in SSO. This WLC is in DNA Center but I think I'll prefer to not use DNAC or ISSU and do this upgrade manually via CLI. 

Its currently in bundle mode which I want to change to Install mode too, so I was going to do the usual prechecks and 'install remove inactive' etc, then copy new image to flash then change to Install mode by amending the boot to packages.conf

9800-40(config)#boot system bootflash:packages.conf

Is the above required to be changed on the standby too?

After above i'll then start upgrade via the usual 

9800-40#install add file bootflash:<image.bin> activate commit

Can anyone confirm if this is correct? or point me in direction of an upgrade guide. Detailing changing from bundle to install mode and the software update process via Cli on HA SSO setup. Many thanks.

[marce1000]

                     >...Is the above required to be changed on the standby too?
  - In my opinion it's a bit the reverse in the sense that you need to break ha-sso first , then change standby to Install Mode and rejoin ; to get on track for the rest of the tasks , 
                       (because otherwise HA-SSO will fall apart , it always assumes complete symmetry)

 M.

[eglinsky2012]

Regarding the boot variable question, that is synced across both units. See syntax and verification below. Notice the "no boot system" prior to resetting the boot variable, or else it appends what you're trying to add after the existing variable rather than replacing it. And the "show boot" command shows the variables on both active and standby for verification.

I can't remember if I went from bundle to install mode on 9800 WLCs, but on IOS-XE switch stacks and chassis with redundant supervisors running in SSO, I've done it the way you outline without issue.

All this said, what Marce advises may be the safest way to do it, especially if this WLC is in production and you have no backup WLC to move APs to during the maintenance.

9800-Lab#show boot
BOOT variable = bootflash:C9800-80-universalk9_wlc.17.09.04a.SPA.bin,12;
CONFIG_FILE variable =
BOOTLDR variable does not exist
Configuration register is 0x102

Standby BOOT variable = bootflash:C9800-80-universalk9_wlc.17.09.04a.SPA.bin,12;
Standby CONFIG_FILE variable =
Standby BOOTLDR variable does not exist
Standby Configuration register is 0x102
9800-Lab#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
9800-Lab(config)#no boot system
9800-Lab(config)#boot system bootflash:packages.conf
9800-Lab(config)#end
9800-Lab#wr
Building configuration...
[OK]
9800-Lab#show boot
BOOT variable = bootflash:packages.conf,12;
CONFIG_FILE variable =
BOOTLDR variable does not exist
Configuration register is 0x102

Standby BOOT variable = bootflash:packages.conf,12;
Standby CONFIG_FILE variable =
Standby BOOTLDR variable does not exist
Standby Configuration register is 0x102
9800-Lab#

For further verification that packages.conf is present and up to date on both units and the .pkg files it points to are actually in each unit's bootflash:

9800-Lab#more packages.conf
[...]
boot   rp 0 0   rp_boot C9800-rpboot.17.09.04a.SPA.pkg

iso   rp 0 0   rp_base C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_base C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_daemons C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_daemons C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_iosd C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_iosd C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_security C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_security C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_webui C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_webui C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_wlc C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_wlc C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   fp 0 0   fp C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   cc 0 0   elc C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   cc 0 0   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   cc 0 1   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   cc 0 2   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   cc 0 3   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg


[...]
# -end- superpackage .pkginfo
#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#dir bootflash:
Directory of bootflash:/

507905  drwx             4096  May 24 2024 13:22:22 -04:00  .dbpersist
20      -rw-              676  May 20 2024 15:36:22 -04:00  vlan.dat
37      -rw-             3194  May 20 2024 15:35:25 -04:00  debugTrace_d6f4.4688.8047.txt
35      -rw-              173  May 20 2024 15:34:26 -04:00  debugTrace_d6f4.4688.8d47.txt
499713  drwx             4096  May 15 2024 10:20:01 -04:00  .installer
278529  drwx             4096  May 15 2024 10:16:54 -04:00  certs
13      -rw-           137940  May 15 2024 10:16:11 -04:00  memleak.tcl
1064961  drwx             4096  May 15 2024 10:15:55 -04:00  dc_profile_dir
1449985  drwx             4096  Jan 16 2024 11:30:43 -05:00  .prst_sync
253953  drwx             4096  Dec 22 2023 13:19:13 -05:00  .PATCH
1490945  drwx             4096  Dec 22 2023 13:19:13 -05:00  .rollback_timer
1335297  drwx             4096  Dec 22 2023 13:12:01 -05:00  .PATCH-backup
34      -rw-        503924124  Dec 22 2023 13:04:42 -05:00  C9800-universalk9_wlc.17.09.04a.CSCwi44524.SPA.apsp.bin
27      -rw-             9652  Dec 22 2023 12:57:46 -05:00  C9800-universalk9_wlc.17.09.04a.CSCwh68219.SPA.smu.bin
36      -rw-             5560  Dec 22 2023 12:44:08 -05:00  packages.conf
409604  -rw-         35918196  Dec 22 2023 12:19:59 -05:00  C9800-rpboot.17.09.04a.SPA.pkg
409602  -rw-             5560  Dec 22 2023 12:19:59 -05:00  C9800-80-universalk9_wlc.17.09.04a.SPA.conf
409603  -rw-       1411884048  Dec 22 2023 12:19:18 -05:00  C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
33      -rw-       1447544256  Dec 22 2023 12:16:17 -05:00  C9800-80-universalk9_wlc.17.09.04a.SPA.bin
1015809  drwx             4096  Dec 15 2023 08:42:18 -05:00  debugBundleCollector
32      -rw-         10072064  Dec 14 2023 15:19:55 -05:00  haIntCaptureLo-standby-Dec14.pcap
24      -rw-            40027  Dec 14 2023 15:12:36 -05:00  stack_mgr_logs.txt
23      -rw-             3659  Dec 14 2023 15:12:03 -05:00  rif_mgr_logs.txt
25      -rw-          9973760  Dec 14 2023 15:10:29 -05:00  haIntCaptureLo-active-Dec14.pcap
31      -rw-            58808  Nov 29 2023 11:23:31 -05:00  C9800-universalk9_wlc.17.09.04.CSCwh87343.SPA.smu.bin
30      -rw-        503895452  Nov 29 2023 10:59:03 -05:00  C9800-universalk9_wlc.17.09.04.CSCwh93727.SPA.apsp.bin
1187841  drwx             4096  Oct 16 2023 12:06:47 -04:00  EFI
29      -rw-         15270932  Oct 16 2023 12:04:20 -04:00  C9800-80-rommon.173-3r.pkg
28      -rw-          4390912  Oct 16 2023 11:50:18 -04:00  haIntCaptureLo-standby-Oct10.pcap
26      -rw-         10985472  Sep 19 2023 15:42:54 -04:00  haIntCaptureLo-standby.pcap
1359873  drwx             4096  Aug 10 2023 15:25:02 -04:00  tracelogs
1089540  -rw-         35923130  Aug 10 2023 14:50:37 -04:00  C9800-rpboot.17.09.04.SPA.pkg
1089538  -rw-             5542  Aug 10 2023 14:50:37 -04:00  C9800-80-universalk9_wlc.17.09.04.SPA.conf
1089539  -rw-       1411884048  Aug 10 2023 14:50:02 -04:00  C9800-mono-universalk9_wlc.17.09.04.SPA.pkg
22      -rw-       1447549190  Aug 10 2023 14:47:03 -04:00  C9800-80-universalk9_wlc.17.09.04.SPA.bin
21      -rw-            30534   Aug 9 2023 18:31:20 -04:00  Lab-1815-9800_cc7f75cf6a80.crash
1482754  -rw-             5541   Apr 6 2023 11:07:44 -04:00  C9800-80-universalk9_wlc.17.09.03.SPA.1.conf
1482756  -rw-         35882684   Apr 6 2023 11:07:44 -04:00  C9800-rpboot.17.09.03.SPA.pkg
1482755  -rw-       1461183504   Apr 6 2023 11:07:09 -04:00  C9800-mono-universalk9_wlc.17.09.03.SPA.pkg
12      -rw-       1496808200   Apr 6 2023 11:03:57 -04:00  C9800-80-universalk9_wlc.17.09.03.SPA.bin
286721  dr-x             4096  Mar 22 2023 18:55:53 -04:00  .wl_persistent_db
1220609  drwx             4096  Mar 22 2023 18:00:11 -04:00  webui_helper
17      -rwx                0  Mar 22 2023 17:54:01 -04:00  mode_event_log
1376257  drwx             4096  Mar 22 2023 17:54:01 -04:00  SHARED-IOX
524290  -rw-             5541  Mar 22 2023 17:27:22 -04:00  C9800-80-universalk9_wlc.17.09.03.SPA.conf
19      -rw-          5242880   Jan 3 2023 16:25:34 -05:00  ssd
737284  -rw-         35935963   Jan 3 2023 16:07:46 -05:00  C9800-rpboot.17.06.03.SPA.pkg
737283  -rw-       1199514640   Jan 3 2023 16:07:20 -05:00  C9800-mono-universalk9_wlc.17.06.03.SPA.pkg
18      -rw-            17286   Jan 3 2023 16:01:25 -05:00  9800-Pair1.cfg
16      -rw-           121907   Jan 3 2023 15:30:17 -05:00  pnp-tech-discovery-summary
15      -rw-               34   Jan 3 2023 15:30:12 -05:00  pnp-tech-time
155649  drwx             4096  May 16 2020 06:44:27 -04:00  onep
14      -rw-                3  May 16 2020 06:44:25 -04:00  .wlc_air_lic
11      drwx            16384  May 16 2020 06:31:36 -04:00  lost+found

24811823104 bytes total (12439400448 bytes free)
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#
9800-Lab#more stby-bootflash:packages.conf
[...]
boot   rp 0 0   rp_boot C9800-rpboot.17.09.04a.SPA.pkg

iso   rp 0 0   rp_base C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_base C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_daemons C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_daemons C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_iosd C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_iosd C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_security C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_security C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_webui C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_webui C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   rp 0 0   rp_wlc C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   rp 0 1   rp_wlc C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   fp 0 0   fp C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   cc 0 0   elc C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

iso   cc 0 0   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   cc 0 1   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   cc 0 2   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
iso   cc 0 3   elc_spa C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg

[...]
# -end- superpackage .pkginfo
#


9800-Lab#dir stby-bootflash:
Directory of stby-bootflash:/

843777  drwx             4096  May 24 2024 13:22:28 -04:00  .dbpersist
20      -rw-              676  May 20 2024 15:36:22 -04:00  vlan.dat
49153   drwx             4096  May 15 2024 10:22:34 -04:00  .installer
1007617  drwx             4096  May 15 2024 10:19:30 -04:00  certs
13      -rw-           137940  May 15 2024 10:16:16 -04:00  memleak.tcl
131073  drwx             4096  May 15 2024 10:16:00 -04:00  dc_profile_dir
1417217  drwx             4096  Jan 17 2024 14:24:36 -05:00  .prst_sync
581633  drwx             4096  Dec 22 2023 13:19:14 -05:00  .rollback_timer
794625  drwx             4096  Dec 22 2023 13:19:14 -05:00  .PATCH
1335297  drwx             4096  Dec 22 2023 13:12:13 -05:00  .PATCH-backup
29      -rw-        503924124  Dec 22 2023 13:04:42 -05:00  C9800-universalk9_wlc.17.09.04a.CSCwi44524.SPA.apsp.bin
26      -rw-             9652  Dec 22 2023 12:57:46 -05:00  C9800-universalk9_wlc.17.09.04a.CSCwh68219.SPA.smu.bin
31      -rw-             5560  Dec 22 2023 12:44:10 -05:00  packages.conf
802820  -rw-         35918196  Dec 22 2023 12:20:52 -05:00  C9800-rpboot.17.09.04a.SPA.pkg
802818  -rw-             5560  Dec 22 2023 12:20:52 -05:00  C9800-80-universalk9_wlc.17.09.04a.SPA.conf
802819  -rw-       1411884048  Dec 22 2023 12:20:04 -05:00  C9800-mono-universalk9_wlc.17.09.04a.SPA.pkg
28      -rw-       1447544256  Dec 22 2023 12:16:17 -05:00  C9800-80-universalk9_wlc.17.09.04a.SPA.bin
23      -rw-         10072064  Dec 14 2023 15:10:46 -05:00  haIntCaptureLo-standby-Dec14.pcap
27      -rw-            58808  Nov 29 2023 11:23:31 -05:00  C9800-universalk9_wlc.17.09.04.CSCwh87343.SPA.smu.bin
25      -rw-        503895452  Nov 29 2023 10:59:03 -05:00  C9800-universalk9_wlc.17.09.04.CSCwh93727.SPA.apsp.bin
286721  drwx             4096  Oct 16 2023 12:15:59 -04:00  EFI
24      -rw-         15270932  Oct 16 2023 12:14:50 -04:00  C9800-80-rommon.173-3r.pkg
22      -rw-         10985472  Sep 19 2023 13:18:34 -04:00  haIntCaptureLo-standby.pcap
196612  -rw-         35923130  Aug 10 2023 14:51:02 -04:00  C9800-rpboot.17.09.04.SPA.pkg
196610  -rw-             5542  Aug 10 2023 14:51:02 -04:00  C9800-80-universalk9_wlc.17.09.04.SPA.conf
196611  -rw-       1411884048  Aug 10 2023 14:50:25 -04:00  C9800-mono-universalk9_wlc.17.09.04.SPA.pkg
12      -rw-       1447549190  Aug 10 2023 14:47:03 -04:00  C9800-80-universalk9_wlc.17.09.04.SPA.bin
917506  -rw-             5541   Apr 6 2023 11:07:46 -04:00  C9800-80-universalk9_wlc.17.09.03.SPA.1.conf
917508  -rw-         35882684   Apr 6 2023 11:07:46 -04:00  C9800-rpboot.17.09.03.SPA.pkg
917507  -rw-       1461183504   Apr 6 2023 11:07:10 -04:00  C9800-mono-universalk9_wlc.17.09.03.SPA.pkg
21      -rw-       1496808200   Apr 6 2023 11:03:57 -04:00  C9800-80-universalk9_wlc.17.09.03.SPA.bin
548865  dr-x             4096  Mar 22 2023 18:55:53 -04:00  .wl_persistent_db
18      -rwx                0  Mar 22 2023 17:52:27 -04:00  mode_event_log
188417  drwx             4096  Mar 22 2023 17:52:27 -04:00  SHARED-IOX
811010  -rw-             5541  Mar 22 2023 17:27:23 -04:00  C9800-80-universalk9_wlc.17.09.03.SPA.conf
19      -rw-          5242880   Jan 3 2023 16:24:13 -05:00  ssd
925700  -rw-         35935963   Jan 3 2023 16:02:19 -05:00  C9800-rpboot.17.06.03.SPA.pkg
925699  -rw-       1199514640   Jan 3 2023 16:01:53 -05:00  C9800-mono-universalk9_wlc.17.06.03.SPA.pkg
17      -rw-            17286   Jan 3 2023 16:00:07 -05:00  9800-Pair1.cfg
16      -rw-           122169   Jan 3 2023 15:29:13 -05:00  pnp-tech-discovery-summary
15      -rw-               34   Jan 3 2023 15:29:08 -05:00  pnp-tech-time
458753  drwx             4096  May 16 2020 06:41:17 -04:00  onep
14      -rw-                3  May 16 2020 06:41:16 -04:00  .wlc_air_lic
11      drwx            16384  May 16 2020 06:28:23 -04:00  lost+found

24811823104 bytes total (12467982336 bytes free)
9800-Lab#

[Leo Laohoo]

What I want to know is this:  Do you want to reboot the WLC pair ASAP or at a later date? 

If you want to reboot the WLC pair at a later date, including unpacking the packages NOW, but rebooting the WLC pair at a later date, then read my NSFW+One-Hit-Wonder upgrade.  

[Marco111]

@Leo Laohoo  - thanks, the guide will definitely come in handy, in this instance I have arranged WLC downtime over a weekend so will just be letting it reboot when asked after 'install add bootflash:xxx.bin active commit' command.

Rich R mentions the install script during the process should set the boot variable automatically, but in your guide it asks this is checked first to ensure pointing to packages.conf, in my case its in bundle mode so currently pointing at the .bin, is it best to change this manually prior to running the install command to ensure everything is correct?

Lastly theres approx 60x 9120ax WAPs at this site, once WLC comes back online are all WAPs immediately upgraded? and roughly how long does that process take? Any precautions to take for this as i could imagine 60x WAPs all wanting to upgrade at same time could cause the controller to grind to a halt. Thanks again for your help, its the second time this week :)..

[Leo Laohoo]

---

@Marco111 wrote:
Rich R mentions the install script during the process should set the boot variable automatically

---

@Rich R is correct.  Technically, it should.  Technically.  However, there is a very well-known bug where the "packages.conf" gets locked up and does not get amended to point to the new version.  This is why, after Step 2, there is a warning to not touch the keyboard and wait for the "y/n" question to appear after about three minutes.  

Cisco's recommended approach was to enter "y" and pray that the bug does not materialize.  My method goes the opposite direction and gives the user the control and manually set (and confirm) the packages.conf is pointing to the new version. 

---

@Marco111 wrote:
but in your guide it asks this is checked first to ensure pointing to packages.conf

---

In my process, checking the boot variable string is Step 9 (out of 11).  

---

@Marco111 wrote:
Lastly theres approx 60x 9120ax WAPs at this site, once WLC comes back online are all WAPs immediately upgraded? and roughly how long does that process take? 

---

That depends on the speed of the WAN link between the site and the WLC.  Manually pushing the firmware alone takes about 60 seconds and an additional 35 seconds to "write" the firmware into the flash of the AP.  However, if the controller is doing the firmware upgrade, it will take around 4 minutes per AP.  

NOTE:  
1.  There is a feature in AireOS that enables manually pushing firmware up to each APs in scale.  Sadly, this feature is disabled in IOS-XE.  I have asked numerous people in WNBU but they were not interested.  

2.  The NSFW + One-Hit-Wonder has been shared to many Cisco staff, including those from WNBU, but no one was interested.

---

@Marco111 wrote:
Any precautions to take for this as i could imagine 60x WAPs all wanting to upgrade at same time could cause the controller to grind to a halt.

---

60 APs joined to a 9800-40 is nothing.  There are only 4 WNCd in a 9800-40 and 60 APs downloading firmware is not going to stress a single one.  However, if there were >2800 APs going to a 9800-80, that is a different story.

[Rich R]

> 1.  There is a feature in AireOS that enables manually pushing firmware up to each APs in scale.  Sadly, this feature is disabled in IOS-XE.  I have asked numerous people in WNBU but they were not interested.  

@Leo Laohoo presume you're aware of the "ap name <apname> tftp-downgrade" command which actually makes it very easy on IOS-XE? (easier than having to use debug commands on AireOS).  It works really well and I've used it a number of times.

[Leo Laohoo]

Thanks, @Rich R.  I will give that a try.

[Rich R]

If the install script works correctly (and it should work in 17.6, earlier versions often didn't) then it will create the packages.conf (copied from the version specific .conf file) and set the boot variable automatically - you should not have to edit that manually at all.