topic Re: WLC 9800CL Upgrade and Image Pre-Download in Wireless

WLC 9800CL Upgrade and Image Pre-Download

jjtech — Mon, 01 Sep 2025 10:59:19 GMT

Hello,

We have a Cisco 9800-CL (as VMs on VSphere, large deployment) configured in HA - SSO with around 980 access points registered.

We want to upgrade it from version 17.3.5b to 17.12.5.

I was checking the pre-download option for the image to minimize downtime, but I found that the image can be pre-downloaded to a max of 100 APs per WCND (our WLC supports 8 WCNDs). However most APs (around 750 APs) use the same site tag which seems to be mapped to the same WCND.

How can this be feasible, what do you recommend for pushing the image using the pre-download option to this large number of APs, especially that most of them use the same site tag?

Thanks,

Re: WLC 9800CL Upgrade and Image Pre-Download

MHM Cisco World — Mon, 01 Sep 2025 11:34:43 GMT

It need some work'

Use wildcard in AP name as example below

ap image predownload primary <image> ap-name Lobby-*

This make WLC send image to these AP not all AP.

MHM

Re: WLC 9800CL Upgrade and Image Pre-Download

balaji.bandi — Mon, 01 Sep 2025 11:41:45 GMT

check any limitation on that version :

Restrictions and best practices for pre-downloading an image to an AP

https://www.cisco.com/c/en/us/td/docs/wireless/controller/9800/17-12/config-guide/b_wl_17_12_cg/m_predwnld_image_ap_ewlc.html

Re: WLC 9800CL Upgrade and Image Pre-Download

Leo Laohoo — Mon, 01 Sep 2025 12:06:44 GMT

What are the AP models?

I've got a better (and proven!) method of "pre-download" and I can do the whole lot in one sweep!

IMPORTANT:

If the APs affected by CSCwe97901, CSCwm08044 &/or CSCwm72142 then the topic of "pre-download" is moot and void because any one of the three bugs will prevent the APs from storing or writing the pre-downloaded firmware into the APs flash since it is affected by the bloatware.

Re: WLC 9800CL Upgrade and Image Pre-Download

jjtech — Mon, 01 Sep 2025 12:19:40 GMT

Hi Leo,

The AP models in use are :

1815I
1832I
1852I
2802I
C9115AXE
C9115AXI

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Mon, 01 Sep 2025 13:48:03 GMT

1. WNCd balancing is already covered in the Best Practices guide (link below) and configuration guides so this is something you need to address regardless because it can cause a lot of problems in normal operation. Note the automatic balancing algorithms have been substantially improved in the latest releases but you still need to pay close attention to your config because inter-WNCd roaming still has restrictions - so as much as possible your APs should be grouped on the same WNCd where clients are expected to roam between the APs.

2. The 100 limit is simultaneous downloads - it just means they are processed in batches. As long as you start the downloads in good time they should all complete in due course. 1st 100, then next 100, then next 100 etc until all 980 complete. Once you're on 17.12.5 you can look at using Out of Band AP Image Download - download over https - much faster and more scalable and reliable (first introduced in 17.11.1) Feature History for Out-of-Band AP Image Download

3. As Leo says you may find you need to reload all the APs before starting pre-downloads to ensure the image is able to be installed otherwise it will just fail constantly because the /tmp partition doesn't have enough free space (reload clears it).

4. Pay close attention to the following field notice and technotes:
Field Notice: FN74109 - Access Point Image Corruption During CAPWAP Upgrade May Result in Boot Failure - Software Upgrade Recommended
Safely Upgrade Access Points, Avoiding Image Corruption That Causes Boot Loop
Recover from a Boot Loop Caused by Image Corruption on Wave 2 and 11ax Access Points (CSCvx32806)

Re: WLC 9800CL Upgrade and Image Pre-Download

eglinsky2012 — Mon, 01 Sep 2025 13:50:50 GMT

You can still do the pre-download per standard procedure, it will just take longer since it will cycle through all the APs starting with an initial 100, then once the first one finishes another one starts, etc. Depending on the throughput available, if there’s a WAN link involved etc., the time will be variable, but it should finish within 30-60 minutes or so and not be service impacting*.

* I’ve come across an issue, I forget if it started with 17 9.X or with 17.12.4, but when pre-downloading an AP service pack in the UI, it causes the APs to reboot. There’s a glitch. If you plan to do that, I can dig up the discussion on this community where we addressed that with a CLI command instead. Or just use the CLI procedure. 🙂

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Mon, 01 Sep 2025 14:26:34 GMT

As I recall the problem with APSP is that there was no option to pre-download on the GUI so the only option was to install and activate which installs and reloads the APs immediately.
From CLI you can install the APSP file, pre-download to APs and then when you are ready activate which then reloads all the APs.

The other problem with APSP is that you cannot install the new main version (17.12.5) and the APSP at the same time so you have to install the main version which pre-downloads and reloads all the APs and then after that is complete you have to install the APSP which then downloads and reloads all the APs again. The only way to avoid this is if you have N+1 WLCs:
- Manually pre-download APSP image to APs using TFTP (explained how to do this on other posts but essentially you can unzip the APSP image to get the AP images) so the APs all have the APSP image installed in backup partition.
- Take one WLC offline so all APs are on other WLC still running old version.
- Upgrade offline WLC then install SMUs and APSP.
- Bring upgraded WLC back online and then take the other WLC offline so all APs move to upgraded WLC where they will now be forced to reboot onto new APSP image to match the WLC. Note I have seen some of them re-download anyway when they shouldn't but hey-ho.
- Upgrade the offline WLC and install SMUs and APSP and then bring it back online.
- This ensures the APs should only have to be reloaded once (provided they don't re-download anyway)

@jjtech you say you're upgrading to 17.12.5 ... I hope you had not forgotten that you will also need to install the SMUs and APSP (current latest is APSP6).

Re: WLC 9800CL Upgrade and Image Pre-Download

Saikat Nandy — Mon, 01 Sep 2025 17:45:03 GMT

let me chime in... whatever @Rich R & @Leo Laohoo said, please pay attention to that. On top of that, couple of things -
1. If you are using 9800-CL, large template, having 7 WNCDs (please validate once, I don't have one immediately to confirm but as far as I recall it should be 7 not 8, 8 is only for 9880) and 750 APs are sitting in one WNCD - that's a problem. You should distribute your APs across different site tags so that you can utilise all the WNCDs. You should keep the total no of site tags to 7 or multiple of 7. Even if you argue that all the APs belong to the same roaming domain, you still should be able to figure out some ways (hallways or floors or buildings) to distribute the APs across site tags/WNCDs.
2. Assuming you are running flex as most of the customers using CL runs Flex. If my assumption is correct, then yes there is a strong possibility that you might hit the corrupted image/bootloop issue if the PMTU between WLC & AP is flapping too much.

So here is what I should have probably done for the upgrade - first upgrade should be from CLI

# Copy the image
# install
# initiate pre-download on site tag basis. (you should do this activity at least a week in advance)
# during MW, just activate and commit.

Most importantly you should keep a syslog server ready as that's the only way to figure out if the AP has downloaded a good or bad image. You will get few days to validate how many APs have downloaded image, how many failed and if you need to take care few APs susceptible for boot loop.

# If you see a couple of APs have ended up downloading corrupted image then you can use archive command on those specific APs.
# If you end up seeing the number is high then WLAN Poller is the option for you.

if you have option, reloading the APs before upgrade could be a good option (because if your APs are having higher uptime the space issue might happen)

Re: WLC 9800CL Upgrade and Image Pre-Download

MHM Cisco World — Mon, 01 Sep 2025 17:50:15 GMT

Sorry but he so clear

All AP use same tag and he need to upgrade group by group

You shift him to WNCD load !!!!

I dont know why.

MHM

Re: WLC 9800CL Upgrade and Image Pre-Download

Leo Laohoo — Mon, 01 Sep 2025 22:06:09 GMT

@jjtech wrote:
The AP models in use are :

1815I

1832I

1852I

2802I

C9115AXE

C9115AXI

Here is a way to "pre-download" in one sweep:

1. Download the 18xx-series AP firmware for 17.12.5 from HERE.

2. Download the 28xx-series AP firmware for 17.12.5 from HERE.

3. Download the 9115-series AP firmware for 17.12.5 from HERE.

4. Put all the TAR files into a TFTP server.

5. Generate a list of APs

show ap cdp neighbors

6. Reboot the APs first because I have a strong suspicion they are affected by CSCwe97901, CSCwm08044 &/or CSCwm72142. The command to do this (from the controller) is:

ap name <NAME> reset

7. Once all the APs have returned, enter the following 9800 commands to initiate the "pre-download" without going through the 9800's WNCD:

ap name <NAME> remote enable ap name <NAME> command "archive download-sw /no-reboot <tftp|sftp|http>://<server_ip>/filename.tar"

8. Give each AP approximately 2 minutes before rebooting them.

ap name <NAME> reset

9. Reboot the controller to 17.12.5.

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Mon, 01 Sep 2025 22:11:27 GMT

You are confusing 2 different things @MHM Cisco World

He never said "he need to upgrade group by group" - he was concerned that only 100 APs could pre-download not realising that that simply means 100 at a time - they will all pre-download eventually, just never more than 100 at the same time.

The issue of WNCd is a separate matter - it is best practice, and absolutely essential to avoid WNCd CPU overload, to ensure that the APs are properly balanced across the WNCd's. This is one area where the 9800 series actually perform significantly worse than an 8540 because of the architecture so spreading the load is an essential partial mitigation for the problems that causes.

Agreed with @Saikat Nandy - split the tags by floor, building, department - whatever works best for @jjtech - in order to achieve good balancing. This needs to be done regardless of anything else so is very important to highlight.

Re: WLC 9800CL Upgrade and Image Pre-Download

MHM Cisco World — Mon, 01 Sep 2025 22:16:14 GMT

Believe me I dont have time

""How can this be feasible, what do you recommend for pushing the image using the pre-download option to this large number of APs, especially that most of them use the same site tag?""

He ask clear Q' he have more than 100 AP and need to do predownload.

Anyway.

I stop answer this post unless he need more info or clear issue.

MHM

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Mon, 01 Sep 2025 22:22:13 GMT

It just takes longer but like @Saikat Nandy suggested you can start that a week before the planned upgrade - leaving plenty of time to deal with any which fail. Normally they will all complete in a few hours (and many of ours are over ADSL lines too) - no big deal. We usually start about 1-2 days before the upgrade.

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Mon, 01 Sep 2025 22:41:23 GMT

But that only gets you to 17.12.5 and then you still need the APSP installed which will mean repeating the process all over again but with pre-download ... OR ...

Download C9800-CL-universalk9.17.12.05.CSCwq58794.SPA.apsp.bin from https://software.cisco.com/download/home/286322605/type/286325254/release/17.12.5

Open the file with something like 7-Zip (works for me) - open as archive
Then extract C9800-CL-mono-patch.17.12.05.CSCwq58794.SPA.apsp.bin from the image.
Then open C9800-CL-mono-patch.17.12.05.CSCwq58794.SPA.apsp.bin with 7-Zip
Navigate down to mnt/images/ap_images/ and there you will find the APSP6 AP images and the md5 hash for each file and you can just extract them all from there The files are just named ap1g4 etc (no file extension) but they are tar files the same as the downloadable images so you can rename as ap1g4-17.12.5.206.tar for example.
If you're not sure which image applies to which AP then use "show ap image file summary" - that shows you the mapping between image names and AP models. 17.12.5.206 is the AP image version number for APSP6.

Now as I said in my previous reply - this is only useful if you're able to upgrade the WLC directly to 17.12.5 + APSP6.

Re: WLC 9800CL Upgrade and Image Pre-Download

jjtech — Tue, 02 Sep 2025 12:24:17 GMT

Thanks for the detailed response Leo. You're suggesting to set the commands in a script and send the commands periodically (every 5 seconds for example)?

And in this case which image will be the backup one, the old backup image or the old primary image?

Re: WLC 9800CL Upgrade and Image Pre-Download

jjtech — Tue, 02 Sep 2025 12:25:25 GMT

Hi Rich,

Thank you for the reply. I initially didn't want to re-change all tags, but if it is the better/safer option I may go the route of re-creating new tags to re-balance the APs distribution.

But in all cases do I still need to download APSP and SMU packages ?

According to a Cisco document, I found:

"Cisco IOS XE 17.12.5 is the fourth bug-fix release in the 17.12 train. It incorporates the fixes in SMUs and APSPs for 17.12.4 and many more bug fixes."

Re: WLC 9800CL Upgrade and Image Pre-Download

MHM Cisco World — Tue, 02 Sep 2025 12:40:02 GMT

That so important point

With predownload the new image will add as backup image

It override old backup image in AP

To make sure select one AP

Check primary and backup image before and after predownload

MHM

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Tue, 02 Sep 2025 15:05:00 GMT

Image install, whether by pre-download or via archive command on CLI, always replaces the backup image.

You can switch the 2 images between primary and backup as needed and the AP does that automatically when you upgrade.
ap name <ap-name> image swap to swap images on a single AP
ap image swap to swap images on all APs
ap image swap completed to swap images on all APs with pre-download completed
You can see the effect of that with "show ap image".

Note that when you install a new image via CLI the AP does not automatically update the WLC which will still show the old backup image. To get that info refreshed you need to do a CAPWAP reset on the AP to force a re-join at which point the WLC will learn about the updated backup image and show it in the "show ap image" output.

Re: WLC 9800CL Upgrade and Image Pre-Download

Rich R — Tue, 02 Sep 2025 15:33:30 GMT

> But in all cases do I still need to download APSP and SMU packages ?
Yes

> Cisco IOS XE 17.12.5 is the fourth bug-fix release in the 17.12 train. It incorporates the fixes in SMUs and APSPs for 17.12.4 and many more bug fixes
The 17.12.5 SMUs and APSP are for bug fixes that have come after 17.12.5 was released.
SMUs fix IOS-XE bugs in the controller code.
APSP fix bugs in the AP-COS code running on the APs.
You ignored the lines that come after that line:
https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#toc-hId-1590314578

SMU CSCwo62157 :Reload SMU, Memory leak observed in tdl_mac_addr object

17.12.5 APSP6 includes fixes listed in release notes

So TAC are specifically recommending those must be applied with 17.12.5 which means they have seen a significant number of customers being affected by those bugs on 17.12.5 <wink>