Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 · ‎04-09-2021

Hello,

We are planning to upgrade our 2x Central WLCs HA SSO from 8.2.130 to 8.2.170 as a 1st step and in a second step from 8.2.170 to 8.5.171.

We have:

3x AIR-CAP2602E-E-K9
21x AIR-CAP3602I/E-E-K9
140x AIR-CAP3702I/E-E-K9
148x AIR-AP3802I/E-E-K9

I'm planning to predownload the firmware to the APs, but I've read few articles saying the 3702 model could have to download twice the image.

We have a critical site 24/7 that have 33x3702 APs and only 10Mbps WAN access (to the remote WLCs), do you think it's more reasonnable to wait their WAN bandwidth upgrade to 100Mbps, because if the 33 APs have to download the image (~260mo), at the same time it could take very long time !

How about 2602, 3602 and 3802 models ?

Leo Laohoo · ‎04-09-2021

2700/3700 will download the same firmware TWICE.

If left alone, the AP will first download the firmware with a prefix of "c3700", reboot, join the controller and then download the second firmware with the prefix "ap3g2", reboot and then join the controller. The entire process will take around 14 minutes (per AP).

The only way to force the AP to boot the 2nd firmware is to force the AP to manually pre-load this firmware. This method will take five to six minutes (per AP).

This is how to do it:

1. Download the firmware for the AP here. Make sure to download the firmware that has the prefix of "ap3g2-k9w8-tar".

2. Upload the WLC firmware but DO NOT reboot the WLC. Do not invoke the "pre-download" for AP3700 (OK to pre-download the rest).

3. Instruct the WLC to "force" the AP to erase the old firmware and load the new firmware:

debug ap enable <AP NAME>
debug ap command "debug capwap console cli" <AP NAME>
debug ap command "delete /f /r flash:ap3g2*" <AP NAME>
debug ap command "archive tar /x tftp://<TFTP_IP_ADDRESS>/<FILENAME.tar>" <AP NAME>

NOTE: Depending on the WAN speed, give each AP 10 minutes to download the firmware before continuing to the last step.

4. Reboot the WLC.

Hope this helps.

NOTE: I used the exact same steps to upgrade a WLC from 8.1.X.X to 8.5.X.X (183 APs) and I only "lost" one AP.

Clem58 · ‎04-09-2021

Thanks Leo.

This workaround is a bit heavy for my side, as I will need to do that for 140 APs ..

When you say "If left alone, the AP will first download the firmware with a prefix of "c3700", reboot, join the controller and then download the second firmware with the prefix "ap3g2", reboot and then join the controller. The entire process will take around 14 minutes (per AP). " does that mean if we predownload the firmware it will have to download only the ap3g2 one ?

The WLC firmware file size is ~260mb, but the APs ones are maybe less ? As I can see on the Cisco site the separate files size are around 15mb.

I mean if the APs reboot twice but only download ~15Mbps each time it's not a big deal.

Could you also confirm it's only concerning the upgrade to 8.5.x ?

How about 2602,3602 & 3802 AP models ?

Leo Laohoo · ‎04-09-2021

@Clem58 wrote:

does that mean if we predownload the firmware it will have to download only the ap3g2 one ?

If you initiate "pre-download", the WLC will push the firmware with a prefix of "c3700" to the AP. When the WLC reboots to load the new firmware, the AP will reboot, join the controller. The controller will then instruct the AP to download, for the 2nd time, the firmware with the prefix "ap3g2" before rebooting the the 2nd time.

Just remember that during this entire process, for >14 minutes the AP is not in service. With my process, the only time the AP is not in service is during the rebooting.

Clem58 · ‎04-09-2021

Ok so that means, on the pre-download the AP will download 15Mb for 1st firmware C3700 then after the 1st WLC reboot will download 15 Mbps of ap3g2 image ?
If it's correct, the downtime would not be that long.

I don't really get the difference of size, WLC firmware 260Mb, and for APs around 30 Mb ? Need clarification here. That means other APs than 3700 will download only one file ? Which size then ?

Scott Fella · ‎04-09-2021

The WLC image contains all the ap images also, that is why the file size is larger. Not all access points do the double download so just read and follow what Leo has mentioned. You will have some downtime and you really need to determine when you can do a change. No matter what, you should take a downtime that takes into account validation and or troubleshooting, even though some of your sites are 24/7.

-Scott
*** Please rate helpful posts ***

Clem58 · ‎04-09-2021

Thanks Scott for your advices !

Yes I will take in account every constraints related with this upgrade but I need to indicate the site admins the average outage time it will take, so I estimated 2 hours (without the predownload time).

Anyway the most difficult part is to find the most suitable date for everyone !

Scott Fella · ‎04-09-2021

That is always the issue, but when I have done cut-overs and or upgrades for a hospital or any customer, you just plan ahead and make sure they know of the downtime. I use to have a section of the hospital that was upgraded for all the departments to test and sign off that all their equipment and applications are still working. That helped with any issue later blaming the network.

-Scott
*** Please rate helpful posts ***

Clem58 · ‎04-09-2021

Haha that's a great idea Scott, but not applicable within our organisation unfortunately.

Anyway thanks for your priceless help Leo and Scott !

Scott Fella · ‎04-09-2021

Too bad... it was something I asked in the past and my customers wanted to make sure that they partnered with other departments and each group was accountable for their testing. Again, it might not be what others want, but it eliminates all these complaints that were probably existing that allows them to raise these after your work is completed.

-Scott
*** Please rate helpful posts ***

Clem58 · ‎04-21-2021

Another question Leo and Scott, if I pre-download firmware on some APs individually, when I will do pre-download all, will the the WLC skip the already downloaded ones ?

Any advices for 2602/3602 models, will the pre-download work for these ones ?

Leo Laohoo · ‎04-21-2021

@Clem58 wrote:

Any advices for 2602/3602 models, will the pre-download work for these ones ?

The double-download bug only affects 2700/3700.

@Clem58 wrote:

when I will do pre-download all, will the the WLC skip the already downloaded ones

What are you trying to do? Indiscriminate pre-download "all", then the 2700/3700 will download the image from the WLC, reboot, download AGAIN from the controller and reboot.

If you want to implement what I have done to "bypass" the double-download bug, then specify which APs will be doing the pre-download.

Clem58 · ‎04-22-2021

Hi Leo, that's not what you think, I just want to be able to pre-download individually on some APs where sites have poor bandwidth (example some have 10Mbps bandwidth), example 5 by 5 instead of All in one time, whatever the models.

Then when I will have finish to do the predownload manually for those sites, I will do "all" for the remaining ones, so I wanted to know if the ones already pre-downloaded will be skipped.

Leo Laohoo · ‎04-22-2021

@Clem58 wrote:

pre-download individually on some APs where sites have poor bandwidth

Oooooo ... I got something up my sleeve.

Ok, so scroll up to my response where I provided the commands to bypass the pre-download? Look at the last command. See it says the "TFTP Server IP Address"?

Ok, answer this: What if that TFTP server is found at the site (which have poor WAN link)? What if ALL the APs at that site is going to download the IOS file from a local server?

Clem58 · ‎04-22-2021

Thanks Leo, indeed that can be a solution, but for example, one site with 30 3802 APs, and 10Mbps bandwidth, with your solution I will have to manually enter commands on each AP and have to find separate firmware image for this model, not sure I want to do that.

I would prefer to prepare a template of commands for launching the pre-download for 5 APs, 6 times, waiting the download is complete each time.

The only thing is I want to be sure when I will download "all", I mean for the other ~270 APs, the 30 APs won't download again the firmware image and will skip it.

Another solution could be also to use flexconnect pre-download but I don't have much info about this feature, and not really trust it, what could happen if the "master" AP fails, doest it revert to classic pre-download ? Not clear.