topic Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs in Wireless

WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Mon, 05 Jul 2021 20:07:33 GMT

Hello,

We are planning to upgrade our 2x Central WLCs HA SSO from 8.2.130 to 8.2.170 as a 1st step and in a second step from 8.2.170 to 8.5.171.

We have:

3x AIR-CAP2602E-E-K9
21x AIR-CAP3602I/E-E-K9
140x AIR-CAP3702I/E-E-K9
148x AIR-AP3802I/E-E-K9

I'm planning to predownload the firmware to the APs, but I've read few articles saying the 3702 model could have to download twice the image.

We have a critical site 24/7 that have 33x3702 APs and only 10Mbps WAN access (to the remote WLCs), do you think it's more reasonnable to wait their WAN bandwidth upgrade to 100Mbps, because if the 33 APs have to download the image (~260mo), at the same time it could take very long time !

How about 2602, 3602 and 3802 models ?

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Leo Laohoo — Fri, 09 Apr 2021 14:50:11 GMT

2700/3700 will download the same firmware TWICE.

If left alone, the AP will first download the firmware with a prefix of "c3700", reboot, join the controller and then download the second firmware with the prefix "ap3g2", reboot and then join the controller. The entire process will take around 14 minutes (per AP).

The only way to force the AP to boot the 2nd firmware is to force the AP to manually pre-load this firmware. This method will take five to six minutes (per AP).

This is how to do it:

1. Download the firmware for the AP here. Make sure to download the firmware that has the prefix of "ap3g2-k9w8-tar".

2. Upload the WLC firmware but DO NOT reboot the WLC. Do not invoke the "pre-download" for AP3700 (OK to pre-download the rest).

3. Instruct the WLC to "force" the AP to erase the old firmware and load the new firmware:

debug ap enable <AP NAME>
debug ap command "debug capwap console cli" <AP NAME>
debug ap command "delete /f /r flash:ap3g2*" <AP NAME>
debug ap command "archive tar /x tftp://<TFTP_IP_ADDRESS>/<FILENAME.tar>" <AP NAME>

NOTE: Depending on the WAN speed, give each AP 10 minutes to download the firmware before continuing to the last step.

4. Reboot the WLC.

Hope this helps.

NOTE: I used the exact same steps to upgrade a WLC from 8.1.X.X to 8.5.X.X (183 APs) and I only "lost" one AP.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Fri, 09 Apr 2021 15:00:52 GMT

Thanks Leo.

This workaround is a bit heavy for my side, as I will need to do that for 140 APs ..

When you say "If left alone, the AP will first download the firmware with a prefix of "c3700", reboot, join the controller and then download the second firmware with the prefix "ap3g2", reboot and then join the controller. The entire process will take around 14 minutes (per AP). " does that mean if we predownload the firmware it will have to download only the ap3g2 one ?

The WLC firmware file size is ~260mb, but the APs ones are maybe less ? As I can see on the Cisco site the separate files size are around 15mb.

I mean if the APs reboot twice but only download ~15Mbps each time it's not a big deal.

Could you also confirm it's only concerning the upgrade to 8.5.x ?

How about 2602,3602 & 3802 AP models ?

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Leo Laohoo — Fri, 09 Apr 2021 15:18:19 GMT

@Clem58 wrote:

does that mean if we predownload the firmware it will have to download only the ap3g2 one ?

If you initiate "pre-download", the WLC will push the firmware with a prefix of "c3700" to the AP. When the WLC reboots to load the new firmware, the AP will reboot, join the controller. The controller will then instruct the AP to download, for the 2nd time, the firmware with the prefix "ap3g2" before rebooting the the 2nd time.

Just remember that during this entire process, for >14 minutes the AP is not in service. With my process, the only time the AP is not in service is during the rebooting.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Fri, 09 Apr 2021 15:22:47 GMT

Ok so that means, on the pre-download the AP will download 15Mb for 1st firmware C3700 then after the 1st WLC reboot will download 15 Mbps of ap3g2 image ?
If it's correct, the downtime would not be that long.

I don't really get the difference of size, WLC firmware 260Mb, and for APs around 30 Mb ? Need clarification here. That means other APs than 3700 will download only one file ? Which size then ?

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Scott Fella — Fri, 09 Apr 2021 16:24:05 GMT

The WLC image contains all the ap images also, that is why the file size is larger. Not all access points do the double download so just read and follow what Leo has mentioned. You will have some downtime and you really need to determine when you can do a change. No matter what, you should take a downtime that takes into account validation and or troubleshooting, even though some of your sites are 24/7.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Fri, 09 Apr 2021 16:31:09 GMT

Thanks Scott for your advices !

Yes I will take in account every constraints related with this upgrade but I need to indicate the site admins the average outage time it will take, so I estimated 2 hours (without the predownload time).

Anyway the most difficult part is to find the most suitable date for everyone ! 🙂

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Scott Fella — Fri, 09 Apr 2021 16:44:05 GMT

That is always the issue, but when I have done cut-overs and or upgrades for a hospital or any customer, you just plan ahead and make sure they know of the downtime. I use to have a section of the hospital that was upgraded for all the departments to test and sign off that all their equipment and applications are still working. That helped with any issue later blaming the network.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Fri, 09 Apr 2021 16:46:21 GMT

Haha that's a great idea Scott, but not applicable within our organisation unfortunately.

Anyway thanks for your priceless help Leo and Scott !

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Scott Fella — Fri, 09 Apr 2021 17:40:05 GMT

Too bad... it was something I asked in the past and my customers wanted to make sure that they partnered with other departments and each group was accountable for their testing. Again, it might not be what others want, but it eliminates all these complaints that were probably existing that allows them to raise these after your work is completed.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Wed, 21 Apr 2021 16:04:29 GMT

Another question Leo and Scott, if I pre-download firmware on some APs individually, when I will do pre-download all, will the the WLC skip the already downloaded ones ?

Any advices for 2602/3602 models, will the pre-download work for these ones ?

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Leo Laohoo — Wed, 21 Apr 2021 23:15:29 GMT

@Clem58 wrote:

Any advices for 2602/3602 models, will the pre-download work for these ones ?

The double-download bug only affects 2700/3700.

@Clem58 wrote:

when I will do pre-download all, will the the WLC skip the already downloaded ones

What are you trying to do? Indiscriminate pre-download "all", then the 2700/3700 will download the image from the WLC, reboot, download AGAIN from the controller and reboot.

If you want to implement what I have done to "bypass" the double-download bug, then specify which APs will be doing the pre-download.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Thu, 22 Apr 2021 09:26:54 GMT

Hi Leo, that's not what you think, I just want to be able to pre-download individually on some APs where sites have poor bandwidth (example some have 10Mbps bandwidth), example 5 by 5 instead of All in one time, whatever the models.

Then when I will have finish to do the predownload manually for those sites, I will do "all" for the remaining ones, so I wanted to know if the ones already pre-downloaded will be skipped.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Leo Laohoo — Thu, 22 Apr 2021 10:55:48 GMT

@Clem58 wrote:

pre-download individually on some APs where sites have poor bandwidth

Oooooo ... I got something up my sleeve.

Ok, so scroll up to my response where I provided the commands to bypass the pre-download? Look at the last command. See it says the "TFTP Server IP Address"?

Ok, answer this: What if that TFTP server is found at the site (which have poor WAN link)? What if ALL the APs at that site is going to download the IOS file from a local server?

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Thu, 22 Apr 2021 12:20:46 GMT

Thanks Leo, indeed that can be a solution, but for example, one site with 30 3802 APs, and 10Mbps bandwidth, with your solution I will have to manually enter commands on each AP and have to find separate firmware image for this model, not sure I want to do that.

I would prefer to prepare a template of commands for launching the pre-download for 5 APs, 6 times, waiting the download is complete each time.

The only thing is I want to be sure when I will download "all", I mean for the other ~270 APs, the 30 APs won't download again the firmware image and will skip it.

Another solution could be also to use flexconnect pre-download but I don't have much info about this feature, and not really trust it, what could happen if the "master" AP fails, doest it revert to classic pre-download ? Not clear.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Leo Laohoo — Thu, 22 Apr 2021 13:06:34 GMT

@Clem58 wrote:

with your solution I will have to manually enter commands on each AP and have to find separate firmware image for this model

Nope. That is definitely NOT my solution.

My solution is to instruct the WLC to instruct the AP to download the firmware from a TFTP server. My solution means I enter the command on the WLC only. Nowhere is my solution "enter commands on each AP".

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Thu, 22 Apr 2021 13:11:08 GMT

Yes sorry, instruct commands on APs via WLC 😊

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Clem58 — Wed, 30 Jun 2021 14:49:22 GMT

Hi Leo,

I've tried your solution on a test 3702 AP, and the firmware your instructed to download, it seems it's not the right one.

Looking at the AP after it downloaded the 8.5.171.0 one from WLC I can see this one in IOS version: 15.3.3-JF14

If I go on Cisco download page I can see these firmwares proposed :

So a recovery and a c3700 one, I'm a bit lost, which do I need to "pre-download" to the AP ?

Thanks by advance.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Scott Fella — Wed, 30 Jun 2021 15:12:38 GMT

The issue with the double downloads is with the 3702's. This is why you have to plan accordingly in case ap's fail or have an issue joining the controller. I always test to see how long it takes and also what issues I have or might run into. In a lab environment, you can always convert back and re-test. This is how you should develop your test plan and your upgrade plan. I know you have remote sites, but the ability to test in your own environment is key. We can always say one thing, but it might be different when you actually try it. Test if you can and take you downtime even if its 24x7. There are a lot of places like hospitals that have to find a good time to make changes.

Re: WLC 5520 upgrade and 2602 3602, 3702 and 3802 APs

Leo Laohoo — Wed, 30 Jun 2021 22:43:09 GMT

@Clem58 wrote:

I've tried your solution on a test 3702 AP, and the firmware your instructed to download, it seems it's not the right one.

I need more info to this: What happened, what went wrong, what error message.