topic Re: AP migration - Stuck in downloading status in Wireless

AP migration - Stuck in downloading status

mkardos1 — Mon, 05 Jul 2021 18:59:21 GMT

Hello all,

We are trying to migrate access-points 2702 from the WLC 5508 (8.0.152.0) to the different controller WLC 5508 (8.5.140.0). Once the AP joins a new controller, it downloads a new softare, reloads and is in the donwloading status again - showing it has a new software.. Have you ever seen such a behaviour?

Many thanks for any advice!

Re: AP migration - Stuck in downloading status

Leo Laohoo — Wed, 29 Apr 2020 08:39:48 GMT

Yes. This is a known issue we call "double-download".
You can stop this from happening by manually uploading the 8.5.140.0 CAPWAP image to the AP prior to moving the APs to the new controllers.

This is how it is done:

1. Download the CAPWAP file "c3700-k9w8-tar.153-3.JF9.tar" and put it in the TFTP server.

2. On the controller, do the following commands:

debug ap enable <AP NAME>
debug command "capwap console cli" <AP NAME> 
debug command "archive download-sw /overwrite tftp://<TFTP_IP_ADDRESS>/c3700-k9w8-tar.153-3.JF9.tar" <AP NAME>

3. Depending on the size of the WAN link(s) to the AP, this will take about 10 minutes.

4. Once the image has been uploaded move the APs to the controller running 8.5.140.0 by configuring the primary controller details:

config ap primary-base <WLC NAME> <WLC IP ADDRESS>

5. Instruct the WLC to reboot the AP:

config ap reset <AP NAME>

WARNING:

Do not allow the APs to move to the controller by themselves. Once the step 4 is entered do step 5 immediately. If an AP moves before step 5, the AP will discard step 3 and double-download will occur.

Hope this helps.

Re: AP migration - Stuck in downloading status

omz — Tue, 28 Apr 2020 09:20:31 GMT

might be worth a try -

ssh to the AP, save config, reload

Re: AP migration - Stuck in downloading status

Rich R — Tue, 28 Apr 2020 12:38:20 GMT

You should always read the release notes ... eg https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn85mr6.html#upgrade-software_85mr6
- The image format of Cisco Aironet 1700, 2700, 3700, and IW3702 APs have been changed from ap3g2 to c3700. Therefore, if you are upgrading to Release 8.5 or a later release from Release 8.3 or an earlier release, these APs will download the image twice and reboot twice.

Re: AP migration - Stuck in downloading status

Leo Laohoo — Tue, 28 Apr 2020 23:37:05 GMT

@Rich R wrote:
The image format of Cisco Aironet 1700, 2700, 3700, and IW3702 APs have been changed from ap3g2 to c3700. Therefore, if you are upgrading to Release 8.5 or a later release from Release 8.3 or an earlier release, these APs will download the image twice and reboot twice.

The workaround I've posted was used to successfully migrate 150 APs to 8.5 and not one AP double-downloaded.

Re: AP migration - Stuck in downloading status

mkardos1 — Wed, 29 Apr 2020 06:13:14 GMT

You are right, the double-donwload is occurring. I will try to predownload first. Many thanks!

Re: AP migration - Stuck in downloading status

Leo Laohoo — Wed, 29 Apr 2020 06:59:42 GMT

@mkardos1 wrote:
I will try to predownload first.

Pre-Download function from the WLC won't work. Double-download will still happen because the WLC will upload the wrong file up to the AP.

Re: AP migration - Stuck in downloading status

mkardos1 — Wed, 29 Apr 2020 07:40:59 GMT

ok ok, i though to do the manual upgrade.. sorry for misunderstanding. anyway, many thanks for your help!

Re: AP migration - Stuck in downloading status

mkardos1 — Thu, 30 Apr 2020 09:42:02 GMT

Hi again!

So this was about the double-download issue, but another issue appeared.. We have now two testing APs registered on the WLC and in case of the WLC reset, those APs will join the WLC after a long time, like 20 mins and then they are in downloading status again! I don't understand this.. I supposed they will join ok if the image is downloaded already..

Re: AP migration - Stuck in downloading status

Rich R — Thu, 30 Apr 2020 10:08:41 GMT

- Are you sure the APs are not roaming to another WLC on different code version before coming back? (meaning they download different version then come back and have to download new version again)
- You might also be suffering side effect of flash corruption since earlier code versions - refer to https://www.cisco.com/c/en/us/support/docs/field-notices/703/fn70330.html and https://www.cisco.com/c/en/us/support/docs/wireless-mobility/wireless-lan-wlan/213317-understanding-various-ap-ios-flash-corru.html

Re: AP migration - Stuck in downloading status

mkardos1 — Thu, 30 Apr 2020 11:02:01 GMT

Well, they use DHCP option 43 to receive the WLC information and they have set the primary base to the new controller only.. But I will check this to make sure. Thanks.

Re: AP migration - Stuck in downloading status

Rich R — Thu, 30 Apr 2020 12:39:15 GMT

Watch the AP console and you'll see exactly what it's doing.
Remember AP can learn other WLC details in other ways even when it gets boot info from option 43 - eg AP HA config, mobility group etc
https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-5/config-guide/b_cg85/ap_connectivity_to_cisco_wlc.html#discover-and-join-controllers

Re: AP migration - Stuck in downloading status

tanner.zaitt — Tue, 30 Aug 2022 19:16:50 GMT

Hi Leo,

Thank you for the solution and guidance.

I could confirm that in my scenario a few access points were stuck in downloading mode.
I resolved my issue only with second reboot.
Then I noticed that the traffic is increased on the ports of the switches where the access points are connected.
Also I checked show system files and I noticed changes on the flash memory of the access points.
After 3 hours with strong patience they were ok and upgraded successfully with 6 kb/s.
WLC and Anchor controllers in Europe and Access points in south America.... You could imagine.

Re: AP migration - Stuck in downloading status

Rich R — Wed, 31 Aug 2022 09:00:55 GMT

If the APs are all at the same location remember you can also use the Flex upgrade feature to improve things a little:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-5/config-guide/b_cg85/configuring_flexconnect_ap_upgrades_for_flexconnect_aps.html

The reason the downloads are so slow is that they still use TFTP which of course performs especially badly with long round trip delay and packet loss.
Also consider if you have a switch or router at the site which can act as TFTP server then download the image to that (you can use scp which will often be 10 times quicker than TFTP) and then get the APs to download the image from there which will be much quicker than crossing the Atlantic.

Re: AP migration - Stuck in downloading status

Louey — Tue, 18 Oct 2022 08:34:06 GMT

Hi,

This is happening on many APs due to new WLC version on our site, it is a nightmare to do this to all APs.

There is not another solution for that ?

Regards

Re: AP migration - Stuck in downloading status

Rich R — Tue, 18 Oct 2022 09:28:14 GMT

No - @Leo Laohoo has described the best possible workaround.

Re: AP migration - Stuck in downloading status

Leo Laohoo — Tue, 18 Oct 2022 19:48:02 GMT

@Louey wrote:
This is happening on many APs due to new WLC version on our site, it is a nightmare to do this to all APs.

I have no idea how many people have implemented my recommendation but this is the first time someone has complained.

I do this all the time and I use an Excel spreadsheet with formulas. Works all the time, never fails.

The last time I did this was on 03 October 2022 when I upgraded to 8.10.181.0. Total number of APs was about 8k.

@Louey wrote:
There is not another solution for that ?

Contact Cisco TAC and see if they have any better solution.

Re: AP migration - Stuck in downloading status

niltonassis — Thu, 16 Feb 2023 13:49:34 GMT

Hello!

After that, was show to me the message: "Could not enable debugging on Cisco AP".

Can you help me with this?

Re: AP migration - Stuck in downloading status

Rich R — Thu, 16 Feb 2023 14:02:27 GMT

@niltonassis what are you talking about? After what?

If you want help with a problem then always provide a full description of the problem, what steps you've taken, any logs, error messages, details of hardware models and software versions in use etc.

If it's not about the topic being discussed here then open a new topic.

Re: AP migration - Stuck in downloading status

niltonassis — Thu, 16 Feb 2023 14:31:55 GMT

Sorry!

The messege post was corrupted.

My WLC is hardware model 5508 and software version 8.5.151.0.

I have many APs with dowloading loop and I can't fix it.

So, after set the command "debug ap enable <AP NAME>" in CLI WLC via SSH, was showed the message: "Could not enable debugging on Cisco AP".

Can you help me with this?