WLC 5508 Upgrade - Ways to Predownload Firmware

endersocorro2368 · ‎12-17-2020

I have a WLC 5508 in HA. Firmware 8.3.143.0 and trying to upgrade to 8.5.140.

We have around 100-105 AP registered in the WLC. I could attach the new image and bundle for the 8.5.140, and now, i'm trying to "predownload" the firmware in all the AP's. In this step, he have had a couple of issues:

- First of all, the 5500 series has a bug reported by Cisco in the version CsCup00927 but they say it affects only the version 7.4.121. But i had the same problem on version 8.3.143.0 as follow:

*spamApTask2: Dec 16 14:43:41.964: %CAPWAP-3-DISC_MAX_DOWNLOAD: [PA]capwap_ac_sm.c:2042 Ignoring discovery request from AP 00:27:90:c9:94:f0 - maximum number of downloads (210) exceeded

*spamApTask2: Dec 16 14:43:41.964: %LOG-3-Q_IND: [PA]capwap_ac_sm.c:2042 Ignoring discovery request from AP 00:27:90:c9:9f:30 - maximum number of downloads (210) exceeded[...It occurred 3 times.!]

That maximum number it's not correct since we have only 100-105 AP (the half of the AP could download the image correctly)

Dec 15 17:40:41 kernel: [*12/15/2020 17:40:41.1404] Image_pre-download task delay 430 seconds.

Dec 15 17:41:09 kernel: [*12/15/2020 17:41:09.4379] Image download did not start for 30 seconds.

Dec 15 17:41:09 kernel: [*12/15/2020 17:41:09.4380] Back-off and retry.

Dec 15 17:47:51 kernel: [*12/15/2020 17:47:51.4915] Image Data Request sent to 172.17.227.21

Dec 15 17:47:52 kernel: [*12/15/2020 17:47:52.3519] AC rejected join request[17], try again later.

Dec 15 17:47:52 kernel: [*12/15/2020 17:47:52.3519]

Dec 15 17:47:52 kernel: [*12/15/2020 17:47:52.3520] Controller rejected image download.Maximum concurrent predownload limit has reached

Dec 15 17:47:52 kernel: [*12/15/2020 17:47:52.4917] Image_pre-download task delay 363 seconds.

Dec 15 17:48:20 kernel: [*12/15/2020 17:48:20.8113] Image download did not start for 30 seconds.

Dec 15 17:48:20 kernel: [*12/15/2020 17:48:20.8113] Back-off and retry.

Dec 15 17:53:55 kernel: [*12/15/2020 17:53:55.2521] Image Data Request sent to 172.17.227.21

Dec 15 17:53:58 kernel: [*12/15/2020 17:53:58.6156] AC rejected join request[17], try again later.

Dec 15 17:53:58 kernel: [*12/15/2020 17:53:58.6156]

Dec 15 17:53:58 kernel: [*12/15/2020 17:53:58.6156] Controller rejected image download.Maximum concurrent predownload limit has reached

Dec 15 17:53:59 kernel: [*12/15/2020 17:53:59.2526] Image_pre-download task delay 238 seconds.

Dec 15 17:54:26 kernel: [*12/15/2020 17:54:26.9786] Image download did not start for 30 seconds.

Dec 15 17:54:26 kernel: [*12/15/2020 17:54:26.9786] Back-off and retry.

Dec 15 17:57:56 kernel: [*12/15/2020 17:57:56.9441] Image Data Request sent to 172.17.227.21

Dec 15 17:57:57 kernel: [*12/15/2020 17:57:57.6813] AC rejected join request[17], try again later.

Dec 15 17:57:57 kernel: [*12/15/2020 17:57:57.6813]

Dec 15 17:57:57 kernel: [*12/15/2020 17:57:57.6814] Controller rejected image download.Maximum concurrent predownload limit has reached

The workaround was making a reload on the WLC. It worked and I could predownload the firmware to some AP. The PROBLEM is that some of this AP have a high latency/delay (1MB throughput) , on those remote sites i cant predownload de firmware, the WLC says it "Failed to predownload" everytime.

I need to know how can I predownload the firmware in those AP, I tried doing by Flexconnect but I hade other issues: For example, some AP said that the firmware was "predownloaded" BUT they didn't SWAP the Primary and Backup Images, if I reload those AP, they don't have the Predownloaded Image. If you do the predownload in the WLC the AP's automaticlly swap the images.

The other option, is to predownload the firmware locally on via TFTP with a PC or the SW connected to the AP, but I need to know how to extract the firmware on ONE of the AP that worked (the firmware isn't in the CISCO web)

Leo Laohoo · ‎12-17-2020

8.5.X.X and 2700/3700 Pre-Download do not work.

endersocorro2368 · ‎12-17-2020

I could predownload on AIR-CAP2702I-A-K9 to 8.5.140.0, I don't have 3700.

Leo Laohoo · ‎12-17-2020

@endersocorro2368 wrote:

I could predownload on AIR-CAP2702I-A-K9 to 8.5.140.0, I don't have 3700.

Give it a try. Pre-Download DO NOT WORK with 2700/3700. It might say that pre-download is "completed" in the status but once the WLC reboots into the new 8.5.XXX.0 firmware the AP2700/AP3700 will download the firmware TWICE. (We call it the "double download".)

The WLC will push a firmware with the prefix filename of "c2700". The AP will reboot into this filename, join the controller, will begin to download the same firmware but with the prefix of "ap3g2" and then reboot again.

The "double download" will take about 22 to 24 minutes (per AP).

Workaround is to MANUALLY force the AP to download the "ap3g2" firmware before the WLC reboots to the new firmware. This method will have an AP outage of about 4 minutes (per AP).

endersocorro2368 · ‎12-17-2020

yeah. I already know that. It's a partial "predownload", but that's not the problem that I have said.

The problem is with the predownload FAILURES on the AP. Not only the 2700 Model, I also have 2800 with the same issue, most of the AP with the predownload problems are in remot locations with low throughput.

How can you MANUALLY for the AP?

What I do is "config ap image predownload primary "ap name" "

Leo Laohoo · ‎12-17-2020

@endersocorro2368 wrote:

How can you MANUALLY for the AP?

There an undocumented method to instruct the AP to download "a firmware" that is not found in the WLC.

Download the correct CAPWAP (not RCV) image from the Cisco website (TAR file).
Put the TAR file in a TFTP server
Delete the all existing CAPWAP & RCV image
Remote into the AP and instruct the AP to download the firmware:

debug capwap console cli
delete /f /r flash:ap3g2*
archive tar /x tftp://<TFTP_IP_ADDDRESS>/IOS_FILENAME.tar flash:

Alternatively, you can "instruct" the WLC to "tell" the AP to do Steps 3 & 4 above:

debug ap enable <AP NAME>
debug ap command "debug capwap console cli" <AP NAME>
debug ap command "del /f /r flash:ap3g2*" <AP NAME>
debug ap command "archive tar /x tftp://<TFTP_IP_ADDDRESS>/IOS_FILENAME.tar flash:" <AP NAME>

Make sure the APs finish downloading the firmware before rebooting the WLC.

NOTE:

I used the same method when moving the <200 x AP3702i from a WLC, running 8.1.X.X, to another running 8.8.X.X. We tried several using the Cisco "recommendation" of pre-downloaded and the APs were out for about 22 to 24 minutes. Then we tried the above method and the APs were back in about <5 minutes. The APs were all in a 24 x 7 site.

endersocorro2368 · ‎12-19-2020

I tried to download the firmware on the AP as shown (MY PC AS TFTP):

archive download-sw /no-reload tftp://10.X.X.X/ap3g3-k9w8-tar.153-3.JF9.tar

curl: (28) Operation too slow. Less than 10 bytes/sec transferred the last 60 seconds
transfer command failed
Image transfer failed

That's what happened. Too slow (it was the only way to know it was a problem with the traffic), no other loggs contained this information.

Then, I started doing it locally on the (LAN), with a PC connected to the AP in the Remote Site and I could do it

I've had certain problems with some AP. they can say they have the image "predownload" but in a couple of hours it appears with the old image (in the WLC GUI)

For the 1602 AP. Sometimes if you try to start the predownload process via WLC, when it ends it shows a "3.0.51.0 " backup version.

CISCO TAC told me it's a problem with the flash memory. But it doesn't happen to me in all the AP (most of the have the new 8.5.140 firmware).

The archive download command doesn't appear on 1602 CLI.

Leo Laohoo · ‎12-19-2020

@endersocorro2368 wrote:

archive download-sw /no-reload tftp://10.X.X.X/ap3g3-k9w8-tar.153-3.JF9.tar

Do NOT use this. Why?

The options you are using is only "/no-reload". What is not seen in this command is "/no-set-boot", which is not available -- This means the process WILL ATTEMPT to set the boot variable string when the entire process completes.

When the process attempts to set the boot variable string, it will crash the AP. And when I mean crash, I mean the AP becomes un-recoverable.

You must use the old and archaic option:

archive tar /x tftp://10.X.X.X/ap3g3-k9w8-tar.153-3.JF9.tar flash:

@endersocorro2368 wrote:

curl: (28) Operation too slow. Less than 10 bytes/sec transferred the last 60 seconds
transfer command failed
Image transfer failed

Next, "keep local traffic local". Find a computer (any computer) in the same location as the APs and install TFTPd64 and point the APs to this server instead of going across the WAN link.

punith-ramesh · ‎12-19-2022

@Leo Laohoo
Hey there Leo Laohoo

I have several 2800 series Access points that are in a remote site.
I also tried the command

archive tar /x tftp://10.X.X.X/ap3g3-k9w8-tar.153-3.JF9.tar flash:

does not seem to be working . it returned back to me with pointing to tar as an invalid command.
We also tried to make the local machine that is consoled into the the AP as the TFTP as well which did not work.

We have a Zscaler installed on the machine which is centrally managed. do you think this could also be a reason for it being accessible.

is there any other way to execute it or perform it .?

any information would be of great help .

Thanks

Leo Laohoo · ‎12-19-2022

@punith-ramesh wrote:
archive download-sw /no-reload tftp://10.X.X.X/ap3g3-k9w8-tar.153-3.JF9.tar

For x800, use the above command.

punith-ramesh · ‎12-19-2022

thank you for the command however I have tried above command.
and encounter this following error .

curl: (28) Operation too slow. Less than 10 bytes/sec transferred the last 60 secondstransfer command failedImage transfer failed

Command : archive download-sw /reload tftp://10.180.X.X/ap3g3-k9w8-tar.153-3.JF14.tar

the TFTP is locally configured and the AP can reach it within one hop when performing a traceroute it.

Still do not understand what could cause this issue

Leo Laohoo · ‎12-19-2022

Slow TFTP from the local LAN? That looks like a routing issue.

punith-ramesh · ‎12-19-2022

Hey Leo,

Thanks for that , We have kind of narrowed it down to Symantec Firewall blocking the connection over UDP , This is centrally managed , Will keep things posted when we have that allowed and tested again.

Solution :
Our infra had a central SEP that was blocking the traffic from the source.
Once that was allowed. the access point was able to download the IOS and two of them successfully connected back to the WLC ,

thank you Leo and to this community.

I hope this helps everyone