WIFI AP Upgrade from Controller takes 3 hours

[billy-kemp]

I am trying to Upgrade code on Cisco APs by moving the AP to a New Controller with Newer Code.  This process is taking three hours per AP.  I have hundreds of APs.  Is this a known issue?  Cisco suggests upgrading via console port on each ap, but thats not feasible.

Any Suggestions?

[balaji.bandi]

  Is this a known issue? 

May be not in General - should 10-15min per AP.

you have not provided enough details ?

what WLC - what Image running ?

what AP model, what Version you upgrading to ?

where is WLC and where is AP located ?

[billy-kemp]

WLC's are 5520.

Moving from code 8.3 to 8.10

APs are distributed across WAN and LAN, some local in the same building as WLCs, others on distant networks

The process WORKS every time, by swapping Primary and Secondary controller IPs on the AP, then rebooting.  Its just taking 3 hours to complete 

[balaji.bandi]

The process WORKS every time, by swapping Primary and Secondary controller IPs on the AP, then rebooting.  Its just taking 3 hours to complete 

what is the requirement to swap the IPs ? So basically you saying you are migrating AP from one WLC to another WLC - the new WLC upgrading the AP, per AP taking 3 hours - personally this also never come across.

What AP Models ?  (for testing i will console to one of the AP see why it taking so Long ?)

In the LAN should be quicker - in the WAN i can understand if any low bandwidth to download image from HQ ? (but 3 hours is very high either case)

other option you can also download the new image to AP before you move the AP to new controller and test it - how long it takes ?

check some release notes :

https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn810.html

Note : never had that big jump version tested, 8.5 to 8.10 is ok for me all 3700AP

[marce1000]

                     >...Moving from code 8.3 to 8.10
 - Are the upgrades (already) done  ? It is important to verify the issue against https://software.cisco.com/download/home/286284738/type/280926587/release/8.10.190.0
  as indicated on https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/200046-tac-recommended-aireos.html

    As the aireos platforms are getting older and out phased , issues should be verified against last/latest available release for the particular model (?)

 M.
  

[billy-kemp]

I have 4 5520 WLCs.   Two running 8.3 and Two running 8.10

Im moving APs over to the 8.10 controllers because we had issues with re-booting the older code WLCs.  We want to Retire Them.

We are also moving wholesale to Meraki, but as I have over 2000 APs, we cant do it all in the same budget year.

I really like the Meraki solution.  I currently have 1300 Meraki APs about about 500 Meraki switches.

[Leo Laohoo]

What are the models of the AP?

I moved about 180 x AP3700 from 8.3.X.X to 8.10.X.X using a different method to Cisco, which I had developed, and it took 4 minutes outage per AP.  If I follow the Cisco-recommended method, the process will take 28 minutes.  

There is also a bug feature exclusive to the 2700/3700 called "double-download":  2700/3700 on 8.3.X.X (and earlier) upgrading to 8.5.X.X (and later) will download the CAPWAP firmware TWICE.  The first will download the TAR file with the prefix of "c3700", reboot, join the controller, download the 2nd CAPWAP TAR file with the prefix of "ap3g2", reboot and then join the controller (hence, the 24 minutes outage).

[billy-kemp]

i have many 3700s to upgrade.  i would love to hear more about your method.  How many did you do at the same time?  Several of my sites have 50+ 3700, 3802s

[Leo Laohoo]

---

@billy-kemp wrote:
How many did you do at the same time?

---

Because I have good experience with others in the past, I did all 180 APs at the same time.  

These are the steps/commands needed to be entered into the WLC:  

1.  Get a list of all the APs
2.  Look at this:  Supported Access Points in Cisco AireOS Wireless Software Releases.  Look under the "Cisco AireOS Controller Release" and match the WLC firmware with the "Access Point IOS Release".  Take note of the suffix.  For example, if the new WLC firmware is going to be 8.10.185.0, the middle column would say "15.3(3)JK9" with JK9 (the suffix) the important bit. 
3.  Go to the Cisco Download portal and look for the version with the exact "suffix".  There are three files presented.  Download the file that starts with "ap3g2-k9w8-tar".  
4.  Put that file into a TFTP server and make sure the AP can reach it.  
5.  WLC Command:  debug ap enable <AP NAME>
6.  WLC Command:  debug ap command "debug capwap console cli" <AP NAME>
7.  WLC Command:  debug ap command "delete /f /r flash:ap3g2*" <AP NAME>
8a.  WLC Command:  debug ap command "archive download-sw tftp://<IP ADDRESS>/filename.tar" <AP NAME>
8b.  If the AP refuses to take the firmware due to FN72524 - During Software Upgrade/Downgrade, Cisco IOS APs Might Remain in Downloading State After December 4, 2022 Due to Certificate Expiration, then use an alternative command to "force feed" the firmware to the AP:  debug ap command "archive tar /x tftp://<IP ADDRESS>/filename.tar flash:" <AP NAME>
9.  WAIT.  Depending on the WAN speed, it takes a good 10 minutes for each AP to download the firmware.  Do not reboot the AP yet.  
10.  Configure the new controller details using the WLC command:  config ap primary-base <WLC NAME> <AP NAME> <WLC IP ADDRESS>
11.  Forcefully reboot the AP (from the controller) is:  config ap reset <AP NAME>.  Enter "Y" immediately.  After step 10, do not hesitate and immediately reboot the AP.  If the AP is not forcefully rebooted, it will join the new controller and download the firmware twice!

NOTE: 

• From Steps 1 until Step 10, the AP is still "in service" -- It is still performing it's regular wireless "duties".  
• In Step 11, the AP reboots but it will load the new firmware.  And because of Step 10, it will immediately swing to the new controller.  And because the AP is booting the same firmware as the WLC, the AP will join the controller and become operational quickly.  

[David Ritter]

at first blush, and I blush alot,  I would have thought you were moving from a 5508/8.3 code to the 5520 on 8.10.  So I guess you have a bunch of stale AP's nolonger supported on 8.10.  bine there/done that.  I have an 8.5 stack (5508) with an 8.10 stack (5520) and 17.9 stack (9800) that I'm moving all the wifi5/6 ap's to from the retiring 5508.   Some of my sites are on a 10g dark fiber and the move/upgrade was rather prompt.  However, others would allow you to have coffee and a smoke before they finished .   

Enjoy!

[Ruben Cocheno]

@billy-kemp 

that is quite odd but as you have it distributed depends on the latency link capacity to get those new images.

Based on the release notes and also upgrade guide it shouldn't matter:

https://www.cisco.com/c/en/us/td/docs/wireless/access_point/mob_exp/810/user_guide/b_ME_User_Guide_810/getg_started.html

https://www.cisco.com/c/en/us/support/wireless/mobility-express/products-release-notes-list.html

From the WLC experience side, I would first go to the latest 8.5.x and then to the latest 8.10.x.

[billy-kemp]

You are correct.  Moving APs from old code controller to another controller with Newer code.  Models are 3800s.  Im doing this by swapping Primary vs Secondary controller IP on the AP, and rebooting.

Looking now how to do a Pre-download. Under AP ADVANCED TAB, Trying to understand "Primary Download" versus

"Interchange Image"  versus " Download Backup " image.    I would like to test downloading the New 8.10 image to the AP, then rebooting.  Controller also looks like its using tftp in Commands TAB..other options are FTP and SFTP.  

Any Suggestions?  any thank you for everyones input

[billy-kemp]

latency is certainly what we want to avoid as some sites have 50+ APs to upgrade, and We are a HOSPITAL.  Downtime is a nono.

We have a Hugh amount of wifi medical devices.

[Rich R]

The pre-download only works for sending an updated image to the AP when you upgrade the WLC: you load the new image on the WLC, then pre-download the update to the APs then when they're all downloaded you switch to the new version and reload WLC and APs.

Moving to a new WLC and new version will always do a download of the new software after joining the new WLC.  The main factor affecting download is round trip delay.  The WLC also limits the number of APs which can download at the same time.
https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-10/config-guide/b_cg810/managing_software.html#ID1225
The rest are kept in a queue and have to keep re-trying.

For those at remote sites the round trip delay and available bandwidth will slow down multiple simultaneous downloads.  One way of improving that (default behaviour on 9800) is to use Flexconnect AP Image Upgrade:
https://www.cisco.com/c/en/us/td/docs/wireless/controller/8-10/config-guide/b_cg810/configuring_flexconnect_ap_upgrades_for_flexconnect_aps.html
But that's for pre-download.  As you're moving to new code on new WLC that won't work for you so your only option will be to manually pre-download from local TFTP server.  Place the image somewhere on the site like router or switch and configure as TFTP server (or on a server if you have one you can use) and then from AP CLI:
AP# archive download-sw /no-reload ?
WORD URL of AP image <tftp|sftp>://<server_ip>/<file_path>
The image will be downloaded and installed to the backup partition then when the AP joins the new WLC it will see it already has the right matching image version on the backup partition and switch to it directly.  You could even switch the backup to active partition before you reload the AP.

You didn't mention what version of code your new WLC is using but hopefully it's 8.10.190.0 in which case the 3800 image will be:
https://software.cisco.com/download/home/286304536/type/286288051/release/15.3.3-JK10

You can take as long as you like downloading the APs in advance and then they should only be down for a few minutes while they reload when you switch to the new WLC.

[JPavonM]

I suffer this issue in some remote location with low reliability circuits and using UDP-based CAPWAP for APs registered on C9800., so may be the same is happening to you. The additional problem with high download times is that the code may be corrupted so APs will enter into a boot loop and finally you'll have to manually download the code via TFTP.

I'm planning moving to long-term code 17.12 so to enable HTTP downloads to decrease download times and avoid code corruption in the proccess.