Large deployment upgrade with CSCwd80290

[CiaranB]

Hi All,

We have a pair of 8540 WLC's (8.10.151.0) in HA with > 2,500 AP's registered. Approx. 2,000 of the APs are 802.11ac wave 1 APs.

To accommodate some new WiFi6 APs we need to upgrade the controller to a min. version of 8.10.183.0.

The upgrade is complicated somewhat by the following issue: https://www.cisco.com/c/en/us/support/docs/wireless/aironet-700-series-access-points/218447-ios-ap-image-download-fails-due-to-expir.html

Unfortunately we currently do not have a test environment in which we can test the upgrade using the workaround of changing disabling NTP and changing the date suggested in the above link.

Has anyone carried out an AireOS software upgrade using this workaround? Particularly in a large scale deployment? I know preloading the software on the .11ac wave1 APs isn't possible, so the upgrade will take longer than normal.

Any advice welcomed.

thanks

[Leo Laohoo]

---

@CiaranB wrote:
I know preloading the software on the .11ac wave1 APs isn't possible

---

It is possible but not the Cisco method.  

What I have been doing is telling the WLC to instruct the AP to delete both IOS directories.  I then tell the WLC to instruct the AP to download the RCV file using an archaic IOS command. 

After this is done, I enter the new WLC details into the AP and immediately force the AP to reboot. 

The AP reboots the RCV file, joins the new controller and then downloads the firmware.  

Done.

[CiaranB]

Hi, thanks for getting back to me. Why use this method instead of the recommended workaround from Cisco?

thanks

Ciarán

[Leo Laohoo]

Because the Cisco method means I have to load the new 8.10.18X.X firmware into the AP before I can move it to the new controller. 

Translation:  Two AP reloads and an outage time of >40 minutes. 

1. WLC reboots with the new firmware (7 minutes)
2. AP downloads the new firmware (14 minutes)
3. AP reboots (5 minutes)
4. AP joins the new controller and downloads the new firmware (14 minutes)
5. AP reboots (5 minutes)

My method means the AP will boot into RCV firmware (5 minutes), joins the controller and downloads the firmware (14 minutes) and reboots with the final firmware (5 minutes).  Grand total of 24 minutes.

[CiaranB]

Thanks for the detailed reply Leo.

I spoke with a rep from Cisco, this issue was actually resolved in 8.10.185.0 code, which is now published on CCO.

https://www.cisco.com/c/en/us/td/docs/wireless/controller/release/notes/crn810mr9.html#resolved-caveats

You’ll see the following is resolved:

CSCwd80290

Cisco IOS AP image validation certificate failed/expired, causing AP join issues.

However – he did say that you should still disable NTP pre-upgrade and re-enable after upgrade, for peace of mind.

Lots of customers have used this image, so he is comfortable that the specific issue won’t impact the upgrade now.

Ciarán

[Rich R]

Yes we've been running 8.10.185.0 for a while now.

Just make sure you have config ap cert-expiry-ignore {mic|ssc} enable configured before the upgrade if not already - that addresses FN63942.
8.10.185.0 fully addresses the Dec 4 2022 issue (FN72524) - no need to change NTP config.

Leo's solution is not really to do with the field notices but more the general problem with double downloads for the 2700/3700 APs.  Manually downloading the correct image ensures that it only needs to do a single download and reboot rather than the stupidly designed double download and reboot which they do by default.