9800 WLC upgrade question

[JaganV]

Hi guys,
I need some advice on this. I need to upgrade my APs with minimal disruption. So usually we upgrade a unused WLC pair and “move” the APs over in a controlled manner like floor by floor or even and odd numbered APs. However, we have seen in the past corruption happening and APs not coming up. This is very troublesome and something I want to avoid. 
Is there any way I can do this via the “Efficient Image Upgrade” process? Any suggestions?

Here is a summary of my scenario:

Source WLC: 9800-CL, version 17.9.5

Target WLC: 9800-CL, version 17.15.3 (fresh, no APs yet)

APs: ~900 total (mix of 9120AXI and 3802i)

AP Mode: FlexConnect (remote sites)

Topology: Centralized controllers in a DC, remote APs over WAN links

Goal: Migrate APs gradually (site-by-site) with minimal WAN impact using Efficient Image Upgrade (FlexConnect Primary/Secondary AP relay model).

[Leo Laohoo]

Throw the idea of "with minimal disruption" out the window because all the APs will need to be rebooted first before the controller reboots (with the new firmware) because I have a strong confidence they are all affected by CSCwe97901, CSCwm08044, CSCwm72142.

These bugs are present from 17.3.X, 17.6.X, 17.9.X, and up to 17.12.3.  If the APs are not rebooted prior to the WLC upgrade, it will constantly be in "Downloading" state.

[Mark Elsen]

  - @JaganV    The more recent versions starting from 17.13.x  have  a complete corruption verification and
prevention system for ap image download over WAN links. The Efficient Image Upgrade  would be advised indeed.
                             Following migration of a site use these commands to verify the health of the access points :
                                         https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/217738-monitor-catalyst-9800-kpis-key-performa.html#toc-hId--1620538988

   Appendix : (always) validate the controller's configuration (also after an upgrade)
                     using the CLI command show tech wireless and feed the output from that into Wireless Config Analyzer
                                (Use the full command as outlined in green, it does not work with show tech-support  )

  M.
                              
                         

[Rich R]

The image corruption issue was a new "feature" of the 9800 code which Cisco battled to understand and fix!  Through a number of different "fixes" and enhancements the problem is supposed to be solved in the latest code versions but it's very much still present in your old 17.9.5 code.

Hopefully you have installed the SMUs and APSP on 17.15.3 before you start moving APs onto it as per the TAC Recommended link below?

As Leo pointed out many of the APs will not be able to complete download at all because of the /tmp bugs - only resolved by reloading the APs before starting the downloads.

As for dealing with the download corruption issue read through the field notice and technotes:
https://www.cisco.com/c/en/us/support/docs/field-notices/741/fn74109.html
https://www.cisco.com/c/en/us/support/docs/wireless/wireless-lan-controller-software/221869-safely-upgrade-access-points-avoiding-i.html
https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/220443-how-to-avoid-boot-loop-due-to-corrupted.html

Key points: the corruption only affects CAPWAP download but does not affect TFTP install and https download (but that's only available on later releases not 17.9.5).  I'm not sure whether it affects efficient image download (flex AP peer to peer sharing) but I've found that to be problematic before anyway, and you must have unique site tags for every site otherwise you'll have APs trying to download from an AP in a different site because it selects 1 AP of each model as the master for each site tag.

If you want to pre-install the AP image using TFTP I've detailed on a few other previous posts how to extract the APSP images for the APs (you basically unzip the APSP twice and then you have the AP images).

[JaganV]

Hi Rich,

Ok I’ll get APSP and SMU updated on the new 17.15.3 beforehand.

I do have unique site tag for every site, so peer-to-peer could be one way to speed things up and avoid corruption.

Alternatively, thinking of TFTP upgrade on each AP and then move them over. I’m avoiding traditional predownload method as that caused many issues before, by timing out, due to large number of sites and distance.

so for installing via TFTP “archive download” on APs, I cannot use the downloaded 17.15.3 firmware? I need to extract it from the APSP?

[Rich R]

> so for installing via TFTP “archive download” on APs, I cannot use the downloaded 17.15.3 firmware? I need to extract it from the APSP?
Correct.
If you install the base 17.15.3 when WLC has APSP installed then as soon as they join they will start downloading the APSP image version from the WLC.
If you only install the APSP later then they'll need to do the download at that point.

Once the APSP is installed you'll see this - 17.15.3.28 is the base image, 17.15.3.203 is the APSP image which the APs will need:
9800#sh ap image file summ
AP Image Active List
============================
Install File Name: base_image.bin
-------------------------------
AP Image Type Capwap Version Size (KB) Supported APs
------------- -------------- --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------
ap1g4 17.15.3.28 40920 AP1852E, AP1852I, AP1832I, AP1830I

ap1g5 17.15.3.28 38820 AP1815W, AP1815T, OEAP1815, AP1815I, AP1800I, AP1800S, AP1815M, 1542D, AP1542I, AP1100AC, AP1101AC, P-WIFI-AC2, AP1840I

ap1g6 17.15.3.28 72290 AP2900I, C9117AXI

ap1g6a 17.15.3.28 87970 C9130AXI, C9130AXE, C9124AXI, C9124AXD, C9124AXE, C9136AXI, C9136I, IW9167EH, CW9164I, CW9166I, CW9166D1, IW9167IH

ap1g6b 17.15.3.28 83900 CW9162I, IW9165E, IW9165DH, CW9163E

ap1g7 17.15.3.28 76890 AP1900I, C9115AXI, AP1900E, C9115AXE, C9120AXE, C9120AXP, C9120AXI, C1115AXI

ap1g8 17.15.3.28 68960 C9105AXI, C9105AXW, C1105AXI, WP-WIFI6, ISR-AP1101AX

ap1g9 17.15.3.28 83160 CW9172I

ap3g2 17.15.3.28 15460 NA

ap3g3 17.15.3.28 57280 AP3802E, AP3802I, AP3802P, AP4800, AP2802E, AP2802I, AP2802H, AP3800, AP1562E, AP1562I, AP1562D, AP1562PS, APVIRTUAL, IW-6300H-DC, IW-6300H-AC, IW-6300H-DCW, ESW-6300

ap3g4 17.15.3.28 79770 CW9178I, CW9176I, CW9176D1

c1570 17.15.3.28 13050 AP1572E, AP1572I

c3700 17.15.3.28 14500 IW3702

AP Image Prepare List**
============================
**Difference of Active and Prepare list gives images being predownloaded to Access Points.

AP Image Active List
============================
Install File Name: C9800-universalk9_wlc.17.15.03.CSCwp18505.SPA.apsp.bin
-------------------------------
AP Image Type Capwap Version Size (KB) Supported APs
------------- -------------- --------- -------------------------------------------------------------------------------------------------------------------------------------------------------------
ap1g4 17.15.3.203 40910 AP1852E, AP1852I, AP1832I, AP1830I

ap1g5 17.15.3.203 38820 AP1815W, AP1815T, OEAP1815, AP1815I, AP1800I, AP1800S, AP1815M, 1542D, AP1542I, AP1100AC, AP1101AC, P-WIFI-AC2, AP1840I

ap1g6 17.15.3.203 72280 AP2900I, C9117AXI

ap1g6a 17.15.3.203 87980 C9130AXI, C9130AXE, C9124AXI, C9124AXD, C9124AXE, C9136AXI, C9136I, IW9167EH, CW9164I, CW9166I, CW9166D1, IW9167IH

ap1g6b 17.15.3.203 83890 CW9162I, IW9165E, IW9165DH, CW9163E

ap1g7 17.15.3.203 76860 AP1900I, C9115AXI, AP1900E, C9115AXE, C9120AXE, C9120AXP, C9120AXI, C1115AXI

ap1g8 17.15.3.203 68970 C9105AXI, C9105AXW, C1105AXI, WP-WIFI6, ISR-AP1101AX

ap1g9 17.15.3.203 83180 CW9172I

ap3g3 17.15.3.203 57270 AP3802E, AP3802I, AP3802P, AP4800, AP2802E, AP2802I, AP2802H, AP3800, AP1562E, AP1562I, AP1562D, AP1562PS, APVIRTUAL, IW-6300H-DC, IW-6300H-AC, IW-6300H-DCW, ESW-6300

ap3g4 17.15.3.203 79770 CW9178I, CW9176I, CW9176D1

AP Image Prepare List**
============================
**Difference of Active and Prepare list gives images being predownloaded to Access Points.

9800#

[JaganV]

There are APSP10 and latest October SMU released for this 17.15.3 .
However, as per TAC doc, it recommends APSP3 and an older SMU. Which one do I install then?

[Parithosh Vema]

APSP10 just released a couple of days ago, so it will take sometime in general for the documentation to get updated, but you can check the "README" which is available on software.cisco.com for the fixed bugs -https://www.cisco.com/web/software/286325254/171535/C9800-CL-universalk9.17.15.03.CSCwr59206.txt 

and APSPs are built in a way where the previous fixes are present on the latest one. 

[Rich R]

TAC does not recommend a specific APSP or SMU - just that they should be installed:
https://www.cisco.com/c/en/us/support/docs/wireless/catalyst-9800-series-wireless-controllers/214749-tac-recommended-ios-xe-builds-for-wirele.html#toc-hId--434755423
The reference to APSP3 and one specific SMU are just suggested considerations.
APSPs are cumulative so as a general rule you should apply the latest APSP.
SMUs are standalone (not cumulative) so generally speaking you should install all of them but if you read the release notes and bug details you might decide you don't need or want a specific SMU, then you don't install that one.  Some SMUs require WLC reload and others do not (non-reload/hitless) so that is also a consideration.  APSP never requires a WLC reload because it is only AP software upgrade but APs always require a reload to activate the new APSP version.

As @Parithosh Vema says TAC will update the doc in due course - it isn't automatically updated the same day that the updates are released.