Weird APSP3 behavior with 17.15.3

[patoberli]

Hi All

I'm running for testing a 9800-CL in VMware Workstation with 17.15.3 (initially installed with 17.9.6). 

There are no APs attached to it and just the absolute minimum configuration (web gui setup wizard, NTP and a valid user).

When I now try to install add file ... the APSP3 via CLI, I get an Error message shown in the wording like: 
%INSTALL-3-OPERATION_ERROR_MESSAGE: R0/0: install_mgr: Failed to install add package bootflash:/...SPA.bin, Error: No error or error string
This error is only visible in the Console and not in "show install log" or "show logg". 

Via SSH I only get the error:
wlc-test#install add file bootflash:C9800-CL-universalk9.17.15.03.CSCwp18505.SPA.apsp.bin activate commit
install_add_activate_commit: START Tue Jun 03 07:24:59 Central 2025
install_add: START Tue Jun 03 07:24:59 Central 2025
install_add: Adding IMG
FAILED: add_activate_commit /bootflash/C9800-CL-universalk9.17.15.03.CSCwp18505.SPA.apsp.bin Tue Jun 03 07:25:16 Central 2025

Now to the very weird part. If I try exactly the same via Web GUI with the file already on the bootflash, it works fine without any error. Any ideas?
I tend to say it's somewhat connected with https://bst.cisco.com/bugsearch/bug/CSCwn42968?rfs=qvred, but the workaround didn't work, as I was unable to delete that directory:
wlc-test#delete /recu /for bootflash:/.images/
wlc-test#delete /rec bootflash:/.images/17.15.03.0.5635.1742971448
Delete filename [/.images/17.15.03.0.5635.1742971448]?
Examine files in directory bootflash:/.images/17.15.03.0.5635.1742971448? [confirm]
%Error deleting bootflash:/.images/17.15.03.0.5635.1742971448/C9800-CL-rpboot.17.15.03.SPA.pkg.lnk (No such file or directory)
wlc-test#

On a separate note, it's still not possible to add an APSP at the same time while doing an IOS XE upgrade?
So let's say, going from 17.9.5 -> 17.15.3 with APSP3 in one ap image pre-download instead of rebooting the APs after first 17.15.3 and then, after the WLC is upgraded, install APSP3?

Thanks
Patrick

[Leo Laohoo]

Next time, try this: 

conf t
 service internal
end
clear install state

The last command will cause the WLC, router or switch to reboot.  After the reboot, re-install the APSP/SMU again.

[patoberli]

Might retry this again, I did create a snapshot while still on the initial 17.9.6 version. What I don't understand, why does it work in the web UI and not CLI. 

[patoberli]

No go, even after doing the clear:

wlc-test#install add file bootflash:C9800-CL-universalk9.17.15.03.CSCwp18505.SPA.apsp.bin
install_add: START Tue Jun 03 10:46:44 Central 2025
install_add: Adding IMG
--- Starting initial file syncing ---
Copying bootflash:C9800-CL-universalk9.17.15.03.CSCwp18505.SPA.apsp.bin from Chassis 1/R0 to Chassis 1/R0
Info: Finished copying to the selected Chassis
Finished initial file syncing

--- Starting SMU Add operation ---
Performing SMU_ADD on all members
Checking status of SMU_ADD on [1/R0]
SMU_ADD: Passed on [1/R0]
Finished SMU Add operation

SUCCESS: install_add /bootflash/C9800-CL-universalk9.17.15.03.CSCwp18505.SPA.apsp.bin Tue Jun 03 10:47:21 Central 2025
wlc-test#insta
wlc-test#install act
wlc-test#install activate
install_activate: START Tue Jun 03 10:47:31 Central 2025
install_activate: Activating IMG
Following packages shall be activated:

This operation may require a reload of the system. Do you want to proceed? [y/n]y

--- Starting Activate ---
Performing Activate on all members
[1] Failed Activate on Chassis 1/R0
Checking status of Activate on [1/R0]
Activate: Passed on []. Failed on [1/R0]
Finished Activate

FAILED: install_activate Tue Jun 03 10:47:35 Central 2025
wlc-test#

[marce1000]

  - @patoberli       >...What I don't understand, why does it work in the web UI and not CLI. 
                        You could for instance CLIcatcher (applet) as explained in : https://goodwi.fi/posts/2025/01/find-9800-cli-cmd/
                        CLIcatcher will log all CLI commands for GUI operations executed in the controller logs.
                                     You can the compare those  against your own CLI commands

  M.

[marce1000]

 - @patoberli       You cannot combine APSP installation(s) with AP pre-image download

 M.

[Rich R]

You can do APSP pre-download - after doing APSP install add - but only from CLI (not possible from GUI).
But agreed with Pat - you cannot install new image version and APSP at the same time.
APSP can only be installed after the new version is running.

As to the original question I found that add_activate_commit didn't work for APSP but that doing it in separate steps did?
Are you sure you were performing the same action in GUI @patoberli ?

However that doesn't explain why your separate activate didn't work - that does look more like the corrupt install database problem.  Did you try the one-shot or separate steps after clearing the database?

I've had trouble trying to manually delete the images folders before (lack of permissions) but install remove inactive does remove them.  But be careful - 17.15.x install remove inactive sometimes wants to delete your active running images!  I reported this in beta but they've not done anything about fixing it because I've seen the released images do exactly the same.

This is a continuation of the saga of the dreadfully unreliable install script!  How difficult can it be?!!!

ps: I've only installed APSP3 on 9800-80 so far as my lab 9800-CL is out of service while lab gets re-arranged (out of my control) so I haven't tested this on 9800-CL specifically.