Catalyst 3850 IOS downgrade

[winstonlimsk]

I have upgraded the CAT environment C3850 (standalone) and the Production Backup Data Center C3850 (stack) from 16.3.6 to 16.9.4 without issues. We have later decided not to proceed with Cisco Smart Licensing and wanted to downgrade to 16.6.8. We tried downgrading the CAT C3850 (standalone) and got the following error.

sev-cr-sw5#request platform software package install switch 1 file flash:cat3k_caa-universalk9.16.06.08.SPA.bin new
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash:cat3k_caa-universalk9.16.06.08.SPA.bin
[1]: Expanding file
[1]: Finished expanding all-in-one software package in switch 1
SUCCESS: Finished expanding all-in-one software package.
[1]: Performing install
SUCCESS: install finished
[1]: install package(s) on switch 1
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1

--- Starting file path checking ---
Finished file path checking

--- Starting image file verification ---
Checking image file names
Locating image files and validating name syntax
Found cat3k_caa-guestshell.16.06.08.SPA.pkg
Found cat3k_caa-rpbase.16.06.08.SPA.pkg
Found cat3k_caa-rpcore.16.06.08.SPA.pkg
Found cat3k_caa-srdriver.16.06.08.SPA.pkg
Found cat3k_caa-webui.16.06.08.SPA.pkg
Verifying image file locations
Inspecting image file types
WARNING: In-service installation of RP Base package
WARNING: requires software reboot of target RP
WARNING: Automatically setting the on-reboot flag
WARNING: In-service installation of guestshell package
WARNING: requires software reboot of target
Processing image file constraints
Creating candidate provisioning file
Finished image file verification

--- Starting candidate package set construction ---
Verifying existing software set
Processing candidate provisioning file
Production Image
Constructing working set for candidate package set
Constructing working set for running package set
Checking command output
Constructing merge of running and candidate packages
Checking if resulting candidate package set would be complete
Finished candidate package set construction

--- Starting ISSU compatibility verification ---
WARNING: Ignoring ISSU version compatibility checks
WARNING: Setting onreboot flag
Verifying image type compatibility
WARNING: Ignoring ISSU subpackage compatibility checks
Checking IPC compatibility for candidate software
[1]: Finished install failed on switch 1
Checking status of install on [1]
[1]: FAILED: install failed in switch 1
SUCCESS: Finished install: Success on []. Failed on [1]

*Feb 3 13:06:39.339: %INSTALL-3-OPERATION_ERROR_MESSAGE: Switch 1 R0/0: provision: Failed to install package flash:cat3k_caa-universalk9.16.06.08.SPA.bin, Error: [1]: FAILED: install failed in switch 1

Directory of flash:/

54210 -rw- 2097152 Feb 3 2021 14:27:19 +00:00 nvram_config
54211 drwx 4096 Dec 8 2020 02:00:56 +00:00 tech_support
54244 drwx 4096 Dec 9 2020 02:08:47 +00:00 .dbpersist
61953 drwx 4096 Dec 8 2020 02:09:58 +00:00 onep
54245 -rw- 0 Dec 8 2020 02:10:03 +00:00 rdope_out.txt
77444 -rw- 19858044 Dec 8 2020 01:53:15 +00:00 cat3k_caa-guestshell.16.09.04.SPA.pkg
77448 -rw- 30894576 Dec 8 2020 01:53:17 +00:00 cat3k_caa-rpbase.16.09.04.SPA.pkg
77445 -rw- 373381748 Dec 8 2020 01:53:17 +00:00 cat3k_caa-rpcore.16.09.04.SPA.pkg
77447 -rw- 3107452 Dec 8 2020 01:53:17 +00:00 cat3k_caa-srdriver.16.09.04.SPA.pkg
54214 -rw- 4774 Dec 8 2020 01:54:30 +00:00 packages.conf
54218 -rw- 481 Dec 8 2020 02:08:19 +00:00 bootloader_evt_handle.log
85185 drwx 4096 Feb 3 2021 14:04:12 +00:00 core
11 drwx 4096 Jul 8 2017 22:43:53 +00:00 .prst_sync
15489 drwx 4096 Jul 8 2017 22:43:55 +00:00 .rollback_timer
23233 drwx 4096 Dec 8 2020 02:08:27 +00:00 dc_profile_dir
23235 drwx 4096 Jul 8 2017 22:44:02 +00:00 gs_script
54219 -rw- 123888 Dec 8 2020 02:08:45 +00:00 memleak.tcl
30977 drwx 4096 Dec 8 2020 02:08:18 +00:00 .installer
54220 -rw- 34 Sep 6 2017 15:00:02 +00:00 pnp-tech-time
54221 -rw- 167270 Sep 6 2017 15:00:06 +00:00 pnp-tech-discovery-summary
54222 -rw- 1516 Jan 29 2018 16:02:05 +00:00 vlan.dat
54217 -rw- 2097152 Feb 3 2021 14:27:20 +00:00 nvram_config_bkup
54233 -rw- 19167872 Feb 3 2021 14:28:48 +00:00 cat3k_caa-guestshell.16.06.08.SPA.pkg
54253 -rw- 30417560 Feb 3 2021 14:28:50 +00:00 cat3k_caa-rpbase.16.06.08.SPA.pkg
54249 -rw- 336503416 Feb 3 2021 14:28:49 +00:00 cat3k_caa-rpcore.16.06.08.SPA.pkg
77446 -rw- 22784632 Dec 8 2020 01:53:17 +00:00 cat3k_caa-webui.16.09.04.SPA.pkg
54252 -rw- 5091968 Feb 3 2021 14:28:50 +00:00 cat3k_caa-srdriver.16.06.08.SPA.pkg
54232 -rw- 4742 Feb 3 2021 14:29:27 +00:00 cat3k_caa-universalk9.16.06.08.SPA.conf
54246 -rw- 90 Dec 8 2020 02:10:17 +00:00 rdope.log
54251 -rw- 16271996 Feb 3 2021 14:28:50 +00:00 cat3k_caa-webui.16.06.08.SPA.pkg

1621966848 bytes total (670203904 bytes free)

Need advice what is wrong.

Thanks

[balaji.bandi]

Try clean before downgrade :  ( cehck do you have enough space or else you can use USB to Downgrade)

#request platform software package clean

reboot the device and try to downgrade and let us know.

use below command give more information

new verbose

[winstonlimsk]

Hi BB,

Below is after issuing #request platform software package clean but did not reboot as I need to inform users in advance. I have also tftp the new bin to the switch. 

I understand that it is recommended to have at least 1GB of free space. From the information below, I believe it has around 1GB which includes the uploaded new *.bin. May I know the recommended 1GB of free space does it include the new bin?  USB downgrade may be an issue as due to Covid-19, this downgrade is supposed to be done remotely and there is no one on-site. 

Directory of flash:/

54210 -rw- 2097152 Feb 8 2021 01:54:24 +00:00 nvram_config
54211 drwx 4096 Dec 8 2020 02:00:56 +00:00 tech_support
54244 drwx 4096 Dec 9 2020 02:08:47 +00:00 .dbpersist
61953 drwx 4096 Dec 8 2020 02:09:58 +00:00 onep
54245 -rw- 0 Dec 8 2020 02:10:03 +00:00 rdope_out.txt
77444 -rw- 19858044 Dec 8 2020 01:53:15 +00:00 cat3k_caa-guestshell.16.09.04.SPA.pkg
77448 -rw- 30894576 Dec 8 2020 01:53:17 +00:00 cat3k_caa-rpbase.16.09.04.SPA.pkg
77445 -rw- 373381748 Dec 8 2020 01:53:17 +00:00 cat3k_caa-rpcore.16.09.04.SPA.pkg
77447 -rw- 3107452 Dec 8 2020 01:53:17 +00:00 cat3k_caa-srdriver.16.09.04.SPA.pkg
54214 -rw- 4774 Dec 8 2020 01:54:30 +00:00 packages.conf
54218 -rw- 481 Dec 8 2020 02:08:19 +00:00 bootloader_evt_handle.log
85185 drwx 4096 Feb 3 2021 14:04:12 +00:00 core
11 drwx 4096 Jul 8 2017 22:43:53 +00:00 .prst_sync
15489 drwx 4096 Jul 8 2017 22:43:55 +00:00 .rollback_timer
23233 drwx 4096 Dec 8 2020 02:08:27 +00:00 dc_profile_dir
23235 drwx 4096 Jul 8 2017 22:44:02 +00:00 gs_script
54219 -rw- 123888 Dec 8 2020 02:08:45 +00:00 memleak.tcl
30977 drwx 4096 Dec 8 2020 02:08:18 +00:00 .installer
54220 -rw- 34 Sep 6 2017 15:00:02 +00:00 pnp-tech-time
54221 -rw- 167270 Sep 6 2017 15:00:06 +00:00 pnp-tech-discovery-summary
54222 -rw- 1516 Jan 29 2018 16:02:05 +00:00 vlan.dat
54213 -rw- 407458043 Feb 4 2021 01:24:59 +00:00 cat3k_caa-universalk9.16.06.08.SPA.bin
54217 -rw- 2097152 Feb 8 2021 01:54:24 +00:00 nvram_config_bkup
77446 -rw- 22784632 Dec 8 2020 01:53:17 +00:00 cat3k_caa-webui.16.09.04.SPA.pkg
54246 -rw- 90 Dec 8 2020 02:10:17 +00:00 rdope.log

1621966848 bytes total (670236672 bytes free)

[balaji.bandi]

I understand that you doing remove upgrade. (just think any issue around ? due to downgrade) - do you have any one can attend site and assits ?

so in that case the version working as expected, (cisco license team can understand this situation). in terms of License.

Once some one available onsite do help you, rather go panic mode if you lost due to some issues.

[winstonlimsk]

I do understand the concern. Prior to the upgrade/downgrade, we will inform our local maintenance vendor to standby. If unfortunately we do need someone onsite, we will alert the vendor to support. Note that these devices are located in Sweden and the remote downgrade/upgrade is carried out from Singapore. 

[balaji.bandi]

Sure - in this kind of situation we always do local simulation rather directly putting pressure on Live devices, so you get more condifdent if you simulated locally you know what to expect what not, technically duarable (cisco does many testings upgrade and downgrade,) ocataionally 1-5% still fails due to unknown test cases.