Solved: Upgrading C3650 stack fails

c0x10 · ‎04-11-2020

Hi have a problem upgrading a WS-C3650-24TS Stack with two switches.
Upgrade is from 16.3.3 to 16.9.5

But is fails with the following usage.

It also complains about a missing file cat3k_caa-rpcore.16.03.03.SPA.pkg,

but when i check the flash content, then the file is present on both switches.

Has someone an idea of how to fix the issue?

request platform software package install switch all file flash:cat3k_caa-universalk9.16.09.05.SPA.bin new auto-copy
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash:cat3k_caa-universalk9.16.09.05.SPA.bin
[1]: Copying flash:cat3k_caa-universalk9.16.09.05.SPA.bin from switch 1 to switch 2
[2]: Finished copying to switch 2
[1 2]: Expanding file
[1 2]: Finished expanding all-in-one software package in switch 1 2
SUCCESS: Finished expanding all-in-one software package.
[1 2]: Performing install
SUCCESS: install Finished
[1]: install package(s) on switch 1
--- Starting list of software package changes ---
Old files list:
Removed cat3k_caa-guestshell.16.03.03.SPA.pkg
Removed cat3k_caa-rpbase.16.03.03.SPA.pkg
Removed cat3k_caa-rpcore.16.03.03.SPA.pkg
Removed cat3k_caa-srdriver.16.03.03.SPA.pkg
Removed cat3k_caa-wcm.16.03.03.SPA.pkg
Removed cat3k_caa-webui.16.03.03.SPA.pkg
New files list:
Added cat3k_caa-guestshell.16.09.05.SPA.pkg
Added cat3k_caa-rpbase.16.09.05.SPA.pkg
Added cat3k_caa-rpcore.16.09.05.SPA.pkg
Added cat3k_caa-srdriver.16.09.05.SPA.pkg
Added cat3k_caa-webui.16.09.05.SPA.pkg
Finished list of software package changes
SUCCESS: Software provisioned. New software will load on reboot.
[1]: Finished install successful on switch 1
[2]: install package(s) on switch 2
--- Starting local lock acquisition on switch 2 ---
Finished local lock acquisition on switch 2

--- Starting file path checking ---
Finished file path checking

--- Starting image file verification ---
Checking image file names
Locating image files and validating name syntax
Found cat3k_caa-guestshell.16.09.05.SPA.pkg
Found cat3k_caa-rpbase.16.09.05.SPA.pkg
Found cat3k_caa-rpcore.16.09.05.SPA.pkg
Found cat3k_caa-srdriver.16.09.05.SPA.pkg
Found cat3k_caa-webui.16.09.05.SPA.pkg
Inspecting image file types
WARNING: In-service installation of RP Base package
WARNING: requires software reboot of target RP
WARNING: Automatically setting the on-reboot flag
WARNING: In-service installation of srdriver package
WARNING: requires software reboot of target
Processing image file constraints
Creating candidate provisioning file
Finished image file verification

--- Starting candidate package set construction ---
Verifying existing software set
FAILED: Package /flash/cat3k_caa-rpcore.16.03.03.SPA.pkg does not exist
[2]: Finished install failed on switch 2
Checking status of install on [1 2]
[1]: Finished install in switch 1
[2]: FAILED: install failed in switch 2
SUCCESS: Finished install: Success on [1]. Failed on [2]

Leo Laohoo · ‎04-12-2020

Ok, all 16.9.5 package files are in.
Boot variable is pointing to the correct "packages.conf" file.
The only thing to do is to reboot the entire stack.
NOTE: After the stack reboots, don't forget to clean the flash. 3650/3850 don't have enough flash space for two (or more) unpacked firmware.

View solution in original post

Leo Laohoo · ‎04-11-2020

Post the complete output to the following commands:

dir flash-2:
sh boot
more flash-2:packages.conf | b for NOVA

c0x10 · ‎04-11-2020

Hi Leo, This is the output of the commands.

switch#dir flash-2:
Directory of flash-2:/

38806 -rw- 264899192 Apr 11 2020 15:40:35 +02:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
7773 -rw- 4737 Apr 27 2017 11:57:41 +02:00 packages.conf
31041 drwx 4096 Sep 1 2019 02:57:26 +02:00 dc_profile_dir
7783 -rw- 2097152 Mar 13 2020 08:08:58 +01:00 nvram_config
7771 -rw- 65301 Sep 1 2019 02:57:45 +02:00 memleak.tcl
7781 drwx 4096 Apr 27 2017 12:02:29 +02:00 .installer
7780 drwx 4096 Sep 1 2019 02:55:07 +02:00 core
7774 -rw- 15954560 Apr 27 2017 11:57:10 +02:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
7775 -rw- 22173639 Apr 27 2017 11:57:10 +02:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
7777 -rw- 9091712 Apr 27 2017 11:57:12 +02:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
7778 -rw- 191324788 Apr 27 2017 11:57:37 +02:00 cat3k_caa-wcm.16.03.03.SPA.pkg
7779 -rw- 13404796 Apr 27 2017 11:57:39 +02:00 cat3k_caa-webui.16.03.03.SPA.pkg
7768 -rw- 296 Sep 1 2019 02:57:06 +02:00 bootloader_evt_handle.log
15521 drwx 4096 Sep 1 2019 03:02:14 +02:00 .prst_sync
31043 drwx 4096 Apr 27 2017 12:01:49 +02:00 .rollback_timer
38802 drwx 4096 Apr 27 2017 12:01:58 +02:00 gs_script
7782 drwx 4096 Apr 27 2017 12:08:50 +02:00 onep
7785 -rw- 796 Sep 1 2019 03:03:05 +02:00 vlan.dat

1624244224 bytes total (754450432 bytes free)
switch#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode= DEVICE
iPXE Timeout= 0
switch#more flash-2:packages.conf | b for NOVA
switch#

c0x10 · ‎04-11-2020

I just reuploaded the bin file and startet the process again.

This is now what i get:

The installation on switch 1 fails and it works now on the second one.

request platform software package install switch all file flash:cat3k_caa-universalk9.16.09.05.SPA.bin new auto-copy
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1

Expanding image file: flash:cat3k_caa-universalk9.16.09.05.SPA.bin
[1]: Copying flash:cat3k_caa-universalk9.16.09.05.SPA.bin from switch 1 to switch 2
[2]: Finished copying to switch 2
[1 2]: Expanding file
[1]: Verifying parameters
[1]: Expanding superpackage flash:cat3k_caa-universalk9.16.09.05.SPA.bin
[1]: Validating package type
[1]: Note: Source file /flash/cat3k_caa-universalk9.16.09.05.SPA.bin will be removed automatically after completion.
[1]: Verifying package signatures
[1]: Signature verified on package cat3k_caa-guestshell.16.09.05.SPA.pkg
[1]: Signature verified on package cat3k_caa-webui.16.09.05.SPA.pkg
[1]: Signature verified on package cat3k_caa-rpcore.16.09.05.SPA.pkg
[1]: Signature verified on package cat3k_caa-srdriver.16.09.05.SPA.pkg
[1]: Signature verified on package cat3k_caa-rpbase.16.09.05.SPA.pkg
[1]: Copying package files
[1]: WARNING: packages.conf not copied. An identical file already exists in flash:
[1]: SUCCESS: Finished expanding all-in-one software package.
[1 2]: Finished expanding all-in-one software package in switch 1 2
SUCCESS: Finished expanding all-in-one software package.
[1 2]: Performing install
  SUCCESS: install Finished
[1]: install package(s) on switch 1
--- Starting local lock acquisition on switch 1 ---
Finished local lock acquisition on switch 1

--- Starting file path checking ---
Finished file path checking

FAILED: A previous provisioning action is pending reboot
[1]: Finished install failed on switch 1
[2]: install package(s) on switch 2
--- Starting list of software package changes ---
Old files list:
  Removed cat3k_caa-guestshell.16.03.03.SPA.pkg
  Removed cat3k_caa-rpbase.16.03.03.SPA.pkg
  Removed cat3k_caa-rpcore.16.03.03.SPA.pkg
  Removed cat3k_caa-srdriver.16.03.03.SPA.pkg
  Removed cat3k_caa-wcm.16.03.03.SPA.pkg
  Removed cat3k_caa-webui.16.03.03.SPA.pkg
New files list:
  Added cat3k_caa-guestshell.16.09.05.SPA.pkg
  Added cat3k_caa-rpbase.16.09.05.SPA.pkg
  Added cat3k_caa-rpcore.16.09.05.SPA.pkg
  Added cat3k_caa-srdriver.16.09.05.SPA.pkg
  Added cat3k_caa-webui.16.09.05.SPA.pkg
Finished list of software package changes
SUCCESS: Software provisioned.  New software will load on reboot.
[2]: Finished install successful on switch 2
Checking status of install on [1 2]
[2]: Finished install in switch 2
[1]: FAILED: install failed in switch 1
SUCCESS: Finished install: Success on [2]. Failed on [1]
switch#

Leo Laohoo · ‎04-11-2020

Do the same commands to both units:

dir flash-1:
dir flash-2:
more flash-1:packages.conf | b for NOVA
more flash-2:packages.conf | b for NOVA

c0x10 · ‎04-12-2020

Thats the output of the commands.

switch1#dir flash-1:
Directory of flash:/

69842 -rw- 15954560 Apr 13 2017 08:49:59 +02:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
69843 -rw- 22173639 Apr 13 2017 08:49:57 +02:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
69844 -rw- 264899192 Apr 13 2017 08:49:59 +02:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
69845 -rw- 9091712 Apr 13 2017 08:49:58 +02:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
69846 -rw- 191324788 Apr 13 2017 08:49:58 +02:00 cat3k_caa-wcm.16.03.03.SPA.pkg
69847 -rw- 13404796 Apr 13 2017 08:49:58 +02:00 cat3k_caa-webui.16.03.03.SPA.pkg
69858 -rw- 4774 Apr 11 2020 13:17:13 +02:00 packages.conf
69849 -rw- 518 Aug 10 2019 14:46:05 +02:00 bootloader_evt_handle.log
7761 drwx 4096 Aug 10 2019 14:44:09 +02:00 core
15521 drwx 4096 Aug 10 2019 14:51:18 +02:00 .prst_sync
31041 drwx 4096 Apr 13 2017 08:52:29 +02:00 .rollback_timer
38801 drwx 4096 Aug 10 2019 14:46:26 +02:00 dc_profile_dir
38803 drwx 4096 Apr 13 2017 08:52:39 +02:00 gs_script
69850 -rw- 65301 Aug 10 2019 14:46:44 +02:00 memleak.tcl
46561 drwx 4096 Apr 13 2017 08:53:09 +02:00 .installer
69854 -rw- 2097152 Apr 11 2020 18:05:44 +02:00 nvram_config
69852 -rw- 35 Apr 27 2017 12:10:20 +02:00 pnp-tech-time
69853 -rw- 28081 Apr 27 2017 12:10:25 +02:00 pnp-tech-discovery-summary
69855 -rw- 796 Aug 10 2019 14:52:29 +02:00 vlan.dat
7763 drwx 4096 May 18 2017 12:21:45 +02:00 onep
69848 -rw- 0 Apr 11 2020 13:17:27 +02:00 .onreboot
69861 -rw- 19860092 Apr 11 2020 17:59:47 +02:00 cat3k_caa-guestshell.16.09.05.SPA.pkg
69865 -rw- 30896047 Apr 11 2020 18:00:46 +02:00 cat3k_caa-rpbase.16.09.05.SPA.pkg
69863 -rw- 373586548 Apr 11 2020 18:00:40 +02:00 cat3k_caa-rpcore.16.09.05.SPA.pkg
69864 -rw- 3136124 Apr 11 2020 18:00:43 +02:00 cat3k_caa-srdriver.16.09.05.SPA.pkg
69862 -rw- 22798968 Apr 11 2020 18:00:42 +02:00 cat3k_caa-webui.16.09.05.SPA.pkg

1624104960 bytes total (151728128 bytes free)
switch2#dir flash-2:
Directory of flash-2:/

7773 -rw- 0 Apr 11 2020 18:03:04 +02:00 .onreboot
38806 -rw- 264899192 Apr 11 2020 15:40:35 +02:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
7766 -rw- 19860092 Apr 11 2020 17:59:52 +02:00 cat3k_caa-guestshell.16.09.05.SPA.pkg
7786 -rw- 30896047 Apr 11 2020 18:02:03 +02:00 cat3k_caa-rpbase.16.09.05.SPA.pkg
7770 -rw- 373586548 Apr 11 2020 18:01:53 +02:00 cat3k_caa-rpcore.16.09.05.SPA.pkg
7763 -rw- 4737 Apr 11 2020 18:02:52 +02:00 packages.conf.00-
7764 -rw- 4774 Apr 11 2020 18:02:52 +02:00 packages.conf
31041 drwx 4096 Sep 1 2019 02:57:26 +02:00 dc_profile_dir
7783 -rw- 2097152 Apr 11 2020 18:05:46 +02:00 nvram_config
7771 -rw- 65301 Sep 1 2019 02:57:45 +02:00 memleak.tcl
7781 drwx 4096 Apr 27 2017 12:02:29 +02:00 .installer
7780 drwx 4096 Sep 1 2019 02:55:07 +02:00 core
7774 -rw- 15954560 Apr 27 2017 11:57:10 +02:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
7775 -rw- 22173639 Apr 27 2017 11:57:10 +02:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
7772 -rw- 3136124 Apr 11 2020 18:01:57 +02:00 cat3k_caa-srdriver.16.09.05.SPA.pkg
7777 -rw- 9091712 Apr 27 2017 11:57:12 +02:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
7778 -rw- 191324788 Apr 27 2017 11:57:37 +02:00 cat3k_caa-wcm.16.03.03.SPA.pkg
7779 -rw- 13404796 Apr 27 2017 11:57:39 +02:00 cat3k_caa-webui.16.03.03.SPA.pkg
7768 -rw- 296 Sep 1 2019 02:57:06 +02:00 bootloader_evt_handle.log
15521 drwx 4096 Sep 1 2019 03:02:14 +02:00 .prst_sync
31043 drwx 4096 Apr 27 2017 12:01:49 +02:00 .rollback_timer
38802 drwx 4096 Apr 27 2017 12:01:58 +02:00 gs_script
7782 drwx 4096 Apr 27 2017 12:08:50 +02:00 onep
7785 -rw- 796 Sep 1 2019 03:03:05 +02:00 vlan.dat
7765 -rw- 4774 Apr 11 2020 18:02:31 +02:00 cat3k_caa-universalk9.16.09.05.SPA.conf
7767 -rw- 22798968 Apr 11 2020 18:01:57 +02:00 cat3k_caa-webui.16.09.05.SPA.pkg

1624244224 bytes total (303562752 bytes free)

---------------

switch#more flash-1:packages.conf | b for NOVA
# This is for NOVA
iso rp 0 0 rp_base cat3k_caa-rpbase.16.09.05.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.09.05.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.09.05.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.09.05.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.09.05.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime: 2020-01-30_18.23
# pkginfo: ReleaseDate: Thu-30-Jan-20-21:08
# pkginfo: .BuildArch: mips64
# pkginfo: BootArchitecture: mips
# pkginfo: .BootArch: mips
# pkginfo: RouteProcessor: edison
# pkginfo: Platform: CAT3K_CAA
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.09.05
# pkginfo: .SupportedBoards: edison
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /scratch/mcpre/release/BLD-V16_09_05_FC1/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64
# pkginfo: .Version: 16.9.5.0.3780.1580437405..Fuji
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

---------------

switch#more flash-2:packages.conf | b for NOVA
# This is for NOVA
iso rp 0 0 rp_base cat3k_caa-rpbase.16.09.05.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.09.05.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.09.05.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.09.05.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.09.05.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.09.05.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime: 2020-01-30_18.23
# pkginfo: ReleaseDate: Thu-30-Jan-20-21:08
# pkginfo: .BuildArch: mips64
# pkginfo: BootArchitecture: mips
# pkginfo: .BootArch: mips
# pkginfo: RouteProcessor: edison
# pkginfo: Platform: CAT3K_CAA
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.09.05
# pkginfo: .SupportedBoards: edison
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /scratch/mcpre/release/BLD-V16_09_05_FC1/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64
# pkginfo: .Version: 16.9.5.0.3780.1580437405..Fuji
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

Thank you

Happy Eastern

Leo Laohoo · ‎04-12-2020

Ok, all 16.9.5 package files are in.
Boot variable is pointing to the correct "packages.conf" file.
The only thing to do is to reboot the entire stack.
NOTE: After the stack reboots, don't forget to clean the flash. 3650/3850 don't have enough flash space for two (or more) unpacked firmware.

c0x10 · ‎04-13-2020

Hi Leo

Thank you for the information's :)

c0x10 · ‎04-11-2020

Have this issue on two locations.

Is there something i can check, or do, before reloading the units?

I am a bit scarred of doing that :)