Re: Cisco 3650 Stack IOS-XE 16.12.x Failed

johnlloyd_13 · ‎06-21-2022

hi,

i'm trying to upgrade a 2-stack 3650 switch 16.8 > 16.12 but failed only in stack switch 2. see output below

i saw 16.12.7 pkg files in flash-2. please advise if i need to perform again transfer .bin IOS-XE and issue the 'request platform' macro command again?

or is there a 'quick fix' on this problem? note it's in a remote site so i don't want the switch to go into ROMMON or brick the entire stack.

#request platform software package install switch all file flash:cat3k_caa-universalk9.16.12.07.SPA.bin auto-copy
--- Starting install local lock acquisition on switch 1 ---
Finished install local lock acquisition on switch 1
Expanding image file: flash:cat3k_caa-universalk9.16.12.07.SPA.bin
[1]: Copying flash:cat3k_caa-universalk9.16.12.07.SPA.bin from switch 1 to switch 2
[2]: Finished copying to switch 2
[1 2]: Expanding file
[1 2]: Finished expanding all-in-one software package in switch 1 2
SUCCESS: Finished expanding all-in-one software package.
[1 2]: Performing install
SUCCESS: install finished
[1]: install package(s) on switch 1
--- Starting list of software package changes ---
Old files list:
Removed cat3k_caa-guestshell.16.06.08.SPA.pkg
Removed cat3k_caa-rpbase.16.06.08.SPA.pkg
Removed cat3k_caa-rpcore.16.06.08.SPA.pkg
Removed cat3k_caa-srdriver.16.06.08.SPA.pkg
Removed cat3k_caa-webui.16.06.08.SPA.pkg
New files list:
Added cat3k_caa-guestshell.16.12.07.SPA.pkg
Added cat3k_caa-rpbase.16.12.07.SPA.pkg
Added cat3k_caa-rpcore.16.12.07.SPA.pkg
Added cat3k_caa-srdriver.16.12.07.SPA.pkg
Added cat3k_caa-webui.16.12.07.SPA.pkg
Finished list of software package changes
SUCCESS: Software provisioned. New software will load on reboot.
[1]: Finished install successful on switch 1
[2]: install package(s) on switch 2
--- Starting local lock acquisition on switch 2 ---
Finished local lock acquisition on switch 2
--- Starting file path checking ---
Finished file path checking
--- Starting image file verification ---
Checking image file names
Locating image files and validating name syntax
Found cat3k_caa-guestshell.16.12.07.SPA.pkg
Found cat3k_caa-rpbase.16.12.07.SPA.pkg
Found cat3k_caa-rpcore.16.12.07.SPA.pkg
Found cat3k_caa-srdriver.16.12.07.SPA.pkg
Found cat3k_caa-webui.16.12.07.SPA.pkg
Inspecting image file types
WARNING: In-service installation of RP Base package
WARNING: requires software reboot of target RP
WARNING: Automatically setting the on-reboot flag
WARNING: In-service installation of guestshell package
WARNING: requires software reboot of target
Processing image file constraints
Creating candidate provisioning file
Finished image file verification
--- Starting candidate package set construction ---
Verifying existing software set
FAILED: Invalid directory: /tmp/sw/mount/cat3k_caa-webui.16.06.08.SPA.pkg
[2]: Finished install failed on switch 2
Checking status of install on [1 2]
[1]: Finished install in switch 1
[2]: FAILED: install failed in switch 2
SUCCESS: Finished install: Success on [1]. Failed on [2]

#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

#more flash-1:packages.conf | b for NOVA
# This is for NOVA
iso rp 0 0 rp_base cat3k_caa-rpbase.16.12.07.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.12.07.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.12.07.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.12.07.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.12.07.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.12.07.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.12.07.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.12.07.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.12.07.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime: 2022-02-02_07.35
# pkginfo: ReleaseDate: Wed-02-Feb-22-04:05
# pkginfo: .BuildArch: mips64
# pkginfo: BootArchitecture: mips
# pkginfo: .BootArch: mips
# pkginfo: RouteProcessor: edison
# pkginfo: Platform: CAT3K_CAA
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.12.07
# pkginfo: .SupportedBoards: edison
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /nobackup/mcpre/release/BLD-V16_12_07_FC2/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64
# pkginfo: .Version: 16.12.07.0.6565.1643816123..Gibraltar
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

#more flash-2:packages.conf | b for NOVA
# <NONE/EMPTY>

#dir flash-2: | i 16.12.07
38762 -rw- 32795564 Jun 21 2022 14:16:08 +00:00 cat3k_caa-rpbase.16.12.07.SPA.pkg
38730 -rw- 4780 Jun 21 2022 11:46:59 +00:00 cat3k_caa-universalk9.16.12.07.SPA.1.conf
38727 -rw- 4780 Jun 21 2022 12:59:49 +00:00 cat3k_caa-universalk9.16.12.07.SPA.2.conf
38755 -rw- 20097668 Jun 21 2022 14:15:24 +00:00 cat3k_caa-guestshell.16.12.07.SPA.pkg
38757 -rw- 400857724 Jun 21 2022 14:15:57 +00:00 cat3k_caa-rpcore.16.12.07.SPA.pkg
38756 -rw- 23018112 Jun 21 2022 14:15:59 +00:00 cat3k_caa-webui.16.12.07.SPA.pkg
38758 -rw- 3414660 Jun 21 2022 14:15:59 +00:00 cat3k_caa-srdriver.16.12.07.SPA.pkg
38754 -rw- 4780 Jun 21 2022 07:59:49 +00:00 cat3k_caa-universalk9.16.12.07.SPA.conf
38731 -rw- 4780 Jun 21 2022 13:38:24 +00:00 cat3k_caa-universalk9.16.12.07.SPA.3.conf
38728 -rw- 4780 Jun 21 2022 14:16:44 +00:00 cat3k_caa-universalk9.16.12.07.SPA.4.conf

Leo Laohoo · ‎06-21-2022

Thanks for the output, @johnlloyd_13. Very useful. Cuts to the chase.

rename flash-2:cat3k_caa-universalk9.16.12.07.SPA.4.conf flash-2:packages.conf

That should work.
NOTE: This behaviour is a "known" bug and will not be fixed.

johnlloyd_13 · ‎06-21-2022

hi leo,

how would you know if file to be renamed is 'cat3k_caa-universalk9.16.12.07.SPA.4.conf' and not cat3k_caa-universalk9.16.12.07.SPA.1.conf or .2.conf or .3.conf?

after renaming the said file, do i just issue a 'reload' in master/switch 1?

Leo Laohoo · ‎06-21-2022

@johnlloyd_13 wrote:

how would you know if file to be renamed is 'cat3k_caa-universalk9.16.12.07.SPA.4.conf' and not cat3k_caa-universalk9.16.12.07.SPA.1.conf or .2.conf or .3.conf?

Look at the time-and-date stamp of those 4 files. They are all "the same" (date-wise). This means the script to unpack the BIN file created them. I suspect someone ran that script four times, hence, four files.

For the "workaround" provided (above), I just picked one of the four.

You have two options:

Before renaming them, run the command "more flash-2:filename.conf | begin for NOVA"
After renaming the file to packages.conf, verify with the command "more flash-2:packages.conf | begin for NOVA".

@johnlloyd_13 wrote:

after renaming the said file, do i just issue a 'reload' in master/switch 1?

No. Never. Always verify the contents of the packages.conf file (more flash:packages.conf | begin for NOVA).

johnlloyd_13 · ‎06-21-2022

hi leo,

i saw there's currently a 'packages.conf' file. i suspect this is from 16.6.8.

will there be any issue/conflict?

or will it just overwrite the said file for 16.12.7?

#dir flash-2:

Directory of flash-2:/

38753 -rw- 2097152 Jun 21 2022 11:57:40 +00:00 nvram_config

38762 -rw- 32795564 Jun 21 2022 14:16:08 +00:00 cat3k_caa-rpbase.16.12.07.SPA.pkg

38723 -rw- 5 Jun 21 2022 14:17:27 +00:00 .issu_loc_lock

38730 -rw- 4780 Jun 21 2022 11:46:59 +00:00 cat3k_caa-universalk9.16.12.07.SPA.1.conf

38727 -rw- 4780 Jun 21 2022 12:59:49 +00:00 cat3k_caa-universalk9.16.12.07.SPA.2.conf

38729 -rw- 76 Jun 2 2022 02:24:17 +00:00 boothelper.log

38732 drwx 4096 May 5 2015 09:45:44 +00:00 virtual-instance

38735 drwx 4096 Sep 24 2021 12:44:13 +00:00 core

38755 -rw- 20097668 Jun 21 2022 14:15:24 +00:00 cat3k_caa-guestshell.16.12.07.SPA.pkg

77441 drwx 4096 May 5 2015 09:45:44 +00:00 vman_fdb

38751 drwx 4096 Oct 6 2021 05:00:30 +00:00 onep

85186 drwx 4096 Jun 2 2022 02:24:15 +00:00 dc_profile_dir

85188 drwx 4096 Mar 28 2017 20:29:45 +00:00 wnweb_store

38747 -rw- 19167872 Sep 24 2021 12:56:02 +00:00 cat3k_caa-guestshell.16.06.08.SPA.pkg

38733 drwx 4096 Jun 21 2022 07:59:50 +00:00 .installer

38734 -rw- 1147 Jun 2 2022 02:23:58 +00:00 bootloader_evt_handle.log

38736 drwx 4096 Sep 24 2021 13:05:19 +00:00 .prst_sync

14 drwx 4096 Sep 24 2021 12:58:49 +00:00 .rollback_timer

7747 drwx 4096 Sep 24 2021 12:40:09 +00:00 gs_script

38737 -rw- 128103 Jun 2 2022 02:24:36 +00:00 memleak.tcl

38738 -rw- 35 Sep 24 2021 13:15:06 +00:00 pnp-tech-time

38740 -rw- 2097152 Jun 21 2022 11:57:41 +00:00 nvram_config_bkup

38749 -rw- 30417560 Sep 24 2021 12:56:26 +00:00 cat3k_caa-rpbase.16.06.08.SPA.pkg

38757 -rw- 400857724 Jun 21 2022 14:15:57 +00:00 cat3k_caa-rpcore.16.12.07.SPA.pkg

77443 -rw- 5091968 Sep 24 2021 12:56:24 +00:00 cat3k_caa-srdriver.16.06.08.SPA.pkg

38756 -rw- 23018112 Jun 21 2022 14:15:59 +00:00 cat3k_caa-webui.16.12.07.SPA.pkg

38746 -rw- 4742 Sep 24 2021 12:56:57 +00:00 packages.conf

38750 -rw- 460 Sep 24 2021 12:59:02 +00:00 .packages.verify.log

38741 -rw- 62276 Sep 24 2021 13:15:09 +00:00 pnp-tech-discovery-summary

38722 -rw- 1936 Jun 2 2022 02:31:17 +00:00 vlan.dat

38758 -rw- 3414660 Jun 21 2022 14:15:59 +00:00 cat3k_caa-srdriver.16.12.07.SPA.pkg

38754 -rw- 4780 Jun 21 2022 07:59:49 +00:00 cat3k_caa-universalk9.16.12.07.SPA.conf

38731 -rw- 4780 Jun 21 2022 13:38:24 +00:00 cat3k_caa-universalk9.16.12.07.SPA.3.conf

38728 -rw- 4780 Jun 21 2022 14:16:44 +00:00 cat3k_caa-universalk9.16.12.07.SPA.4.conf

Leo Laohoo · ‎06-21-2022

@johnlloyd_13 wrote:

i saw there's currently a 'packages.conf' file. i suspect this is from 16.6.8.

Based on my experience, you are 100% correct. The date-and-time stamp of flash-2:packages.conf points to 16.6.8.

@johnlloyd_13 wrote:

will there be any issue/conflict?

No, this is a very well-known bug and the developers will not be fixing it because of DNAC.

The workaround is just as simple:

rename flash-2:packages.conf flash-2:packages.conf.00-
rename flash-2:cat3k_caa-universalk9.16.12.07.SPA.4.conf flash-2:packages.conf

Check and verify the contents of the packages.conf file are pointing to the new packages.

Check the boot-variable string is pointing to packages.conf (and not package.conf, another bug).

Reboot.

After the reboot, clean up the old files or else the switch will constantly spam the logs about not enough flash space.

Anyone who is reading this response, please refer to Cisco 3850: IOS-XE/Firmware Upgrade and scroll down to the GOTCHA! section.

johnlloyd_13 · ‎06-21-2022

hi leo,

i managed to rename packages.conf in flash-2.

i now see packages.conf points to 16.12.7 files. can i issue a reload at this point?

#dir flash-2:

Directory of flash-2:/

38753 -rw- 2097152 Jun 21 2022 11:57:40 +00:00 nvram_config

38762 -rw- 32795564 Jun 21 2022 14:16:08 +00:00 cat3k_caa-rpbase.16.12.07.SPA.pkg

38723 -rw- 5 Jun 21 2022 14:17:27 +00:00 .issu_loc_lock

38746 -rw- 4742 Sep 24 2021 12:56:57 +00:00 packages-old.conf

38730 -rw- 4780 Jun 21 2022 11:46:59 +00:00 cat3k_caa-universalk9.16.12.07.SPA.1.conf

38754 -rw- 4780 Jun 21 2022 07:59:49 +00:00 packages.conf

38727 -rw- 4780 Jun 21 2022 12:59:49 +00:00 cat3k_caa-universalk9.16.12.07.SPA.2.conf

38729 -rw- 76 Jun 2 2022 02:24:17 +00:00 boothelper.log

38732 drwx 4096 May 5 2015 09:45:44 +00:00 virtual-instance

38735 drwx 4096 Sep 24 2021 12:44:13 +00:00 core

38755 -rw- 20097668 Jun 21 2022 14:15:24 +00:00 cat3k_caa-guestshell.16.12.07.SPA.pkg

77441 drwx 4096 May 5 2015 09:45:44 +00:00 vman_fdb

38751 drwx 4096 Oct 6 2021 05:00:30 +00:00 onep

85186 drwx 4096 Jun 2 2022 02:24:15 +00:00 dc_profile_dir

85188 drwx 4096 Mar 28 2017 20:29:45 +00:00 wnweb_store

38747 -rw- 19167872 Sep 24 2021 12:56:02 +00:00 cat3k_caa-guestshell.16.06.08.SPA.pkg

38733 drwx 4096 Jun 21 2022 07:59:50 +00:00 .installer

38734 -rw- 1147 Jun 2 2022 02:23:58 +00:00 bootloader_evt_handle.log

38736 drwx 4096 Sep 24 2021 13:05:19 +00:00 .prst_sync

14 drwx 4096 Sep 24 2021 12:58:49 +00:00 .rollback_timer

7747 drwx 4096 Sep 24 2021 12:40:09 +00:00 gs_script

38737 -rw- 128103 Jun 2 2022 02:24:36 +00:00 memleak.tcl

38738 -rw- 35 Sep 24 2021 13:15:06 +00:00 pnp-tech-time

38740 -rw- 2097152 Jun 21 2022 11:57:41 +00:00 nvram_config_bkup

38749 -rw- 30417560 Sep 24 2021 12:56:26 +00:00 cat3k_caa-rpbase.16.06.08.SPA.pkg

38757 -rw- 400857724 Jun 21 2022 14:15:57 +00:00 cat3k_caa-rpcore.16.12.07.SPA.pkg

77443 -rw- 5091968 Sep 24 2021 12:56:24 +00:00 cat3k_caa-srdriver.16.06.08.SPA.pkg

38756 -rw- 23018112 Jun 21 2022 14:15:59 +00:00 cat3k_caa-webui.16.12.07.SPA.pkg

38750 -rw- 460 Sep 24 2021 12:59:02 +00:00 .packages.verify.log

38741 -rw- 62276 Sep 24 2021 13:15:09 +00:00 pnp-tech-discovery-summary

38722 -rw- 1936 Jun 2 2022 02:31:17 +00:00 vlan.dat

38758 -rw- 3414660 Jun 21 2022 14:15:59 +00:00 cat3k_caa-srdriver.16.12.07.SPA.pkg

38731 -rw- 4780 Jun 21 2022 13:38:24 +00:00 cat3k_caa-universalk9.16.12.07.SPA.3.conf

38728 -rw- 4780 Jun 21 2022 14:16:44 +00:00 cat3k_caa-universalk9.16.12.07.SPA.4.conf

1622147072 bytes total (660078592 bytes free)

#more flash-2:packages.conf | b for NOVA

# This is for NOVA

iso rp 0 0 rp_base cat3k_caa-rpbase.16.12.07.SPA.pkg

iso rp 0 0 rp_core cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 rp_webui cat3k_caa-webui.16.12.07.SPA.pkg

iso rp 0 0 srdriver cat3k_caa-srdriver.16.12.07.SPA.pkg

iso rp 0 0 rp_security cat3k_caa-rpcore.16.12.07.SPA.pkg

iso rp 0 0 guestshell cat3k_caa-guestshell.16.12.07.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.12.07.SPA.pkg

#

# -start- superpackage .pkginfo

#

# pkginfo: Name: rp_super

# pkginfo: BuildTime: 2022-02-02_07.35

# pkginfo: ReleaseDate: Wed-02-Feb-22-04:05

# pkginfo: .BuildArch: mips64

# pkginfo: BootArchitecture: mips

# pkginfo: .BootArch: mips

# pkginfo: RouteProcessor: edison

# pkginfo: Platform: CAT3K_CAA

# pkginfo: User: mcpre

# pkginfo: PackageName: universalk9

# pkginfo: Build: 16.12.07

# pkginfo: .SupportedBoards: edison

# pkginfo: .InstallModel:

# pkginfo: .PackageRole: rp_super

# pkginfo: .RestartRole: rp_super

# pkginfo: CardTypes:

# pkginfo: .CardTypes:

# pkginfo: .BuildPath: /nobackup/mcpre/release/BLD-V16_12_07_FC2/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64

# pkginfo: .Version: 16.12.07.0.6565.1643816123..Gibraltar

# pkginfo: .InstallVersion: 1.0.0

# pkginfo: .InstallCapCommitSupport: yes

#

# -end- superpackage .pkginfo

#

#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable = flash:packages.conf
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0

Leo Laohoo · ‎06-21-2022

That looks good to reboot, @johnlloyd_13.

Just a reminder/warning, going to 16.12.7 will include a micro-code upgrade. Give it 14 minutes to complete the reboot.

Let us know how you go.