Downgrading 3650 from 16.3.8 to 16.3.3 after a previous unpacking to 16.3.8? (filesystem errors)

JimFranklin6775 · ‎06-20-2019

Hello...we have some 3650 stacks that we're upgrading from 16.3.3 to 16.3.8. One stack had some large files on switch 4 that prevented the successful unpacking of the 16.3.8 bundle. In trying to resolve that by backing out of the unpack, I've come across an issue. Running:

request platform software package clean switch all (while still running the 16.3.3 code)

and also renaming all the packages.conf files, trying to reinstall the 16.3.8 bundle using:

request platform software package install switch all file flash:old/cat3k_caa-universalk9.16.03.08.SPA.bin auto-copy

generated the following error on all 4 switches:

FAILED: A previous provisioning action is pending reboot

I get zero hits on Google for that error message, thus I'm here :-) It seems some flag is set that tells the system I've already "scheduled" an upgrade to 16.3.8, even though I did a "no boot system" and "wr mem". What am I missing in getting the stack to accept the unpacking? I'm 90% sure if I reloaded, it would work, since I once copied the packages.conf file to a switch 4 in another site and ignored the error, and it reloaded fine. But it concerns me that the error exists. Also the site in question is 100 miles away and I don't want to brick it and drive there.

thanks,

jim

nazimkha · ‎06-20-2019

Can you paste the output of the following
show boot
dir flash-<switch-no>:

more packages.conf

JimFranklin6775 · ‎06-20-2019

Here you go!

SPR3550M-01CSC01#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable does not exist

Boot Variables on next reload:
BOOT variable does not exist
Manual Boot = no
Enable Break = no
Boot Mode= DEVICE
iPXE Timeout= 0

SPR3550M-01CSC01#dir flash-1:
Directory of flash:/

127234 -rw- 15954560 Dec 31 1969 19:01:45 -05:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
127235 -rw- 22173639 Dec 31 1969 19:01:45 -05:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
127236 -rw- 264899192 Dec 31 1969 19:01:46 -05:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
127237 -rw- 9091712 Dec 31 1969 19:01:46 -05:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
127238 -rw- 191324788 Dec 31 1969 19:01:46 -05:00 cat3k_caa-wcm.16.03.03.SPA.pkg
127239 -rw- 13404796 Dec 31 1969 19:01:46 -05:00 cat3k_caa-webui.16.03.03.SPA.pkg
127250 drwx 4096 Jun 20 2019 11:07:13 -04:00 old
127241 -rw- 259 Apr 23 2019 08:37:40 -04:00 bootloader_evt_handle.log
127242 drwx 4096 Oct 7 2017 16:05:10 -04:00 core
166993 drwx 4096 Oct 7 2017 15:59:43 -04:00 .prst_sync
182897 drwx 4096 Oct 7 2017 15:59:45 -04:00 .rollback_timer
190849 drwx 4096 Apr 23 2019 08:37:54 -04:00 dc_profile_dir
190851 drwx 4096 Oct 7 2017 15:59:51 -04:00 gs_script
127243 -rw- 65301 Apr 23 2019 08:38:08 -04:00 memleak.tcl
127244 drwx 4096 Oct 7 2017 16:00:04 -04:00 .installer
127245 -rw- 2097152 Jun 20 2019 11:02:58 -04:00 nvram_config
127247 -rw- 35 Apr 18 2018 15:37:13 -04:00 pnp-tech-time
127248 -rw- 101649 Apr 18 2018 15:37:16 -04:00 pnp-tech-discovery-summary
127249 -rw- 856 Apr 23 2019 08:41:33 -04:00 vlan.dat
127240 -rw- 0 Jun 19 2019 17:53:38 -04:00 .onreboot
127254 -rw- 4737 Jun 20 2019 07:28:32 -04:00 packages.conf.old
127261 -rw- 15938176 Jun 20 2019 11:06:47 -04:00 cat3k_caa-guestshell.16.03.08.SPA.pkg
127266 -rw- 22306936 Jun 20 2019 11:07:13 -04:00 cat3k_caa-rpbase.16.03.08.SPA.pkg
127262 -rw- 264563320 Jun 20 2019 11:06:51 -04:00 cat3k_caa-rpcore.16.03.08.SPA.pkg
127264 -rw- 9671296 Jun 20 2019 11:06:53 -04:00 cat3k_caa-srdriver.16.03.08.SPA.pkg
127265 -rw- 209654388 Jun 20 2019 11:07:12 -04:00 cat3k_caa-wcm.16.03.08.SPA.pkg
127263 -rw- 13439612 Jun 20 2019 11:06:52 -04:00 cat3k_caa-webui.16.03.08.SPA.pkg
127260 -rw- 4737 Jun 20 2019 11:07:34 -04:00 packages.conf

SPR3550M-01CSC01#dir flash-2:
Directory of flash-2:/

143138 -rw- 15954560 Dec 31 1969 19:01:06 -05:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
143139 -rw- 22173639 Dec 31 1969 19:01:06 -05:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
143140 -rw- 264899192 Dec 31 1969 19:01:07 -05:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
143141 -rw- 9091712 Dec 31 1969 19:01:07 -05:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
143142 -rw- 191324788 Dec 31 1969 19:01:08 -05:00 cat3k_caa-wcm.16.03.03.SPA.pkg
143143 -rw- 13404796 Dec 31 1969 19:01:08 -05:00 cat3k_caa-webui.16.03.03.SPA.pkg
143156 drwx 4096 Jun 20 2019 11:07:13 -04:00 old
143145 -rw- 370 Apr 23 2019 04:36:22 -04:00 bootloader_evt_handle.log
143146 drwx 4096 Dec 29 2018 10:46:58 -05:00 core
182897 drwx 4096 Dec 29 2018 10:52:52 -05:00 .prst_sync
198801 drwx 4096 May 4 2017 11:37:30 -04:00 .rollback_timer
11 drwx 4096 Apr 23 2019 04:36:36 -04:00 dc_profile_dir
13 drwx 4096 May 4 2017 11:37:37 -04:00 gs_script
143147 -rw- 65301 Apr 23 2019 04:36:49 -04:00 memleak.tcl
143148 drwx 4096 May 4 2017 11:37:48 -04:00 .installer
143150 -rw- 2097152 Jun 20 2019 11:03:02 -04:00 nvram_config
143151 -rw- 35 Apr 18 2018 14:59:10 -04:00 pnp-tech-time
143152 -rw- 21276 Apr 18 2018 14:59:18 -04:00 pnp-tech-discovery-summary
143154 drwx 4096 Apr 18 2018 16:27:41 -04:00 onep
143153 -rw- 856 Apr 23 2019 08:44:50 -04:00 vlan.dat
143144 -rw- 0 Jun 19 2019 17:53:33 -04:00 .onreboot
23859 -rw- 4737 Jun 20 2019 07:28:32 -04:00 packages.conf.old
143165 -rw- 15938176 Jun 20 2019 11:06:49 -04:00 cat3k_caa-guestshell.16.03.08.SPA.pkg
143170 -rw- 22306936 Jun 20 2019 11:07:13 -04:00 cat3k_caa-rpbase.16.03.08.SPA.pkg
143166 -rw- 264563320 Jun 20 2019 11:06:52 -04:00 cat3k_caa-rpcore.16.03.08.SPA.pkg
143168 -rw- 9671296 Jun 20 2019 11:06:54 -04:00 cat3k_caa-srdriver.16.03.08.SPA.pkg
143169 -rw- 209654388 Jun 20 2019 11:07:11 -04:00 cat3k_caa-wcm.16.03.08.SPA.pkg
143167 -rw- 13439612 Jun 20 2019 11:06:53 -04:00 cat3k_caa-webui.16.03.08.SPA.pkg
143163 -rw- 4737 Jun 20 2019 11:07:35 -04:00 packages.conf

SPR3550M-01CSC01#dir flash-3:

Directory of flash-3:/

23286 -rw- 4737 Jun 20 2019 07:28:49 -04:00 packages.conf.old
23282 drwx 4096 Jun 20 2019 11:07:04 -04:00 old
23290 -rw- 4737 Jun 20 2019 11:07:49 -04:00 packages.conf
23299 -rw- 15938176 Jun 20 2019 11:06:51 -04:00 cat3k_caa-guestshell.16.03.08.SPA.pkg
23302 -rw- 22306936 Jun 20 2019 11:07:04 -04:00 cat3k_caa-rpbase.16.03.08.SPA.pkg
23306 -rw- 264563320 Jun 20 2019 11:06:57 -04:00 cat3k_caa-rpcore.16.03.08.SPA.pkg
46561 drwx 4096 Sep 29 2017 12:17:30 -04:00 .installer
23288 -rw- 259 Apr 23 2019 08:37:45 -04:00 bootloader_evt_handle.log
54321 drwx 4096 Jun 19 2019 18:16:26 -04:00 core
62081 drwx 4096 Sep 29 2017 12:29:17 -04:00 .prst_sync
77601 drwx 4096 Sep 29 2017 12:18:09 -04:00 .rollback_timer
85362 drwx 4096 Apr 23 2019 08:38:06 -04:00 dc_profile_dir
85364 drwx 4096 Sep 29 2017 12:18:18 -04:00 gs_script
23289 -rw- 65301 Apr 23 2019 08:38:23 -04:00 memleak.tcl
23283 -rw- 2097152 Jun 20 2019 11:02:58 -04:00 nvram_config
23291 -rw- 2097152 Sep 29 2017 12:38:07 -04:00 nvram_config_bkup
23292 -rw- 35 Sep 29 2017 12:30:00 -04:00 pnp-tech-time
23293 -rw- 45777 Sep 29 2017 12:30:08 -04:00 pnp-tech-discovery-summary
23294 -rw- 15954560 Dec 31 1969 19:01:06 -05:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
23295 -rw- 22173639 Dec 31 1969 19:01:45 -05:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
23296 -rw- 264899192 Dec 31 1969 19:01:07 -05:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
23297 -rw- 9091712 Dec 31 1969 19:01:46 -05:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
23298 -rw- 191324788 Dec 31 1969 19:01:46 -05:00 cat3k_caa-wcm.16.03.03.SPA.pkg
15521 -rw- 13404796 Dec 31 1969 19:01:46 -05:00 cat3k_caa-webui.16.03.03.SPA.pkg
15525 -rw- 856 Apr 23 2019 08:41:33 -04:00 vlan.dat
23301 -rw- 9671296 Jun 20 2019 11:06:57 -04:00 cat3k_caa-srdriver.16.03.08.SPA.pkg
23307 -rw- 209654388 Jun 20 2019 11:07:04 -04:00 cat3k_caa-wcm.16.03.08.SPA.pkg
23300 -rw- 13439612 Jun 20 2019 11:06:57 -04:00 cat3k_caa-webui.16.03.08.SPA.pkg
15522 -rw- 0 Jun 19 2019 17:53:43 -04:00 .onreboot

SPR3550M-01CSC01#dir flash-4:
Directory of flash-4:/

15526 -rw- 4737 Jun 20 2019 07:28:49 -04:00 packages.conf.old
15522 drwx 4096 Jun 20 2019 11:07:04 -04:00 old
15535 -rw- 4737 Jun 20 2019 11:07:48 -04:00 packages.conf
15543 -rw- 15938176 Jun 20 2019 11:06:51 -04:00 cat3k_caa-guestshell.16.03.08.SPA.pkg
23282 -rw- 22306936 Jun 20 2019 11:07:04 -04:00 cat3k_caa-rpbase.16.03.08.SPA.pkg
15552 -rw- 264563320 Jun 20 2019 11:07:00 -04:00 cat3k_caa-rpcore.16.03.08.SPA.pkg
15528 drwx 4096 Sep 29 2017 12:42:14 -04:00 .installer
15529 -rw- 259 Apr 23 2019 08:37:42 -04:00 bootloader_evt_handle.log
15530 drwx 4096 Jun 19 2019 18:18:10 -04:00 core
46561 drwx 4096 Sep 29 2017 12:53:55 -04:00 .prst_sync
62081 drwx 4096 Sep 29 2017 12:42:52 -04:00 .rollback_timer
69841 drwx 4096 Apr 23 2019 08:38:03 -04:00 dc_profile_dir
69843 drwx 4096 Sep 29 2017 12:43:01 -04:00 gs_script
15531 -rw- 65301 Apr 23 2019 08:38:21 -04:00 memleak.tcl
15523 -rw- 2097152 Jun 20 2019 11:02:58 -04:00 nvram_config
15536 -rw- 2097152 Sep 29 2017 13:02:51 -04:00 nvram_config_bkup
15532 -rw- 35 Sep 29 2017 12:54:39 -04:00 pnp-tech-time
15533 -rw- 45567 Sep 29 2017 12:54:46 -04:00 pnp-tech-discovery-summary
15537 -rw- 15954560 Dec 31 1969 19:01:06 -05:00 cat3k_caa-guestshell.16.03.03.SPA.pkg
15538 -rw- 22173639 Dec 31 1969 19:01:45 -05:00 cat3k_caa-rpbase.16.03.03.SPA.pkg
15539 -rw- 264899192 Dec 31 1969 19:01:07 -05:00 cat3k_caa-rpcore.16.03.03.SPA.pkg
15540 -rw- 9091712 Dec 31 1969 19:01:46 -05:00 cat3k_caa-srdriver.16.03.03.SPA.pkg
15541 -rw- 191324788 Dec 31 1969 19:01:46 -05:00 cat3k_caa-wcm.16.03.03.SPA.pkg
15542 -rw- 13404796 Dec 31 1969 19:01:46 -05:00 cat3k_caa-webui.16.03.03.SPA.pkg
15544 -rw- 856 Apr 23 2019 08:41:33 -04:00 vlan.dat
15546 -rw- 9671296 Jun 20 2019 11:07:01 -04:00 cat3k_caa-srdriver.16.03.08.SPA.pkg
15553 -rw- 209654388 Jun 20 2019 11:07:04 -04:00 cat3k_caa-wcm.16.03.08.SPA.pkg
15545 -rw- 13439612 Jun 20 2019 11:07:01 -04:00 cat3k_caa-webui.16.03.08.SPA.pkg
15527 -rw- 0 Jun 19 2019 17:53:43 -04:00 .onreboot

SPR3550M-01CSC01#more packages.conf
#! /usr/binos/bin/packages_conf.sh

sha1sum: 3a3d10a164890a92d9d4b572616378d30a62f631

# sha1sum above - used to verify that this file is not corrupted.

#
# package.conf: provisioned software file for build 2019-02-13_03.31
#

# NOTE: Editing this file by hand is not recommended. It is generated
# as part of the build process, and is subject to boot-time
# consistency checks. Automatically-produced package files are
# guaranteed to pass those checks. Manually-maintained ones are
# not. Because "nfs" and "mount" directives are processed first,
# regardless of their position in the file, the recommended
# approach is to keep a separate file containing JUST your
# personal "nfs" and "mount" directives, and to append it to the
# automatically-generated file.
#
# Note further that when SHA-1 checksum verification is enabled,
# you will NOT be able to alter this file without updating the
# SHA-1 sum.

#
# This file can contain three types of entries:
#

#
# NFS directives (optional)
# notes: NFS directives are processed before all others (mount, iso).
# Multiple NFS directives may appear so long as they do not
# conflict -- that is, specify the same source or mountpoint.
# syntax: nfs <IP ADDRESS>:<REMOTE_PATH> <LOCAL_MOUNTPOINT>
# example: nfs 127.0.0.1:/auto/some/nfs/path /auto/some/nfs/path
#

#
# mount directives (optional)
# notes: mount directives are processed after 'nfs' and before 'iso'.
# One mount directive may appear for each F/S/B/P tuple
# syntax: mount FRU SLOT BAY PACKAGE_NAME LINUX_PATH
# example: mount rp 0 0 rp_base /auto/some/nfs/path/abs_soft/rp_base.ppc
#
# The specified LINUX_PATH may be local [sata disk] or
# reference the NFS mounts since they are processed first.
#
# Mount directives cause the package-specific mount link to
# be set to the specified path instead of to the mountpoint
# in sw for the corresponding ISO.
#

#
# iso directives (mandatory)
# notes: iso directives are processed last: any package for which
# a 'mount' directive does not appear will be mounted.
# One iso directive may appear for each F/S/B/P tuple.
# syntax: iso FRU SLOT BAY PACKAGE_NAME PACKAGE_FILE.bin
# example: iso rp 0 0 rp_base rp_base.ppc.bin
#
# PACKAGE_FILE.bin is a path relative to the packages.conf
# file. Although it supports sub-directories for development
# purposes, in deployment the files will always be managed
# as in the same directory as packages.conf so as to
# guarantee that name collisions cannot occur.
#
# Note that the RP 0/1 distinction is a convenience for development
# and testing as it allows us to have a packages.conf describe a
# SW load that varies depending on whether the RP finds itself in
# slot 0 or 1.
#
# The ISSU process *must* update *both* RP slots simultaneously so that
# the RP will behave predictably whichever slot it finds itself on [e.g.,
# if package X is upgraded, and the RP is ejected and put into either
# slot of a new chassis, we expect to see the upgraded X without regard
# to slot].
#

iso rp 0 0 rp_base cat3k_caa-rpbase.16.03.08.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 rp_wcm cat3k_caa-wcm.16.03.08.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.03.08.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.03.08.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.03.08.SPA.pkg

iso fp 0 0 fp cat3k_caa-rpcore.16.03.08.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime:
# pkginfo: ReleaseDate: Wed-13-Feb-19-07:00
# pkginfo: .BuildArch: mips64
# pkginfo: BootArchitecture: mips
# pkginfo: .BootArch: mips
# pkginfo: RouteProcessor: edison
# pkginfo: Platform: CAT3K_CAA
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.03.08
# pkginfo: .SupportedBoards: edison
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /auto/mcpbuilds16/release/16.03.08/BLD-V16_03_08_FC3/binos/linkfarm/stage-edison/hard/rp_super_universalk9.mips64
# pkginfo: .Version: 16.3.8.0.2042.1550057506..Denali
#
# -end- superpackage .pkginfo
#

Leo Laohoo · ‎06-21-2019

What is the uptime of the stack?
I saw a similar error message to a stack that had an uptime of >2 years. The solution was to reboot the entire stack to the same IOS and then do the upgrade script. And this method worked.

JimFranklin6775 · ‎06-21-2019

It's only been up 8 weeks. I've seen this on all the stacks where we didn't have a clean unpacking on 16.3.8 (I'm still learning the package system).

thanks,

jim

nazimkha · ‎06-21-2019

Looks like the packages.conf file is still based out of 16.3.8 (Assuming the more packages.conf output is from the switch under consideration)

For your query whether reboot will make the switch come up with 16.3.3 and the answer is NO.

Before rebooting the switch and ensuring it comes up with the correct version you need to verify two things

in the show boot output it should point to

Something like this

3850-SWITCH#sh boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable =

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Allow Dev Key = yes
Manual Boot = no
Enable Break = no
3850-SWITCH#
and when you more packages.conf it should show files for 16.3.3 (target version)

In your output it

iso rp 0 0 rp_base cat3k_caa-rpbase.16.03.08.SPA.pkg
iso rp 0 0 rp_core cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 rp_daemons cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 rp_iosd cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 rp_wcm cat3k_caa-wcm.16.03.08.SPA.pkg
iso rp 0 0 rp_webui cat3k_caa-webui.16.03.08.SPA.pkg
iso rp 0 0 srdriver cat3k_caa-srdriver.16.03.08.SPA.pkg
iso rp 0 0 rp_security cat3k_caa-rpcore.16.03.08.SPA.pkg
iso rp 0 0 guestshell cat3k_caa-guestshell.16.03.08.SPA.pkg

JimFranklin6775 · ‎06-21-2019

Right, because the last thing I did was unpack 16.3.8, so I expect it to look "ready" for that. However, my question is why it displayed the FAILED error in my initial post, and does that matter. What is flagging the fact that I previously unpacked something and it's ready to boot into that? Deleting packages.conf, and the setting the boot variable to nothing, did not erase the "flag". How do I get the stack back to where I can unpack 16.3.8 cleanly with no errors, and without having to reload to get it there?

thanks,

jim