Solved: Re: C9200L-24T-4X downgraded itself after upgrade

support1@lima.co.uk · ‎06-25-2021

Hi,

I have just upgraded a C9200L-24T-4X from cat9k_lite_iosxe.16.12.04.SPA.bin to cat9k_lite_iosxe.16.12.05b.SPA.bin

Using the steps below.

After the upgrade the switch was online and operating the new firmware as expected. Then around the hour mark it rebooted and defaulted back to the old IOS.

boot system flash:packages.conf this command instructs the switch to boot from the package files that came with the firmware

Write mem to save

Show boot system Use this command to verify the boot variable is set to flash:packages.conf

install add file flash: cat9k_lite_iosxe.16.12.05b.SPA.bin

activate commit this instructs the switch stack to upgrade to the new firmware

When complete, This will prompt you for a reboot to be allowed, type yes

Once reboot is complete, type show version to confirm previous firmware has been applied to all members of the stack.

Write mem to save.

After logging off I had some alerts come in. The switch had reload itself and defaulted back to the previous version.

Switch uptime : 11 hours, 19 minutes

Base Ethernet MAC Address : xxxxxxxx
Motherboard Assembly Number : 73-19244-04
Motherboard Serial Number : xxxxxxxxx
Model Revision Number : A0
Motherboard Revision Number : A0
Model Number : C9200L-24T-4X
System Serial Number : xxxxxxxxx
Last reload reason : Image Install Abort

I have checked the dir and it is showing both pkg files.

Directory of flash:/

72866 -rw- 463474829 Jun 24 2021 19:10:35 +00:00 cat9k_lite_iosxe.16.12.05b.SPA.bin
72867 drwx 4096 Jun 24 2021 21:27:53 +00:00 .installer
72869 -rw- 407 Jun 24 2021 21:27:55 +00:00 bootloader_evt_handle.log
72870 drwx 4096 Jun 24 2021 19:26:57 +00:00 core
97153 drwx 4096 Jun 24 2021 21:29:26 +00:00 .prst_sync
113345 drwx 4096 Jun 24 2021 21:24:22 +00:00 .rollback_timer
11 drwx 4096 Jun 24 2021 22:00:09 +00:00 tech_support
8098 drwx 4096 Jun 24 2021 21:28:13 +00:00 dc_profile_dir
72868 -rw- 2131 Jun 24 2021 21:28:15 +00:00 boothelper.log
72872 -rw- 132108 Jun 24 2021 21:28:27 +00:00 memleak.tcl
72874 drwx 4096 Feb 13 2019 00:56:38 +00:00 .dbpersist
8100 drwx 4096 Feb 13 2019 00:55:58 +00:00 onep
72880 -rw- 2097152 Jun 24 2021 21:29:37 +00:00 nvram_config
72877 -rw- 2097152 Jun 24 2021 21:29:37 +00:00 nvram_config_bkup
72871 -rw- 2137 Jun 24 2021 19:29:52 +00:00 boothelper.log.old
72876 -rw- 856 Aug 15 2020 13:24:17 +00:00 vlan.dat
72884 -rw- 856 Jun 24 2021 21:31:48 +00:00 stby-vlan.dat
72890 -rw- 4883 Jun 24 2021 21:25:48 +00:00 packages.conf
72879 drwx 4096 Jun 24 2021 21:29:32 +00:00 license_evlog
72912 -rw- 0 Jun 24 2021 21:29:59 +00:00 dope_hist
72902 -rw- 391255060 Dec 20 2020 12:29:52 +00:00 cat9k_lite-rpbase.16.12.04.SPA.pkg
72905 -rw- 40633301 Dec 20 2020 12:30:33 +00:00 cat9k_lite-rpboot.16.12.04.SPA.pkg
72903 -rw- 4576276 Dec 20 2020 12:29:53 +00:00 cat9k_lite-srdriver.16.12.04.SPA.pkg
72904 -rw- 10171408 Dec 20 2020 12:29:54 +00:00 cat9k_lite-webui.16.12.04.SPA.pkg
72899 -rw- 862 Jun 24 2021 21:29:59 +00:00 rdope_out.txt
72911 -rw- 89 Jun 24 2021 21:29:56 +00:00 rdope.log
72893 -rw- 391455768 Jun 24 2021 19:15:15 +00:00 cat9k_lite-rpbase.16.12.05b.SPA.pkg
72913 -rw- 40602733 Jun 24 2021 19:16:12 +00:00 cat9k_lite-rpboot.16.12.05b.SPA.pkg
72894 -rw- 21500952 Jun 24 2021 19:15:16 +00:00 cat9k_lite-srdriver.16.12.05b.SPA.pkg
72895 -rw- 10191896 Jun 24 2021 19:15:17 +00:00 cat9k_lite-webui.16.12.05b.SPA.pkg
72892 -rw- 4894 Jun 24 2021 19:16:12 +00:00 cat9k_lite_iosxe.16.12.05b.SPA.conf

Any ideas what would cause this?

balaji.bandi · ‎06-25-2021

copy flash:/packages.conf flash:/packages.back

copy flash:/cat9k_lite_iosxe.16.12.05b.SPA.conf flash:/packages.conf

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

View solution in original post

Sergiu.Daniluk · ‎06-25-2021

~~You are in the wrong community section. This is for Nexus Dashboard. You should ask the question in Switching.~~

Edit: post was moved to appropriate subsection.

Stay safe,

Sergiu

balaji.bandi · ‎06-25-2021

#more flash:packages.conf

This file might be pointing to OLD images still i guess. rename that in to different name

move cat9k_lite_iosxe.16.12.05b.SPA.conf to packages.conf and test it.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Leo Laohoo · ‎06-25-2021

support1@lima.co.uk wrote:

boot system flash:packages.conf

Post the complete output to the following commands:

sh version
sh boot
more flash-1:packages.conf (page two only)

I have a feeling there is another boot variable statement ABOVE the packages.conf.

support1@lima.co.uk · ‎06-25-2021

L1C9KSTK01-P#
L1C9KSTK01-P#
L1C9KSTK01-P#show ver
L1C9KSTK01-P#show version
Cisco IOS XE Software, Version 16.12.04
Cisco IOS Software [Gibraltar], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 16.12.4, RELEASE SOFTWARE (fc5)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 09-Jul-20 19:31 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.1r [FC7], RELEASE SOFTWARE (P)

L1C9KSTK01-P uptime is 13 hours, 13 minutes
Uptime for this control processor is 13 hours, 15 minutes
System returned to ROM by Image Install Abort
System image file is "flash:packages.conf"
Last reload reason: Image Install Abort

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-essentials Smart License network-essentials
dna-essentials Subscription Smart License dna-essentials

Smart Licensing Status: REGISTERED/AUTHORIZED

cisco C9200L-24T-4X (ARM64) processor with 521022K/3071K bytes of memory.
Processor board ID JAE230709GV
2 Virtual Ethernet interfaces
48 Gigabit Ethernet interfaces
8 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
2015268K bytes of physical memory.
819200K bytes of Crash Files at crashinfo:.
819200K bytes of Crash Files at crashinfo-2:.
1941504K bytes of Flash at flash:.
1941504K bytes of Flash at flash-2:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 6c:8b:d3:53:f4:00
Motherboard Assembly Number : 73-19244-04
Motherboard Serial Number : JAE230709GV
Model Revision Number : A0
Motherboard Revision Number : A0
Model Number : C9200L-24T-4X
System Serial Number : JAE230709GV

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 C9200L-24T-4X 16.12.4 CAT9K_LITE_IOSXE INSTALL
2 28 C9200L-24T-4X 16.12.4 CAT9K_LITE_IOSXE INSTALL

Switch 02
---------
Switch uptime : 13 hours, 15 minutes

Base Ethernet MAC Address : 6c:8b:d3:6e:67:00
Motherboard Assembly Number : 73-19244-04
Motherboard Serial Number : JAE230709B0
Model Revision Number : A0
Motherboard Revision Number : A0
Model Number : C9200L-24T-4X
System Serial Number : JAE230709B0
Last reload reason : Image Install Abort

Configuration register is 0x102

L1C9KSTK01-P#
L1C9KSTK01-P#
L1C9KSTK01-P#
L1C9KSTK01-P#show bo
L1C9KSTK01-P#show boot
---------------------------
Switch 1
---------------------------
Current Boot Variables:
BOOT variable = flash:packages.conf;

Boot Variables on next reload:
BOOT variable = flash:packages.conf;
Manual Boot = no
Enable Break = no
Boot Mode = DEVICE
iPXE Timeout = 0
L1C9KSTK01-P#
L1C9KSTK01-P#
L1C9KSTK01-P#
L1C9KSTK01-P#mor
L1C9KSTK01-P#more fl
L1C9KSTK01-P#more flash-1
L1C9KSTK01-P#more flash-1:pa
L1C9KSTK01-P#more flash-1:packages.conf
#! /usr/binos/bin/packages_conf.sh

sha1sum: 378148a671da4ea15d2ae653a71b4053e4a29b90

# sha1sum above - used to verify that this file is not corrupted.

#
# package.conf: provisioned software file for build 2020-07-09_19.43
#

# NOTE: Editing this file by hand is not recommended. It is generated
# as part of the build process, and is subject to boot-time
# consistency checks. Automatically-produced package files are
# guaranteed to pass those checks. Manually-maintained ones are
# not. Because "nfs" and "mount" directives are processed first,
# regardless of their position in the file, the recommended
# approach is to keep a separate file containing JUST your
# personal "nfs" and "mount" directives, and to append it to the
# automatically-generated file.
#
# Note further that when SHA-1 checksum verification is enabled,
# you will NOT be able to alter this file without updating the
# SHA-1 sum.

#
# This file can contain three types of entries:
#

#
# NFS directives (optional)
# notes: NFS directives are processed before all others (mount, iso).
# Multiple NFS directives may appear so long as they do not
# conflict -- that is, specify the same source or mountpoint.
# syntax: nfs <IP ADDRESS>:<REMOTE_PATH> <LOCAL_MOUNTPOINT>
# example: nfs 127.0.0.1:/auto/some/nfs/path /auto/some/nfs/path
#

#
# mount directives (optional)
# notes: mount directives are processed after 'nfs' and before 'iso'.
# One mount directive may appear for each F/S/B/P tuple
# syntax: mount FRU SLOT BAY PACKAGE_NAME LINUX_PATH
# example: mount rp 0 0 rp_base /auto/some/nfs/path/abs_soft/rp_base.ppc
#
# The specified LINUX_PATH may be local [sata disk] or
# reference the NFS mounts since they are processed first.
#
# Mount directives cause the package-specific mount link to
# be set to the specified path instead of to the mountpoint
# in sw for the corresponding ISO.
#

#
# iso directives (mandatory)
# notes: iso directives are processed last: any package for which
# a 'mount' directive does not appear will be mounted.
# One iso directive may appear for each F/S/B/P tuple.
# syntax: iso FRU SLOT BAY PACKAGE_NAME PACKAGE_FILE.bin
# example: iso rp 0 0 rp_base rp_base.ppc.bin
#
# PACKAGE_FILE.bin is a path relative to the packages.conf
# file. Although it supports sub-directories for development
# purposes, in deployment the files will always be managed
# as in the same directory as packages.conf so as to
# guarantee that name collisions cannot occur.
#
# Note that the RP 0/1 distinction is a convenience for development
# and testing as it allows us to have a packages.conf describe a
# SW load that varies depending on whether the RP finds itself in
# slot 0 or 1.
#
# The ISSU process *must* update *both* RP slots simultaneously so that
# the RP will behave predictably whichever slot it finds itself on [e.g.,
# if package X is upgraded, and the RP is ejected and put into either
# slot of a new chassis, we expect to see the upgraded X without regard
# to slot].
#
boot rp 0 0 rp_boot cat9k_lite-rpboot.16.12.04.SPA.pkg
iso rp 0 0 rp_base cat9k_lite-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_daemons cat9k_lite-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_iosd cat9k_lite-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_security cat9k_lite-rpbase.16.12.04.SPA.pkg
iso rp 0 0 rp_webui cat9k_lite-webui.16.12.04.SPA.pkg
iso rp 0 0 srdriver cat9k_lite-srdriver.16.12.04.SPA.pkg
iso fp 0 0 fp cat9k_lite-rpbase.16.12.04.SPA.pkg
iso cc 0 0 cc cat9k_lite-rpbase.16.12.04.SPA.pkg
iso cc 0 0 cc_srdriver cat9k_lite-srdriver.16.12.04.SPA.pkg

#
# -start- superpackage .pkginfo
#
# pkginfo: Name: rp_super
# pkginfo: BuildTime: 2020-07-09_19.43
# pkginfo: ReleaseDate: Thu-09-Jul-20-15:11
# pkginfo: .BuildArch: arm32
# pkginfo: BootArchitecture: arm32
# pkginfo: .BootArch: arm32
# pkginfo: RouteProcessor: quake
# pkginfo: Platform: CAT9K_LITE
# pkginfo: User: mcpre
# pkginfo: PackageName: universalk9
# pkginfo: Build: 16.12.04
# pkginfo: .SupportedBoards: quake
# pkginfo: .InstallModel:
# pkginfo: .PackageRole: rp_super
# pkginfo: .RestartRole: rp_super
# pkginfo: .UnifiedPlatformList: quake1,quake2
# pkginfo: CardTypes:
# pkginfo: .CardTypes:
# pkginfo: .BuildPath: /scratch/mcpre/release/BLD-V16_12_04_FC5/binos/linkfarm/stage-quake/hard/rp_super_universalk9.arm32
# pkginfo: .Version: 16.12.4.0.4480.1594349007..Gibraltar
# pkginfo: .InstallVersion: 1.0.0
# pkginfo: .InstallCapCommitSupport: yes
#
# -end- superpackage .pkginfo
#

L1C9KSTK01-P#

balaji.bandi · ‎06-25-2021

Packages.conf has still old IOS XE image :

rp_boot cat9k_lite-rpboot.16.12.04.SPA.pkg

if you do more same way with cat9k_lite_iosxe.16.12.05b.SPA.conf - you see laest version

i take the backup of old packages.conf and move cat9k_lite_iosxe.16.12.05b.SPA.conf to packages conf. save and reload and test it.

Once it boot to new, clean up the install to remove old files.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

support1@lima.co.uk · ‎06-25-2021

Whats the commands for moving the new ios to the packages.conf and then removing the old ones?

balaji.bandi · ‎06-25-2021

copy flash:/packages.conf flash:/packages.back

copy flash:/cat9k_lite_iosxe.16.12.05b.SPA.conf flash:/packages.conf

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Leo Laohoo · ‎06-25-2021

support1@lima.co.uk wrote:

cat9k_lite_iosxe.16.12.05b.SPA.conf

Ok, I know how to fix this. Easy.

copy flash-1:packages.conf flash-1:packages.conf.00-
copy flash-2:packages.conf flash-2:packages.conf.00-
copy flash-2:cat9k_lite_iosxe.16.12.05b.SPA.conf flash-2:packages.conf
copy flash-1:cat9k_lite_iosxe.16.12.05b.SPA.conf flash-1:packages.conf

Ok, this is done, inspect each "packages.conf" file and make sure they are pointing to 16.12.5b.

Once confirm, reboot the stack.

support1@lima.co.uk · ‎06-25-2021

Thanks for this, what is the best command to inspect the packages.conf once completed.

Leo Laohoo · ‎06-25-2021

support1@lima.co.uk wrote:

Thanks for this, what is the best command to inspect the packages.conf once completed.

more flash-1:packages.conf
more flash-2:packages.conf