Re: Reload - reason reload command on c9300

Netmart · ‎07-09-2022

Hello,

We noted reload of c9300.

When issuing :show version, the following appears:

System returned to ROM by Reload Command

System restarted at ***** EDT *****

System image file is "flash:packages.conf"

Last reload reason: Reload Command

Neither reload message in sylslog server nor any user activity were seen in aaa authentication logs at the reported time.

No crash info, but trace logs have been generated short after the reload took place.

Thanks,

Netmart

Leo Laohoo · ‎07-09-2022

@Netmart wrote:

Last reload reason: Reload Command

There is a very long list of IOS-XE bugs that give out "blank" reload reasons &/or does not leave any crashfinfo log file(s).

Kindly post the complete output to the following commands:

sh version
dir flash:
dir flash:core
dir crashinfo:
sh log on switch act active up detail

Netmart · ‎07-11-2022

Hello Leo,

This is the extract of "show log on switch active up detail":

07/05/2022 21:24:19 Reload 2 48 4 4 5

#dir flash:core
Directory of flash:/core/

385025 drwx 4096 Aug 27 2018 16:49:10 -04:00 modules
360450 -rw- 1 Jul 11 2022 23:54:52 -04:00 .callhome

dir flash: | i Jul
319490 -rwx 2097152 Jul 5 2022 17:24:46 -04:00 nvram_config
319491 -rwx 2097152 Jul 5 2022 17:24:46 -04:00 nvram_config_bkup
368657 -rw- 7554 Jul 31 2019 12:46:56 -04:00 packages.conf
352257 drwx 4096 Jul 5 2022 17:23:40 -04:00 .installer
368647 -rw- 222 Jul 5 2022 17:23:41 -04:00 bootloader_evt_handle.log
376833 drwx 4096 Jul 31 2019 12:54:55 -04:00 .prst_sync
393217 drwx 4096 Jul 31 2019 12:47:09 -04:00 .rollback_timer
409601 drwx 4096 Jul 5 2022 17:23:54 -04:00 dc_profile_dir
368648 -rw- 252 Jul 5 2022 17:23:55 -04:00 boothelper.log
368649 -rw- 123888 Jul 5 2022 17:24:05 -04:00 memleak.tcl
368650 drwx 60 Jul 5 2022 17:23:57 -04:00 NVRAM
368651 -rw- 35 Jul 30 2019 14:43:38 -04:00 pnp-tech-time
368652 -rw- 108423 Jul 30 2019 14:43:40 -04:00 pnp-tech-discovery-summary
368654 -rw- 796 Jul 5 2022 17:24:40 -04:00 vlan.dat
360453 -rw- 27149312 Jul 31 2019 12:33:56 -04:00 cat9k-cc_srdriver.16.09.03.SPA.pkg
352258 drwx 4096 Jul 31 2019 12:49:00 -04:00 tech_support
360454 -rw- 81187836 Jul 31 2019 12:33:58 -04:00 cat9k-espbase.16.09.03.SPA.pkg
360455 -rw- 1647612 Jul 31 2019 12:33:58 -04:00 cat9k-guestshell.16.09.03.SPA.pkg
360456 -rw- 422978552 Jul 31 2019 12:34:10 -04:00 cat9k-rpbase.16.09.03.SPA.pkg
360462 -rw- 34562710 Jul 31 2019 12:34:34 -04:00 cat9k-rpboot.16.09.03.SPA.pkg
360457 -rw- 28914684 Jul 31 2019 12:34:11 -04:00 cat9k-sipbase.16.09.03.SPA.pkg
360458 -rw- 54989816 Jul 31 2019 12:34:12 -04:00 cat9k-sipspa.16.09.03.SPA.pkg
360459 -rw- 34886652 Jul 31 2019 12:34:13 -04:00 cat9k-srdriver.16.09.03.SPA.pkg
360460 -rw- 15856632 Jul 31 2019 12:34:14 -04:00 cat9k-webui.16.09.03.SPA.pkg
360461 -rw- 9208 Jul 31 2019 12:34:14 -04:00 cat9k-wlc.16.09.03.SPA.pkg
360452 -rw- 7554 Jul 31 2019 12:34:34 -04:00 cat9k_iosxe.16.09.03.SPA.conf
368656 -rw- 6628 Jul 31 2019 12:46:56 -04:00 packages.conf.00-
368645 -rw- 720 Jul 5 2022 17:23:55 -04:00 fpgahelper.log
376834 drwx 4096 Jul 31 2019 12:54:58 -04:00 onep

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Extract fpagahelper - created close to reload
Wed Jul 31 16:49:00 Universal 2019 env set. starting auto upgrade
Wed Jul 31 16:49:01 Universal 2019 checking for secure boot rommon
checking if upgrade required: Strutt
FPGA is running version 0x00000206
Version_old 0x00000206
Version_new 0x00000206
fpga version is up to date, return
Wed Jul 31 16:49:01 Universal 2019 no upgrade required aborting upgrade

Tue Jul 5 21:23:54 Universal 2022 env set. starting auto upgrade
Tue Jul 5 21:23:55 Universal 2022 checking for secure boot rommon
checking if upgrade required: Strutt
FPGA is running version 0x00000206
Version_old 0x00000206
Version_new 0x00000206
fpga version is up to date, return
Tue Jul 5 21:23:55 Universal 2022 no upgrade required aborting upgrade

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

show version

Cisco IOS XE Software, Version 16.09.03
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Wed 20-Mar-19 08:02 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.9.1r [FC2], RELEASE SOFTWARE (P)

********* uptime is 6 days, 6 hours, 31 minutes
Uptime for this control processor is 6 days, 6 hours, 32 minutes
System returned to ROM by Reload Command
System restarted at *********************
System image file is "flash:packages.conf"
Last reload reason: Reload Command

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-advantage Smart License network-advantage
dna-advantage Subscription Smart License dna-advantage

Smart Licensing Status: REGISTERED/OUT OF COMPLIANCE

cisco C9300-48UXM (X86) processor with 1419496K/6147K bytes of memory.
Processor board ID ******
36 Ethernet interfaces
6 Virtual Ethernet interfaces
4 Gigabit Ethernet interfaces
20 Ten Gigabit Ethernet interfaces
2 TwentyFive Gigabit Ethernet interfaces
2 Forty Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
1638400K bytes of Crash Files at crashinfo:.
11264000K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : *******
Motherboard Assembly Number : *****
Motherboard Serial Number : *******
Model Revision Number : A0
Motherboard Revision Number : A0
Model Number : C9300-48UXM
System Serial Number : *****

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 64 C9300-48UXM 16.9.3 CAT9K_IOSXE INSTALL

Leo Laohoo · ‎07-11-2022

@Netmart wrote:

This is the extract of "show log on switch active up detail":

07/05/2022 21:24:19 Reload 2 48 4 4 5

I want to see the complete output. The command "sh log on switch active up detail" has a very long output. And the output you've provided does not match the other output.

I also want to see the complete output to the command "sh env power".

Netmart · ‎07-12-2022

Hello Leo,

Here we go:

sh environment power

SW PID Serial# Status Sys Pwr PoE Pwr Watts

-- ------------------ ---------- --------------- ------- ------- -----

1A PWR-C1-1100WAC LIT22203FGR OK Good Good 1100

1B PWR-C1-1100WAC LIT22203FEV OK Good Good 1100

sh log on switch active uptime detail

--------------------------------------------------------------------------------

UPTIME SUMMARY INFORMATION

--------------------------------------------------------------------------------

First customer power on : 08/13/2018 17:57:52

Total uptime : 2 years 49 weeks 4 days 22 hours 55 minutes

Total downtime : 2127 years 25 weeks 6 days 17 hours 50 minutes

Number of resets : 11

Number of slot changes : 2

Current reset reason : Gt^@^U]U

Current reset timestamp : 01/01/1970 09:06:06

Current slot : 6

Chassis type : 0

Current uptime : 0 years 0 weeks 6 days 18 hours 5 minutes

--------------------------------------------------------------------------------

UPTIME CONTINUOUS INFORMATION

--------------------------------------------------------------------------------

Time Stamp | Reset | Uptime

MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minut es

--------------------------------------------------------------------------------

08/13/2018 17:57:52 Reload 0 0 0 0 0

08/13/2018 18:11:24 Reload 0 0 0 0 3

08/14/2018 04:35:50 Reload 0 0 0 0 5

08/14/2018 04:51:01 Reload 0 0 0 0 3

08/15/2018 00:39:44 Reload 0 0 0 0 5

08/15/2018 00:52:07 Reload 0 0 0 0 3

08/27/2018 20:49:50 Reload 0 0 0 0 5

08/27/2018 20:57:46 Reload 0 0 0 0 5

08/27/2018 21:08:50 Reload 0 0 0 0 5

07/30/2019 17:54:55 Reload 0 0 0 0 0

07/31/2019 16:54:47 Image Install 0 0 0 21 56

07/05/2022 21:24:19 Reload 2 48 4 4 5

Leo Laohoo · ‎07-12-2022

Nothing points to a crash.

The latest reboot occurred on 05 July 2022. Someone entered the reason "Gt^@^U]U" for rebooting the switch.

lagerplane · ‎07-12-2022

That reload reason showing as "Current reset reason : Gt^@^U]U" looks bug / software related.

You mentioned "No crash info, but trace logs have been generated short after the reload took place."

You mean a trace file was generated on your crashinfo: directory? Or an actual trace log / traceback message showed up on your terminal / show logging output ?

Leo Laohoo · ‎07-12-2022

@lagerplane wrote:

System returned to ROM by Reload Command

There are no signs of crashes but all the data provided showed someone issued a "reload" command.

The output for the command "sh log on switch active up detail" also confirms this as well.

lagerplane · ‎07-12-2022

Hi Leo, exactly, on one hand we have the show version showing "System returned to ROM by Reload Command" which is simple as. Somebody went into the device and typed reload.

Then on the "sh log on switch active up detail" we have those weird outputs of "Current reset reason : Gt^@^U]U" and "Current reset timestamp : 01/01/1970 09:06:06" and also the reload reason listed as only "reload" instead of "reload command". So those few entries on the output of "sh log on switch active up detail" look a bit buggy. That device running an early 16.9.X code also could explain those weird outputs. On 17.3.X or 17.6.X that "sh log on switch active up detail" output is improved (shows reasons more accurately). But yes, at the end of the day, all points to be as you mentioned, looks like somebody accessed the Switch a did a reload command.

Leo Laohoo · ‎07-12-2022

There are several known bugs (plural) of IOS-XE crashes and do not leave any crashinfo/coredump files and it could, potentially, affect something is "old" as 16.9.3, however, the other output does not point that way.

Everything points to someone invoking the "reload" command, like this:

reload r Gt^@^U]U