C3650 Unknown Post-Reboot Issues

[ohchan]

hi

"start-up config" was re-entered after an unknown reboot.

I've never seen anything like this before.

What's wrong with this?
- Is it an archive problem?

Furthermore, it appears that the user who accessed it remotely entered the command.

We've never been remotely approached.

 

[Flavio Miranda]

Hi

 There´s a bug, which does not apply to your device apparently, but can give you some direction. In case you are worried about unauthorized access.

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvr89973 

[Jitendra Kumar]

Can you let us know here where this C3650  switch is included in the bug ID??

How do you know that running firmware is the same ?? even there no firmware details are provided by the requester?

your common sense is too high..

https://quickview.cloudapps.cisco.com/quickview/bug/CSCvr89973

thanks,

Jitendra

[ohchan]

hi 

Thank you for your reply.

In my case, not only the interface but also the entire start-up config is entered.

[marce1000]

          - What do you mean by 'the entire start-up config is entered.'

 M.

[ohchan]

Commands such as access-list, snmp, and banner were also re-entered.

It's like pasting the startup configuration stored in flash memory.

[Flavio Miranda]

But, as I said, this bug was not meant to explain your case as the platform is different. It was just to give you an idea that things like that happens and not necessarily it was an invasion.

If you can, open a TAC so that you can be more backed up.  If you have tacacs and syslog, dig on it and try to get more evidencies.

[Jitendra Kumar]

@Flavio Miranda  go through your comment you have directly declared it a bug without verifying required details like firmware and modal number. think about what has made comments.

Thanks,

Jitendra

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
sh platform resource
sh platform software status con brief

[ohchan]

It's a log from the past. There is no latest log.

#show platform resource
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 6.59% 100% 5% 10% H
DRAM 1568MB(40%) 3884MB 90% 95% H

#show ver
Cisco IOS XE Software, Version 16.09.04
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:33 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 4.66, RELEASE SOFTWARE (P)

uptime is 2 years, 29 weeks, 4 days, 21 hours, 8 minutes
Uptime for this control processor is 2 years, 29 weeks, 4 days, 21 hours, 11 minutes
System returned to ROM by Power Failure or Unknown at 13:33:12 KST Thu Oct 31 2019
System reed at 17:09:36 KST Thu Oct 31 2019
System image file is "flash:packages.conf"
Last reload reason: Power Failure or Unknown

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
lanbasek9 Smart License lanbasek9
None Subscription Smart License None

cisco WS-C3650-24TS (MIPS) processor (revision K0) with 832441K/6147K bytes of memory.
Processor board ID
5 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
1609272K bytes of Flash at flash:.
0K bytes of WebUI ODM Files at webui:.

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24TS 16.9.4 CAT3K_CAA-UNIVERSALK9 INSTALL

If it's a version problem, it's serious.

Unknown reboot issues include:
Last reload reason: Power Failure or Unknown
I don't know what this is.

[Leo Laohoo]

Post the "top" four lines to the command "sh run".  

[ohchan]

show run
Building configuration...

Current configuration : 15658 bytes
!
! Last configuration change at 10:34:35 KST Fri Mar 11 2022
! NVRAM config last updated at 13:52:16 KST Thu Apr 21 2022
!
version 16.9

For your information, an unknown reboot occurred on June 7.

[Leo Laohoo]

---

@ohchan wrote:

! Last configuration change at 10:34:35 KST Fri Mar 11 2022
! NVRAM config last updated at 13:52:16 KST Thu Apr 21 2022

---

The system was is the one who shut those ports down.  If it was someone else, it would show up here. 

Next, the switch is running 16.9.4.  This means that could also be affected by FN-72323 (Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality).  Either upgrade the firmware or apply the workaround.  If not, watch the control-plane memory utilization of the switch/stack.  Refer to this.

[ohchan]

Thank you for your kind answer.

! ! Last configuration change at 10:34:35 KST Fri Mar 11 2022
! ! NVRAM config last updated at 13:52:16 KST Thu Apr 21 2022

The system was is the one who shut those ports down. If it was someone else, it would show up here.
- Can I ask you in detail about this?

We upgraded the firmware to 16.09.04 two years ago due to a memory leak bug.

If there is a bug related to a memory leak even in version 16.09.04, it becomes serious.

[Leo Laohoo]

---

@ohchan wrote:

If there is a bug related to a memory leak even in version 16.09.04, it becomes serious.

---

IOS-XE memory leaks like a sieve.  Read the link I have provided.