3650 stack - Single Switch Crashed

[tinywashbear]

Hello,

We have a stack of 5 Cisco 3650's that were installed back in September of 2023. They have been running fine, no issues.

Monday morning 3/11/24, at apparently 6:21, our logging server reports that all the interfaces went down and nothing else. Our DC was on this switch which caused a DHCP/DNS outage for obvious reasons.

Now the issue I have is finding out why the switch went down out of the whole stack. Like I said, 5 3650 switches, running version 16.6, and none of the other switches went down. Rebooting fixed it, but rebooting also appeared to clear the buffer log, so all I have is our log file that shows the interfaces going down: I do notice that with the list of interfaces going down that gi3/0/34 went down on our 3rd switch in the stack at the same exact time, but no other interface did.

I have very little hope that I'll find a root cause, but any direction I can be pointed in would be appreciated. I'm "intermediate" with Cisco infrastructure, not an amateur but not an expert, so bear with me. Also if someone could provide a method to make logs show more than just interfaces going up and down, that would be appreciated.

Thank you.

[nict]

Hello @tinywashbear 

Maybe this could be interesting to you Capture Information from the Crashinfo File 

[tinywashbear]

Got a "opening crashinfo:crashinfo_RP_00_00 (No such file or directory)" unfortunately.

[nict]

Where did you get the name "crashinfo:crashinfo_RP_00_00" from?

Could you please provide "dir flash:" output?

[tinywashbear]

dir flash:
Directory of flash:/

54210 -rw- 15938176 Mar 18 2019 02:39:24 +00:00 cat3k_caa-guestshell.16.03.08.SPA.pkg
54211 -rw- 22306936 Mar 18 2019 02:39:22 +00:00 cat3k_caa-rpbase.16.03.08.SPA.pkg
54212 -rw- 264563320 Mar 18 2019 02:39:24 +00:00 cat3k_caa-rpcore.16.03.08.SPA.pkg
54213 -rw- 9671296 Mar 18 2019 02:39:23 +00:00 cat3k_caa-srdriver.16.03.08.SPA.pkg
54214 -rw- 209654388 Mar 18 2019 02:39:23 +00:00 cat3k_caa-wcm.16.03.08.SPA.pkg
54215 -rw- 13439612 Mar 18 2019 02:39:23 +00:00 cat3k_caa-webui.16.03.08.SPA.pkg
54225 -rw- 4742 Sep 18 2020 23:07:18 +00:00 packages.conf
54217 -rw- 777 Aug 21 2023 00:18:08 +00:00 bootloader_evt_handle.log
85185 drwx 4096 Nov 30 2023 23:16:20 +00:00 core
11 drwx 4096 Mar 18 2019 02:42:14 +00:00 .prst_sync
15489 drwx 4096 Mar 18 2019 02:42:16 +00:00 .rollback_timer
23233 drwx 4096 Aug 21 2023 00:18:25 +00:00 dc_profile_dir
23235 drwx 4096 Mar 18 2019 02:42:24 +00:00 gs_script
54218 -rw- 128103 Aug 21 2023 00:18:46 +00:00 memleak.tcl
30977 drwx 4096 Aug 21 2023 00:18:04 +00:00 .installer
54219 -rw- 2097152 Feb 26 2024 20:35:04 +00:00 nvram_config
54220 -rw- 2097152 Feb 26 2024 20:35:05 +00:00 nvram_config_bkup
54221 -rw- 35 Sep 26 2019 19:54:51 +00:00 pnp-tech-time
54222 -rw- 71711 Sep 26 2019 19:54:54 +00:00 pnp-tech-discovery-summary
54223 -rw- 2056 Feb 26 2024 20:34:33 +00:00 vlan.dat
54216 -rw- 1 Jan 8 2021 16:23:01 +00:00 cs_verify_rc.txt
54227 -rw- 5383060 Jan 8 2021 20:43:16 +00:00 105.pcap
54234 -rw- 452384 Jan 8 2021 20:50:10 +00:00 1047.pcap
54229 -rw- 19167872 Oct 9 2019 16:28:08 +00:00 cat3k_caa-guestshell.16.06.06.SPA.pkg
54233 -rw- 27926203 Oct 9 2019 16:28:11 +00:00 cat3k_caa-rpbase.16.06.06.SPA.pkg
54230 -rw- 336437880 Oct 9 2019 16:28:11 +00:00 cat3k_caa-rpcore.16.06.06.SPA.pkg
54232 -rw- 5051008 Oct 9 2019 16:28:11 +00:00 cat3k_caa-srdriver.16.06.06.SPA.pkg
54228 -rw- 4742 Oct 9 2019 16:28:44 +00:00 cat3k_caa-universalk9.16.06.06.SPA.conf
54231 -rw- 16263804 Oct 9 2019 16:28:11 +00:00 cat3k_caa-webui.16.06.06.SPA.pkg
54224 -rw- 4742 Sep 18 2020 23:07:18 +00:00 packages.conf.00-
61955 -rw- 19167872 Sep 18 2020 23:04:00 +00:00 cat3k_caa-guestshell.16.06.08.SPA.pkg
85187 -rw- 30417560 Sep 18 2020 23:04:43 +00:00 cat3k_caa-rpbase.16.06.08.SPA.pkg
15490 -rw- 336503416 Sep 18 2020 23:04:36 +00:00 cat3k_caa-rpcore.16.06.08.SPA.pkg
61957 -rw- 5091968 Sep 18 2020 23:04:40 +00:00 cat3k_caa-srdriver.16.06.08.SPA.pkg
61954 -rw- 4742 Sep 18 2020 23:05:07 +00:00 cat3k_caa-universalk9.16.06.08.SPA.conf
61956 -rw- 16271996 Sep 18 2020 23:04:40 +00:00 cat3k_caa-webui.16.06.08.SPA.pkg
54226 -rw- 4737 Oct 9 2019 16:29:07 +00:00 packages.conf.01-

1621966848 bytes total (177037312 bytes free)

The crash info was after I ran the show stack command, at the end of it.

[balaji.bandi · Accepted solution]

first i would suggest to clean up unwanted packages :

check :

#show install committed  or #show version what version active now.

#show install inactive

clean up

#install remove inactive

check crash files :

#dir flash:core/

Suggest to upgrade latest IOX XE suggested by Cisco TAC

some reference to check crash issue :

https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/201070-Troubleshooting-3650-3850-reloads-by-sta.html

[Leo Laohoo]

For consistency reasons, different firmware versions store their crashinfo and crashlogs in different locations.  In some cases, crashinfo and crashlogs do not get generated because of several known bugs.

Post the complete output to the following commands: 

dir flash-1:core
dir flash-2:core
dir flash-3:core
dir flash-4:core
dir flash-5:core
dir crashinfo-1:
dir crashinfo-2:
dir crashinfo-3:
dir crashinfo-4:
dir crashinfo-5:
sh log on switch active uptime detail

Were there any log entries about failed Smart License &/or Call Home?  If no one has done any workaround, the entire logs would have been filled with CSL & Call Home error messages.  All this is related to:   FN72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality?

Failure to do any workaround will cause the "keyman" process to memory leak.  And because the 3650 has a very tiny memory, it will take about 6 to 8 months a switch to crash.