3850 Last switchover reason = active unit removed

[chrisc100]

We had a failover event on the 3850 stack where switch 2 became active. Switch1 is now member and switch3 standby.

There were no system reports generated, how can I see more reason for the failover?

The session command to connect to member switches does not exist in 16.09.06, but according to here it should be there:

https://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3850/software/release/16-8/command_reference/b_168_3850_cr/b_168_3850_cr_chapter_01110.html

#show redundancy
Redundant System Information :
------------------------------
Available system uptime = 14 weeks, 3 days, 13 minutes
Switchovers system experienced = 1
Standby failures = 0
Last switchover reason = active unit removed

Software version is 16.09.06

The last message was :

CFG-MGR:ERROR: could not send cfg sync msg

Thanks in advance

[Leo Laohoo]

Post the complete output to the following commands:

sh version
dir flash-1:
dir flash-2:
dir flash-3:
dir crashinfo-1:
dir crashinfo-2:
dir crashinfo-3:
dir flash-1:/core
dir flash-2:/core
dir flash-3:/core
sh log on switch 1 up details
sh log on switch 3 up details
sh logs

[chrisc100]

STACKPROD#show version
Cisco IOS XE Software, Version 16.09.06
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.9.6, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2020 by Cisco Systems, Inc.
Compiled Thu 27-Aug-20 01:57 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2020 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.58, RELEASE SOFTWARE (P)

STACKPROD uptime is 19 weeks, 6 days, 18 hours, 8 minutes
Uptime for this control processor is 19 weeks, 6 days, 18 hours, 15 minutes
System returned to ROM by SSO Switchover
System restarted at 15:51:30 CET Fri Feb 26 2021
System image file is "flash:packages.conf"
Last reload reason: Image Install

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
ipbasek9 Smart License ipbasek9
None Subscription Smart License None

Smart Licensing Status: REGISTERED/AUTHORIZED

cisco WS-C3850-48P (MIPS) processor (revision AB0) with 832327K/6147K bytes of memory.
2 Virtual Ethernet interfaces
156 Gigabit Ethernet interfaces
12 Ten Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
250456K bytes of Crash Files at crashinfo:.
250456K bytes of Crash Files at crashinfo-3:.
1609272K bytes of Flash at flash:.
1609272K bytes of Flash at flash-3:.
0K bytes of WebUI ODM Files at webui:.
252000K bytes of Crash Files at crashinfo-1:.
1611414K bytes of Flash at flash-1:.

Base Ethernet MAC Address : 00:f8:2c:fc:01:00
Motherboard Assembly Number : 73-15800-08
Model Revision Number : AB0
Motherboard Revision Number : A0
Model Number : WS-C3850-48P

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 56 WS-C3850-48P 16.09.06 CAT3K_CAA-UNIVERSALK9 INSTALL
* 2 56 WS-C3850-48P 16.09.06 CAT3K_CAA-UNIVERSALK9 INSTALL
3 56 WS-C3850-48P 16.09.06 CAT3K_CAA-UNIVERSALK9 INSTALL

Switch 01
---------
Switch uptime : 1 day, 2 hours, 37 minutes

Base Ethernet MAC Address : 00:59:dc:a4:de:00
Motherboard Assembly Number : 73-15800-08
Model Revision Number : AB0
Motherboard Revision Number : A0
Model Number : WS-C3850-48P

Switch 03
---------
Switch uptime : 19 weeks, 6 days, 18 hours, 9 minutes

Base Ethernet MAC Address : 70:18:a7:4f:7f:00
Motherboard Assembly Number : 73-15800-08
Model Revision Number : AC0
Motherboard Revision Number : B0
Model Number : WS-C3850-48P

Configuration register is 0x102

STACKPROD# dir flash-1:
Directory of flash-1:/

77602 -rw- 2097152 Jul 16 2021 10:17:53 +02:00 nvram_config
77649 -rw- 509 Feb 26 2021 15:46:44 +01:00 .packages.verify.log
77612 drwx 4096 Jul 15 2021 08:21:01 +02:00 .installer
77604 -rw- 76 Jul 15 2021 08:21:18 +02:00 boothelper.log
77613 -rw- 259 Jul 15 2021 08:21:03 +02:00 bootloader_evt_handle.log
77648 -rw- 4776 Feb 26 2021 15:44:53 +01:00 packages.conf
11 drwx 4096 Jul 15 2021 08:21:15 +02:00 dc_profile_dir
77607 -rw- 916 Feb 26 2021 15:59:40 +01:00 vlan.dat
77614 drwx 4096 Jul 15 2021 08:19:07 +02:00 core
85361 drwx 4096 Feb 26 2021 15:55:35 +01:00 .prst_sync
12 drwx 4096 Feb 26 2021 15:45:51 +01:00 .rollback_timer
77620 drwx 4096 Feb 8 2020 06:38:10 +01:00 onep
77645 -rw- 22809212 Feb 26 2021 15:42:25 +01:00 cat3k_caa-webui.16.09.06.SPA.pkg
77643 -rw- 19862144 Feb 26 2021 15:42:20 +01:00 cat3k_caa-guestshell.16.09.06.SPA.pkg
7761 drwx 4096 Feb 8 2020 06:27:07 +01:00 gs_script
15521 drwx 4096 Feb 8 2020 06:27:08 +01:00 tech_support
77615 -rw- 123888 Jul 15 2021 08:21:39 +02:00 memleak.tcl
77616 drwx 4096 Mar 13 2020 12:16:51 +01:00 .dbpersist
77617 -rw- 0 Jul 15 2021 08:23:05 +02:00 rdope_out.txt
77618 -rw- 90 Jul 15 2021 08:23:18 +02:00 rdope.log
77619 -rw- 2097152 Feb 26 2021 18:57:06 +01:00 nvram_config_bkup
77622 -rw- 916 Feb 26 2021 15:58:36 +01:00 stby-vlan.dat
77627 -rw- 66937 Sep 17 2020 06:56:26 +02:00 backup_20200917.txt
77647 -rw- 30872219 Feb 26 2021 15:42:25 +01:00 cat3k_caa-rpbase.16.09.06.SPA.pkg
77644 -rw- 373863032 Feb 26 2021 15:42:23 +01:00 cat3k_caa-rpcore.16.09.06.SPA.pkg
77646 -rw- 3156608 Feb 26 2021 15:42:25 +01:00 cat3k_caa-srdriver.16.09.06.SPA.pkg
77637 -rw- 23 Feb 26 2021 15:46:42 +01:00 .bootpackages.verify.log
77654 -rw- 107 Feb 26 2021 15:46:44 +01:00 .fpga_upg_run.log

1624244224 bytes total (905969664 bytes free)
STACKPROD#dir flash-2:
Directory of flash:/

54218 -rw- 2097152 Jul 16 2021 10:17:53 +02:00 nvram_config
54255 -rw- 22809212 Feb 26 2021 15:42:28 +01:00 cat3k_caa-webui.16.09.06.SPA.pkg
54212 -rw- 916 Jul 15 2021 08:17:01 +02:00 stby-vlan.dat
54216 -rw- 23 Feb 26 2021 15:46:45 +01:00 .bootpackages.verify.log
54258 -rw- 4776 Feb 26 2021 15:44:56 +01:00 packages.conf
77441 drwx 4096 Feb 26 2021 15:49:13 +01:00 dc_profile_dir
54217 drwx 4096 Feb 26 2021 16:03:01 +01:00 .installer
54259 -rw- 916 Feb 26 2021 15:59:40 +01:00 vlan.dat
54253 -rw- 19862144 Feb 26 2021 15:42:24 +01:00 cat3k_caa-guestshell.16.09.06.SPA.pkg
54247 -rw- 509 Feb 26 2021 15:46:47 +01:00 .packages.verify.log
54228 drwx 4096 Feb 8 2020 07:30:48 +01:00 core
54227 -rw- 222 Feb 26 2021 15:49:02 +01:00 bootloader_evt_handle.log
61953 drwx 4096 Feb 26 2021 15:58:39 +01:00 .prst_sync
77443 drwx 4096 Feb 26 2021 15:45:54 +01:00 .rollback_timer
85185 drwx 4096 Feb 8 2020 06:37:27 +01:00 gs_script
92929 drwx 4096 Feb 8 2020 06:37:27 +01:00 tech_support
54229 -rw- 123888 Feb 26 2021 15:49:38 +01:00 memleak.tcl
54230 drwx 4096 Mar 13 2020 12:16:50 +01:00 .dbpersist
54231 -rw- 0 Feb 26 2021 15:55:39 +01:00 rdope_out.txt
54232 -rw- 90 Feb 26 2021 15:55:52 +01:00 rdope.log
54219 -rw- 2097152 Jul 16 2021 10:17:53 +02:00 nvram_config_bkup
54234 drwx 4096 Feb 8 2020 06:51:16 +01:00 onep
54257 -rw- 30872219 Feb 26 2021 15:42:28 +01:00 cat3k_caa-rpbase.16.09.06.SPA.pkg
54254 -rw- 373863032 Feb 26 2021 15:42:26 +01:00 cat3k_caa-rpcore.16.09.06.SPA.pkg
54256 -rw- 3156608 Feb 26 2021 15:42:28 +01:00 cat3k_caa-srdriver.16.09.06.SPA.pkg
54250 -rw- 107 Feb 26 2021 15:46:47 +01:00 .fpga_upg_run.log
54221 -rw- 66 Feb 26 2021 15:49:31 +01:00 boothelper.log

1621966848 bytes total (1077641216 bytes free)
STACKPROD#dir flash-3:
Directory of flash-3:/

7746 -rw- 2097152 Jul 16 2021 10:17:50 +02:00 nvram_config
7771 -rw- 4776 Feb 26 2021 15:44:55 +01:00 packages.conf
30977 drwx 4096 Feb 26 2021 16:02:59 +01:00 .installer
7753 -rw- 333 Feb 26 2021 15:49:03 +01:00 bootloader_evt_handle.log
38721 drwx 4096 Jan 29 2019 22:41:43 +01:00 core
46465 drwx 4096 Jul 15 2021 08:17:05 +02:00 .prst_sync
61953 drwx 4096 Feb 26 2021 15:45:54 +01:00 .rollback_timer
69697 drwx 4096 Jan 29 2019 22:32:03 +01:00 gs_script
77441 drwx 4096 Feb 26 2021 15:49:14 +01:00 dc_profile_dir
7754 -rw- 123888 Feb 26 2021 15:49:39 +01:00 memleak.tcl
7755 -rw- 2097152 Jul 16 2021 10:17:50 +02:00 nvram_config_bkup
7756 -rw- 35 Feb 26 2021 13:26:42 +01:00 pnp-tech-time
7757 -rw- 47748 Feb 26 2021 13:26:55 +01:00 pnp-tech-discovery-summary
30978 drwx 4096 Feb 26 2021 13:45:59 +01:00 tech_support
31001 drwx 4096 Feb 26 2021 13:58:54 +01:00 .dbpersist
7769 -rw- 23 Feb 26 2021 15:46:45 +01:00 .bootpackages.verify.log
7758 -rw- 509 Feb 26 2021 15:46:47 +01:00 .packages.verify.log
7752 -rw- 0 Feb 26 2021 15:55:37 +01:00 rdope_out.txt
7767 -rw- 90 Feb 26 2021 15:55:51 +01:00 rdope.log
38723 drwx 4096 Feb 26 2021 13:58:14 +01:00 onep
7760 -rw- 66 Feb 26 2021 15:49:32 +01:00 boothelper.log
7747 -rw- 916 Jul 15 2021 08:18:00 +02:00 vlan.dat
7773 -rw- 19862144 Feb 26 2021 15:42:24 +01:00 cat3k_caa-guestshell.16.09.06.SPA.pkg
7777 -rw- 30872219 Feb 26 2021 15:42:31 +01:00 cat3k_caa-rpbase.16.09.06.SPA.pkg
7775 -rw- 373863032 Feb 26 2021 15:42:30 +01:00 cat3k_caa-rpcore.16.09.06.SPA.pkg
7776 -rw- 3156608 Feb 26 2021 15:42:31 +01:00 cat3k_caa-srdriver.16.09.06.SPA.pkg
7774 -rw- 22809212 Feb 26 2021 15:42:31 +01:00 cat3k_caa-webui.16.09.06.SPA.pkg
7770 -rw- 107 Feb 26 2021 15:46:47 +01:00 .fpga_upg_run.log

1622147072 bytes total (1080557568 bytes free)
STACKPROD#dir crashinfo-1:
Directory of crashinfo-1:/

11 -rw- 0 Jan 1 1970 01:00:06 +01:00 koops.dat
12 -rw- 0 Feb 2 2019 11:20:52 +01:00 cilogs
44881 drwx 11264 Jul 16 2021 10:56:34 +02:00 tracelogs
14 -rw- 2455376 Jun 23 2017 19:44:32 +02:00 switch_report_1_20170623-174321.gz

250085376 bytes total (227540992 bytes free)
STACKPROD#dir crashinfo-2:
Directory of crashinfo:/

11 -rw- 272 Apr 21 2017 08:35:58 +02:00 last_systemreport_log
12 -rw- 0 Jan 1 1970 01:00:05 +01:00 koops.dat
13 -rw- 0 Feb 2 2019 11:21:05 +01:00 cilogs
28337 drwx 21504 Jul 16 2021 10:57:10 +02:00 tracelogs
14 -rw- 1140624 Jul 15 2021 08:14:40 +02:00 STACKPROD_2_RP_0_trace_archive_0-20210715-081437.tar.gz

248354816 bytes total (229367808 bytes free)
STACKPROD#dir crashinfo-3:
Directory of crashinfo-3:/

4049 drwx 34816 Jul 16 2021 10:47:40 +02:00 tracelogs
11 -rw- 0 Jan 29 2019 22:55:37 +01:00 koops.dat
18 -rw- 1969607 Jul 15 2021 08:14:39 +02:00 STACKPROD_RP_0_trace_archive_0-20210715-081436.tar.gz

248512512 bytes total (228065280 bytes free)
STACKPROD#dir flash-1:/core
Directory of flash-1:/core/

93121 drwx 4096 Feb 8 2020 06:27:03 +01:00 modules
77621 -rw- 1 Jul 15 2021 08:09:55 +02:00 .callhome
77605 -rw- 38123 Jul 15 2021 08:14:35 +02:00 kernel.rp_RP-EDISON_0_20210715061435.txt
77606 -rw- 173827290 Jul 15 2021 08:19:07 +02:00 kernel.rp_RP-EDISON_0_20210715061435.core.flat.gz

1624244224 bytes total (905969664 bytes free)
STACKPROD#dir flash-2:/core
Directory of flash:/core/

69697 drwx 4096 Feb 8 2020 06:37:23 +01:00 modules
54235 -rw- 1 Jul 16 2021 10:49:19 +02:00 .callhome

1621966848 bytes total (1077641216 bytes free)
STACKPROD#dir flash-3:/core
Directory of flash-3:/core/

54209 drwx 4096 Jan 29 2019 22:31:54 +01:00 modules
38722 -rw- 1 Jul 16 2021 10:48:13 +02:00 .callhome

1622147072 bytes total (1080557568 bytes free)
STACKPROD#sh log on switch 1 up details
^
% Invalid input detected at '^' marker.

STACKPROD#sh log on switch 3 up details
^
% Invalid input detected at '^' marker.

STACKPROD#sh log
Syslog logging: enabled (0 messages dropped, 5024 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

No Active Message Discriminator.

No Inactive Message Discriminator.

Console logging: disabled
Monitor logging: level debugging, 0 messages logged, xml disabled,
filtering disabled
Buffer logging: level debugging, 444 messages logged, xml disabled,
filtering disabled
Exception Logging: size (4096 bytes)
Count and timestamp logging messages: disabled
File logging: disabled
Persistent logging: disabled

No active filter modules.

Trap logging: level informational, 405 message lines logged
Logging to 10.1.1.10 (udp port 514, audit disabled,
link up),
358 message lines logged,
0 message lines rate-limited,
0 message lines dropped-by-MD,
xml disabled, sequence number disabled
filtering disabled
Logging Source-Interface: VRF Name:
Vlan13

Log Buffer (100000 bytes):

Jul 15 08:14:38.562 cest: CFG-MGR:ERROR: could not send cfg sync msg
Jul 15 08:14:39.439 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi19/3, changed state to up
Jul 15 08:14:39.440 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC19/1, changed state to up
Jul 15 08:14:39.440 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN19/2, changed state to up
Jul 15 08:14:39.622 cest: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair has been generated or imported by SLA-KeyPair
Jul 15 08:14:39.767 cest: %SYS-6-LOGGINGHOST_STARTSTOP: Logging to host 10.1.1.10 port 514 started - CLI initiated
Jul 15 08:14:39.816 cest: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to down
Jul 15 08:14:39.897 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface Null0, changed state to up
Jul 15 08:14:39.898 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan13, changed state to up
Jul 15 08:14:39.905 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/1, changed state to up
Jul 15 08:14:39.906 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/2, changed state to up
Jul 15 08:14:39.910 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to up
Jul 15 08:14:39.925 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/4, changed state to up
Jul 15 08:14:39.926 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/5, changed state to up
Jul 15 08:14:39.926 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/6, changed state to up
Jul 15 08:14:39.930 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Jul 15 08:14:39.931 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/8, changed state to up
Jul 15 08:14:39.933 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/10, changed state to up
Jul 15 08:14:39.967 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/11, changed state to up
Jul 15 08:14:39.967 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to up
Jul 15 08:14:40.031 cest: %SMART_LIC-6-EXPORT_CONTROLLED: Usage of export controlled features is allowed
Jul 15 08:14:40.033 cest: %CALL_HOME-6-CALL_HOME_ENABLED: Call-home is enabled by Smart Agent for Licensing.
Jul 15 08:14:40.036 cest: %SMART_LIC-5-COMM_RESTORED: Communications with the Cisco Smart Software Manager (CSSM) restored
Jul 15 08:14:40.123 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/3, changed state to up
Jul 15 08:14:40.136 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to up
Jul 15 08:14:46.518 cest: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Jul 15 08:14:47.127 cest: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Jul 15 08:15:10.491 cest: %SMART_LIC-5-END_POINT_RESET: End Point list reset
Jul 15 08:15:10.492 cest: %SMART_LIC-6-AUTH_RENEW_SUCCESS: Authorization renewal successful. State=authorized for udi PID:WS-C3850-48P,SN:xx
Jul 15 08:15:41.383 cest: %SMART_LIC-6-AUTH_RENEW_SUCCESS: Authorization renewal successful. State=authorized for udi PID:WS-C3850-48P,SN:xx
Jul 15 08:16:36.207 cest: %IOSXE_REDUNDANCY-6-PEER: Active detected switch 3 as standby.
Jul 15 08:16:36.203 cest: %STACKMGR-6-STANDBY_ELECTED: Switch 2 R0/0: stack_mgr: Switch 3 has been elected STANDBY.
Jul 15 08:16:51.269 cest: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))

Jul 15 08:16:51.269 cest: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

Jul 15 08:18:01.446 cest: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
Jul 15 08:18:02.454 cest: %RF-5-RF_TERMINAL_STATE: Terminal state reached for (SSO)
Jul 15 08:18:20.937 cest: %SW_MATM-4-MACFLAP_NOTIF: Host 0022.bd1b.d56b in vlan 33 is flapping between port Po1 and port Gi2/0/8
Jul 15 08:19:08.580 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 3 R0/0: stack_mgr: Stack port 1 on Switch 3 is down
Jul 15 08:19:08.582 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 2 on Switch 2 is down
Jul 15 08:22:38.816 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 2 on Switch 2 is up
Jul 15 08:22:39.158 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 3 R0/0: stack_mgr: Stack port 1 on Switch 3 is up
Jul 15 08:22:39.181 cest: %STACKMGR-4-SWITCH_ADDED: Switch 3 R0/0: stack_mgr: Switch 1 has been added to the stack.
Jul 15 08:22:39.184 cest: %STACKMGR-4-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 1 has been added to the stack.
Jul 15 08:22:42.290 cest: %STACKMGR-4-SWITCH_ADDED: Switch 3 R0/0: stack_mgr: Switch 1 has been added to the stack.
Jul 15 08:22:42.294 cest: %STACKMGR-4-SWITCH_ADDED: Switch 2 R0/0: stack_mgr: Switch 1 has been added to the stack.
Jul 15 08:22:48.659 cest: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair2 has been removed from key storage
Jul 15 08:22:50.006 cest: %LINK-3-UPDOWN: Interface GigabitEthernet1/1/1, changed state to down
Jul 15 08:22:50.007 cest: %LINK-3-UPDOWN: Interface GigabitEthernet1/1/2, changed state to down
Jul 15 08:22:50.007 cest: %LINK-3-UPDOWN: Interface GigabitEthernet1/1/3, changed state to down
Jul 15 08:22:50.008 cest: %LINK-3-UPDOWN: Interface GigabitEthernet1/1/4, changed state to down
Jul 15 08:22:50.022 cest: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to down
Jul 15 08:22:52.364 cest: %ILPOWER-7-DETECT: Interface Gi1/0/2: Power Device detected: Cisco PD
Jul 15 08:22:52.365 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/2: inline power denied. Reason: insufficient power
Jul 15 08:22:52.367 cest: %ILPOWER-7-DETECT: Interface Gi1/0/6: Power Device detected: Cisco PD
Jul 15 08:22:52.369 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/6: inline power denied. Reason: insufficient power
Jul 15 08:22:52.371 cest: %ILPOWER-7-DETECT: Interface Gi1/0/10: Power Device detected: Cisco PD
Jul 15 08:22:52.373 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/10: inline power denied. Reason: insufficient power
Jul 15 08:22:52.375 cest: %ILPOWER-7-DETECT: Interface Gi1/0/14: Power Device detected: Cisco PD
Jul 15 08:22:52.377 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/14: inline power denied. Reason: insufficient power
Jul 15 08:22:52.379 cest: %ILPOWER-7-DETECT: Interface Gi1/0/16: Power Device detected: Cisco PD
Jul 15 08:22:52.381 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/16: inline power denied. Reason: insufficient power
Jul 15 08:22:52.383 cest: %ILPOWER-7-DETECT: Interface Gi1/0/25: Power Device detected: Cisco PD
Jul 15 08:22:52.385 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/25: inline power denied. Reason: insufficient power
Jul 15 08:22:52.387 cest: %ILPOWER-7-DETECT: Interface Gi1/0/27: Power Device detected: Cisco PD
Jul 15 08:22:52.389 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/27: inline power denied. Reason: insufficient power
Jul 15 08:22:52.391 cest: %ILPOWER-7-DETECT: Interface Gi1/0/35: Power Device detected: Cisco PD
Jul 15 08:22:52.393 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/35: inline power denied. Reason: insufficient power
Jul 15 08:22:52.395 cest: %ILPOWER-7-DETECT: Interface Gi1/0/39: Power Device detected: Cisco PD
Jul 15 08:22:52.397 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/39: inline power denied. Reason: insufficient power
Jul 15 08:22:52.399 cest: %ILPOWER-7-DETECT: Interface Gi1/0/46: Power Device detected: Cisco PD
Jul 15 08:22:52.401 cest: %ILPOWER-5-ILPOWER_POWER_DENY: Interface Gi1/0/46: inline power denied. Reason: insufficient power
Jul 15 08:22:52.460 cest: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 1: EMP_RELAY: Channel UP!
Jul 15 08:22:52.474 cest: %HMANRP-6-EMP_NO_ELECTION_INFO: Could not elect active EMP switch, setting emp active switch to 0: EMP_RELAY: Could not elect switch with mgmt port UP
Jul 15 08:22:53.413 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is down
Jul 15 08:22:53.413 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is down
Jul 15 08:22:53.413 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 1 on Switch 1 is up
Jul 15 08:22:53.413 cest: %STACKMGR-6-STACK_LINK_CHANGE: Switch 1 R0/0: stack_mgr: Stack port 2 on Switch 1 is up
Jul 15 08:22:53.413 cest: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
Jul 15 08:22:53.413 cest: %STACKMGR-4-SWITCH_ADDED: Switch 1 R0/0: stack_mgr: Switch 1 has been added to the stack.
Jul 15 08:22:55.083 cest: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 2 inserted
Jul 15 08:22:55.090 cest: %PLATFORM_STACKPOWER-6-LINK_EVENT: Switch 1 stack power protocol is up on cable 2
Jul 15 08:22:55.404 cest: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair2 has been generated or imported by crypto-engine
Jul 15 08:22:55.705 cest: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAM
Jul 15 08:22:57.761 cest: %ILPOWER-7-DETECT: Interface Gi1/0/1: Power Device detected: IEEE PD
Jul 15 08:22:57.767 cest: %ILPOWER-7-DETECT: Interface Gi1/0/4: Power Device detected: IEEE PD
Jul 15 08:22:57.772 cest: %ILPOWER-7-DETECT: Interface Gi1/0/5: Power Device detected: IEEE PD
Jul 15 08:22:57.778 cest: %ILPOWER-7-DETECT: Interface Gi1/0/7: Power Device detected: IEEE PD
Jul 15 08:22:57.784 cest: %ILPOWER-7-DETECT: Interface Gi1/0/8: Power Device detected: IEEE PD
Jul 15 08:22:57.789 cest: %ILPOWER-7-DETECT: Interface Gi1/0/11: Power Device detected: IEEE PD
Jul 15 08:22:57.794 cest: %ILPOWER-7-DETECT: Interface Gi1/0/12: Power Device detected: IEEE PD
Jul 15 08:22:57.799 cest: %ILPOWER-7-DETECT: Interface Gi1/0/13: Power Device detected: IEEE PD
Jul 15 08:22:57.876 cest: %ILPOWER-7-DETECT: Interface Gi1/0/17: Power Device detected: IEEE PD
Jul 15 08:22:57.881 cest: %ILPOWER-7-DETECT: Interface Gi1/0/18: Power Device detected: IEEE PD
Jul 15 08:22:57.886 cest: %ILPOWER-7-DETECT: Interface Gi1/0/22: Power Device detected: IEEE PD
Jul 15 08:22:57.891 cest: %ILPOWER-7-DETECT: Interface Gi1/0/23: Power Device detected: IEEE PD
Jul 15 08:22:57.898 cest: %ILPOWER-7-DETECT: Interface Gi1/0/24: Power Device detected: IEEE PD
Jul 15 08:22:57.904 cest: %ILPOWER-7-DETECT: Interface Gi1/0/26: Power Device detected: IEEE PD
Jul 15 08:22:57.910 cest: %ILPOWER-7-DETECT: Interface Gi1/0/28: Power Device detected: IEEE PD
Jul 15 08:22:57.916 cest: %ILPOWER-7-DETECT: Interface Gi1/0/29: Power Device detected: IEEE PD
Jul 15 08:22:57.923 cest: %ILPOWER-7-DETECT: Interface Gi1/0/31: Power Device detected: IEEE PD
Jul 15 08:22:57.929 cest: %ILPOWER-7-DETECT: Interface Gi1/0/32: Power Device detected: IEEE PD
Jul 15 08:22:57.936 cest: %ILPOWER-7-DETECT: Interface Gi1/0/36: Power Device detected: IEEE PD
Jul 15 08:22:57.943 cest: %ILPOWER-7-DETECT: Interface Gi1/0/40: Power Device detected: IEEE PD
Jul 15 08:22:57.948 cest: %ILPOWER-7-DETECT: Interface Gi1/0/45: Power Device detected: IEEE PD
Jul 15 08:22:57.953 cest: %ILPOWER-7-DETECT: Interface Gi1/0/47: Power Device detected: IEEE PD
Jul 15 08:23:02.590 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/1: Power granted
Jul 15 08:23:02.590 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/4: Power granted
Jul 15 08:23:02.591 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/5: Power granted
Jul 15 08:23:02.591 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/7: Power granted
Jul 15 08:23:02.592 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/8: Power granted
Jul 15 08:23:02.593 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/11: Power granted
Jul 15 08:23:02.593 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/12: Power granted
Jul 15 08:23:02.594 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/13: Power granted
Jul 15 08:23:02.620 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/17: Power granted
Jul 15 08:23:02.621 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/18: Power granted
Jul 15 08:23:02.621 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/22: Power granted
Jul 15 08:23:02.622 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/23: Power granted
Jul 15 08:23:02.623 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/24: Power granted
Jul 15 08:23:02.623 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/26: Power granted
Jul 15 08:23:02.624 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/28: Power granted
Jul 15 08:23:02.625 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/29: Power granted
Jul 15 08:23:02.625 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/31: Power granted
Jul 15 08:23:02.626 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/32: Power granted
Jul 15 08:23:02.639 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/36: Power granted
Jul 15 08:23:02.640 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/40: Power granted
Jul 15 08:23:02.640 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/45: Power granted
Jul 15 08:23:02.641 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/47: Power granted
Jul 15 08:23:04.202 cest: %SMART_LIC-3-ID_CERT_RENEW_FAILED: Automatic registration renewal failed: Error received from Smart Software Manager: invalid product_instance_id
Jul 15 08:23:04.203 cest: %SMART_LIC-3-ID_CERT_RENEW_FAILED: Automatic registration renewal failed: FAILED
Jul 15 08:23:07.370 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/2: PD removed
Jul 15 08:23:07.375 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/6: PD removed
Jul 15 08:23:07.380 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/10: PD removed
Jul 15 08:23:07.384 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/14: PD removed
Jul 15 08:23:07.389 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/16: PD removed
Jul 15 08:23:07.394 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/25: PD removed
Jul 15 08:23:07.398 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/27: PD removed
Jul 15 08:23:07.403 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/35: PD removed
Jul 15 08:23:07.408 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/39: PD removed
Jul 15 08:23:07.414 cest: %ILPOWER-5-IEEE_DISCONNECT: Interface Gi1/0/46: PD removed
Jul 15 08:23:18.230 cest: %SMART_LIC-6-AUTH_RENEW_SUCCESS: Authorization renewal successful. State=authorized for udi PID:WS-C3850-48P,SN:xx
Jul 15 08:23:20.787 cest: %ILPOWER-7-DETECT: Interface Gi1/0/2: Power Device detected: Cisco PD
Jul 15 08:23:20.792 cest: %ILPOWER-7-DETECT: Interface Gi1/0/6: Power Device detected: Cisco PD
Jul 15 08:23:20.797 cest: %ILPOWER-7-DETECT: Interface Gi1/0/10: Power Device detected: Cisco PD
Jul 15 08:23:20.801 cest: %ILPOWER-7-DETECT: Interface Gi1/0/16: Power Device detected: Cisco PD
Jul 15 08:23:20.805 cest: %ILPOWER-7-DETECT: Interface Gi1/0/25: Power Device detected: Cisco PD
Jul 15 08:23:20.810 cest: %ILPOWER-7-DETECT: Interface Gi1/0/27: Power Device detected: Cisco PD
Jul 15 08:23:20.814 cest: %ILPOWER-7-DETECT: Interface Gi1/0/35: Power Device detected: Cisco PD
Jul 15 08:23:20.818 cest: %ILPOWER-7-DETECT: Interface Gi1/0/39: Power Device detected: Cisco PD
Jul 15 08:23:21.125 cest: %ILPOWER-7-DETECT: Interface Gi1/0/46: Power Device detected: Cisco PD
Jul 15 08:23:21.136 cest: %ILPOWER-7-DETECT: Interface Gi1/0/14: Power Device detected: Cisco PD
Jul 15 08:23:22.944 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/2: Power granted
Jul 15 08:23:22.945 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/6: Power granted
Jul 15 08:23:22.945 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/10: Power granted
Jul 15 08:23:22.946 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/14: Power granted
Jul 15 08:23:22.947 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/16: Power granted
Jul 15 08:23:23.585 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/25: Power granted
Jul 15 08:23:23.586 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/27: Power granted
Jul 15 08:23:23.603 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/35: Power granted
Jul 15 08:23:23.603 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/39: Power granted
Jul 15 08:23:23.604 cest: %ILPOWER-5-POWER_GRANTED: Interface Gi1/0/46: Power granted
Jul 15 08:23:28.173 cest: %PLATFORM_PM-6-FRULINK_INSERTED: 4x10G uplink module inserted in the switch 1 slot 1
Jul 15 08:23:30.421 cest: %PLATFORM_PM-6-MODULE_INSERTED: SFP module inserted with interface name Te1/1/3
Jul 15 08:23:34.465 cest: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/1/3, changed state to up
Jul 15 08:23:36.418 cest: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/3, changed state to up
Jul 15 08:27:08.526 cest: %PLATFORM-4-ELEMENT_WARNING: Switch 2 R0/0: smand: 1/RP/0: limited space - copy corefiles/switch-reports out of flash:core & crashinfo: directories. flash:core value 11% (169 MB) exceeds warning level 5% (79 MB)
Jul 15 08:41:00.207 cest: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair2 has been removed from key storage
Jul 15 08:41:04.432 cest: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair2 has been generated or imported by crypto-engine
Jul 15 08:41:04.713 cest: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAM
Jul 15 08:41:12.863 cest: %SMART_LIC-3-ID_CERT_RENEW_FAILED: Automatic registration renewal failed: Error received from Smart Software Manager: invalid product_instance_id
Jul 15 08:41:12.864 cest: %SMART_LIC-3-ID_CERT_RENEW_FAILED: Automatic registration renewal failed: FAILED
Jul 15 08:57:24.867 cest: %CRYPTO_ENGINE-5-KEY_DELETED: A key named SLA-KeyPair2 has been removed from key storage
Jul 15 08:57:31.160 cest: %CRYPTO_ENGINE-5-KEY_ADDITION: A key named SLA-KeyPair2 has been generated or imported by crypto-engine
Jul 15 08:57:31.434 cest: %PKI-6-CONFIGAUTOSAVE: Running configuration saved to NVRAM
Jul 15 08:57:39.787 cest: %SMART_LIC-3-ID_CERT_RENEW_FAILED: Automatic registration renewal failed: Error received from Smart Software Manager: invalid product_instance_id
Jul 15 08:57:39.788 cest: %SMART_LIC-3-ID_CERT_RENEW_FAILED: Automatic registration renewal failed: FAILED

Couple of observations:

- There were some trace archives generated at the time of the failover (no system reports)

- Messages related to stack power further down on switch1:

Jul 15 08:22:55.083 cest: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 2 inserted

switch1 has dual power supplies. show stack-power looks OK, not sure if this was the cause/related.

- As mentioned earlier, the session command does not work to log in to individual switches to check the reload reason.

Thanks in advance

[Leo Laohoo]

---

@chrisc100 wrote:

STACKPROD_2_RP_0_trace_archive_0-20210715-081437.tar.gz
STACKPROD_RP_0_trace_archive_0-20210715-081436.tar.gz
kernel.rp_RP-EDISON_0_20210715061435.txt
kernel.rp_RP-EDISON_0_20210715061435.core.flat.gz

---

We want to see these four files.  Kindly export them from the switch and attach them to the thread so we can have a peek inside.  

Looking at the "sh version" output, it looks like switch 1 crashed.  Kindly post the complete output to the following commands: 

• sh log on switch 1 up detail
• sh platform software status con brief

---

@chrisc100 wrote:

77605 -rw- 38123 Jul 15 2021 08:14:35

---

I am interested to see what happened during this time.  

[chrisc100]

Hello, please find the outputs below, unfortunately due to policy I'm unable to upload all the files requested, however a looking at the kernel.rp file looks like evidence of the following bug:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvw20578/?rfs=iqvred

Therefore we will upgrade to a suggested release.

#sh log on switch 1 up detail
--------------------------------------------------------------------------------
UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 02/08/2020 05:37:17
Total uptime : 1 years 23 weeks 4 days 23 hours 51 minutes
Total downtime : 0 years 0 weeks 0 days 8 hours 59 minutes
Number of resets : 5
Number of slot changes : 0
Current reset reason : Reload
Current reset timestamp : 07/15/2021 06:22:41
Current slot : 1
Chassis type : 0
Current uptime : 0 years 1 weeks 1 days 8 hours 5 minutes
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp | Reset | Uptime
MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes
--------------------------------------------------------------------------------
02/08/2020 05:37:17 Reload 0 0 0 0 0
02/08/2020 10:19:02 Reload 0 0 0 1 38
02/06/2021 05:51:37 Reload 0 51 6 19 5
02/06/2021 10:43:17 Reload 0 0 0 0 5
02/26/2021 14:54:49 Image Install 0 2 6 3 5
07/15/2021 06:22:41 Reload 0 19 5 15 7
--------------------------------------------------------------------------------

#sh platform software status con brief
Load Average
Slot Status 1-Min 5-Min 15-Min
1-RP0 Healthy 0.57 0.44 0.42
2-RP0 Healthy 1.01 0.75 0.64
3-RP0 Healthy 0.25 0.21 0.22

Memory (kB)
Slot Status Total Used (Pct) Free (Pct) Committed (Pct)
1-RP0 Healthy 3977740 1156304 (29%) 2821436 (71%) 1510408 (38%)
2-RP0 Healthy 3977740 1822944 (46%) 2154796 (54%) 2726996 (69%)
3-RP0 Healthy 3977740 1741312 (44%) 2236428 (56%) 2689488 (68%)

CPU Utilization
Slot CPU User System Nice Idle IRQ SIRQ IOwait
1-RP0 0 3.70 1.20 0.00 95.09 0.00 0.00 0.00
1 4.39 1.29 0.00 94.30 0.00 0.00 0.00
2 15.00 1.40 0.00 83.60 0.00 0.00 0.00
3 6.20 1.30 0.00 92.50 0.00 0.00 0.00
2-RP0 0 10.68 2.29 0.00 86.81 0.00 0.19 0.00
1 11.30 1.40 0.00 87.30 0.00 0.00 0.00
2 12.31 2.00 0.00 85.68 0.00 0.00 0.00
3 11.20 1.50 0.00 87.20 0.00 0.10 0.00
3-RP0 0 2.49 0.89 0.00 96.60 0.00 0.00 0.00
1 3.89 0.89 0.00 95.20 0.00 0.00 0.00
2 3.10 0.90 0.00 95.99 0.00 0.00 0.00
3 2.99 0.49 0.00 96.50 0.00 0.00 0.00

kernel.rp log:

11978653.407000] oobhal: rx_write: PID = 12055 TID = 13996 invalid skb (80000000d10df380) handle (0x1a10000d10df380) magic/users = deadbeefdeadbeef/1
[11978655.220000] Unhandled kernel unaligned access or invalid instruction[#1]:
[11978655.220000] CPU: 0 PID: 12305 Comm: fed main event Tainted: G O 3.10.104-rt117 #1
[11978655.220000] task: 80000000e949d6d8 ti: 80000000ea268000 task.ti: 80000000ea268000
[11978655.220000] $ 0 : 0000000000000000 ffffffff812c220c 0000000000000000 ffffffff81a5fa70
[11978655.220000] $ 4 : 0000000000000050 ffffffff819404f0 ffffffff816078e4 800000000ac40460
[11978655.220000] $ 8 : ffffffff819404f0 ffffffff819404f0 80000000d10de780 000000002cc70e2a
[11978655.220000] $12 : 80000000ea26b9d8 ffffffff8145682c ffffffff815fdc68 0000000009000000
[11978655.220000] $16 : 8000000108002500 a5be00aa00000000 0000000000000280 00000000000106d0
[11978655.220000] $20 : ffffffff816078e4 0000000000000280 80000000ebc09800 80000000ebc09974
[11978655.220000] $24 : 0000000000000002 ffffffff813f5fe8
[11978655.220000] $28 : 80000000ea268000 80000000ea26b9e0 0000000000000000 ffffffff812c220c
[11978655.220000] Hi : 0000000000007ab8
[11978655.220000] Lo : 0000000000000128
[11978655.220000] epc : ffffffff812c2254 __kmalloc_track_caller+0x8c/0x250
[11978655.220000] Tainted: G O
[11978655.220000] ra : ffffffff812c220c __kmalloc_track_caller+0x44/0x250
[11978655.220000] Status: 10008ce3 KX SX UX KERNEL EXL IE
[11978655.220000] Cause : 00800010
[11978655.220000] BadVA : a5be00aa00000000
[11978655.220000] PrId : 000d900a (Cavium Octeon II)
[11978655.220000] Modules linked in: bridge stp llc contdev_generic(O) dplr_oobhal(O) dplr_oobnd(O) dplr_pci(O) tipc lsmpilodrv(O) liin(O) lfts(O) lsmpi(O) mtd_map(O) rmonbifo_mod(O) cvmx_gpio(O) cvmx_mdio mtdblock mtd_blkdevs cpldvirt(O) cpldha(O) nfsd ipv6 cfi_cmdset_0002 cfi_probe cfi_util gen_probe
[11978655.220000] Process fed main event (pid: 12305, threadinfo=80000000ea268000, task=80000000e949d6d8, tls=000000ffe7e86700)
[11978655.220000] Stack : 0000000000000001 80000000d10de600 80000000ea26ba60 00000000000004d0
ffffffff816078e4 8000000109264000 0000000000000280 ffffffff816077f0
0000000000000000 80000000d10de600 0000000000000131 0000000000000000
00000000000004d0 8000000109264000 0000000000000000 ffffffff816078e4
ffffffff81b56600 ffffffff816ee454 00000000000000b2 ffffffffffffffff
ffffffff811c9db0 0000000000000000 0000000000000000 00000000000004d0
0000000000000000 ffffffff81600674 00000000000000c3 ffffffff811d3d24
80000000e92bd080 0000000000000001 0000000000000001 00000000000000c3
80000000ea26bb60 00000000000000b2 80000000ebc09400 0000000000000000
80000000ea26bc98 80000000ea26bdc0 00000000000000b2 80000000e3e8a580
...
[11978655.220000] Call Trace:
[11978655.220000] [<ffffffff812c2254>] __kmalloc_track_caller+0x8c/0x250
[11978655.220000] [<ffffffff816077f0>] __kmalloc_reserve.isra.3+0x50/0xb8
[11978655.220000] [<ffffffff816078e4>] __alloc_skb+0x8c/0x1a0
[11978655.220000] [<ffffffff81600674>] sock_alloc_send_pskb+0x264/0x3a8
[11978655.220000] [<ffffffff816b7fbc>] unix_stream_sendmsg+0x21c/0x400
[11978655.220000] [<ffffffff815fb640>] sock_sendmsg+0xc8/0x100
[11978655.220000] [<ffffffff815fdd48>] SyS_sendto+0xe0/0x180
[11978655.220000] [<ffffffff8117f1a4>] handle_sys64+0x44/0x68

[Leo Laohoo]

---

@chrisc100 wrote:

unfortunately due to policy I'm unable to upload all the files requested

---

Raise a TAC Case.

[marce1000]

 - FYI : https://www.cisco.com/c/en/us/support/docs/switches/catalyst-3850-series-switches/201070-Troubleshooting-3650-3850-reloads-by-sta.html

 M.

[Mohsin Alam]

it could be due to a memory leak which would force the switch to reload.

Can you monitor the same and see if any process is holding on to memory? 

I would suggest you to upgrade to a safe hub so avoid any bugs affecting older versions. 

## Make sure to mark post as helpful, If it resolved your issue. ##