Cisco 3850 Stack member reload

[FuadG]

Hello,

We have a stack of these switches and member 2 started to reload frequently with no logs available.

I have crashinfo files but there is no support contract for these switches? Can anyone assist with crashinfo analysing if I provide them?

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 16 WS-C3850-12XS 16.6.8 CAT3K_CAA-UNIVERSALK9 INSTALL
2 16 WS-C3850-12XS 16.6.8 CAT3K_CAA-UNIVERSALK9 INSTALL

Thank you
 

[Leo Laohoo]

Post the complete output to the following command: 

dir flash-2:
dir crashinfo-2:
dir flash-2:core
sh log on switch 2 uptime detail

[FuadG]

Hello Leo:

Directory of flash-2:/

88203 -rw- 2097152 Jan 7 2024 18:19:00 +04:00 nvram_config
88178 -rw- 76 Jan 7 2024 18:14:32 +04:00 boothelper.log
88182 -rw- 35 May 12 2020 15:04:06 +04:00 pnp-tech-time
88179 -rw- 15736 Jan 7 2024 18:19:01 +04:00 vlan.dat
88188 -rw- 4742 May 13 2020 16:44:00 +04:00 packages.conf
88180 -rw- 478277 Dec 30 2023 10:43:57 +04:00 crashinfo_20231230-064348-UTC
88193 -rw- 19167872 May 13 2020 16:43:23 +04:00 cat3k_caa-guestshell.16.06.08.SPA.pkg
88202 -rw- 30417560 May 13 2020 16:43:26 +04:00 cat3k_caa-rpbase.16.06.08.SPA.pkg
112225 drwx 4096 Jan 7 2024 18:14:19 +04:00 .installer
88185 -rw- 3145 Jan 7 2024 18:14:19 +04:00 bootloader_evt_handle.log
120241 drwx 4096 Jan 7 2024 18:12:36 +04:00 core
128257 drwx 4096 Jan 7 2024 18:18:17 +04:00 .prst_sync
144289 drwx 4096 Dec 21 2019 23:56:07 +04:00 .rollback_timer
152305 drwx 4096 Dec 21 2019 23:56:13 +04:00 gs_script
160321 drwx 4096 Jan 7 2024 18:14:30 +04:00 dc_profile_dir
88186 -rw- 128103 Jan 7 2024 18:14:46 +04:00 memleak.tcl
88187 -rw- 2097152 Jan 7 2024 18:19:00 +04:00 nvram_config_bkup
88183 -rw- 63698 May 12 2020 15:04:11 +04:00 pnp-tech-discovery-summary
88199 -rw- 336503416 May 13 2020 16:43:25 +04:00 cat3k_caa-rpcore.16.06.08.SPA.pkg
112226 drwx 4096 Apr 22 2020 16:54:40 +04:00 tech_support
120243 drwx 4096 May 7 2020 11:43:06 +04:00 .dbpersist
88190 -rw- 0 May 7 2020 12:23:17 +04:00 rdope_out.txt
128258 drwx 4096 Apr 22 2020 16:56:03 +04:00 onep
88201 -rw- 5091968 May 13 2020 16:43:25 +04:00 cat3k_caa-srdriver.16.06.08.SPA.pkg
88189 -rw- 4742 May 13 2020 16:43:43 +04:00 cat3k_caa-universalk9.16.06.08.SPA.conf
88200 -rw- 16271996 May 13 2020 16:43:25 +04:00 cat3k_caa-webui.16.06.08.SPA.pkg
88184 -rw- 4742 May 13 2020 16:44:00 +04:00 packages.conf.00-
88181 -rw- 358522 Jan 7 2024 18:10:24 +04:00 crashinfo_20240107-181010-UTC
88191 -rw- 71 May 7 2020 12:23:31 +04:00 rdope.log
88194 -rw- 19167872 May 12 2020 17:29:30 +04:00 cat3k_caa-guestshell.16.06.06.SPA.pkg
88198 -rw- 27926203 May 12 2020 17:29:32 +04:00 cat3k_caa-rpbase.16.06.06.SPA.pkg
88195 -rw- 336437880 May 12 2020 17:29:31 +04:00 cat3k_caa-rpcore.16.06.06.SPA.pkg
88197 -rw- 5051008 May 12 2020 17:29:32 +04:00 cat3k_caa-srdriver.16.06.06.SPA.pkg
88196 -rw- 16263804 May 12 2020 17:29:32 +04:00 cat3k_caa-webui.16.06.06.SPA.pkg

Directory of crashinfo-2:/

58 -rw- 19588208 Dec 1 2023 22:26:57 +04:00 system-report_2_20231201-182628-UTC.tar.gz
59 -rw- 31784800 Dec 26 2023 13:05:33 +04:00 system-report_2_20231226-090514-UTC.tar.gz
60 -rw- 38672072 Dec 27 2023 12:29:03 +04:00 system-report_2_20231227-082849-UTC.tar.gz
61 -rw- 30928379 Dec 28 2023 12:01:11 +04:00 system-report_2_20231228-080058-UTC.tar.gz
62 -rw- 12051925 Dec 28 2023 18:14:40 +04:00 GYD-ATS-SW_2_RP_0_trace_archive_0-20231228-141321.tar.gz
63 -rw- 29801415 Jan 7 2024 18:12:47 +04:00 system-report_2_20240107-181235-UTC.tar.gz
64 -rw- 37541271 Dec 30 2023 10:46:36 +04:00 system-report_2_20231230-064622-UTC.tar.gz
11 -rw- 0 Dec 22 2019 00:23:12 +04:00 koops.dat
20 -rw- 117016 May 12 2020 16:47:40 +04:00 system-report_local_20200512-124738-Universal.tar.gz
43177 drwx 1024 May 12 2020 16:27:38 +04:00 ap_crash
34 -rw- 59788 May 13 2020 08:51:51 +04:00 system-report__20200513-045132-UTC.gz
25 -rw- 126464 May 12 2020 18:00:33 +04:00 system-report_local_20200512-140032-Universal.tar.gz
16 -rw- 130555 May 12 2020 15:09:14 +04:00 system-report_local_20200512-110914-Universal.tar.gz
19 -rw- 127957 May 12 2020 15:39:15 +04:00 system-report_local_20200512-113914-Universal.tar.gz
47 -rwx 932465 May 13 2020 09:24:00 +04:00 crashinfo_platform_mgr_20200513-052347-UTC
30 -rw- 152 May 13 2020 15:20:21 +04:00 last_systemreport_log
48 -rw- 59792 May 13 2020 09:24:06 +04:00 system-report__20200513-052347-UTC.gz
51 -rwx 935472 May 13 2020 14:56:43 +04:00 crashinfo_platform_mgr_20200513-105631-UTC
35 -rw- 38 May 13 2020 15:20:19 +04:00 last_systemreport
12 -rw- 129555 May 12 2020 13:27:44 +04:00 system-report_local_20200512-092743-Universal.tar.gz
40 -rwx 933367 May 13 2020 15:15:31 +04:00 crashinfo_platform_mgr_20200513-111518-UTC
45 -rwx 938191 May 13 2020 09:19:18 +04:00 crashinfo_platform_mgr_20200513-051903-UTC
36 -rwx 935446 May 13 2020 15:06:05 +04:00 crashinfo_platform_mgr_20200513-110552-UTC
15 -rw- 128371 May 12 2020 13:47:22 +04:00 system-report_local_20200512-094722-Universal.tar.gz
14 -rw- 127798 May 12 2020 13:41:27 +04:00 system-report_local_20200512-094127-Universal.tar.gz
37 -rw- 59928 May 13 2020 08:56:35 +04:00 system-report__20200513-045615-UTC.gz
46 -rw- 59817 May 13 2020 09:19:24 +04:00 system-report__20200513-051903-UTC.gz
26 -rw- 126400 May 12 2020 18:03:56 +04:00 system-report_local_20200512-140355-Universal.tar.gz
43 -rwx 937082 May 13 2020 09:14:32 +04:00 crashinfo_platform_mgr_20200513-051419-UTC
44 -rw- 62251 May 13 2020 09:14:39 +04:00 system-report__20200513-051419-UTC.gz
28 -rw- 128648 May 12 2020 18:14:04 +04:00 system-report_local_20200512-141404-Universal.tar.gz
13 -rw- 128040 May 12 2020 13:31:04 +04:00 system-report_local_20200512-093103-Universal.tar.gz
17 -rw- 127449 May 12 2020 15:12:34 +04:00 system-report_local_20200512-111233-Universal.tar.gz
56 -rw- 59874 May 13 2020 15:15:40 +04:00 system-report__20200513-111518-UTC.gz
33 -rw- 52 May 13 2020 15:20:13 +04:00 last_crashinfo
54 -rw- 59921 May 13 2020 15:06:15 +04:00 system-report__20200513-110552-UTC.gz
21 -rw- 112909 May 12 2020 16:51:09 +04:00 system-report_local_20200512-125108-Universal.tar.gz
39 -rw- 59934 May 13 2020 09:01:15 +04:00 system-report__20200513-050056-UTC.gz
31 -rw- 384 May 13 2020 15:20:13 +04:00 deleted_crash_files
18 -rw- 128191 May 12 2020 15:35:55 +04:00 system-report_local_20200512-113554-Universal.tar.gz
42 -rwx 931619 May 13 2020 15:20:13 +04:00 crashinfo_platform_mgr_20200513-112000-UTC
38 -rwx 937206 May 13 2020 15:10:49 +04:00 crashinfo_platform_mgr_20200513-111036-UTC
50 -rw- 59900 May 13 2020 13:46:15 +04:00 system-report__20200513-094556-UTC.gz
49 -rwx 930200 May 13 2020 13:46:09 +04:00 crashinfo_platform_mgr_20200513-094556-UTC
32 -rwx 934610 May 13 2020 15:01:26 +04:00 crashinfo_platform_mgr_20200513-110113-UTC
53 -rw- 59854 May 13 2020 15:01:32 +04:00 system-report__20200513-110113-UTC.gz
22 -rw- 111684 May 12 2020 16:54:40 +04:00 system-report_local_20200512-125438-Universal.tar.gz
47289 drwx 271360 Jan 8 2024 10:38:26 +04:00 tracelogs
24 -rw- 125926 May 12 2020 17:53:21 +04:00 system-report_local_20200512-135320-Universal.tar.gz
55 -rw- 59865 May 13 2020 15:10:55 +04:00 system-report__20200513-111036-UTC.gz
57 -rw- 59886 May 13 2020 15:20:19 +04:00 system-report__20200513-112000-UTC.gz
27 -rw- 125061 May 12 2020 18:07:19 +04:00 system-report_local_20200512-140718-Universal.tar.gz
41 -rw- 59845 May 13 2020 09:05:55 +04:00 system-report__20200513-050536-UTC.gz
52 -rw- 59808 May 13 2020 14:56:50 +04:00 system-report__20200513-105631-UTC.gz
23 -rw- 128897 May 12 2020 17:49:59 +04:00 system-report_local_20200512-134959-Universal.tar.gz
29 -rw- 0 May 13 2020 15:19:50 +04:00 cilogs

Directory of flash-2:/core/

136273 drwx 4096 Dec 21 2019 23:56:07 +04:00 modules
120242 -rw- 1 Jan 8 2024 10:33:58 +04:00 .callhome

UPTIME SUMMARY INFORMATION
--------------------------------------------------------------------------------
First customer power on : 12/21/2019 20:11:49
Total uptime : 3 years 33 weeks 3 days 19 hours 14 minutes
Total downtime : 0 years 21 weeks 1 days 14 hours 54 minutes
Number of resets : 73
Number of slot changes : 4
Current reset reason : Reload reason content is absent
Current reset timestamp : 01/07/2024 14:15:30
Current slot : 255
Chassis type : 0
Current uptime : 0 years 0 weeks 0 days 16 hours 5 minutes
--------------------------------------------------------------------------------

--------------------------------------------------------------------------------
UPTIME CONTINUOUS INFORMATION
--------------------------------------------------------------------------------
Time Stamp | Reset | Uptime
MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes
--------------------------------------------------------------------------------
12/21/2019 20:11:49 Reload 0 0 0 0 0
12/21/2019 20:17:01 Reload 0 0 0 0 0
12/21/2019 20:24:46 Reload 0 0 0 0 0
02/25/2020 02:02:24 Reload 0 0 0 0 0
02/25/2020 02:48:25 Reload 0 0 0 0 5
04/22/2020 09:56:21 Reload 0 0 0 0 0
04/22/2020 12:55:36 Reload 0 0 0 2 5
04/28/2020 09:52:26 Reload 0 0 0 0 5
05/07/2020 06:24:00 Reload 0 0 0 2 5
05/07/2020 07:45:59 Reload 0 0 0 1 5
05/07/2020 08:01:13 Reload 0 0 0 0 0
05/07/2020 08:17:37 Reload 0 0 0 0 5
05/07/2020 08:22:52 Reload 0 0 0 0 0
05/07/2020 08:46:21 Reload 0 0 0 0 5
05/07/2020 08:55:53 Reload 0 0 0 0 5
05/12/2020 09:12:17 Reload 0 0 0 4 5
05/12/2020 09:26:55 Reload 0 0 0 0 5
05/12/2020 09:30:16 Reload reason content is absent 0 0 0 0 0
05/12/2020 09:40:40 Reload reason content is absent 0 0 0 0 0
05/12/2020 09:46:35 Reload reason content is absent 0 0 0 0 0
05/12/2020 09:49:55 Reload reason content is absent 0 0 0 0 0
05/12/2020 09:56:17 Reload reason content is absent 0 0 0 0 0
05/12/2020 10:34:06 Reload 0 0 0 0 0
05/12/2020 11:02:18 Reload 0 0 0 0 5
05/12/2020 11:08:26 Reload 0 0 0 0 0
05/12/2020 11:11:46 Reload reason content is absent 0 0 0 0 0
05/12/2020 11:35:07 Reload reason content is absent 0 0 0 0 0
05/12/2020 11:38:27 Reload reason content is absent 0 0 0 0 0
05/12/2020 11:41:47 Reload reason content is absent 0 0 0 0 0
05/12/2020 11:45:37 Reload reason content is absent 0 0 0 0 0
05/12/2020 11:56:04 Reload 0 0 0 0 5
05/12/2020 12:00:59 Reload 0 0 0 0 0
05/12/2020 12:05:08 Reload 0 0 0 0 0
05/12/2020 12:27:01 Reload 0 0 0 0 0
05/12/2020 12:46:54 Reload 0 0 0 0 5
05/12/2020 12:50:26 Reload 0 0 0 0 0
05/12/2020 12:53:56 Reload 0 0 0 0 0
05/12/2020 13:20:18 Reload 0 0 0 0 0
05/12/2020 13:41:28 Reload 0 0 0 17 50
05/12/2020 13:49:08 Reload 0 0 0 0 0
05/12/2020 13:52:30 Reload reason content is absent 0 0 0 0 0
05/12/2020 13:59:42 Reload reason content is absent 0 0 0 0 0
05/12/2020 14:03:05 Reload reason content is absent 0 0 0 0 0
05/12/2020 14:06:28 Reload reason content is absent 0 0 0 0 0
05/12/2020 14:13:11 Reload reason content is absent 0 0 0 0 0
05/12/2020 14:18:47 Reload reason content is absent 0 0 0 0 0
05/13/2020 04:40:10 Reload 0 0 0 0 0
05/13/2020 09:55:07 Reload 0 0 0 0 5
05/13/2020 11:26:16 Reload 0 0 0 0 5
05/13/2020 11:46:25 Reload 0 0 0 0 5
05/13/2020 11:56:50 Reload 0 0 0 0 5
05/13/2020 12:07:05 Reload 0 0 0 0 5
05/13/2020 12:31:07 Reload 0 0 0 0 3
05/13/2020 12:57:32 Reload Command 0 0 0 0 3
05/13/2020 13:38:41 Reload 0 0 0 0 4
05/13/2020 13:44:19 Reload 0 0 0 0 0
05/13/2020 13:49:11 Reload 0 0 0 0 0
05/13/2020 13:59:34 Reload 0 0 0 0 5
05/19/2020 08:02:26 Reload 0 0 0 0 5
05/19/2020 12:04:00 Reload 0 0 0 3 4
05/19/2020 12:40:48 Reload 0 0 0 0 0
05/19/2020 12:56:50 Reload Command 0 0 0 0 0
05/20/2020 04:40:24 Reload Command 0 0 0 0 5
05/20/2020 12:42:34 Reload 0 0 0 7 5
05/20/2020 13:23:11 Reload Command 0 0 0 0 5
05/20/2020 14:41:24 Reload Command 0 0 0 0 0
12/01/2023 18:29:40 Reload reason content is absent 3 27 6 2 45
12/26/2023 09:08:16 Reload reason content is absent 0 3 3 14 5
12/27/2023 08:31:48 Reload reason content is absent 0 0 0 23 5
12/28/2023 08:03:53 Reload reason content is absent 0 0 0 23 5
12/29/2023 06:05:29 Reload reason content is absent 0 0 0 21 5
12/30/2023 06:49:21 Reload reason content is absent 0 0 1 0 5
12/30/2023 16:21:37 Reload Command 0 0 0 9 5
01/07/2024 14:15:30 Reload reason content is absent 0 1 0 21 5
--------------------------------------------------------------------------------

[Leo Laohoo]

---

@FuadG wrote:

88181 -rw- 358522 Jan 7 2024 18:10:24 +04:00 crashinfo_20240107-181010-UTC

---

Please share this file.

[FuadG]

Hello Lea,

pls see attached

[Leo Laohoo]

---

@FuadG wrote:

-Traceback= 1#98a0c9ffdd99f9f477a56440fdf2d874  :AAAE653000+4A6C240 :AAAE653000+4A6B3AC :AAAE653000+4A6B75C :AAAE653000+193F010 :AAAE653000+81176BC :AAAE653000+3663B3C :AAAE653000+35CAE24 :AAAE653000+35CB150 :AAAE653000+35C6750 :AAAE653000+3644E2C :AAAE653000+6CD1608
IOSD-WATCHDOG: Process = REDUNDANCY peer monitor

---

I see this Traceback generated before the crash.  

Please post the complete output to the command "sh platform resources".  

[FuadG]

Pls see below

sh platform resources
**State Acronym: H - Healthy, W - Warning, C - Critical
Resource Usage Max Warning Critical State
----------------------------------------------------------------------------------------------------
Control Processor 28.05% 100% 90% 95% H
DRAM 2205MB(56%) 3887MB 90% 95% H

[Leo Laohoo]

---

@FuadG wrote:

DRAM 2205MB(56%)

---

That's high.

Please post the first page output to the command "sh process memory sort platform location switch 2 rp0".

[FuadG]

Hello Leo,

There is no such command:

sh process memory sorted ?
allocated Sort based on total allocated memory
getbufs Sort based on buffers got
holding Sort based on holding memory
| Output modifiers
<cr>

[Leo Laohoo]

Apologies, I am doing all this on memory.  

Try this:  sh process memory platform sort location switch 2 rp0

[FuadG]

Hello Leo,

No problem, however this command gets rejected after switch 2

sh process memory platform sort location switch 2 ?
0 SPA-Inter-Processor slot 0
F0 Embedded-Service-Processor slot 0
R0 Route-Processor slot 0

[Leo Laohoo]

Try:  sh process memory platform sort location switch 2 r0

[FuadG]

sh process memory platform sort location switch 2 r0
System memory: 3980524K total, 3261784K used, 718740K free,
Lowest: 718740K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
20987 117 363224 136 71576 363224 2564608 fed main event
9728 177486 686044 136 120 686044 2081476 linux_iosd-imag
30471 8824 1305832 136 1150684 1305832 1967796 fman_fp_image
20112 312 181992 136 2752 181992 1575348 sif_mgr
18097 1015 184252 136 2520 184252 1304836 platform_mgr
8155 287 183728 136 7492 183728 895996 cli_agent
11534 649 173488 136 19028 173488 864540 smand
8710 170 186588 136 3332 186588 861624 dbm
3139 127 176372 0 268 176372 772216 smd
8977 8482 168380 136 1808 168380 740368 fman_rp
2337 430 131840 136 1632 131840 732060 repm
12557 251 126712 136 6580 126712 713680 tms
13982 38 121672 136 1196 121672 710828 bt_logger
15057 521 124308 136 2144 124308 705956 hman
17205 200 115392 136 268 115392 702412 nif_mgr
10714 40 122488 136 1152 122488 702380 psd
21066 418 122576 136 1632 122576 701664 stack_mgr
8317 114 118244 136 1084 118244 701212 cmm
16168 147 120196 136 1780 120196 701156 lman
1798 45 118900 136 1892 118900 698888 plogd
24283 61 117988 136 940 117988 698504 epc_ws_liaison
15904 73 116420 136 432 116420 693176 keyman
3436 604 3952 132 132 3952 212368 libvirtd
3426 753 1920 132 132 1920 17168 virtlogd
13718 7 3140 136 148 3140 16820 auto_upgrade_se
32152 118 2732 136 172 2732 12880 btelnet
95 314 3736 132 132 3736 12428 systemd-journal
16345 974 7552 136 5908 7552 10248 ncd.sh
15433 974 7456 136 5784 7456 10124 issu_stack.sh
13208 974 7256 136 5648 7256 9988 auto_upgrade_cl
25945 974 6508 136 5640 6508 9980 issu_stack.sh
25939 974 6420 136 5640 6420 9980 issu_stack.sh
30958 7 1848 132 148 1848 9820 rotee
30501 7 1848 132 148 1848 9820 rotee
29903 7 1852 136 148 1852 8928 rotee
21014 7 1928 136 148 1928 8928 rotee
20420 7 1848 136 148 1848 8928 rotee
20290 7 1860 136 148 1860 8928 rotee
19802 7 1888 136 148 1888 8928 rotee
18912 7 1932 136 148 1932 8928 rotee
17506 7 1860 136 148 1860 8928 rotee
16988 7 1904 136 148 1904 8928 rotee
16361 7 1852 136 148 1852 8928 rotee
15894 7 1856 136 148 1856 8928 rotee
15531 7 1848 136 148 1848 8928 rotee
14825 7 1960 136 148 1960 8928 rotee
14816 7 1852 136 148 1852 8928 rotee
14250 7 1848 136 148 1848 8928 rotee
13602 7 1856 136 148 1856 8928 rotee
12249 7 1880 136 148 1880 8928 rotee
11159 7 1888 136 148 1888 8928 rotee
9292 7 1912 136 148 1912 8928 rotee
8603 7 1880 136 148 1880 8928 rotee
8162 7 1880 136 148 1880 8928 rotee
7710 7 1880 136 148 1880 8928 rotee
7451 7 1880 136 148 1880 8928 rotee
1801 7 1852 136 148 1852 8928 rotee
18453 7 1872 132 148 1872 8924 rotee
12148 7 1876 132 148 1876 8924 rotee
12018 7 1848 132 148 1848 8924 rotee
24024 7 1852 136 148 1852 8808 rotee
19458 7 1852 136 148 1852 8808 rotee
18307 7 1844 136 148 1844 8808 rotee
17224 7 1848 136 148 1848 8808 rotee
16689 7 1848 136 148 1848 8808 rotee
15166 7 1848 136 148 1848 8808 rotee
14325 7 1856 136 148 1856 8808 rotee
13229 7 1848 136 148 1848 8808 rotee
12931 7 1852 136 148 1852 8808 rotee
12604 7 1856 136 148 1856 8808 rotee
10393 7 1888 136 148 1888 8808 rotee
2498 7 1852 136 148 1852 8808 rotee

[Leo Laohoo · Accepted solution]

---

@FuadG wrote:

20987 117 363224 136 71576 363224 2564608 fed main event

---

Raise a TAC Case and ask them to verify if this is CSCvn46171.