Solved: Silent reload C3850

Graham Murison · ‎09-25-2023

I am building a new network to run in parallel during the transition to new building addition.
I have purchased two WS-C3850-48XS, however due to strict budget I purchased them refurbished and no Smartnet
I deployed them in StackWsie Vrtual.
Over the last couple of months, there has been two slient reload of one of the switches.
Both are running on 16.06.10
I find that when the event occurs, one switch is powering down and the other is unreachable.
I have issued several commands based on other posts I have read, so I hope this gives enough information.
I see the last power on reason is reload, but I didn't issue that command in these cases.

A bug search showed a few similar result for different versions, but that it should be fixed in 16.6.10.
CSCvi15897
CSCvn22162
CSCvj72706

Any ideas for the next step to check?
Is IOS version upgrade the only thing to try (Other than Smartnet)?

"Edited to include another bud id"

Leo Laohoo · ‎09-26-2023

@Graham Murison wrote:
Will be upgrading over the next few days and report back.

I would recommend performing Workaround 2 (manually downloading the certificate) instead of upgrading the firmware -- Less painful in the long run.

View solution in original post

adarsing · ‎09-25-2023

do you have:

>crash file firing the issue under the crashinfo?

>any logs prior to the reload ?

balaji.bandi · ‎09-25-2023

i would try to upgrade to 16.12.X version and check.

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

Graham Murison · ‎09-25-2023

Thanks for the quick replies!

@adarsing
I see this in the logs around the time of the event:
*Sep 22 2023 21:10:23.782: %NIF_MGR-6-PORT_LINK_DOWN: Switch 2 R0/0: nif_mgr: Port 0, LPN 65 on front side stack link 0 is DOWN.
*Sep 22 2023 21:10:23.782: %NIF_MGR-6-PORT_CONN_DISCONNECTED: Switch 2 R0/0: nif_mgr: Port 0, LPN 65 on front side stack link 0 connection has DISCONNECTED: CONN_ERR_PORT_LINK_DOWN_EVENT
*Sep 22 2023 21:10:24.790: %LINEPROTO-5-UPDOWN: Line protocol on Interface FortyGigabitEthernet2/1/1, changed state to down
*Sep 22 2023 21:10:24.871: %LINEPROTO-5-UPDOWN: Line protocol on Interface FortyGigabitEthernet2/1/2, changed state to down
*Sep 22 2023 21:10:24.878: %NIF_MGR-6-STACK_LINK_DOWN: Switch 2 R0/0: nif_mgr: Front side stack link 0 is DOWN.
*Sep 22 2023 21:10:24.878: %STACKMGR-6-STACK_LINK_CHANGE: Switch 2 R0/0: stack_mgr: Stack port 1 on Switch 2 is down
*Sep 22 2023 21:10:24.962: %LINEPROTO-5-UPDOWN: Line protocol on Interface FortyGigabitEthernet2/1/4, changed state to down
*Sep 22 2023 21:10:24.963: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_NOT_PRESENT)
*Sep 22 2023 21:10:24.964: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_DOWN)
*Sep 22 2023 21:10:24.964: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_REDUNDANCY_STATE_CHANGE)
*Sep 22 2023 21:10:25.212: %HMANRP-5-CHASSIS_DOWN_EVENT: Chassis 1 gone DOWN!
*Sep 22 2023 21:10:25.792: %LINK-3-UPDOWN: Interface FortyGigabitEthernet2/1/1, changed state to down
*Sep 22 2023 21:10:25.873: %LINK-3-UPDOWN: Interface FortyGigabitEthernet2/1/2, changed state to down
*Sep 22 2023 21:10:25.898: %STACKMGR-1-DUAL_ACTIVE_CFG_MSG: Switch 2 R0/0: stack_mgr: Dual Active Detection links are not available anymore
*Sep 22 2023 21:10:25.899: %LINK-3-UPDOWN: Interface FortyGigabitEthernet2/1/4, changed state to down

@adarsing I only see trace archives in the crashinfo directory.

@balaji.bandi What would be the impact on the feature set if I upgrade to that version without Smart licensing?

Thanks.

balaji.bandi · ‎09-26-2023

upgrade should not cause the issue (make sure you read the release notes before upgrading)

BB

***** Rate All Helpful Responses *****

How to Ask The Cisco Community for Help

adarsing · ‎09-25-2023

We need to collect the trace archive and decode it to know more. Bit tedious 🥶.

If you upgrade it to 16.9.x version or above, the smart license feature will be enabled automatically with an evaluation period of 90 days. post that even if it expired there will be NO impact on ANY functionality of the switch.

Graham Murison · ‎09-25-2023

@adarsing Great thanks. Right now only have console access, so will get the trace archive uploaded tomorrow when I have physical access again. Also thank for clarifying the second point, which I guess is due to perpetual license from previous version of IOS.

Leo Laohoo · ‎09-25-2023

Post the complete output to the following commands:

dir crashinfo-1:tracelogs
dir crashinfo-2:tracelogs

Graham Murison · ‎09-25-2023

This I can do today.
The event occurred Sept 22 at ~21:00 UTC

sw-fl2-aggr-01#dir crashinfo-1:tracelogs
Directory of crashinfo-1:/tracelogs/

36180 -rwx 1726 Sep 22 2023 21:20:44 +00:00 auto_upgrade_client_sh_pmanlog_R0-0.12730_0.20230922212043.bin.gz
36201 -rwx 1509 Sep 22 2023 21:20:45 +00:00 auto_upgrade_client_sh_pmanlog_R0-0.12730_1.20230922212045.bin.gz
36204 -rwx 381 Sep 22 2023 21:20:26 +00:00 binos_R0-0.4543_0.20230922212026.bin.gz
36205 -rwx 1642 Sep 22 2023 21:20:38 +00:00 flashutil_R0-0.10521_0.20230922212038.bin.gz
36206 -rwx 1653 Sep 22 2023 21:20:38 +00:00 flashutil_R0-0.10554_0.20230922212038.bin.gz
36207 -rwx 1637 Sep 22 2023 21:20:38 +00:00 flashutil_R0-0.10595_0.20230922212038.bin.gz
36208 -rwx 551 Sep 22 2023 21:20:40 +00:00 inst_issu_check_R0-0.11966_0.20230922212040.bin.gz
36209 -rwx 403 Sep 22 2023 21:20:28 +00:00 inst_publish_retrieve_sw_sets_R0-0.4866_0.20230922212028.bin.gz
36210 -rwx 395 Sep 22 2023 21:20:32 +00:00 inst_rollback_timer_R0-0.6366_0.20230922212032.bin.gz
36211 -rwx 694 Sep 22 2023 21:20:19 +00:00 install_engine_R0-0.1294_0.20230922212019.bin.gz
36212 -rwx 1559 Sep 22 2023 21:20:55 +00:00 run_ioxn_secure_dir_sh_pmanlog_R0-0.18515_0.20230922212053.bin.gz
36213 -rwx 2282 Sep 22 2023 21:21:28 +00:00 sif_mgr_pmanlog_R0-0.18974_0.20230922212054.bin.gz
36214 -rwx 726 Sep 22 2023 21:20:26 +00:00 tdl_boottime_R0-0.4719_0.20230922212026.bin.gz
36215 -rwx 897 Sep 22 2023 21:20:55 +00:00 reflector_R0-0.4070_0.20230922212024.bin.gz
36216 -rwx 9251 Sep 22 2023 21:21:59 +00:00 platform_mgr_R0-0.18193_0.20230922212110.bin.gz
36189 -rwx 22951 Sep 24 2023 13:54:46 +00:00 IOSRP_R0-0.10677_23.20230924121005.bin.gz
36161 -rwx 22967 Sep 25 2023 20:08:50 +00:00 IOSRP_R0-0.10677_40.20230925182343.bin.gz
36178 -rwx 11527 Sep 20 2023 13:16:36 +00:00 btrace_rotate_sh_R0-0.14935_4.20230916060623.bin.gz
36158 -rwx 23098 Sep 25 2023 16:38:30 +00:00 IOSRP_R0-0.10677_38.20230925145400.bin.gz
36199 -rwx 51545 Sep 20 2023 17:23:29 +00:00 sec_key_agent_R0-0.15986_63.20230920091008.bin.gz
36200 -rwx 23054 Sep 25 2023 11:18:51 +00:00 IOSRP_R0-0.10677_35.20230925093301.bin.gz
36198 -rwx 51362 Sep 18 2023 07:36:19 +00:00 sec_key_agent_R0-0.15986_56.20230917232254.bin.gz
36166 -rwx 51531 Sep 22 2023 10:40:04 +00:00 sec_key_agent_R0-0.15986_68.20230922022640.bin.gz
36181 -rwx 22990 Sep 25 2023 00:37:53 +00:00 IOSRP_R0-0.10677_29.20230924225208.bin.gz
36190 -rwx 51561 Sep 19 2023 08:22:13 +00:00 sec_key_agent_R0-0.15986_59.20230919000853.bin.gz
36153 -rwx 23134 Sep 24 2023 17:29:13 +00:00 IOSRP_R0-0.10677_25.20230924154216.bin.gz
36154 -rwx 23004 Sep 25 2023 13:05:49 +00:00 IOSRP_R0-0.10677_36.20230925111942.bin.gz
36182 -rwx 51592 Sep 22 2023 18:55:21 +00:00 sec_key_agent_R0-0.15986_69.20230922104200.bin.gz
36152 -rwx 51420 Sep 20 2023 00:52:51 +00:00 sec_key_agent_R0-0.15986_61.20230919163930.bin.gz
36148 -rwx 22980 Sep 24 2023 03:13:16 +00:00 IOSRP_R0-0.10677_17.20230924012835.bin.gz
36217 -rwx 1276 Sep 22 2023 21:21:44 +00:00 epc_ws_liaison_F0-0.24526_0.20230922212138.bin.gz
36218 -rwx 883 Sep 22 2023 21:22:23 +00:00 epc_ws_liaison_F0-0.24526_1.20230922212208.bin.gz
36219 -rwx 819 Sep 22 2023 21:22:18 +00:00 reflector_R0-0.4070_1.20230922212143.bin.gz
36157 -rwx 22990 Sep 25 2023 14:53:08 +00:00 IOSRP_R0-0.10677_37.20230925130638.bin.gz
36156 -rwx 51624 Sep 21 2023 18:09:27 +00:00 sec_key_agent_R0-0.15986_66.20230921095602.bin.gz
36179 -rwx 51716 Sep 17 2023 23:20:58 +00:00 sec_key_agent_R0-0.15986_55.20230917150738.bin.gz
36162 -rwx 51406 Sep 21 2023 01:38:49 +00:00 sec_key_agent_R0-0.15986_64.20230920172525.bin.gz
36176 -rwx 51895 Sep 19 2023 00:06:57 +00:00 sec_key_agent_R0-0.15986_58.20230918155332.bin.gz
36191 -rwx 51667 Sep 17 2023 15:05:42 +00:00 sec_key_agent_R0-0.15986_54.20230917065217.bin.gz
36185 -rwx 9371 Sep 21 2023 03:57:48 +00:00 btrace_rotate_sh_F0-0.21176_3.20230915164008.bin.gz
36155 -rwx 23062 Sep 24 2023 08:34:04 +00:00 IOSRP_R0-0.10677_20.20230924064921.bin.gz
36168 -rwx 51587 Sep 21 2023 09:54:06 +00:00 sec_key_agent_R0-0.15986_65.20230921014045.bin.gz
36171 -rwx 23050 Sep 24 2023 05:01:46 +00:00 IOSRP_R0-0.10677_18.20230924031407.bin.gz
36160 -rwx 23013 Sep 24 2023 22:51:20 +00:00 IOSRP_R0-0.10677_28.20230924210528.bin.gz
36150 -rwx 23053 Sep 24 2023 06:48:29 +00:00 IOSRP_R0-0.10677_19.20230924050238.bin.gz
36184 -rwx 2092 Sep 22 2023 21:24:41 +00:00 plogd_R0-0.2563_0.20230922212441.bin.gz
36220 -rwx 1529 Sep 22 2023 21:21:09 +00:00 bt_logger_R0-0.14108_0.20230922212059.bin.gz
36221 -rwx 883 Sep 22 2023 21:23:05 +00:00 epc_ws_liaison_F0-0.24526_2.20230922212247.bin.gz
36222 -rwx 1518 Sep 22 2023 21:20:58 +00:00 droputil_R0-0.4063_0.20230922212024.bin.gz
36159 -rwx 22975 Sep 24 2023 10:20:30 +00:00 IOSRP_R0-0.10677_21.20230924083455.bin.gz
36174 -rwx 23011 Sep 24 2023 15:41:29 +00:00 IOSRP_R0-0.10677_24.20230924135536.bin.gz
36175 -rwx 23168 Sep 25 2023 02:26:23 +00:00 IOSRP_R0-0.10677_30.20230925003845.bin.gz
36165 -rwx 11940 Sep 16 2023 02:58:50 +00:00 btrace_rotate_sh_R0-0.14935_3.20230912012515.bin.gz
36183 -rwx 22938 Sep 25 2023 04:11:25 +00:00 IOSRP_R0-0.10677_31.20230925022711.bin.gz
36167 -rwx 51615 Sep 22 2023 02:24:44 +00:00 sec_key_agent_R0-0.15986_67.20230921181123.bin.gz
36163 -rwx 23120 Sep 25 2023 21:56:23 +00:00 IOSRP_R0-0.10677_41.20230925200942.bin.gz
36164 -rwx 23113 Sep 25 2023 07:45:20 +00:00 IOSRP_R0-0.10677_33.20230925055826.bin.gz
36146 -r-x 27042 Sep 18 2023 00:21:52 +00:00 dmesg
36147 -rw- 11 Sep 18 2023 00:21:52 +00:00 timestamp
36186 -rwx 22991 Sep 25 2023 09:32:12 +00:00 IOSRP_R0-0.10677_34.20230925074611.bin.gz
36187 -rwx 51794 Sep 20 2023 09:08:12 +00:00 sec_key_agent_R0-0.15986_62.20230920005447.bin.gz
36169 -rwx 23144 Sep 24 2023 12:09:17 +00:00 IOSRP_R0-0.10677_22.20230924102122.bin.gz
36172 -rwx 51425 Sep 18 2023 15:51:36 +00:00 sec_key_agent_R0-0.15986_57.20230918073815.bin.gz
36151 -rwx 51530 Sep 24 2023 04:39:43 +00:00 sec_key_agent_R0-0.16014_3.20230923202623.bin.gz
36223 -rwx 1415 Sep 22 2023 21:23:39 +00:00 reflector_R0-0.4070_2.20230922212305.bin.gz
36224 -rwx 41169 Sep 22 2023 21:24:41 +00:00 fed_F0-0.21185_0.20230922212118.bin.gz
36225 -rwx 680 Sep 22 2023 21:24:37 +00:00 plogd_F0-0.32111_0.20230922212437.bin.gz
36227 -rwx 5864 Sep 22 2023 21:24:42 +00:00 fed_F0-0.21185_1.20230922212441.bin.gz
36173 -rwx 51654 Sep 24 2023 12:55:04 +00:00 sec_key_agent_R0-0.16014_4.20230924044139.bin.gz
36149 -rwx 22921 Sep 25 2023 18:22:53 +00:00 IOSRP_R0-0.10677_39.20230925163917.bin.gz
36195 -rwx 51510 Sep 25 2023 21:56:19 +00:00 sec_key_agent_R0-0.16014_8.20230925134255.bin.gz
36192 -rwx 51478 Sep 19 2023 16:37:34 +00:00 sec_key_agent_R0-0.15986_60.20230919082409.bin.gz
36197 -rwx 22974 Sep 24 2023 19:17:55 +00:00 IOSRP_R0-0.10677_26.20230924173003.bin.gz
36248 -rwx 51435 Sep 23 2023 20:24:27 +00:00 sec_key_agent_R0-0.16014_2.20230923121102.bin.gz
36177 -rwx 51656 Sep 25 2023 05:25:42 +00:00 sec_key_agent_R0-0.16014_6.20230924211217.bin.gz
36202 -rwx 23071 Sep 24 2023 21:04:38 +00:00 IOSRP_R0-0.10677_27.20230924191848.bin.gz
36226 -rwx 1529 Sep 22 2023 21:24:37 +00:00 plogd_pmanlog_F0-0.31711_0.20230922212428.bin.gz
36228 -rwx 9355 Sep 22 2023 21:26:27 +00:00 fed_F0-0.21185_2.20230922212442.bin.gz
36229 -rwx 2780 Sep 22 2023 21:24:32 +00:00 console_relay_get_data_R0-0.1582_0.20230922212432.bin.gz
36230 -rwx 2779 Sep 22 2023 21:24:33 +00:00 console_relay_get_data_R0-0.2191_0.20230922212433.bin.gz
36231 -rwx 2786 Sep 22 2023 21:24:34 +00:00 console_relay_get_data_R0-0.2887_0.20230922212434.bin.gz
36232 -rwx 2696 Sep 22 2023 21:24:30 +00:00 console_relay_get_data_R0-0.32578_0.20230922212430.bin.gz
36233 -rwx 2781 Sep 22 2023 21:24:35 +00:00 console_relay_get_data_R0-0.3410_0.20230922212435.bin.gz
36239 -rwx 12000 Sep 22 2023 21:26:36 +00:00 pvp_R0-0.12273_0.20230922212042.bin.gz
36234 -rwx 2778 Sep 22 2023 21:24:36 +00:00 console_relay_get_data_R0-0.3837_0.20230922212436.bin.gz
36235 -rwx 2792 Sep 22 2023 21:24:37 +00:00 console_relay_get_data_R0-0.4183_0.20230922212437.bin.gz
36236 -rwx 2783 Sep 22 2023 21:24:38 +00:00 console_relay_get_data_R0-0.4553_0.20230922212438.bin.gz
36237 -rwx 2781 Sep 22 2023 21:24:39 +00:00 console_relay_get_data_R0-0.4919_0.20230922212439.bin.gz
36238 -rwx 2730 Sep 22 2023 21:24:31 +00:00 console_relay_get_data_R0-0.975_0.20230922212431.bin.gz
36241 -rwx 479 Sep 22 2023 21:27:43 +00:00 cfgwr_R0-0.19109_0.20230922212743.bin.gz
36242 -rwx 479 Sep 22 2023 21:27:43 +00:00 cfgwr_R0-0.19110_0.20230922212743.bin.gz
36243 -rwx 482 Sep 22 2023 21:27:43 +00:00 cfgwr_R0-0.19111_0.20230922212743.bin.gz
36244 -rwx 478 Sep 22 2023 21:27:43 +00:00 cfgwr_R0-0.19112_0.20230922212743.bin.gz
36245 -rwx 482 Sep 22 2023 21:27:43 +00:00 cfgwr_R0-0.19113_0.20230922212743.bin.gz
36246 -rwx 15700 Sep 22 2023 21:27:40 +00:00 linux_iosd_image_pmanlog_R0-0.10244_0.20230922212632.bin.gz
36170 -rwx 63037 Sep 23 2023 03:53:49 +00:00 sec_key_agent_R0-0.16014_0.20230922212058.bin.gz
36194 -rwx 51594 Sep 23 2023 12:09:06 +00:00 sec_key_agent_R0-0.16014_1.20230923035545.bin.gz
36240 -rwx 51666 Sep 24 2023 21:10:21 +00:00 sec_key_agent_R0-0.16014_5.20230924125700.bin.gz
36193 -rwx 22991 Sep 25 2023 05:57:36 +00:00 IOSRP_R0-0.10677_32.20230925041213.bin.gz
36203 -rwx 51449 Sep 25 2023 13:40:59 +00:00 sec_key_agent_R0-0.16014_7.20230925052738.bin.gz

254803968 bytes total (236978176 bytes free)
sw-fl2-aggr-01#dir crashinfo-2:tracelogs
Directory of crashinfo:/tracelogs/

30123 -r-x 27735 Sep 23 2023 07:26:04 +00:00 dmesg
30125 -rw- 11 Sep 23 2023 07:26:04 +00:00 timestamp
30183 -rwx 815 Sep 22 2023 21:24:24 +00:00 flashutil_R0-0.30146_0.20230922212424.bin.gz
30140 -rwx 22788 Sep 24 2023 14:22:12 +00:00 IOSRP_R0-0.5735_1668.20230924123716.bin.gz
30173 -rwx 22882 Sep 25 2023 07:58:44 +00:00 IOSRP_R0-0.5735_1678.20230925061356.bin.gz
30136 -rwx 22931 Sep 24 2023 23:10:28 +00:00 IOSRP_R0-0.5735_1673.20230924212540.bin.gz
30156 -rwx 22850 Sep 25 2023 00:56:04 +00:00 IOSRP_R0-0.5735_1674.20230924231116.bin.gz
30139 -rwx 22788 Sep 24 2023 10:50:44 +00:00 IOSRP_R0-0.5735_1666.20230924090556.bin.gz
30127 -rwx 22806 Sep 25 2023 09:44:28 +00:00 IOSRP_R0-0.5735_1679.20230925075940.bin.gz
30131 -rwx 22851 Sep 24 2023 16:07:48 +00:00 IOSRP_R0-0.5735_1669.20230924142300.bin.gz
30122 -rwx 22874 Sep 24 2023 17:53:24 +00:00 IOSRP_R0-0.5735_1670.20230924160836.bin.gz
30143 -rwx 22814 Sep 24 2023 21:24:52 +00:00 IOSRP_R0-0.5735_1672.20230924193956.bin.gz
30157 -rwx 22800 Sep 24 2023 05:33:48 +00:00 IOSRP_R0-0.5735_1663.20230924034900.bin.gz
30177 -rwx 22844 Sep 25 2023 20:17:16 +00:00 IOSRP_R0-0.5735_1685.20230925183228.bin.gz
30153 -rwx 22796 Sep 24 2023 03:48:04 +00:00 IOSRP_R0-0.5735_1662.20230924020316.bin.gz
30134 -rwx 22853 Sep 24 2023 09:05:08 +00:00 IOSRP_R0-0.5735_1665.20230924072020.bin.gz
30141 -rwx 22849 Sep 25 2023 11:30:12 +00:00 IOSRP_R0-0.5735_1680.20230925094516.bin.gz
30162 -rwx 22825 Sep 25 2023 02:41:48 +00:00 IOSRP_R0-0.5735_1675.20230925005700.bin.gz
30158 -rwx 22821 Sep 24 2023 07:19:32 +00:00 IOSRP_R0-0.5735_1664.20230924053436.bin.gz
30164 -rwx 22851 Sep 25 2023 13:15:48 +00:00 IOSRP_R0-0.5735_1681.20230925113100.bin.gz
30171 -rwx 22777 Sep 25 2023 04:27:32 +00:00 IOSRP_R0-0.5735_1676.20230925024236.bin.gz
30138 -rwx 23010 Sep 24 2023 12:36:28 +00:00 IOSRP_R0-0.5735_1667.20230924105140.bin.gz
30148 -rwx 23074 Sep 25 2023 16:47:00 +00:00 IOSRP_R0-0.5735_1683.20230925150220.bin.gz
30130 -rwx 24376 Sep 25 2023 18:31:32 +00:00 IOSRP_R0-0.5735_1684.20230925164756.bin.gz
30149 -rwx 22875 Sep 25 2023 06:13:08 +00:00 IOSRP_R0-0.5735_1677.20230925042820.bin.gz
30145 -rwx 22865 Sep 25 2023 22:02:52 +00:00 IOSRP_R0-0.5735_1686.20230925201804.bin.gz
30135 -rwx 22783 Sep 25 2023 15:01:24 +00:00 IOSRP_R0-0.5735_1682.20230925131636.bin.gz
30137 -rwx 11477 Sep 19 2023 05:23:29 +00:00 btrace_rotate_sh_R0-0.14998_27.20230914235000.bin.gz
30165 -rwx 50142 Sep 17 2023 17:52:01 +00:00 sec_key_agent_R0-0.16109_336.20230917093840.bin.gz
30160 -rwx 22894 Sep 24 2023 19:39:08 +00:00 IOSRP_R0-0.5735_1671.20230924175420.bin.gz
30176 -rwx 50225 Sep 23 2023 14:12:24 +00:00 sec_key_agent_R0-0.16109_353.20230923055900.bin.gz
30154 -rwx 50256 Sep 25 2023 15:44:18 +00:00 sec_key_agent_R0-0.16109_359.20230925073053.bin.gz
30144 -rwx 50238 Sep 17 2023 09:36:44 +00:00 sec_key_agent_R0-0.16109_335.20230917012319.bin.gz
30172 -rwx 50218 Sep 18 2023 02:07:22 +00:00 sec_key_agent_R0-0.16109_337.20230917175357.bin.gz
30170 -rwx 50143 Sep 21 2023 04:25:10 +00:00 sec_key_agent_R0-0.16109_346.20230920201149.bin.gz
30147 -rwx 50270 Sep 18 2023 10:22:39 +00:00 sec_key_agent_R0-0.16109_338.20230918020918.bin.gz
30124 -rwx 50191 Sep 19 2023 11:08:38 +00:00 sec_key_agent_R0-0.16109_341.20230919025513.bin.gz
30167 -rwx 50179 Sep 24 2023 23:13:40 +00:00 sec_key_agent_R0-0.16109_357.20230924150015.bin.gz
30152 -rwx 11797 Sep 23 2023 12:02:56 +00:00 btrace_rotate_sh_R0-0.14998_28.20230919084448.bin.gz
30169 -rwx 50132 Sep 18 2023 18:38:00 +00:00 sec_key_agent_R0-0.16109_339.20230918102435.bin.gz
30129 -rwx 50250 Sep 20 2023 20:09:53 +00:00 sec_key_agent_R0-0.16109_345.20230920115628.bin.gz
30178 -rwx 50286 Sep 21 2023 12:40:31 +00:00 sec_key_agent_R0-0.16109_347.20230921042706.bin.gz
30191 -rwx 50159 Sep 22 2023 21:41:47 +00:00 sec_key_agent_R0-0.16109_351.20230922132822.bin.gz
30174 -rwx 50266 Sep 23 2023 05:57:04 +00:00 sec_key_agent_R0-0.16109_352.20230922214343.bin.gz
30155 -rwx 9322 Sep 20 2023 04:25:38 +00:00 btrace_rotate_sh_F0-0.21236_20.20230914170741.bin.gz
30133 -rwx 50320 Sep 22 2023 05:11:09 +00:00 sec_key_agent_R0-0.16109_349.20230921205744.bin.gz
30150 -rwx 50106 Sep 25 2023 07:28:57 +00:00 sec_key_agent_R0-0.16109_358.20230924231536.bin.gz
30175 -rwx 50232 Sep 19 2023 02:53:17 +00:00 sec_key_agent_R0-0.16109_340.20230918183956.bin.gz
30179 -rwx 1249 Sep 22 2023 21:10:25 +00:00 auto_upgrade_client_R0-0.14510_86.20230922211024.bin.gz
30132 -rwx 50114 Sep 19 2023 19:23:55 +00:00 sec_key_agent_R0-0.16109_342.20230919111034.bin.gz
30161 -rwx 50417 Sep 20 2023 03:39:15 +00:00 sec_key_agent_R0-0.16109_343.20230919192551.bin.gz
30166 -rwx 50334 Sep 20 2023 11:54:32 +00:00 sec_key_agent_R0-0.16109_344.20230920034111.bin.gz
30151 -rwx 50222 Sep 21 2023 20:55:48 +00:00 sec_key_agent_R0-0.16109_348.20230921124227.bin.gz
30163 -rwx 50247 Sep 22 2023 13:26:26 +00:00 sec_key_agent_R0-0.16109_350.20230922051305.bin.gz
30128 -rwx 50214 Sep 23 2023 22:27:41 +00:00 sec_key_agent_R0-0.16109_354.20230923141420.bin.gz
30126 -rwx 50271 Sep 24 2023 06:43:02 +00:00 sec_key_agent_R0-0.16109_355.20230923222937.bin.gz
30159 -rwx 50113 Sep 24 2023 14:58:19 +00:00 sec_key_agent_R0-0.16109_356.20230924064458.bin.gz
30142 -rwx 1130 Sep 22 2023 21:24:09 +00:00 auto_upgrade_client_R0-0.14510_87.20230922212409.bin.gz
30168 -rwx 1150 Sep 22 2023 21:10:23 +00:00 auto_upgrade_client_R0-0.14510_85.20230922211023.bin.gz
30182 -rwx 1113 Sep 22 2023 21:24:15 +00:00 auto_upgrade_client_R0-0.14510_88.20230922212412.bin.gz
30185 -rwx 1336 Sep 22 2023 21:24:21 +00:00 auto_upgrade_client_R0-0.14510_89.20230922212417.bin.gz
30187 -rwx 15278 Sep 22 2023 21:27:48 +00:00 linux_iosd_image_pmanlog_R0-0.5318_26.20230922212651.bin.gz

Leo Laohoo · ‎09-25-2023

How many power supplies does each switch have?

Is there a power-stack cable attached?

Graham Murison · ‎09-25-2023

Each switch has two PSU's and no stack power involved.

Leo Laohoo · ‎09-25-2023

Post the complete output to the command "sh crypto pki trustpool | include cn=".

If the first two lines come up as QuoVadis, then I suspect the crash is due to FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality, which, 16.6.10 is in the list of Known Affected Version.

Graham Murison · ‎09-26-2023

Thanks @Leo Laohoo Looks like this is what I am facing. Will be upgrading over the next few days and report back.

Leo Laohoo · ‎09-26-2023

@Graham Murison wrote:
Will be upgrading over the next few days and report back.

I would recommend performing Workaround 2 (manually downloading the certificate) instead of upgrading the firmware -- Less painful in the long run.

Graham Murison · ‎09-28-2023

Thanks all, the field noticed fixed my issue.