CSCvk36507 - Fman-fp in one c9300 crashed unexpectedly and took down the rest of stack members without failover

[adujeu]

We had similar issue this morning (13th september)

ST002A-IXCMB uptime is 10 hours, 35 minutes

Uptime for this control processor is 10 hours, 37 minutes

System returned to ROM by LocalSoft

System image file is "flash:packages.conf"

Last reload reason: LocalSoft

Any progress from Cisco Tac so far ?

[Leo Laohoo]

What firmware is the stack running?
Post the complete output to the command "dir crashinfo-1:".

[adujeu]

Cisco IOS XE Software, Version 16.09.01

ST002A-IXCMB#dir crashinfo-1:
Directory of crashinfo:/

47329 drwx 45056 Sep 16 2019 09:39:55 +00:00 tracelogs
11 -rw- 0 Dec 22 2017 04:27:00 +00:00 koops.dat
12 -rw- 2286969 Aug 29 2019 15:22:48 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_0-20190829-152246.tar.gz
13 -rw- 236089 Sep 6 2019 19:01:05 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_0-20190906-190105.tar.gz
14 -rw- 237798 Sep 8 2019 22:36:22 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_1-20190908-223622.tar.gz
15 -rw- 233034 Sep 11 2019 16:36:16 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_2-20190911-163615.tar.gz
16 -rw- 242885 Sep 12 2019 20:50:18 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_3-20190912-205017.tar.gz

1651314688 bytes total (1559867392 bytes free)
ST002A-IXCMB#dir crashinfo-2:
Directory of crashinfo-2:/

47329 drwx 49152 Sep 16 2019 09:42:07 +00:00 tracelogs
11 -rw- 0 Jun 23 2018 12:43:39 +00:00 koops.dat
12 -rw- 1165544 Aug 30 2019 12:53:17 +00:00 ST002A-IXCMB_2_RP_0_trace_archive_0-20190830-125316.tar.gz
13 -rw- 165710 Sep 6 2019 19:01:03 +00:00 ST002A-IXCMB_2_RP_0_trace_archive_0-20190906-190103.tar.gz
14 -rw- 206148 Sep 8 2019 22:36:19 +00:00 ST002A-IXCMB_2_RP_0_trace_archive_0-20190908-223619.tar.gz
15 -rw- 181696 Sep 11 2019 16:36:17 +00:00 ST002A-IXCMB_2_RP_0_trace_archive_0-20190911-163617.tar.gz
16 -rw- 176085 Sep 12 2019 20:50:18 +00:00 ST002A-IXCMB_2_RP_0_trace_archive_0-20190912-205018.tar.gz
17 -rw- 353389 Sep 13 2019 23:17:12 +00:00 ST002A-IXCMB_2_RP_0_trace_archive_0-20190913-231711.tar.gz

1651507200 bytes total (1560281088 bytes free)
ST002A-IXCMB#dir crashinfo:
Directory of crashinfo:/

47329 drwx 45056 Sep 16 2019 09:49:55 +00:00 tracelogs
11 -rw- 0 Dec 22 2017 04:27:00 +00:00 koops.dat
12 -rw- 2286969 Aug 29 2019 15:22:48 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_0-20190829-152246.tar.gz
13 -rw- 236089 Sep 6 2019 19:01:05 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_0-20190906-190105.tar.gz
14 -rw- 237798 Sep 8 2019 22:36:22 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_1-20190908-223622.tar.gz
15 -rw- 233034 Sep 11 2019 16:36:16 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_2-20190911-163615.tar.gz
16 -rw- 242885 Sep 12 2019 20:50:18 +00:00 ST002A-IXCMB_1_RP_0_trace_archive_3-20190912-205017.tar.gz

[Leo Laohoo]

---

@adujeu wrote:
ST002A-IXCMB_2_RP_0_trace_archive_0-20190913-231711.tar.gz

---

Download this file and attached to the thread. 

Next, 16.9.4 was recently released.  I suggest you think about upgrading the stack to this version and get off 16.9.1 which is very buggy.