Re: CSCvb26072 - Denali 3850 crashinfo file location - add tracebacks to show tech - 1

engineer467 · ‎12-30-2019

Hello,

Our C9200L-48P-4G switch running on Fuji 16.9.4 restarted today with the reboot reason as -

Critical software exception, check /crashinfo/system-report_2_20191231-081556-IST.tar.gz

I have downloaded the file, but not sure what to look for.

Any help from the community here is much appreciated.

Regards,

Abhishek

Leo Laohoo · ‎12-30-2019

Attach the crashinfo file into this thread and we will have a look.

Post the complete output to the following commands:

1. sh version;

2. dir crashinfo-<switch that crashed>:

engineer467 · ‎12-31-2019

PFA the files..

Show version-

Cisco IOS XE Software, Version 16.09.04
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_LITE_IOSXE), Version 16.9.4, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2019 by Cisco Systems, Inc.
Compiled Thu 22-Aug-19 17:30 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2019 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: System Bootstrap, Version 16.12.1r [FC2], RELEASE SOFTWARE (P)

uptime is 5 hours, 41 minutes
Uptime for this control processor is 5 hours, 44 minutes
System returned to ROM by Critical software exception, check /crashinfo/system-report_2_20191231-081556-IST.tar.gz at 09:06:54 IST Fri Nov 8 2019
System restarted at 08:21:00 IST Tue Dec 31 2019
System image file is "flash:packages.conf"
Last reload reason: Critical software exception, check /crashinfo/system-report_2_20191231-081556-IST.tar.gz

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

------------------------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------------------
network-essentials Smart License network-essentials
dna-essentials Subscription Smart License dna-essentials

Smart Licensing Status: UNREGISTERED/EVAL MODE

cisco C9200L-48P-4G (ARM64) processor with 519008K/3071K bytes of memory.
Processor board ID JAE233415MR
3 Virtual Ethernet interfaces
104 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
1998948K bytes of physical memory.
819200K bytes of Crash Files at crashinfo:.
819200K bytes of Crash Files at crashinfo-1:.
1941504K bytes of Flash at flash:.
1941504K bytes of Flash at flash-1:.
0K bytes of WebUI ODM Files at webui:.

Base Ethernet MAC Address : 7c:21:0e:21:71:00
Motherboard Assembly Number : 73-18365-06
Motherboard Serial Number : JAE233415MR
Model Revision Number : E0
Motherboard Revision Number : A0
Model Number : C9200L-48P-4G
System Serial Number : JAE233415MR

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
1 52 C9200L-48P-4G 16.9.4 CAT9K_LITE_IOSXE INSTALL
* 2 52 C9200L-48P-4G 16.9.4 CAT9K_LITE_IOSXE INSTALL

Switch 01
---------
Switch uptime : 5 hours, 46 minutes

Base Ethernet MAC Address : 7c:21:0e:21:71:80
Motherboard Assembly Number : 73-18365-06
Motherboard Serial Number : JAE233415MS
Model Revision Number : E0
Motherboard Revision Number : A0
Model Number : C9200L-48P-4G
System Serial Number : JAE233415MS

Configuration register is 0x102

dir crashinfo-2:
Directory of crashinfo:/

11 -rw- 0 Jan 1 1970 05:30:53 +05:30 koops.dat
14657 drwx 12288 Dec 31 2019 14:01:21 +05:30 tracelogs
12 -rw- 1787380 Nov 8 2019 07:33:11 +05:30 KMI_BAN_RO_SW_2_RP_0_trace_archive_0-20191108-073307.tar.gz
13 -rw- 1983828 Nov 8 2019 08:05:30 +05:30 KMI_BAN_RO_SW_2_RP_0_trace_archive_0-20191108-080525.tar.gz
14 -rw- 3493802 Dec 10 2019 08:14:05 +05:30 system-report_2_20191210-081404-IST.tar.gz
15 -rw- 428395 Dec 31 2019 08:14:30 +05:30 KMI_BAN_RO_BB_2_RP_0_trace_archive_0-20191231-081429.tar.gz
16 -rw- 25938158 Dec 31 2019 08:16:01 +05:30 system-report_2_20191231-081556-IST.tar.gz

825638912 bytes total (747003904 bytes free)

Thank you.

Leo Laohoo · ‎12-31-2019

Please attach "KMI_BAN_RO_BB_2_RP_0_trace_archive_0-20191231-081429.tar.gz" this too.

Did the entire stack crashed or just the stack master?

engineer467 · ‎01-01-2020

I believe it was switch-2.

PFA the file you requested.

Leo Laohoo · ‎01-01-2020

Thanks for this.
Give me a few days to check it out.

engineer467 · ‎01-01-2020

Thank you Leo.

Leo Laohoo · ‎01-06-2020

Apologies for the delay, I had a look at all the crashinfo attachments.

Dec 31 08:14:28.983: %REDUNDANCY-3-SWITCHOVER: RP switchover (KEEPALIVE_FAILURE)
Dec 31 08:14:29.087: %PLATFORM-6-HASTATUS: RP switchover, sent message became active. IOS is ready to switch to primary after chassis confirmation
Dec 31 08:14:29.090: %PLATFORM-6-HASTATUS: RP switchover, received chassis event became active
Dec 31 08:14:29.091: %IOSXE_REDUNDANCY-6-PEER: Active detected switch 2 as standby.
Dec 31 08:14:29.497: %HMANRP-6-HMAN_IOS_CHANNEL_INFO: HMAN-IOS channel event for switch 2: EMP_RELAY: Channel UP!
Dec 31 08:14:29.524: %IOSXE_REDUNDANCY-6-PEER_LOST: Active detected switch 2 is no longer standby
Dec 31 08:14:29.554: %PLATFORM-4-HASTATUS_WARN: RP HA status warning,  received peer RP not present event) -Process= "IOSD chasfs task", ipl= 0, pid= 85
-Traceback= 1#12346ab448d7749fcfe74ad40f3d8987   :400000+36E2630 :400000+1F14520 :400000+1261668 :400000+125BD6C :400000+131F7FC :400000+130A418 :400000+2468208

I suspect the crash is due to CSCvg22277.

Kindly raise a TAC Case and get them to verify.

engineer467 · ‎01-06-2020

Thank you for taking time and reviewing the logs.

I checked info on this bug (https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvg22277/?rfs=iqvred),

but it seems it's not updated.

They have mentioned Fuji-16.8.1 in both known affected and known fixed releases.

Not sure which version to upgrade to now.

But you are right, opening a case with TAC would confirm things.

Thanks again Leo.

Regards

Leo Laohoo · ‎01-06-2020

If you're able to raise a TAC Case, let us know what they come back with.