Cisco 3650 Switch | crashinfo:crashinfo_RP_00_00

[indunilrasss]

One of our Cisco 3650 switches rebooted unexpectedly. However, before rebooted the switch we could not log in to the switch. After rebooting it will show the Last reload reason: Critical software exception, check crashinfo:crashinfo_RP_00_00_20201128-132212-GMT error. Could anyone explain why this error shows? Do we need to replace the new switch? And also how to resolve this issue. Thank you! 

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
dir crashinfo:

[indunilrasss]

Dear Leo,

Thanks for your prompt support! Please find the show version and dir crashinfo as per your request. 

Cisco IOS Software [Denali], Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 16.3.5b, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2017 by Cisco Systems, Inc.
Compiled Thu 02-Nov-17 11:07 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2017 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 3.56, RELEASE SOFTWARE (P)

BlackMX-SW-3650 uptime is 2 hours, 38 minutes
Uptime for this control processor is 2 hours, 42 minutes
System returned to ROM by reload
System restarted at 13:29:30 GMT Sat Nov 28 2020
System image file is "flash:packages.conf"
Last reload reason: Critical software exception, check crashinfo:crashinfo_RP_00_00_20201128-132212-GMT

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Technology Package License Information:

-----------------------------------------------------------------
Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
lanbasek9 Permanent lanbasek9

cisco WS-C3650-24TS (MIPS) processor (revision R0) with 865070K/6147K bytes of memory.
Processor board ID FDO2203E1G5
2 Virtual Ethernet interfaces
28 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
252000K bytes of Crash Files at crashinfo:.
1611414K bytes of Flash at flash:.
0K bytes of at webui:.

Base Ethernet MAC Address : 70:69:5a:a9:13:00
Motherboard Assembly Number : 73-15898-06
Motherboard Serial Number : FDO22031BS4
Model Revision Number : R0
Motherboard Revision Number : A0
Model Number : WS-C3650-24TS
System Serial Number : FDO2203E1G5

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 28 WS-C3650-24TS 16.3.5b CAT3K_CAA-UNIVERSALK9 INSTALL

Configuration register is 0x102

Directory of crashinfo:/

28561 drwx 36864 Nov 28 2020 16:11:43 +05:30 tracelogs
11 -rw- 0 Jan 21 2018 04:35:45 +05:30 koops.dat
12 -rw- 12598644 Jun 6 2019 00:59:43 +05:30 system-report_1_20190605-192932-UTC.tar.gz
13 -rw- 23455929 Nov 28 2020 13:25:04 +05:30 system-report_1_20201128-132450-GMT.tar.gz

249872384 bytes total (197130240 bytes free)

[Leo Laohoo]

---

@indunilrasss wrote:

13 -rw- 23455929 Nov 28 2020 13:25:04 +05:30 system-report_1_20201128-132450-GMT.tar.gz

---

Can you attach this file to this thread so we can have a look? 

---

@indunilrasss wrote:

16.3.5b

---

I would strongly recommend upgrading to something more recent.  "5b" means it is a "re-build" and, more than often, a lot buggier.

[indunilrasss]

Dear Leo,

Could you please tel me how to download this file from the switch?

Sure. I'll update the switch OS. 

[Leo Laohoo]

---

@indunilrasss wrote:

Could you please tel me how to download this file from the switch?

---

If you have a supported USB then the command is very simple: 

copy crashinfo:system-report_1_20201128-132450-GMT.tar.gz usbflash0:

If you do not have a supported USB, then use TFTP: 

copy crashinfo:system-report_1_20201128-132450-GMT.tar.gz tftp://<TFTP_IP_ADDRESS>

[indunilrasss]

Dear Leo,

Thanks for your support as always. I'll upload it asap.

[indunilrasss]

Dear Leo,

I downloaded crashinfo file. could you please tell me, which file do I need to upload?

[Leo Laohoo]

When you reply, there is an option below to "Drag and drop here or browse files to attach".  

[indunilrasss]

Dear Leo,

Please find the attached crashinfo file as per your request. 

[Leo Laohoo · Accepted solution]

1.  The crash generated a "crashinfo" file.  In more recent IOS-XE releases, crashes hit a known bug/feature where it no longer creates a crashinfo file.  This one did.  Very helpful. 

2.  The crash tells me the firmware is indeed buggy.  Very buggy:  I can identify six Bug IDs and they are: 

• The file "platform_mgr_R0-0.15800_8644.20201128132112.bin" points to a not-so-critical CSCvh95375. 
• The file "crashinfo_RP_00_00_20201128-132212-GMT" points to several bugs:  CSCvd07505, CSCve95037, CSCvh56835, CSCvk1248.  These bugs are "not that critical" because they just spam the logs about the memory leak -- Yes, there is a slow memory leak bug (unlucky) and triggers after 3 weeks of uptime. 
• However, the crash was due to a different bug:  CSCvt59178

Nov 28 13:20:58.399 GMT: %PLATFORM_INFRA-5-IOS_INTR_OVER_LIMIT: IOS thread disabled interrupt for 19 msec
-Traceback= 1#924a998d29c6b41cd4d8f2471c7daed8  :AAADACE000+4590C20 :AAADACE000+458FD8C :AAADACE000+459013C :AAADACE000+1761A18 :AAADACE000+45852E0 :AAADACE000+67F40F0 :AAADACE000+67F4A9C :AAADACE000+67F2B7C :AAADACE000+20B193C :AAADACE000+208E10C :AAADACE000+6801288
IOSXE-WATCHDOG: Process = Crypto IKEv2
-Traceback= 1#924a998d29c6b41cd4d8f2471c7daed8   c:FFF3B3C000+153032 c:FFF3B3C000+150AC8 c:FFF3B3C000+153BB0 gcc_s:FFD7468000+51514 gcc_s:FFD7468000+5195C gcc_s:FFD7468000+523C0 c:FFF3B3C000+124A00 btrace:FFF6975000+3ECC :AAADACE000+177EA34 :AAADACE000+45A9928

To be sure, raise a Cisco TAC Case and get them to verify. 

In the meantime, I recommend upgrading the firmware to the latest 16.3.X or 16.6.X.  
NOTE:  This memory leak crashes the switch/stack every 3 weeks.  

[indunilrasss]

Dear Leo,

Thank you for your support as always. I will check and update you! 

[indunilrasss]

Dear Leo,

Could you please tell me how to identify bug ID from crashinfo file? Because I need to explain to our management. Thank you ! 

[Leo Laohoo]

---

@indunilrasss wrote:

Could you please tell me how to identify bug ID from crashinfo file? Because I need to explain to our management.

---

IF management wants it "official", then raise a TAC Case.  I DO NOT work for Cisco.

[indunilrasss]

Dear Leo,

Sorry, I appreciate your help so much.