ISR 4331 Crash

[Elopower123]

Hi,

So I just got a new ISR4331 router for connection with my ISP, the router is meant to have 4GB default RAM size. I intend to use the router to receive full BGP table from my ISP and I believe that I should be able to do this seamlessly as theoretically 4GB of RAM should be more than capable of comfortably handling a full BGP table.

But once my ISP advertises their full BGP table, the router begins to receive the routes seamlessly until at some point it just abruptly freezes until my ISP reverts back to advertising just default route and then the router reloads.

After the router comes back up and I issue the "sh ver" command I get the following output

Uptime for this control processor is 25 minutes
System returned to ROM by Critical software exception, check bootflash:crashinfo _RP_00_00_20150103-030218-WAT
System image file is "bootflash:isr4300-universalk9.16.06.04.SPA.bin"
Last reload reason: Critical software exception, check bootflash:crashinfo_RP_00 _00_20150103-030218-WAT

Also a look at the router's memory also gives me the below

cisco ISR4331/K9 (1RU) processor with 1795999K/6147K bytes of memory.
Processor board ID MYPROCESSORID
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
3207167K bytes of flash memory at bootflash:.
0K bytes of WebUI ODM Files at webui:.

I see that the total memory in use is about 1.8GB and not the default 4GB which is pretty confusing to me.

So please can someone help me understand why this could be happening and what I could do to get the router to function appropriately

[Leo Laohoo]

---

@Elopower123 wrote:

Last reload reason: Critical software exception, check bootflash:crashinfo_RP_00 _00_20150103-030218-WAT

---

Kindly post the complete output to the following commands: 

sh version
sh clock
dir bootflash:
dir crashinfo:
dir crashinfo:/tracelogs

[Elopower123]

The output of the first three commands:

"sh ver"

Cisco IOS XE Software, Version 16.06.04
Cisco IOS Software [Everest], ISR Software (X86_64_LINUX_IOSD-UNIVERSALK9-M), Ve rsion 16.6.4, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2018 by Cisco Systems, Inc.
Compiled Sun 08-Jul-18 04:33 by mcpre

Cisco IOS-XE software, Copyright (c) 2005-2018 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0. The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY. You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0. For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.

ROM: IOS-XE ROMMON

Edge_Router uptime is 14 hours, 46 minutes
Uptime for this control processor is 14 hours, 49 minutes
System returned to ROM by Critical software exception, check bootflash:crashinfo_RP_00_00_20150103-030218-WAT
System image file is "bootflash:isr4300-universalk9.16.06.04.SPA.bin"
Last reload reason: Critical software exception, check bootflash:crashinfo_RP_00_00_20150103-030218-WAT

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Suite License Information for Module:'esg'

--------------------------------------------------------------------------------
Suite Suite Current Type Suite Next reboot
--------------------------------------------------------------------------------
FoundationSuiteK9 None None None
securityk9
appxk9

AdvUCSuiteK9 None None None
uck9
cme-srst
cube

Technology Package License Information:

-----------------------------------------------------------------
Technology Technology-package Technology-package
Current Type Next reboot
------------------------------------------------------------------
appxk9 None None None
uck9 None None None
securityk9 securityk9 Permanent securityk9
ipbase ipbasek9 Permanent ipbasek9

cisco ISR4331/K9 (1RU) processor with 1795999K/6147K bytes of memory.
Processor board ID "BOARDID"
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
3207167K bytes of flash memory at bootflash:.
0K bytes of WebUI ODM Files at webui:.

Configuration register is 0x2102

"sh clock"

*17:57:52.528 WAT Sat Jan 3 2015

"dir bootflash:"

Directory of bootflash:/

11 -rw- 550119351 Feb 28 2020 15:23:59 +01:00 isr4300-universalk9.16.06.04.SPA.bin
88353 drwx 4096 Jan 3 2015 03:07:11 +01:00 .installer
12 -rw- 34 Jan 1 2015 01:10:04 +01:00 pnp-tech-time
13 -rw- 49967 Jan 1 2015 01:10:07 +01:00 pnp-tech-discovery-summary
80321 drwx 4096 Jan 3 2015 03:02:52 +01:00 core
40161 drwx 4096 Jan 3 2015 03:09:19 +01:00 .prst_sync
48193 drwx 4096 Feb 28 2020 15:31:10 +01:00 .rollback_timer
14 -rw- 0 Feb 28 2020 15:31:23 +01:00 tracelogs.02M
56225 drwx 20480 Jan 3 2015 17:45:04 +01:00 tracelogs
15 -rw- 30 Jan 3 2015 03:09:20 +01:00 throughput_monitor_params
16 drwx 4096 Jan 1 2015 01:02:14 +01:00 lost+found
17 -rw- 173978 Jan 2 2015 01:32:43 +01:00 crashinfo_RP_00_00_20150102-013231-WAT
18 -rw- 176048 Jan 3 2015 03:02:20 +01:00 crashinfo_RP_00_00_20150103-030218-WAT

3258179584 bytes total (2249625600 bytes free)

The last two commands didn't execute, responded as unrecognised commands. I typed a question mark after the dir keyword and got the following output

"dir ?"
/all List all files
/recursive List files recursively
all-filesystems List files on all filesystems
bootflash: Directory or file name
cns: Directory or file name
flash: Directory or file name
null: Directory or file name
nvram: Directory or file name
system: Directory or file name
tar: Directory or file name
tmpsys: Directory or file name
webui: Directory or file name
| Output modifiers

[Leo Laohoo]

Ok, so the router is not configured with NTP. 

Can I see the output to the command: 

dir bootflash:core
dir bootflash:tracelogs
sh log on uptime detail

You need to export those two crashinfo files off the router and send them off to TAC.

[Elopower123]

Hi @Leo Laohoo 

Below is the output of the first command

Directory of bootflash:/core/

88354 drwx 4096 Feb 28 2020 15:31:09 +01:00 modules
80322 -rw- 1 Jan 4 2015 04:54:41 +01:00 .callhome
80325 -rw- 2409830 Jan 3 2015 03:01:38 +01:00 TFnet_Edge_Router_RP_ 0_smd_6648_20150103-030046-WAT.core.gz
80324 -rw- 2408532 Jan 2 2015 01:37:36 +01:00 TFnet_Edge_Router_RP_ 0_smd_6854_20150102-013734-WAT.core.gz
80323 -rw- 143012666 Jan 2 2015 01:37:39 +01:00 TFnet_Edge_Router_RP_ 0_x86_64_crb_linux_iosd_crankshaft-universalk9-ms_3494_20150102-013253-WAT.core. gz
80326 -rw- 143385806 Jan 3 2015 03:03:48 +01:00 TFnet_Edge_Router_RP_ 0_x86_64_crb_linux_iosd_crankshaft-universalk9-ms_2955_1420250541.gz.TEMP_IN_PRO GRESS.

3258179584 bytes total (2249490432 bytes free)

I have attached a file containing the output of the dir bootflash:tracelogs

The last command didn't yield any output

[Leo Laohoo]

Without knowing the correct time-and-date it is next to impossible to determine: 

1. How many times have this router crashed?
2. What is the latest crashinfo files

Unfortunately, the crashinfo files contain non-readable files.  Only Cisco TAC have the necessary tools to diagnose them. 

NOTE:  They may recommend updating the firmware.

[Elopower123 · Accepted solution]

Hi @Leo Laohoo 

Actually the router has only crashed twice. All crashes on the router are recent and have only happended during the BGP route learning

But can you explain the disparity in the memory. I can see 4GB of ram as the physical memory, yet what is in use by the control plane and data plane is just about 1.8GB.

Is there anything I need to do to utilise the full memory?

[Elopower123]

Hi @Leo Laohoo 

I've been trying to open a case with TAC but when I open the cisco support case manager, the @"open new case" tab is not highlighted so I can't click on it.

I already logged in with my email and passwords and still the same thing.

I have attached a screenshot of the page

[marce1000 · Accepted solution]

 -   Check if there are any more and or advisory  software releases for your platform, consider upgrading if available , feasible and or applicable.

 M.

[Elopower123]

Hi @marce1000 

Could you direct me to the platform where I can check for the software releases information and how I might go about the upgrade.

[Georg Pauwen · Accepted solution]

Hello,

below is the link. Release Gibraltar-16.12.4 MD is currently the recommended release. Keep in mind that starting with 16.9.1, SMARTNet is required.

https://software.cisco.com/download/home/285018115/type/282046477/release/Gibraltar-16.12.4