[ASR1001] Reboot for no known reason

[fabienfs]

Hello,

I have a router Cisco ASR1001 that rebooted on its own last night and I don't know the cause.

How can I find a log of what happened? I tried with "show log" but I only have recent logs from the last minutes.

I also looked at the alarms "show facility-alarm status", but nothing to report

Thanks for your help

[balaji.bandi]

we may need to consider crash logs :  show logs will be gone once the device rebooted, until you have any syslog facility configured to offload.

below document universal to help you to find some information, if you have contract, suggest to raise an TAC case so most of the information collected and correctly, this may help for TAC if any one having same issue with that code to fix the bugs if any.

https://www.cisco.com/c/en/us/support/docs/universal-gateways-access-servers/90-series-customer-premises-equipment/7900-crashes-router-troubleshooting.html

[Richard Burts]

show log reports data from the logging buffer maintained by the router. But when a router reboots the logging buffer is initialized at boot time and any log messages generated by the event that caused the reboot are lost. Do you by any chance send log messages to a logging server? If so there might be helpful information from that logging server. Do you by any chance have a device connected on the console which might have received log messages about the failure event?

It might be possible that the output of show version could have some information about the recent reboot. Or it is possible that the output of show tech might have helpful information - but interpreting that output is quite complex. 

[fabienfs]

Thanks for your messages

Unfortunately I don't have a syslog server, I'm looking to set it up

With show version command, I have this indication :

Last reload reason: Critical software exception, check bootflash:crashinfo_RP_00_00_20201217-044005-UTC

I can't find the details of this crash log anywhere if I don't have a syslog?

[Leo Laohoo]

Post the complete output to the following commands: 

sh version
dir bootflash:

[fabienfs]

#sh version
Cisco IOS Software, IOS-XE Software (X86_64_LINUX_IOSD-UNIVERSAL-M), Version 15.1(1)S, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Mon 22-Nov-10 12:32 by mcpre


Cisco IOS-XE software, Copyright (c) 2005-2010 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.  For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.


ROM: IOS-XE ROMMON

edge.bru uptime is 6 hours, 18 minutes
Uptime for this control processor is 6 hours, 19 minutes
System returned to ROM by reload
System image file is "bootflash:/asr1001-universal.03.02.00.S.151-1.S.bin"
Last reload reason: Critical software exception, check bootflash:crashinfo_RP_00_00_20201217-044005-UTC



License Info:

License UDI:
Device#	PID     		SN      		UDI
--------------------------------------------------------------------------------
*0  	ASR1001                 JAE154100QP             ASR1001:JAE154100QP


License Package Information for Module:'asr1001'

Module name   Image level          Priority   Configured   Valid license       

asr1001       adventerprise        1          NO           adventerprise       
              advipservices        2          NO           advipservices       
              ipbase               3          NO           ipbase              

Current License Level: ipbase

cisco ASR1001 (1RU) processor with 3880312K/6147K bytes of memory.
4 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8388608K bytes of physical memory.
7782399K bytes of eUSB flash at bootflash:.

Configuration register is 0x2102

dir bootflash:
Directory of bootflash:/

   11  drwx       16384  Oct 11 2011 08:44:41 +00:00  lost+found
   12  -rw-   314874112  Oct 11 2011 08:47:29 +00:00  asr1001-universal.03.02.00.S.151-1.S.bin
827425  drwx        4096  Oct 11 2011 08:46:42 +00:00  .installer
697633  drwx        4096  Dec 17 2020 04:44:06 +00:00  core
194689  drwx        4096  Dec 17 2020 04:47:07 +00:00  .prst_sync
681409  drwx        4096  Oct 11 2011 09:16:49 +00:00  .rollback_timer
   13  -rw-           0  Oct 11 2011 09:17:04 +00:00  tracelogs.386
64897  drwx      176128  Dec 17 2020 11:06:13 +00:00  tracelogs
   14  -rw-      124805   Apr 6 2012 09:14:45 +00:00  crashinfo_SIP_00_00_20120406-091445-UTC
   15  -rw-      114966  Apr 28 2015 01:42:05 +00:00  crashinfo_SIP_00_00_20150428-014205-UTC
   16  -rw-      289854   Jan 3 2018 22:16:02 +00:00  crashinfo_RP_00_00_20180103-221601-UTC
   17  -rw-      286297   Mar 6 2018 08:32:19 +00:00  crashinfo_RP_00_00_20180306-083218-UTC
   18  -rw-      302552   Mar 6 2018 13:47:11 +00:00  crashinfo_RP_00_00_20180306-134710-UTC
   19  -rw-      266362  Mar 30 2018 17:14:34 +00:00  crashinfo_RP_00_00_20180330-171433-UTC
   20  -rw-      320565  Apr 15 2018 06:03:27 +00:00  crashinfo_RP_00_00_20180415-060326-UTC
   21  -rw-      299044  Apr 16 2018 03:00:52 +00:00  crashinfo_RP_00_00_20180416-030052-UTC
   22  -rw-      300523  Oct 21 2018 23:32:34 +00:00  crashinfo_RP_00_00_20181021-233234-UTC
   23  -rw-      309959  Aug 24 2020 06:33:50 +00:00  crashinfo_RP_00_00_20200824-063350-UTC
   24  -rw-      110718  Sep 15 2020 05:40:36 +00:00  crashinfo_SIP_00_00_20200915-054036-UTC
   25  -rw-      329769  Dec 17 2020 04:40:06 +00:00  crashinfo_RP_00_00_20201217-044005-UTC

7844007936 bytes total (5343993856 bytes free)

[Leo Laohoo]

Geez whiz!

Look at them crash logs. 

Are you sure you do not want to upgrade the firmware?

[balaji.bandi]

Offload all the Crashlogs and trace to TFTP to be safe, and if get chance upgrade to latest stable version of code and check.

what kind of load you see of CPU normally.

[Georg Pauwen]

Hello,

on a side note, it looks like you have the ASR1001 (and not the X or HX). The ASR1001 is end of life as of 2016. Only 3.1x versions are available (check the link below). Still better than, as others have noted, the extremely outdated IOS version you are running.

https://www.cisco.com/c/en/us/support/routers/asr-1001-router/model.html#~tab-downloads

[paul driver]

Hello
Funnily enough I replaced an AS1002rp 1 running Avd/Enterprise 02.06 code (10 years old IOS !) last week and when its replacement from cisco arrived it had ipbase ios on it , So it needed upgraded however whatever I tried I couldn't get any 3.x code to load, it wasn't down to flash memory or dram it just wouldn't take and due to a very short time window I eventually admitted defeat and used the IOS off the old rtr, (not good but having this rtr down was affecting a large part of the company so it had to be brought back up ASAP)
Anyway whilst in the middle of TS this rtr, it kept reloading intermittently and my thought was I have been send faulty hardware until a light bulb moment and I cannibalised a second PSU off the old rtr and put into the chassis of the new one then the reloading stopped, It turns out these rtrs require BOTH PSU's to be functioning otherwise they will continually reload to save from overheating and thus create crash logs.
Still not had the opportunity to find the reason for the IOS install failure but at least I learnt something new about an old piece of kit anyway I just thought I would share this with you all as it may save others from my own heartache and I guess its kind of relevant to this OP!

Editied - @fabienfs  forgot to mention - review your crash logs it should give you the explanation as to your reload.
dir bootflash:crashinfo_RP_00_00_20201217-044005-UTC

[Richard Burts]

Looking at the number of crashinfo files we can see that this router has had a number of crashes over the years (1 crash in 2012, 1 crash in 2015, 7 crashes in 2018, and 3 crashes in 2020). So crashes are not real frequent but the number of crashes certainly seems to point to a problem with the software that it is running. I think addressing this will be difficult. If there is an active service contract on this router than a case with Cisco TAC and a code upgrade would be possible. But I am guessing that this router does not have an active service contract. Without a contract how would a code upgrade be possible? It looks to me that either they need to live with an occasional crash or they need to replace the router.

[Leo Laohoo]

If the OP does not have a valid Service Contract, then they need to first read this:  Cisco IOS XE Software for Cisco ASR 1000 Series 20-Gbps Embedded Services Processor IP ARP Denial of Service Vulnerability

Next, the OP needs to scroll down to the "Customers Without Service Contracts" section and read the two paragraphs very, very carefully.  

NOTE:  It would be very, very helpful if the OP already has the newer IOS-XE filename ready give to the Cisco Entitlements Team for publication.

[Leo Laohoo]

Reviewing the crash log(s) is a waste of time because: 

1. The router is running a "version 0" of 15.1(1)S.  Version "0"!  
2. If TAC gets hold of this case, guess what is the first thing TAC is going to recommend:  Upgrade the router firmware