01-03-2016 05:53 AM - edited 03-05-2019 03:02 AM
I have two location HO & DR. Both are having ASR-1001 to connect to Service Provider with "asr1001x-universalk9.03.13.02.S.154-3.S2-ext.SPA.bin" software.
Both the Router simultaneously reloaded at the same time with the "Last reload reason: <NULL>" message in the "show version:. There is no crashinfo generated but we have some tracelogs available which do not contain enough information. Below are the logs after the router came back after the reload. How can we find out the root cause as we want to do RCA. Initially it looked like a licensing issue as it happened on new year night but as per my understanding reload should not be caused by any licensing.
*Jan 1 00:06:02.967: %SMART_LIC-6-AGENT_READY: Smart Agent for Licensing is initialized
*Jan 1 00:06:04.020: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = asr1001x Next reboot level = adventerprise and License = adventerprise
*Jan 1 00:06:05.579: %IOSXE_THROUGHPUT-6-LEVEL: Throughput level has been set to 2500000 kbps
*Jan 1 00:06:12.517: pak_debug_init: Successfully initialized pak debug trace buffer
*Jan 1 00:06:16.702: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jan 1 00:06:20.027: %VOICE_HA-7-STATUS: CUBE HA-supported platform detected.
*Jan 1 00:06:20.993: %LINK-3-UPDOWN: Interface Lsmpi0, changed state to up
*Jan 1 00:06:20.993: %LINK-3-UPDOWN: Interface EOBC0, changed state to up
*Jan 1 00:06:20.993: %LINEPROTO-5-UPDOWN: Line protocol on Interface VoIP-Null0, changed state to up
*Jan 1 00:06:20.993: %LINEPROTO-5-UPDOWN: Line protocol on Interface LI-Null0, changed state to up
*Jan 1 00:06:20.993: %LINK-3-UPDOWN: Interface GigabitEthernet0, changed state to down
*Jan 1 00:06:20.996: %LINK-3-UPDOWN: Interface LIIN0, changed state to up
*Jan 1 00:06:23.028: %IOSXE_MGMTVRF-6-CREATE_SUCCESS_INFO: Management vrf Mgmt-intf created with ID 1, ipv4 table-id 0x1, ipv6 table-id 0x1E000001
*Jan 1 00:06:23.142: %LINEPROTO-5-UPDOWN: Line protocol on Interface Lsmpi0, changed state to up
*Jan 1 00:06:23.142: %LINEPROTO-5-UPDOWN: Line protocol on Interface EOBC0, changed state to up
*Jan 1 00:06:23.142: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet0, changed state to down
*Jan 1 00:06:23.142: %LINEPROTO-5-UPDOWN: Line protocol on Interface LIIN0, changed state to up
*Jan 1 00:06:07.450: %CMLIB-6-THROUGHPUT_VALUE: R0/0: cmand: Throughput license found, throughput set to 2500000 kbps
*Jan 1 00:06:08.972: %CPPHA-7-START: F0: cpp_ha: CPP 0 preparing ucode
*Jan 1 00:06:09.148: %CPPHA-7-START: F0: cpp_ha: CPP 0 startup init
*Jan 1 00:06:12.678: %IOSXE-4-PLATFORM: R0/0: kernel: OCTEON[0]: Received active indication from core
*Jan 1 00:06:12.678: %IOSXE-4-PLATFORM: R0/0: kernel: octeon_core_drv_init: ctx_chunks = 14000
*Jan 1 00:06:12.678: %IOSXE-4-PLATFORM: R0/0: kernel: octeon_core_drv_init: boot_log_ptr=12d580
*Jan 1 00:06:12.678: %IOSXE-4-PLATFORM: R0/0: kernel: octeon_core_drv_init: Build = N2-Like IPSec Version: 2.3.0-101-03-0402-91831-ilkx-CR N2-Like SSL Version:
2.3.0-101-03-0402-9
*Jan 1 00:06:12.678: %IOSXE-4-PLATFORM: R0/0: kernel: OCTEON[0] is running BASE application (core clock: 1100000000 Hz)
*Jan 1 00:06:12.941: %CPPHA-7-START: F0: cpp_ha: CPP 0 running init
*Jan 1 00:06:13.282: %CPPHA-7-READY: F0: cpp_ha: CPP 0 loading and initialization complete
*Jan 1 00:06:13.403: %IOSXE-6-PLATFORM: F0: cpp_cp: Process CPP_PFILTER_EA_EVENT__API_CALL__REGISTER
*Jan 1 00:06:14.393: %IOSXE-4-PLATFORM: R0/0: kernel: OCTEON[0] Poll Function (Module Starter arg: 0x0) completed (status: Finished)
*Jan 1 00:06:25.170 UTC: %SYS-5-LOG_CONFIG_CHANGE: Buffer logging: level debugging, xml disabled, filtering disabled, size (64000)
*Jan 1 00:06:25.171 UTC: %SYS-5-LOG_CONFIG_CHANGE: Console logging disabled
*Jan 1 00:06:25.172 UTC: %SYS-5-LOG_CONFIG_CHANGE: Monitor logging disabled
01-03-2016 12:45 PM
These log messages seem to indicate a normal boot sequence and do not provide identification of what caused the reload. It does seem odd that both routers would have rebooted simultaneously. And if the bootup is happening at 0:06::02 then the reboot would have come very soon after midnight which perhaps does suggest that the reboot might have been time related. But I do not know anything that would provide an explanation of why reboot at midnight.
Do you have any logs that were generated by the routers in the period before they rebooted? Do you send syslog messages to a server where there might be additional log messages?
HTH
Rick
01-04-2016 06:37 AM
01-04-2016 09:06 AM
Thank you for providing the additional information. I do not see anything in it that identifies the cause of the reboot. But there is much in the show tech that I do not have proper tools to evaluate but which Cisco TAC would have. Perhaps the best thing to do would be to open a case with Cisco TAC about this.
HTH
Rick
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide