Cisco 2821 Random System Reloads

[Alexander Arnold]

Good Afternoon, 

I am having an issue with our Cisco 2821 Voice Router randomly reloading. Please see the below show stack command output and hopefully someone can help me determine what is going on!

vr1# sh stacks
Minimum process stacks:
Free/Size Name
11016/12000 MRIB IPv4 Init Process
11112/12000 MRIB IPv6 Init Process
4100/6000 EEM Shell Director
5140/6000 Inspect Init Msg
5092/6000 VPM_CABLE_STAT_POLL_BACKGROUND
5108/6000 SPAN Subsystem
58912/60000 script background loader
15316/24000 Init
58644/60000 EEM Auto Registration Proc
5016/6000 Auto Upgrade Startup Process
4780/6000 DIB error message
5144/6000 SASL MAIN
2140/3000 allegro libretto init
5048/6000 LICENSE AGENT DEFAULT
5144/6000 FLEX DSPRM boot download main
4924/6000 RADIUS INITCONFIG
2128/3000 Rom Random Update Process
5056/6000 URPF stats
9712/12000 SSH Process
5140/6000 IP SLAs Deferred Schedule Processor
Interrupt level stacks:
Level Called Unused/Size Name
1 190696 6800/9000 Network interfaces
2 542175 8544/9000 DMA/Timer Interrupt
3 1 8376/9000 PA Management Int Handler
4 587 8616/9000 Console Uart
5 0 9000/9000 External Interrupt
7 259762 8544/9000 NMI Interrupt Handler
System was restarted by error - a System Error, PC 0x4007B7B0 at 12:26:46 MST Fri Feb 17 2017
2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Compiled Tue 24-Mar-15 09:00 by prod_rel_team
Image text-base: 0x400184F0, data-base: 0x45168C20
Stack trace from system failure:
FP: 0x48FF64E0, RA: 0x4007B7B0
FP: 0x48FF64E0, RA: 0x43F0BC08
FP: 0x48FF6500, RA: 0x42AF7F2C
FP: 0x48FF6530, RA: 0x42AF84C4
FP: 0x48FF6570, RA: 0x42AF5F64
FP: 0x48FF6590, RA: 0x44B38BB0
FP: 0x48FF65A8, RA: 0x41E678A4
FP: 0x48FF65C0, RA: 0x41E651DC
***************************************************
******* Information of Last System Crash **********
***************************************************
Using flash:crashinfo.
%Error opening flash:crashinfo (File not found)

Thank you in advance!

[Georg Pauwen]

Hello,

post the output of 'show version', this is necessary for the CLI analyzer...

[Alexander Arnold]

Here is the sh version:

vr1#sh ver
Cisco IOS Software, 2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M1 0, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Tue 24-Mar-15 09:00 by prod_rel_team

ROM: System Bootstrap, Version 12.4(13r)T11, RELEASE SOFTWARE (fc1)

vr1 uptime is 1 hour, 21 minutes
System returned to ROM by error - a System Error, PC 0x4007B7B0 at 12:26:46 MST Fri Feb 17 2017
System restarted at 12:28:25 MST Fri Feb 17 2017
System image file is "flash:c2800nm-adventerprisek9-mz.151-4.M10.bin"
Last reload type: Normal Reload

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

Cisco 2821 (revision 1.0) with 509952K/14336K bytes of memory.
Processor board ID FTX1423AJ46
2 Gigabit Ethernet interfaces
28 Serial interfaces
1 terminal line
2 Channelized T1/PRI ports
1 Virtual Private Network (VPN) Module
1 cisco service engine(s)
DRAM configuration is 64 bits wide with parity enabled.
239K bytes of non-volatile configuration memory.
500472K bytes of ATA CompactFlash (Read/Write)

License Info:

License UDI:

-------------------------------------------------
Device# PID SN
-------------------------------------------------
*0 CISCO2821 FTX1423AJ46

Configuration register is 0x2102

[Georg Pauwen]

Hello,

I can't find a specific bug, but try and uprade to Version 15.1(4)M12a. These kind of crashes are typically caused by bugs.

[Alexander Arnold]

Here is a copy of the show context:

vr1# sh context

System was restarted by error - a System Error, PC 0x4007B7B0 at 12:26:46 MST Fri Feb 17 2017
2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Compiled Tue 24-Mar-15 09:00 by prod_rel_team
Image text-base: 0x400184F0, data-base: 0x45168C20

Stack trace from system failure:
FP: 0x48FF64E0, RA: 0x4007B7B0
FP: 0x48FF64E0, RA: 0x43F0BC08
FP: 0x48FF6500, RA: 0x42AF7F2C
FP: 0x48FF6530, RA: 0x42AF84C4
FP: 0x48FF6570, RA: 0x42AF5F64
FP: 0x48FF6590, RA: 0x44B38BB0
FP: 0x48FF65A8, RA: 0x41E678A4
FP: 0x48FF65C0, RA: 0x41E651DC

Fault History Buffer:
2800 Software (C2800NM-ADVENTERPRISEK9-M), Version 15.1(4)M10, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Compiled Tue 24-Mar-15 09:00 by prod_rel_team
Signal = 0, Code = 0x0, Uptime 00:28:43
$0 : 00000000, AT : 00000000, v0 : 00000000, v1 : 00000000
a0 : 00000000, a1 : 00000000, a2 : 00000000, a3 : 00000000
t0 : 00000000, t1 : 00000000, t2 : 00000000, t3 : 00000000
t4 : 00000000, t5 : 00000000, t6 : 00000000, t7 : 00000000
s0 : 00000000, s1 : 00000000, s2 : 00000000, s3 : 00000000
s4 : 00000000, s5 : 00000000, s6 : 00000000, s7 : 00000000
t8 : 00000000, t9 : 00000000, k0 : 00000000, k1 : 00000000
gp : 00000000, sp : 00000000, s8 : 00000000, ra : 00000000
EPC : 00000000, SREG : 00000000, Cause : 00000000
Error EPC : 00000000, BadVaddr : 00000000
CacheErr : 00000000, DErrAddr0 : 00000000, DErrAddr1 : 00000000
TEXT_START : 0x400184F0
DATA_START : 0x45168C20