08-22-2013 01:20 AM - edited 03-04-2019 08:50 PM
Hi,
I have an ISR 2911. Every now and then, it crashes saying "Possible software fault". and the system Constantly reload
The following information is presented under the CONSOL
%Software-forced reload
: Breakpoint exception, CPU signal 23, PC = 0x219CC00C
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
-Traceback= 0x219C1D6Cz 0x219C0248z 0x2115BF1Cz 0x219C6F6Cz 0x219C01E4z 0x2115EB0Cz
$0 : 00000000 00000000, AT : 00000000 28600000, v0 : 00000000 287EB3B8, v1 : 00000000 00000000
a0 : 00000000 00000001, a1 : 00000000 2640DAF3, a2 : 00000000 00000000, a3 : 00000000 00000000
t0 : 00000000 00009F00, t1 : 00000000 50009FE1, t2 : 00000000 219D1B28, t3 : 00000000 27EC0000
t4 : 00000000 28600000, t5 : 00000000 00000000, t6 : 00000000 28600000, t7 : 00000000 00000C28
s0 : 00000000 00000000, s1 : 00000000 27EB0000, s2 : 00000000 00000200, s3 : FFFFFFFF 80010700
s4 : 00000000 0000002E, s5 : 00000000 28A6A9E4, s6 : 00000000 00000000, s7 : FFFFFFFF 80010700
t8 : 00000000 00000C40, t9 : 00000000 00800118, k0 : 00000000 00000000, k1 : 00000000 00000000
gp : 00000000 28603FE0, sp : 00000000 28FB5C18, s8 : 00000000 00000000, ra : 00000000 219CA4E8
EPC : 219CC00C, ErrorEPC : 80412BEC, SREG : 50009FE3
MDLO : 00000000, MDHI : 00000002, BadVaddr : 00000000
CacheErr_Icache : 0x007F7F00 - 000003F8, CacheErr_Dcache : 0x00000000 - 00000D60
TagLo_Icache : 0x00000000 - 21164F83, DataLo_Icache : 0x02518025 - 00007821
DataHi_Icache: 0x00000000 - 00000000
TagLo_Dcache : 0x00000000 - 285FC903, TagHi_Dcache : 0x00000000 - 085FC101
DataLo_Dcache: 0x00000001 - 2114AA84, DataHi_Dcache : 0x00000000 - 00000010
TEXT_START : 0x2100AB78
DATA_START : 0x259362A0
Cause 00000024 (Code 0x9): Breakpoint exception
: Breakpoint exception, CPU signal 23, PC = 0x219CC00C
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
Anyone ever encountered such a problem?
Thanks,
Solved! Go to Solution.
08-23-2013 02:20 AM
Hello,
I agree with Paolo - if you have tried different IOS versions and each of them behaves in the same way then I am afraid the hardware may be faulty.
As a last resort, try pulling out all added interfaces, modules, AIMs, PVDMs etc. from your router, leaving only a barebone router. Then try booting it up. Perhaps the problem is limited to a particular module you could then try to isolate.
If this does not help, and using tftpdnld -r to download and boot an IOS from TFTP without writing it into FLASH does not help either (try either 15.2(4)M4 or 15.3(3)M) then I am afraid we are out of luck, and you should RMA the device.
Just curious, was there any incident you could trace back to when the problem started appearing? Was there a power surge, or a power brownout/blackout, anything unusual that happened around the time the router became unable to boot? When did you first notice these problems and what preceded them?
Best regards,
Peter
08-22-2013 01:51 AM
Hi,
This suggests at a bug in IOS, or to a hardware fault - most probably in the RAM. Let's hope it is the IOS.
What IOS version are you currently running? Can you afford to upgrade to a different version? Avoid the 'T' versions if possible.
Also, can you post the complete configuration and the crashinfo file that was stored into FLASH when your router reloaded? Use the advanced editor when responding here - it allows attaching files.
Best regards,
Peter
08-22-2013 01:58 AM
i Have been re-replaced two IOS, and neither is "T" version, the phenomenon is still the same
08-22-2013 02:15 AM
Hi,
Would you mind posting the complete configuration and the crashinfo files?
Best regards,
Peter
08-22-2013 02:20 AM
it continuously reboot, i can not enter configuration mode, so i have no configuration
All tips are as follows
System Bootstrap, Version 15.0(1r)M16, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 2012 by cisco Systems, Inc.
Total memory size = 512 MB - On-board = 512 MB, DIMM0 = 0 MB
CISCO2911/K9 platform with 524288 Kbytes of main memory
Main memory is configured to 72/-1(On-board/DIMM0) bit mode with ECC enabled
Readonly ROMMON initialized
program load complete, entry point: 0x80803000, size: 0x1b340
program load complete, entry point: 0x80803000, size: 0x1b340
IOS Image Load Test
___________________
Digitally Signed Release Software
program load complete, entry point: 0x81000000, size: 0x3bc25d0
Self decompressing the image : ################################################################################################################################################################################################################################################################################################################################################################################ [OK]
Smart Init is enabled
smart init is sizing iomem
TYPE MEMORY_REQ
Onboard devices &
buffer pools 0x0228F000
-----------------------------------------------
TOTAL: 0x0228F000
Rounded IOMEM up to: 36Mb.
Using 7 percent iomem. [36Mb/512Mb]
Restricted Rights Legend
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
cisco Systems, Inc.
170 West Tasman Drive
San Jose, California 95134-1706
Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.0(1)M3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Sun 18-Jul-10 03:32 by prod_rel_team
Image text-base: 0x2100AB78, data-base: 0x259362A0
%Software-forced reload
: Breakpoint exception, CPU signal 23, PC = 0x219CC00C
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
-Traceback= 0x219C1D6Cz 0x219C0248z 0x2115BF1Cz 0x219C6F6Cz 0x219C01E4z 0x2115EB0Cz
$0 : 00000000 00000000, AT : 00000000 28600000, v0 : 00000000 287EB3B8, v1 : 00000000 00000000
a0 : 00000000 00000001, a1 : 00000000 2640DAF3, a2 : 00000000 00000000, a3 : 00000000 00000000
t0 : 00000000 00009F00, t1 : 00000000 50009FE1, t2 : 00000000 219D1B28, t3 : 00000000 27EC0000
t4 : 00000000 28600000, t5 : 00000000 00000000, t6 : 00000000 28600000, t7 : 00000000 00000C28
s0 : 00000000 00000000, s1 : 00000000 27EB0000, s2 : 00000000 00000200, s3 : FFFFFFFF 80010700
s4 : 00000000 0000002E, s5 : 00000000 28A6A9E4, s6 : 00000000 00000000, s7 : FFFFFFFF 80010700
t8 : 00000000 00000C40, t9 : 00000000 00800118, k0 : 00000000 00000000, k1 : 00000000 00000000
gp : 00000000 28603FE0, sp : 00000000 28FB5C18, s8 : 00000000 00000000, ra : 00000000 219CA4E8
EPC : 219CC00C, ErrorEPC : 80412BEC, SREG : 50009FE3
MDLO : 00000000, MDHI : 00000002, BadVaddr : 00000000
CacheErr_Icache : 0x007F7F00 - 000003F8, CacheErr_Dcache : 0x00000000 - 00000D60
TagLo_Icache : 0x00000000 - 21164F83, DataLo_Icache : 0x02518025 - 00007821
DataHi_Icache: 0x00000000 - 00000000
TagLo_Dcache : 0x00000000 - 285FC903, TagHi_Dcache : 0x00000000 - 085FC101
DataLo_Dcache: 0x00000001 - 2114AA84, DataHi_Dcache : 0x00000000 - 00000010
TEXT_START : 0x2100AB78
DATA_START : 0x259362A0
Cause 00000024 (Code 0x9): Breakpoint exception
: Breakpoint exception, CPU signal 23, PC = 0x219CC00C
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
-Traceback= 0x219C1D6Cz 0x219C0248z 0x2115BF1Cz 0x219C6F6Cz 0x219C01E4z 0x2115EB0Cz
$0 : 00000000 00000000, AT : 00000000 28600000, v0 : 00000000 287EB3B8, v1 : 00000000 00000000
a0 : 00000000 00000001, a1 : 00000000 2640DAF3, a2 : 00000000 00000000, a3 : 00000000 00000000
t0 : 00000000 00009F00, t1 : 00000000 50009FE1, t2 : 00000000 219D1B28, t3 : 00000000 27EC0000
t4 : 00000000 28600000, t5 : 00000000 00000000, t6 : 00000000 28600000, t7 : 00000000 00000C28
s0 : 00000000 00000000, s1 : 00000000 27EB0000, s2 : 00000000 00000200, s3 : FFFFFFFF 80010700
s4 : 00000000 0000002E, s5 : 00000000 28A6A9E4, s6 : 00000000 00000000, s7 : FFFFFFFF 80010700
t8 : 00000000 00000C40, t9 : 00000000 00800118, k0 : 00000000 00000000, k1 : 00000000 00000000
gp : 00000000 28603FE0, sp : 00000000 28FB5C18, s8 : 00000000 00000000, ra : 00000000 219CA4E8
EPC : 219CC00C, ErrorEPC : 80412BEC, SREG : 50009FE3
MDLO : 00000000, MDHI : 00000002, BadVaddr : 00000000
CacheErr_Icache : 0x007F7F00 - 000003F8, CacheErr_Dcache : 0x00000000 - 00000D60
TagLo_Icache : 0x00000000 - 21164F83, DataLo_Icache : 0x02518025 - 00007821
DataHi_Icache: 0x00000000 - 00000000
TagLo_Dcache : 0x00000000 - 285FC903, TagHi_Dcache : 0x00000000 - 085FC101
DataLo_Dcache: 0x00000001 - 2114AA84, DataHi_Dcache : 0x00000000 - 00000010
TEXT_START : 0x2100AB78
DATA_START : 0x259362A0
Cause 00000024 (Code 0x9): Breakpoint exception
: Breakpoint exception, CPU signal 23, PC = 0x219CC00C
--------------------------------------------------------------------
Possible software fault. Upon reccurence, please collect
crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------
-Traceback= 0x219C1D6Cz 0x219C0248z 0x2115BF1Cz 0x219C6F6Cz 0x219C01E4z 0x2115EB0Cz
$0 : 00000000 00000000, AT : 00000000 28600000, v0 : 00000000 287EB3B8, v1 : 00000000 00000000
a0 : 00000000 00000001, a1 : 00000000 2640DAF3, a2 : 00000000 00000000, a3 : 00000000 00000000
t0 : 00000000 00009F00, t1 : 00000000 50009FE1, t2 : 00000000 219D1B28, t3 : 00000000 27EC0000
t4 : 00000000 28600000, t5 : 00000000 00000000, t6 : 00000000 28600000, t7 : 00000000 00000C28
s0 : 00000000 00000000, s1 : 00000000 27EB0000, s2 : 00000000 00000200, s3 : FFFFFFFF 80010700
s4 : 00000000 0000002E, s5 : 00000000 28A6A9E4, s6 : 00000000 00000000, s7 : FFFFFFFF 80010700
t8 : 00000000 00000C40, t9 : 00000000 00800118, k0 : 00000000 00000000, k1 : 00000000 00000000
gp : 00000000 28603FE0, sp : 00000000 28FB5C18, s8 : 00000000 00000000, ra : 00000000 219CA4E8
EPC : 219CC00C, ErrorEPC : 80412BEC, SREG : 50009FE3
MDLO : 00000000, MDHI : 00000002, BadVaddr : 00000000
CacheErr_Icache : 0x007F7F00 - 000003F8, CacheErr_Dcache : 0x00000000 - 00000D60
TagLo_Icache : 0x00000000 - 21164F83, DataLo_Icache : 0x02518025 - 00007821
DataHi_Icache: 0x00000000 - 00000000
TagLo_Dcache : 0x00000000 - 285FC903, TagHi_Dcache : 0x00000000 - 085FC101
DataLo_Dcache: 0x00000001 - 2114AA84, DataHi_Dcache : 0x00000000 - 00000010
TEXT_START : 0x2100AB78
DATA_START : 0x259362A0
Cause 00000024 (Code 0x9): Breakpoint exception
-Traceback= 0x219C1D6Cz 0x219C0248z 0x2115BF1Cz 0x219C6F6Cz 0x219C01E4z 0x2115EB0Cz
08-22-2013 03:48 AM
The IOS you are using is quite old, please update from rommon unsing tftpdnld command.
Anyway, from address present the failure, it is possible that this is an hardware fault requiring device replacement.
08-23-2013 02:20 AM
Hello,
I agree with Paolo - if you have tried different IOS versions and each of them behaves in the same way then I am afraid the hardware may be faulty.
As a last resort, try pulling out all added interfaces, modules, AIMs, PVDMs etc. from your router, leaving only a barebone router. Then try booting it up. Perhaps the problem is limited to a particular module you could then try to isolate.
If this does not help, and using tftpdnld -r to download and boot an IOS from TFTP without writing it into FLASH does not help either (try either 15.2(4)M4 or 15.3(3)M) then I am afraid we are out of luck, and you should RMA the device.
Just curious, was there any incident you could trace back to when the problem started appearing? Was there a power surge, or a power brownout/blackout, anything unusual that happened around the time the router became unable to boot? When did you first notice these problems and what preceded them?
Best regards,
Peter
08-23-2013 02:40 AM
This equipment is the customer's, customer can't provide the events before the failure
i will try using tftpdnld to download an IOS later
as you say,may be it's hardware fault
Thank you very much for help
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide