System returned to ROM by s/w reset (SP by error - a Software forced crash)

Suputhear · ‎01-16-2012

Hello

I have my router 7606-S rebooted by itself. According to output of show version command it's rebooted by software bug.

What is the actual cause? Is there any workarounds for this problem? Thanks.

================ Snipped from crashinfo file =================

Jan 14 05:33:50.822 GMT+7: %C7600_MEM_ECC-2-MBE: Multiple bit error detected at 0xC09F4D38

Jan 14 05:33:50.822 GMT+7: %C7600_MEM_ECC-3-SYNDROME_MBE: 8-bit Syndrome for the detected Multi-bit error: 0x0

%Software-forced reload

05:33:50 GMT+7 Sat Jan 14 2012: Unexpected exception to CPU: vector 1500, PC = 0xB7922BC , LR = 0xB792250

==================== Show Version ======================

Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVENTERPRISEK9-M), Version 15.1(1)S1, RELEASE SOFTWARE (fc1)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 10-Feb-11 19:27 by prod_rel_team

ROM: System Bootstrap, Version 12.2(33r)SRD5, RELEASE SOFTWARE (fc1)

BOOTLDR: Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVENTERPRISEK9-M), Version 15.1(1)S1, RELEASE SOFTWARE (fc1)

R-01-BVT-CORE uptime is 3 days, 4 hours, 18 minutes

Uptime for this control processor is 3 days, 4 hours, 18 minutes

System returned to ROM by s/w reset (SP by error - a Software forced crash, PC 0x83AF9B8)

System restarted at 05:36:31 GMT+7 Sat Jan 14 2012

System image file is "disk0:c7600rsp72043-adventerprisek9-mz.151-1.S1.bin"

Last reload type: Normal Reload

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

Cisco CISCO7606-S (M8500) processor (revision 1.1) with 3670016K/262144K bytes of memory.

Processor board ID FOX1403G3E6

BASEBOARD: RSP720

CPU: MPC8548_E, Version: 2.0, (0x80390020)

CORE: E500, Version: 2.0, (0x80210020)

CPU:1200MHz, CCB:400MHz, DDR:200MHz,

L1: D-cache 32 kB enabled

I-cache 32 kB enabled

Last reset from power-on

1 Virtual Ethernet interface

98 Gigabit Ethernet interfaces

3964K bytes of non-volatile configuration memory.

507024K bytes of Internal ATA PCMCIA card (Sector size 512 bytes).

Configuration register is 0x2102

Richard Burts · ‎01-16-2012

It appears that the real cause of the reload is a software bug. We do not have enough information to identify the specific bug. If the router is covered by a maintenance contract you could open a case with Cisco TAC and they have resources to identify the problem.

With a software bug like this the workaround is to change the software version. If you stay with the same software you might be lucky and it might be a long time before the problem happens again. But it could also happen again soon. So my advice is to plan to change the software version.

HTH

Rick

HTH

Rick

Suputhear · ‎01-16-2012

Thank Richard Burts.

This is the second times that the problem occurred. I have 2 routers 7606-S series supplied with the same memory (3GB) and IOS version. I really wonder why only this one has such a problem, while the other one is working fine without any problems.

I'd better contact my vendor to open a TAC case.

Richard Burts · ‎01-16-2012

I agree that it would be good to open a TAC case and to have Cisco work with you on this problem.

HTH

Rick

HTH

Rick

antonkolev · ‎01-31-2015

Richard ,

well I have the same error - its non S chassis - standard 7606 router and ver 12

with RSP 720 3cxl card

Seems like to me its a hardware issue -

000750: Jan 31 23:39:09.454 EST: %ECC-3-SBE_LIMIT: Single bit error detected and corrected
at 0x08771BE0
000751: Jan 31 23:39:09.454 EST: %ECC-3-SYNDROME_SBE_LIMIT: 8-bit Syndrome for the detected Single-bit error: 0x7C
000752: Jan 31 23:39:11.154 EST: %ECC-3-SBE_HARD: Single bit *hard* error detected at 0x08771BE0
000753: Jan 31 23:44:11.665 EST: %ECC-3-SBE_LIMIT: Single bit error detected and corrected
at 0x08771FE0
000754: Jan 31 23:44:11.665 EST: %ECC-3-SYNDROME_SBE_LIMIT: 8-bit Syndrome for the detected Single-bit error: 0xF
000755: Jan 31 23:45:32.685 EST: %ECC-3-SBE_LIMIT: Single bit error detected and corrected
at 0x08771FE0
000756: Jan 31 23:45:32.685 EST: %ECC-3-SYNDROME_SBE_LIMIT: 8-bit Syndrome for the detected Single-bit error: 0xF
000757: Jan 31 23:46:08.521 EST: %C7600_MEM_ECC-2-MBE: Multiple bit error detected at 0x08771000
000758: Jan 31 23:46:08.521 EST: %C7600_MEM_ECC-3-SYNDROME_MBE: 8-bit Syndrome for the detected Multi-bit error: 0xF9

%Software-forced reload

23:46:08 EST Sat Jan 31 2015: Unexpected exception to CPU: vector 1500, PC = 0xAFC0844 , LR = 0xAFC0808

ECC Single Bit error log
------------------------
Single Bit error detected and corrected at 0x00000000
- Occured 0 times
- Whether a scrub was attempted at this address: No
- Syndrome of the last error at this address: 0x0
- Error detected on a read-modify-write cycle ? No
- Address region classification: Unknown
- Address media classification : Unknown
Single Bit error detected and corrected at 0x087715F0
- Occured 12 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0xED
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08771000
- Occured 15 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0xF9
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x087714E0
- Occured 8 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x25
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08770F40
- Occured 13 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x32
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08771500
- Occured 12 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x61
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08771500
- Occured 1 time
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x61
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08770CA0
- Occured 11 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x69
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09999280
- Occured 25061 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x36
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09999280
- Occured 1 time
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x36
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09998A40
- Occured 44612 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x96
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09998A40
- Occured 1 time
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x96
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09999EE0
- Occured 7328 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x66
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09999EE0
- Occured 1 time
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x66
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08770BA0
- Occured 5 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0xD1
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08770BA0
- Occured 1 time
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0xD1
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x09998A50
- Occured 1 time
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0x96
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only
Single Bit error detected and corrected at 0x08771140
- Occured 3 times
- Whether a scrub was attempted at this address: Yes
- Syndrome of the last error at this address: 0xEF
- Error detected on a read-modify-write cycle ? No
- Address region classification: text
- Address media classification : Read Only

Total Single Bit errors thus far: 77086

Richard Burts · ‎02-01-2015

This does suggest a hardware problem. So I will repeat the advice that I gave for the original question in this thread - it would be good to open a case with Cisco TAC to get this figured out.

HTH

Rick

HTH

Rick

antonkolev · ‎02-09-2015

Before I go with TAC

I did remove that router out of production and running without load

- And haven't rebooted yet -

Richard Burts · ‎02-09-2015

It is interesting that when running without load that it does not reboot. If you believe that this indicates that it is a software issue then you might think about trying to use a different version of code on the switch.

I continue to believe that opening a case with Cisco TAC is the way to get the most authoritative analysis of the problem and whether it is hardware or software.

HTH

Rick

HTH

Rick