3550 crashing - Software Forced Crash Exception

lily.chang · ‎03-14-2011

Hi,

I have a 3550 that has been crashing/reloading intermittently. The switch was upgraded to the latest IOS after a crash last week and it just crashed again today. Does anyone have any ideas of what else to do? Otherwise, I'm looking at replacing it with another 3550 from our storage. Below is a bunch of info from the crash. Any help would be appreciated!

Thanks,

Lily

From sh ver:

System returned to ROM by error - Software Forced Crash Exception (0x0700)!, PC 0x3EE914
System restarted at 14:37:07 CDT Mon Mar 14 2011
System image file is "flash:c3550-ipbasek9-mz.122-44.SE6/c3550-ipbasek9-mz.122-44.SE6.bin"

From sh logg:

.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: System previously crashed with the following message:
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: Cisco IOS Software, C3550 Software (C3550-IPBASEK9-M), Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: Copyright (c) 1986-2009 by Cisco Systems, Inc.
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: Compiled Mon 09-Mar-09 20:26 by gereddy
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED:
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: Software Forced Crash Exception (0x0700)!
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: CPU Register Context:
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: Vector = 0x00000700 PC = 0x003EE914 MSR = 0x0002B030 CR = 0x24444024
.Mar 14 14:38:51.877 CDT: %PLATFORM_CAT3550-1-CRASHED: LR = 0x003EE8C4 CTR = 0x00459FF4 XER = 0x00000000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R0 = 0x003EE8C4 R1 = 0x01892470 R2 = 0x00000000 R3 = 0x00000000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R4 = 0xFFFFFFFE R5 = 0xFEC0FEC0 R6 = 0xFEE00000 R7 = 0x00003030
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R8 = 0x0000B030 R9 = 0x014A0000 R10 = 0x01614200 R11 = 0x00000000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R12 = 0x44444028 R13 = 0xDF6AACF7 R14 = 0x00000000 R15 = 0x00000000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R16 = 0x016A827C R17 = 0x016A8244 R18 = 0x016B0000 R19 = 0x01670000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R20 = 0x01670000 R21 = 0x00000000 R22 = 0x016B0000 R23 = 0x00000000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R24 = 0x00000000 R25 = 0x016B0000 R26 = 0x00000000 R27 = 0x016B0000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: R28 = 0x00000000 R29 = 0x016B0000 R30 = 0x00000000 R31 = 0x00000000
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED:
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: Stack trace:
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: PC = 0x003EE914, SP = 0x01892470
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: Frame 00: SP = 0x01892480    PC = 0x003EE8C4
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: Frame 01: SP = 0x018924E0    PC = 0x003EB478
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED: Frame 02: SP = 0xFFFFFFFF    PC = 0x003E97C4
.Mar 14 14:38:51.881 CDT: %PLATFORM_CAT3550-1-CRASHED:

From sh stack:

Minimum process stacks:
Free/Size   Name
4668/6000   vegas_flash init
3604/6000   SaveCrashBuffer
5740/6000   CDP BLOB
8568/9000   IP Background
5652/6000   vqpc_shim_create_addr_tbl
5640/6000   SPAN Subsystem
5684/6000   stp_change_tree_type
5000/6000   vegas IPC process
5580/6000   SASL MAIN
6992/12000 Init
8724/9000   cdp init process
5480/6000   RADIUS INITCONFIG
4940/6000   Vegas CrashBuffer
34984/36000 TCP Command
5688/6000   URPF stats
2596/3000   Rom Random Update Process

Interrupt level stacks:
Level    Called Unused/Size Name
2        1090   8872/9000 Tweetie pwr controller chips
4       39759   8372/9000 CPU NETWORK INTERFACE CHIP
5          53   8900/9000 Vegas miami tty xmit hack timer
6      509694   8932/9000 Clock Timer Vector

Spurious interrupts: 269

System was restarted by error - Software Forced Crash Exception (0x0700)!, PC 0x3EE914
C3550 Software (C3550-IPBASEK9-M), Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)
Compiled Mon 09-Mar-09 20:26 by gereddy (current version)
Image text-base: 0x00003000, data-base: 0x00EA3DB8

Stack trace from system failure:
FP: 0x1892470, RA: 0x3EE914
FP: 0x1892480, RA: 0x3EE8C4
FP: 0x18924E0, RA: 0x3EB478
FP: 0xFFFFFFFF, RA: 0x3E97C4

***************************************************
******* Information of Last System Crash **********
***************************************************

Using flash:/crashinfo/crashinfo_4.
Cisco IOS Software, C3550 Software (C3550-IPBASEK9-M), Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 09-Mar-09 20:26 by gereddy

Software Forced Crash Exception (0x0700)!
CPU Register Context:
Vector = 0x00000700 PC = 0x003EE914 MSR = 0x0002B030 CR = 0x24444024
LR = 0x003EE8C4 CTR = 0x00459FF4 XER = 0x00000000
R0 = 0x003EE8C4 R1 = 0x01892470 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0xFEC0FEC0 R6 = 0xFEE00000 R7 = 0x00003030
R8 = 0x0000B030 R9 = 0x014A0000 R10 = 0x01614200 R11 = 0x00000000
R12 = 0x44444028 R13 = 0xDF6AACF7 R14 = 0x00000000 R15 = 0x00000000
R16 = 0x016A827C R17 = 0x016A8244 R18 = 0x016B0000 R19 = 0x01670000
R20 = 0x01670000 R21 = 0x00000000 R22 = 0x016B0000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x016B0000 R26 = 0x00000000 R27 = 0x016B0000
R28 = 0x00000000 R29 = 0x016B0000 R30 = 0x00000000 R31 = 0x00000000

Stack trace:
PC = 0x003EE914, SP = 0x01892470
Frame 00: SP = 0x01892480    PC = 0x003EE8C4
Frame 01: SP = 0x018924E0    PC = 0x003EB478
Frame 02: SP = 0xFFFFFFFF    PC = 0x003E97C4

Leo Laohoo · ‎03-14-2011

Using flash:/crashinfo/crashinfo_4.

Can you please post the crashinfo files? Thanks.

lily.chang · ‎03-14-2011

Here is crashinfo_4:

Cisco IOS Software, C3550 Software (C3550-IPBASEK9-M), Version 12.2(44)SE6, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Mon 09-Mar-09 20:26 by gereddy

Software Forced Crash Exception (0x0700)!
CPU Register Context:
Vector = 0x00000700 PC = 0x003EE914 MSR = 0x0002B030 CR = 0x24444024
LR = 0x003EE8C4 CTR = 0x00459FF4 XER = 0x00000000
R0 = 0x003EE8C4 R1 = 0x01892470 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0xFEC0FEC0 R6 = 0xFEE00000 R7 = 0x00003030
R8 = 0x0000B030 R9 = 0x014A0000 R10 = 0x01614200 R11 = 0x00000000
R12 = 0x44444028 R13 = 0xDF6AACF7 R14 = 0x00000000 R15 = 0x00000000
R16 = 0x016A827C R17 = 0x016A8244 R18 = 0x016B0000 R19 = 0x01670000
R20 = 0x01670000 R21 = 0x00000000 R22 = 0x016B0000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x016B0000 R26 = 0x00000000 R27 = 0x016B0000
R28 = 0x00000000 R29 = 0x016B0000 R30 = 0x00000000 R31 = 0x00000000

Stack trace:
PC = 0x003EE914, SP = 0x01892470
Frame 00: SP = 0x01892480    PC = 0x003EE8C4
Frame 01: SP = 0x018924E0    PC = 0x003EB478
Frame 02: SP = 0xFFFFFFFF    PC = 0x003E97C4

Crashinfo_3:

Cisco IOS Software, C3550 Software (C3550-IPBASEK9-M), Version 12.2(25)SEE3, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 22-Feb-07 15:35 by myl

Software Forced Crash Exception (0x0700)!
CPU Register Context:
Vector = 0x00000700 PC = 0x0043A910 MSR = 0x0002B030 CR = 0x24444024
LR = 0x0043A848 CTR = 0x0049C7A0 XER = 0x00000000
R0 = 0x00000000 R1 = 0x01405B20 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0xFEC0FEC0 R6 = 0xFEE00000 R7 = 0x00003030
R8 = 0x0000B030 R9 = 0x010F0000 R10 = 0x0124B7A0 R11 = 0x00000000
R12 = 0x1A6F6FA4 R13 = 0xDF6AACF7 R14 = 0x01405B38 R15 = 0x00F90000
R16 = 0x00D60000 R17 = 0x00F90000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x012B0000 R22 = 0x00000000 R23 = 0x01280000
R24 = 0x00000000 R25 = 0x012B0000 R26 = 0x00000000 R27 = 0x01280000
R28 = 0x00000001 R29 = 0x010718D8 R30 = 0x00000000 R31 = 0x01280000

Stack trace:
PC = 0x0043A910, SP = 0x01405B20
Frame 00: SP = 0x01405B30    PC = 0x0043A848
Frame 01: SP = 0x01405CB0    PC = 0x00436DEC
Frame 02: SP = 0xFFFFFFFF    PC = 0x004348A4

Crashinfo_2:

Cisco IOS Software, C3550 Software (C3550-IPBASEK9-M), Version 12.2(25)SEE3, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 22-Feb-07 15:35 by myl

Software Forced Crash Exception (0x0700)!
CPU Register Context:
Vector = 0x00000700 PC = 0x0043A910 MSR = 0x0002B030 CR = 0x24444024
LR = 0x0043A848 CTR = 0x0049C7A0 XER = 0x00000000
R0 = 0x00000000 R1 = 0x01405B20 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0xFEC0FEC0 R6 = 0xFEE00000 R7 = 0x00003030
R8 = 0x0000B030 R9 = 0x010F0000 R10 = 0x0124B7A0 R11 = 0x00000000
R12 = 0x074300AE R13 = 0xDF6AACF7 R14 = 0x01405B38 R15 = 0x00F90000
R16 = 0x00D60000 R17 = 0x00F90000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x012B0000 R22 = 0x00000000 R23 = 0x01280000
R24 = 0x00000000 R25 = 0x012B0000 R26 = 0x00000000 R27 = 0x01280000
R28 = 0x00000001 R29 = 0x010718D8 R30 = 0x00000000 R31 = 0x01280000

Stack trace:
PC = 0x0043A910, SP = 0x01405B20
Frame 00: SP = 0x01405B30    PC = 0x0043A848
Frame 01: SP = 0x01405CB0    PC = 0x00436DEC
Frame 02: SP = 0xFFFFFFFF    PC = 0x004348A4

Crashinfo_1:

Cisco IOS Software, C3550 Software (C3550-IPBASEK9-M), Version 12.2(25)SEE3, RELEASE SOFTWARE (fc2)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Thu 22-Feb-07 15:35 by myl

Software Forced Crash Exception (0x0700)!
CPU Register Context:
Vector = 0x00000700 PC = 0x0043A910 MSR = 0x0002B030 CR = 0x22444024
LR = 0x0043A848 CTR = 0x0049C7A0 XER = 0x00000000
R0 = 0x00000000 R1 = 0x01405B20 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0xFEC0FEC0 R6 = 0xFEE00000 R7 = 0x00003030
R8 = 0x0000B030 R9 = 0x010F0000 R10 = 0x0124B7A0 R11 = 0x00000000
R12 = 0xD9FFF7B0 R13 = 0xDF6AACF7 R14 = 0x01405B38 R15 = 0x00F90000
R16 = 0x00D60000 R17 = 0x00F90000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x012B0000 R22 = 0x00000000 R23 = 0x01280000
R24 = 0x00000000 R25 = 0x01280000 R26 = 0x00000000 R27 = 0x01280000
R28 = 0x00000001 R29 = 0x01071168 R30 = 0x00000000 R31 = 0x01280000

Stack trace:
PC = 0x0043A910, SP = 0x01405B20
Frame 00: SP = 0x01405B30    PC = 0x0043A848
Frame 01: SP = 0x01405CB0    PC = 0x00436DEC
Frame 02: SP = 0xFFFFFFFF    PC = 0x004348A4

Leo Laohoo · ‎03-14-2011

I am suspecting you have a failed component.

Erase the config and observe the switch. If it crashes then you know it's a failed component.