Hello Cisco forums!
Our switch had a little crash-fest this morning at 2:30 AM, and I was hoping to see if anyone here might have some insight into it. I did find a web page about diagnosing Software Forced Crash Exceptions, but it did not look like ours was one of the more easily-identifiable ones.
It may be worth noting that we've only used this switch for about a month, everything seemed fine until now. When we got the switch it did not have any GigE modules, and this week we put 2 into it and have been using them for 2 servers.
It looks like the switch was crashing repeatedly over a period of 20 minutes, and then it stopped and normalized. In the logs of the router that this switch uplinks into, we could see the ethernet port flapping during the time that the switch wasn't reachable.
Any help or insight you might have into it would be appreciated!
Here's the Show Stack on the switch:
Sfld_3550# show stack
Minimum process stacks:
Free/Size Name
4404/6000 vegas_flash init
3352/6000 SaveCrashBuffer
5716/6000 CDP BLOB
8512/9000 IP Background
5596/6000 vqpc_shim_create_addr_tbl
5584/6000 SPAN Subsystem
5552/6000 SASL MAIN
4944/6000 vegas IPC process
8704/9000 cdp init process
5404/6000 RADIUS INITCONFIG
4928/6000 Vegas CrashBuffer
5664/6000 URPF stats
2536/3000 Rom Random Update Process
9080/12000 Init
34860/36000 TCP Command
Interrupt level stacks:
Level Called Unused/Size Name
4 41913 8324/9000 CPU NETWORK INTERFACE CHIP
5 9591 8900/9000 Vegas miami tty xmit hack timer
6 432879 8932/9000 Clock Timer Vector
Spurious interrupts: 237
System was restarted by error - Software Forced Crash Exception (0x0700)!, PC 0x4FD5A4
C3550 Software (C3550-IPBASE-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Compiled Fri 25-Sep-09 03:41 by sasyamal (current version)
Image text-base: 0x00003000, data-base: 0x00FCF56C
Stack trace from system failure:
FP: 0x29EE0A8, RA: 0x4FD5A4
FP: 0x29EE0B8, RA: 0x4FD5A4
FP: 0x29EE0D8, RA: 0x305A70
FP: 0x29EE0F0, RA: 0x57DDFC
FP: 0x29EE0F8, RA: 0x4FF530
FP: 0x0, RA: 0x4F6000
***************************************************
******* Information of Last System Crash **********
***************************************************
Using flash:/crashinfo/crashinfo_1.
Cisco IOS Software, C3550 Software (C3550-IPBASE-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 03:41 by sasyamal
Software Forced Crash Exception (0x0700)!
CPU Register Context:
Vector = 0x00000700 PC = 0x004FD5A4 MSR = 0x0002B030 CR = 0x20000002
LR = 0x004FD5A4 CTR = 0x0056C3AC XER = 0x20000000
R0 = 0x004FD5A4 R1 = 0x029EE0A8 R2 = 0x00000000 R3 = 0x00000000
R4 = 0x00EED1DC R5 = 0x0000B030 R6 = 0xFEE00000 R7 = 0xBEEFCAFE
R8 = 0x00000000 R9 = 0x016A0000 R10 = 0x015D9AB0 R11 = 0x015D98C0
R12 = 0x000DDD1C R13 = 0x1CE6E9F3 R14 = 0x0057DD20 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000
R28 = 0x014C97D0 R29 = 0x016DFAB0 R30 = 0x014C97D0 R31 = 0x00000001
Stack trace:
PC = 0x004FD5A4, SP = 0x029EE0A8
Frame 00: SP = 0x029EE0B8 PC = 0x004FD5A4
Frame 01: SP = 0x029EE0D8 PC = 0x00305A70
Frame 02: SP = 0x029EE0F0 PC = 0x0057DDFC
Frame 03: SP = 0x029EE0F8 PC = 0x004FF530
Frame 04: SP = 0x00000000 PC = 0x004F6000
Here's our Show Ver on it:
Sfld_3550>sh ver
Cisco IOS Software, C3550 Software (C3550-IPBASE-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 03:41 by sasyamal
Image text-base: 0x00003000, data-base: 0x00FCF56C
ROM: Bootstrap program is C3550 boot loader
Sfld_3550 uptime is 3 hours, 51 minutes
System returned to ROM by error - Software Forced Crash Exception (0x0700)!, PC 0x4FD5A4
System restarted at 02:54:35 EST Fri Feb 8 2013
System image file is "flash:c3550-ipbase-mz.122-52.SE/c3550-ipbase-mz.122-52.SE.bin"
Cisco WS-C3550-48 (PowerPC) processor (revision E0) with 65526K/8192K bytes of memory.
Processor board ID CHK0635W16V
Last reset from warm-reset
Running Layer2/3 Switching Image
Ethernet-controller 1 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 2 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 3 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 4 has 12 Fast Ethernet/IEEE 802.3 interfaces
Ethernet-controller 5 has 1 Gigabit Ethernet/IEEE 802.3 interface
Ethernet-controller 6 has 1 Gigabit Ethernet/IEEE 802.3 interface
48 FastEthernet interfaces
2 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
384K bytes of flash-simulated NVRAM.
Base ethernet MAC Address: 00:0A:B7:12:56:00
Motherboard assembly number: 73-5701-06
Power supply part number: 34-0967-01
Motherboard serial number: CAT063103KB
Power supply serial number: DCA06341JHC
Model revision number: E0
Motherboard revision number: D0
Model number: WS-C3550-48-SMI
System serial number: CHK0635W16V
Configuration register is 0x10FSfld_3550>sh ver
Cisco IOS Software, C3550 Software (C3550-IPBASE-M), Version 12.2(52)SE, RELEASE SOFTWARE (fc3)
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Fri 25-Sep-09 03:41 by sasyamal
Image text-base: 0x00003000, data-base: 0x00FCF56C
