I have a stack of two 3750X's (WS-C3750X-24P-S) running c3750e-universalk9-mz.150-1.SE3.bin.
One of them (the master at the time) crashed on 27/28th Feb & appears to have kept restarting and crashing again.
Here is the syslog...
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: System previously crashed with the following message: (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Cisco IOS Software, C3750E Software (C3750E-UNIVERSALK9-M), Version 15.0(1)SE3, RELEASE SOFTWARE (fc1) (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Technical Support: http://www.cisco.com/techsupport (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Copyright (c) 1986-2012 by Cisco Systems, Inc. (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: Compiled Wed 30-May-12 13:41 by prod_rel_team (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: (QG-BLUECP-
Feb 28 04:11:28.770: %PLATFORM-1-CRASHED: ESR: 0x00000000 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: SRR0: 0x01B97A9C SRR1: 0x00029230 SRR2: 0x01B80DAC SRR3: 0x00029230 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: BESRA: 0x00000000, BESRB: 0x00000000, BEAR: 0x00000000 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: CPU Register Context: (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Vector = 0x00000200 PC = 0x01B80DAC MSR = 0x00029230 CR = 0x30009005 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: LR = 0x01B80C90
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R0 = 0x00000000 R1 = 0x06341CC8 R2 = 0x00000000 R3 = 0x00000001 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R4 = 0x045A18E8 R5 = 0x00008002 R6 = 0x00000000 R7 = 0x0F000000 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0x01B7AB64 R10 = 0x00000000 R11 = 0x00178298 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R12 = 0x30000003 R13 = 0x0D98279C R14 = 0x00000000 R15 = 0x00000000 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R16 = 0x01E8078C R17 = 0x03C00000 R18 = 0x00000000 R19 = 0x00000000 (QG-BLUECP-3750X-01-1)
Feb 28 04:11
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R24 = 0x045A1BB0 R25 = 0x045A1DCC R26 = 0x00000350 R27 = 0x045A18E8 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: R28 = 0x00008002 R29 = 0x00000001 R30 = 0x045A18E8 R31 = 0x0ED0FAA8 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Stack trace: (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: PC = 0x01B80DAC, SP = 0x06341CC8 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 00: SP = 0x06341CE0 PC = 0x045A18E8 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 01: SP =
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 03: SP = 0x06341DF0 PC = 0x01E80790 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 04: SP = 0x06341E10 PC = 0x01E80E5C (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 05: SP = 0x06341E58 PC = 0x01E7DB4C (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 06: SP = 0x06341EC0 PC = 0x022DF73C (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 07: SP = 0x06341F20 PC = 0x022E00E8 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 08: SP = 0x06341F40 PC = 0x022E0218 (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: Frame 10: SP = 0x00000000 PC = 0x028478FC (QG-BLUECP-3750X-01-1)
Feb 28 04:11:28.778: %PLATFORM-1-CRASHED: (QG-BLUECP-3750X-01-1)
Mar 3 08:38:41.986: %SYS-5-PRIV_AUTH_FAIL: Authentication to privilege level 15 failed by admin on vty0 (10.191.6.1)
The following files were created in flash...
QG-BLUECP-3750X-01-1#dir
Directory of flash:/crashinfo_ext/
475 -rwx 329296 Feb 27 2014 23:41:04 +00:00 crashinfo_ext_1
476 -rwx 260421 Feb 28 2014 03:07:18 +00:00 crashinfo_ext_2
477 -rwx 320129 Feb 28 2014 03:12:33 +00:00 crashinfo_ext_3
478 -rwx 290248 Feb 28 2014 03:16:54 +00:00 crashinfo_ext_4
479 -rwx 284496 Feb 28 2014 03:21:18 +00:00 crashinfo_ext_5
480 -rwx 65 Feb 28 2014 03:25:34 +00:00 crashinfo_ext_6
481 -rwx 276767 Feb 28 2014 03:29:52 +00:00 crashinfo_ext_7
482 -rwx 271592 Feb 28 2014 03:34:18 +00:00 crashinfo_ext_8
483 -rwx 65 Feb 28 2014 03:39:26 +00:00 crashinfo_ext_9
484 -rwx 65 Feb 28 2014 03:43:49 +00:00 crashinfo_ext_10
485 -rwx 65 Feb 28 2014 03:48:14 +00:00 crashinfo_ext_11
486 -rwx 65 Feb 28 2014 03:52:33 +00:00 crashinfo_ext_12
487 -rwx 65 Feb 28 2014 03:56:47 +00:00 crashinfo_ext_13
488 -rwx 65 Feb 28 2014 04:01:29 +00:00 crashinfo_ext_14
489 -rwx 65 Feb 28 2014 04:06:30 +00:00 crashinfo_ext_15
57671680 bytes total (13599744 bytes free)
QG-BLUECP-3750X-01-1#cd ..
QG-BLUECP-3750X-01-1#cd crashinfo
QG-BLUECP-3750X-01-1#dir
Directory of flash:/crashinfo/
491 -rwx 1761 Mar 1 1993 00:00:08 +00:00 crashinfo_1
492 -rwx 1446 Mar 1 1993 00:00:08 +00:00 crashinfo_2
493 -rwx 1761 Mar 1 1993 00:00:08 +00:00 crashinfo_3
494 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_4
495 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_5
496 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_6
497 -rwx 1761 Mar 1 1993 00:00:08 +00:00 crashinfo_7
498 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_8
499 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_9
500 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_10
501 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_11
502 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_12
503 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_13
504 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_14
505 -rwx 1671 Mar 1 1993 00:00:08 +00:00 crashinfo_15
57671680 bytes total (13599744 bytes free)
The second switch took over as the stack master as expected...
QG-BLUECP-3750X-01#sh switch
Switch/Stack Mac Address : 4403.a727.4980
H/W Current
Switch# Role Mac Address Priority Version State
----------------------------------------------------------
1 Member 6c41.6a05.c780 15 1 Ready
*2 Master 4403.a727.4980 1 1 Ready
I have attached the crashdump files from flash1 that I listed above.
Is it possible for anyone to decode why the switch crashed both initially and then repeatedly when trying to come back up.
btw, It has been up for three days now since these events, but I would like to get to the root cause.
Many thanks
Steve
