A few weeks ago I was called to help with a switch crash. There was a running stack of 4 3750 switches with ios c3750-ipbasek9-mz.122-35.SE2.bin. It crashed when adding a new 3750 to the stack. The stack keeps crashing until the new switch was disconnected. The members in the stack keep rebooting as long as the new switch was connected to the stack.
When the stack was up and running again (I only needed to disconnect the new switch) I downgraded the new 3750 to the same ios. Because the crashlogs where pointing me to a bad stack cable I used another one to connect the new switch which then went all well.
But before I added the new switch I tested the cable between 2 new 3750's. I first booted one of the new switches (not connected to the stack), added some config and then connected a second new 3750 to it with the same stack cable that first seems to cause some trouble. It al went well.
Then I switched it all off. I cleared the configs on the new switches. I downgraded one of them to c3750-ipbasek9-mz.122-35.SE2.bin. I booted that one, gave it some config and then added the second one. It then came up with a message that the IOS versions where differend and suggested a way to correct it. This is what they expected the first time when adding a new switch to a running stack. But it did not do that, it chooses to crash
The crashdumps pointed me to a bad stack cable but when I tested it seperately between two switches it worked fine. What could have caused this? Could it be the stack cable or was it the difference in IOS what seems to be not a problem when I tested it with the 2 new switches?
This is the information from the crash dumps:
Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(35)SE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Tue 06-Feb-07 05:07 by antonino
Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!
SRR0 = 0x0099B0B4 SRR1 = 0x00029210 SRR2 = 0x00647984 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x01000000
CPU Register Context:
Vector = 0x00002000 PC = 0x00564524 MSR = 0x00029210 CR = 0x33555039
LR = 0x005644F4 CTR = 0x005644A8 XER = 0xC0000033
R0 = 0x00000000 R1 = 0x0237B8A0 R2 = 0x00000000 R3 = 0x02AC9764
R4 = 0x00000000 R5 = 0x0237B8A8 R6 = 0x00000002 R7 = 0x00000003
R8 = 0x00000001 R9 = 0x00000000 R10 = 0x01C841DC R11 = 0x01BF0000
R12 = 0x6E4D9459 R13 = 0x00110000 R14 = 0x00000000 R15 = 0x00000000
R16 = 0x01DF47B4 R17 = 0x01DF477C R18 = 0x01E60000 R19 = 0x01BF0000
R20 = 0x01BF0000 R21 = 0x00000000 R22 = 0x01DF0000 R23 = 0x01BF0000
R24 = 0x0004073F R25 = 0x00000001 R26 = 0xD0000000 R27 = 0x00000005
R28 = 0x02AC9764 R29 = 0x01810000 R30 = 0x02AC9764 R31 = 0x02AC9764
Stack trace:
PC = 0x00564524, SP = 0x0237B8A0
Frame 00: SP = 0x0237B8C0 PC = 0x005644DC
Frame 01: SP = 0x0237B8D8 PC = 0x00490160
Frame 02: SP = 0x0237B900 PC = 0x006CC79C
Frame 03: SP = 0x0237B930 PC = 0x006CFF84
Frame 04: SP = 0x0237B950 PC = 0x006D0364
Frame 05: SP = 0x01D0DEAC PC = 0x00646E08
Frame 06: SP = 0x01D0DEBC PC = 0x00646DB0
Frame 07: SP = 0x01D0DEDC PC = 0x006491FC
Frame 08: SP = 0x01D0DEE4 PC = 0x00649734
Frame 09: SP = 0x01D0DF30 PC = 0x00646EB8
Frame 10: SP = 0x01E8A8F0 PC = 0x0099B0B4
Frame 11: SP = 0x01E8A950 PC = 0x0099B08C
Frame 12: SP = 0xFFFFFFFF PC = 0x00998138
