Re: C3750 crash - every few days

g.leonard · ‎02-16-2011

Hi

2nd switch in 2 switch stack keeps crashing every few days or so. Please see lastest crash file below:

Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Fri 28-Jul-06 08:46 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x0056B958 SRR1 = 0x00029210 SRR2 = 0x0056A1E8 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000 PC = 0x00906338 MSR = 0x00029210 CR = 0x30000005
LR = 0x00906260 CTR = 0x00000000 XER = 0x80000047
R0 = 0x00000000 R1 = 0x026EB788 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x026EB760 R7 = 0x00000000
R8 = 0x00029210 R9 = 0x01660000 R10 = 0x0197F080 R11 = 0x00000000
R12 = 0xA0000000 R13 = 0x00110000 R14 = 0x00578E94 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x026E9F60
R24 = 0x00000000 R25 = 0x00000001 R26 = 0x004DD7D8 R27 = 0x00000000
R28 = 0x02635FB0 R29 = 0x00000030 R30 = 0x00000000 R31 = 0x0000000F

Stack trace:
PC = 0x00906338, SP = 0x026EB788
Frame 00: SP = 0x026EB798    PC = 0x00906238
Frame 01: SP = 0x026EB7A0    PC = 0x005747F4
Frame 02: SP = 0x026EB7D8    PC = 0x00578C98
Frame 03: SP = 0x026EB7F8    PC = 0x00578F48
Frame 04: SP = 0x026EB800    PC = 0x00908064
Frame 05: SP = 0x00000000    PC = 0x008FE62C

Can anybody tell me what is causing this?

blacktrack · ‎02-16-2011

It seem to be an hardware failure.

Can you post show tech-support ?

regards

Hicham Azarou

g.leonard · ‎02-16-2011

Hi Hicham

Thanks for the response. What makes you think it is hardware?

Please find sh tech attached as requested

Regards

Gary

johnlloyd_13 · ‎02-16-2011

hi gary,

as per your show version output, you would need to upgrade your IOS on both your 3750s.

blacktrack · ‎02-16-2011

Try to check stack cable.

then

Uprgade version

Cisco bug ID CSCsa72400

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

As a workaround, do not connect 802.3af non-standard class PDs, or even bad or loopback cables, to the switch because the switch can detect the class incorrectly.

In order to resolve this issue, upgrade to Cisco IOS Software Release 12.2(25)SEA on the switch. Alternatively, upgrade to the latest maintenance release, which can be downloaded from Cisco Downloads.

Refer to the Debug Exception (Could be NULL pointer dereference) section of Catalyst 3750 Series Switches Troubleshoot Common Issues for more information.

blacktrack · ‎02-16-2011

if you need image give your email.

g.leonard · ‎02-17-2011

Thanks Hicham

As you can see from the files I have posted the switch is already on 12.2(25)SEE2. I will check the stack cable as there are no PoE devices attached to the switch.

G

Leo Laohoo · ‎02-16-2011

If it keeps crashing every few days then there should be traces left. Check the flash directory of the switch and look for "crashinfo" files/subdirectories. Post the last 3 files.

g.leonard · ‎02-17-2011

Hi

See my first post for latest crash file. Please see below for previous two:

Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Fri 28-Jul-06 08:46 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x0056B958 SRR1 = 0x00029210 SRR2 = 0x0056A1E8 SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000 PC = 0x00906338 MSR = 0x00029210 CR = 0x30000005
LR = 0x00906260 CTR = 0x00000000 XER = 0x80000047
R0 = 0x00000000 R1 = 0x02A445D0 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x02A445A8 R7 = 0x00000000
R8 = 0x00029210 R9 = 0x01660000 R10 = 0x0197F080 R11 = 0x00000000
R12 = 0xA0000000 R13 = 0x00110000 R14 = 0x00578E94 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000
R24 = 0x00000001 R25 = 0x00000001 R26 = 0x0053E27C R27 = 0x00000000
R28 = 0x02635F38 R29 = 0x000008C8 R30 = 0x00000000 R31 = 0x0000000F

Stack trace:
PC = 0x00906338, SP = 0x02A445D0
Frame 00: SP = 0x02A445E0    PC = 0x00906238
Frame 01: SP = 0x02A445E8    PC = 0x005747F4
Frame 02: SP = 0x02A44620    PC = 0x00578C98
Frame 03: SP = 0x02A44640    PC = 0x00578F48
Frame 04: SP = 0x02A44648    PC = 0x00908064
Frame 05: SP = 0x00000000    PC = 0x008FE62C

Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Fri 28-Jul-06 08:46 by yenanh

Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

SRR0 = 0x0056B958 SRR1 = 0x00029210 SRR2 = 0x0056A1EC SRR3 = 0x00021000
ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000

CPU Register Context:
Vector = 0x00002000 PC = 0x00906338 MSR = 0x00029210 CR = 0x30000005
LR = 0x00906260 CTR = 0x00000000 XER = 0x80000047
R0 = 0x00000000 R1 = 0x026EB878 R2 = 0x00000000 R3 = 0x00000000
R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x026EB850 R7 = 0x00000000
R8 = 0x00029210 R9 = 0x01660000 R10 = 0x0197F080 R11 = 0x00000000
R12 = 0xA0000000 R13 = 0x00110000 R14 = 0x00578E94 R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x026EA050
R24 = 0x00000000 R25 = 0x00000001 R26 = 0x004DD7D8 R27 = 0x00000000
R28 = 0x026365C8 R29 = 0x00000030 R30 = 0x00000000 R31 = 0x0000000F

Stack trace:
PC = 0x00906338, SP = 0x026EB878
Frame 00: SP = 0x026EB888    PC = 0x00906238
Frame 01: SP = 0x026EB890    PC = 0x005747F4
Frame 02: SP = 0x026EB8C8    PC = 0x00578C98
Frame 03: SP = 0x026EB8E8    PC = 0x00578F48
Frame 04: SP = 0x026EB8F0    PC = 0x00908064
Frame 05: SP = 0x00000000    PC = 0x008FE62C

Many thanks

Gary

Leo Laohoo · ‎02-17-2011

Vector = 0x00002000 PC = 0x00906338 MSR = 0x00029210 CR = 0x30000005

Hmmmm ... Vector 2K.

Can I request if you reboot this switch and post the entire bootup process? Please don't forget to add the "sh log" too?

g.leonard · ‎02-18-2011

Hey

It's difficult to reboot this switch as it has the majority of our call centre attached to it. Funnily enough it has just crashed again causing them problems! The log buffer on this stack member is only 4096Bytes as we log to a Syslog server.

Feb 18 2009 09:56:21 SYS 2 MALLOCFAIL Memory allocation of 36688 bytes failed from 0x702348, alignment 8 (EGH-Floor2-1-2) *
Feb 18 2009 09:57:02 STACKMGR 4 STACK_LINK_CHANGE Stack Port 1 Switch 1 has changed to state DOWN *
Feb 18 2009 09:57:02 STACKMGR 4 STACK_LINK_CHANGE Stack Port 2 Switch 1 has changed to state DOWN *
Feb 18 2009 09:57:02 STACKMGR 4 SWITCH_REMOVED Switch 2 has been REMOVED from the stack *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED (EGH-Floor2-1-2) *
Feb 18 2009 09:57:55 STACKMGR 4 STACK_LINK_CHANGE Stack Port 1 Switch 1 has changed to state UP *
Feb 18 2009 09:57:55 STACKMGR 4 STACK_LINK_CHANGE Stack Port 2 Switch 1 has changed to state UP *
Feb 18 2009 10:00:12 STACKMGR 4 STACK_LINK_CHANGE Stack Port 1 Switch 2 has changed to state UP *
Feb 18 2009 10:00:12 STACKMGR 4 STACK_LINK_CHANGE Stack Port 2 Switch 2 has changed to state UP *
Feb 18 2009 10:00:12 STACKMGR 4 SWITCH_ADDED Switch 1 has been ADDED to the stack (Switch-2) *
Feb 18 2009 10:00:12 STACKMGR 4 SWITCH_ADDED Switch 2 has been ADDED to the stack (Switch-2) *
Feb 18 2009 10:00:12 SPANTREE 5 EXTENDED_SYSID Extended SysId enabled for type vlan (Switch-2) *
Feb 18 2009 10:00:12 STACKMGR 5 MASTER_READY Master Switch 1 is READY (Switch-2) *
Feb 18 2009 10:00:12 ENT_API 4 NOPORT Physical entity does not have a Port PhysicalClass when *

Feb 18 2009 10:01:03 PLATFORM 1 CRASHED System previously crashed with the following message: (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Cisco IOS Software, C3750 Software (C3750-IPBASE-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1) (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Copyright (c) 1986-2006 by Cisco Systems, Inc. (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Compiled Fri 28-Jul-06 08:46 by yenanh (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Debug Exception (Could be NULL pointer dereference) Exception (0x2000)! (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED SRR0 = 0x0056B958 SRR1 = 0x00029210 SRR2 = 0x0056A1EC SRR3 = 0x00021000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x10000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED CPU Register Context: (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Vector = 0x00002000 PC = 0x00906338 MSR = 0x00029210 CR = 0x30000005 (EGH-Floor2-1-2)

Feb 18 2009 10:01:03 PLATFORM 1 CRASHED LR = 0x00906260 CTR = 0x00000000 XER = 0x80000047 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R0 = 0x00000000 R1 = 0x026EE158 R2 = 0x00000000 R3 = 0x00000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R4 = 0xFFFFFFFE R5 = 0x00000000 R6 = 0x026EE130 R7 = 0x00000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R8 = 0x00029210 R9 = 0x01660000 R10 = 0x0197F080 R11 = 0x00000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R12 = 0xA0000000 R13 = 0x00110000 R14 = 0x00578E94 R15 = 0x00000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x026EC930 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R24 = 0x00000000 R25 = 0x00000001 R26 = 0x004DD7D8 R27 = 0x00000000 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED R28 = 0x02638638 R29 = 0x0000002C R30 = 0x00000000 R31 = 0x0000000F (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Stack trace: (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED PC = 0x00906338, SP = 0x026EE158 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Frame 00:00 SP = 0x026EE168 PC = 0x00906238 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Frame 01:00 SP = 0x026EE170 PC = 0x005747F4 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Frame 02:00 SP = 0x026EE1A8 PC = 0x00578C98 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Frame 03:00 SP = 0x026EE1C8 PC = 0x00578F48 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Frame 04:00 SP = 0x026EE1D0 PC = 0x00908064 (EGH-Floor2-1-2) *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED Frame 05:00 SP = 0x00000000 PC = 0x008FE62C (EGH-Floor2-1-2)

Is the Vector 2k message significant?

Leo Laohoo · ‎02-18-2011

A word of warning, you are upgrading from 12.2(25). If you upgrade the IOS to 12.2(50) and later, the switch in the stack will take double the time (one time only) for the bootup because the ROMmon/Bootstrap will also be upgraded. Once the ROMmon/Bootstrap gets upgraded succeeding reload/reboot time will be normal.

kapathak · ‎02-17-2011

Hello Gary,

The crash decodes are very generic. This usually happens when a switch crashes due to memory leaks/fragmentations etc. The switch crash is just a side-effect of the actual problem at hand. Memory issues are usually caused due to software bugs. It seems that your second switch is running 12.2(25)SEE2 which to be honest, is very old and probably deferred. The resolution to such problems is to upgrade to newer IOS releases since they incorporate fixes for such issues. It also seems that Switch 1 is running 12.2(25)SEE4, maybe the problem that we are seeing with switch 2 (12.2(25)SEE2) has been fixed in 12.2(25)SEE4 and thats probably why the Switch1 is unaffected.

I would not really consider this a hardware failure. Most likely, an simple IOS upgrade to a newer release should fix this. If that does not resolve the problem, please contact Cisco TAC :-)

Leo Laohoo · ‎02-18-2011

Hmmm ... Sorry Kapil, but I will have to disagree on the theory of an IOS bug. I have yet to see a bug that can take down a switch within a stack regularly.

I am suspecting something is wrong with this switch. That's why I'm requesting for a reboot of the switch and post the complete bootup process to see if something has gone amiss.

Besides, the OP did mention that he can't reboot the switch due to the importance of the clients so what are the chances of the user upgrading the IOS?

The next and only choice is for the problematic switch to be replaced.

kapathak · ‎02-18-2011

Feb 18 2009 09:56:21 SYS 2 MALLOCFAIL Memory allocation of 36688 bytes failed from 0x702348, alignment 8 (EGH-Floor2-1-2) *
Feb 18 2009 09:57:02 STACKMGR 4 STACK_LINK_CHANGE Stack Port 1 Switch 1 has changed to state DOWN *
Feb 18 2009 09:57:02 STACKMGR 4 STACK_LINK_CHANGE Stack Port 2 Switch 1 has changed to state DOWN *
Feb 18 2009 09:57:02 STACKMGR 4 SWITCH_REMOVED Switch 2 has been REMOVED from the stack *
Feb 18 2009 10:01:03 PLATFORM 1 CRASHED (EGH-Floor2-1-2) *

MALLOC failures will end crashing the device. As I suspected earlier, the memory allocation problem needs to be fixed. This is causing the crash with regularity.