Cisco 3750 Switch crushed

fgasimzade · ‎11-17-2014

Hello,

Recently one of our 3750 switches crushed 3 times during 2 hours with the following syslog messages:

04:21 PM (switch-name) Alert 98: 00:01:39:

04:21 PM (switch-name) Alert 97: 00:01:39: Frame 05: SP = 0x00000000 PC = 0x0039A5E4

04:21 PM (switch-name) Alert 96: 00:01:39: Frame 04: SP = 0x03C817A8 PC = 0x003A2FF8

04:21 PM (switch-name) Alert 95: 00:01:39: Frame 03: SP = 0x03C817A0 PC = 0x01616314

04:21 PM (switch-name) Alert 94: 00:01:39: Frame 02: SP = 0x03C81780 PC = 0x00212684

04:21 PM (switch-name) Alert 93: 00:01:39: Frame 01: SP = 0x03C81758 PC = 0x01613250

04:21 PM (switch-name) Alert 92: 00:01:39: Frame 00: SP = 0x03C81728 PC = 0x0161E9D0

04:21 PM (switch-name) Alert 89: 00:01:39:

04:21 PM (switch-name) Alert 90: 00:01:39: Stack trace:

04:21 PM (switch-name) Alert 87: 00:01:39: R24 = 0x00000000 R25 = 0x00000000 R26 = 0x00000000 R27 = 0x00000000

04:21 PM (switch-name) Alert 91: 00:01:39: PC = 0x0161E9D4, SP = 0x03C81718

04:21 PM (switch-name) Alert 88: 00:01:39: R28 = 0x03A73F0C R29 = 0x03C74258 R30 = 0x00000000 R31 = 0x03C74258

04:21 PM (switch-name) Alert 84: 00:01:39: R12 = 0x000433A9 R13 = 0x00110000 R14 = 0x01615FF4 R15 = 0x00000000

04:21 PM (switch-name) Alert 85: 00:01:39: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000

04:21 PM (switch-name) Alert 86: 00:01:39: R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x00000000

04:21 PM (switch-name) Alert 83: 00:01:39: R8 = 0x00000000 R9 = 0x03C81705 R10 = 0x03A4DDE6 R11 = 0x03A4DDE5

04:21 PM (switch-name) Alert 82: 00:01:39: R4 = 0x03C81706 R5 = 0x00000000 R6 = 0x025D632C R7 = 0xBEEFCAFE

04:21 PM (switch-name) Alert 79: 00:01:39: Vector = 0x00002000 PC = 0x0161E9D4 MSR = 0x00029210 CR = 0x33000005

04:21 PM (switch-name) Alert 81: 00:01:39: R0 = 0x0161E9D0 R1 = 0x03C81718 R2 = 0x00000000 R3 = 0x03C74258

04:21 PM (switch-name) Alert 80: 00:01:39: LR = 0x0161E9D0 CTR = 0x00000000 XER = 0xE0000000

04:21 PM (switch-name) Alert 75: 00:01:39: SRR0 = 0x00393704 SRR1 = 0x00029210 SRR2 = 0x004CF998 SRR3 = 0x00021000

04:21 PM (switch-name) Alert 78: 00:01:39: CPU Register Context:

04:21 PM (switch-name) Alert 77: 00:01:39:

04:21 PM (switch-name) Alert 76: 00:01:39: ESR = 0x00000000 DEAR = 0x00000000 TSR = 0x8C000000 DBSR = 0x01000000

04:21 PM (switch-name) Alert 74: 00:01:39:

04:21 PM (switch-name) Alert 73: 00:01:39: Debug Exception (Could be NULL pointer dereference) Exception (0x2000)!

04:21 PM (switch-name) Alert 72: 00:01:39:

04:21 PM (switch-name) Alert 71: 00:01:39: Compiled Tue 06-Feb-07 05:20 by antonino

04:21 PM (switch-name) Alert 69: 00:01:39: Cisco IOS Software, C3750ME Software (C3750ME-I5K91-M), Version 12.2(35)SE2, RELEASE SOFTWARE (fc1)

04:21 PM (switch-name) Alert 68: 00:01:39: System previously crashed with the following message:

Also, in show version I have this:

System returned to ROM by address error at PC 0x161E9D4, address 0x0

Switch   Ports Model              SW Version              SW Image
------   ----- -----              ----------              ----------
*    1   28     ME-C3750-24TE      12.2(35)SE2             C3750ME-I5K91-M

After 3rd crush the switch is working fine for several hours, however I dont get syslogs anymore

Any clue why it could happen?

Thank you in advance

Joseph W. Doherty · ‎12-02-2014

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Your question is the kind more suitable for TAC, but if you have maintenance, you might consider upgrading your IOS version and see if the problem goes away.

(NB: Unsure it's as applicable to the ME variant, but on "regular" 3750s, 55[SE9] is very solid.)

fgasimzade · ‎12-02-2014

Dear All,

We have already upgraded the IOS, it helped

Thank you for your comments

InayathUlla Sharieff · ‎12-02-2014

I can try helping you if you could provide me show tech output from the box?