Cisco IOS System CRASHED

fredgraham19 · ‎01-02-2007

My Catalyst 3750 Gig 24 port switch incurred an error of %PLATFORM-1-CRASHED:Program Exception (0x0700)! I was wondering what this messege means. My attached devices are up and running and the switch is now normal. Could this be a prelude of a future hardware problem?

Richard Burts · ‎01-02-2007

Fred

It seems quite possible that this is a prelude of some future problems. But at this point I do not know that we can say whether they are more likely a software issue or a hardware issue. Program Exception (0x0700) does not tell me much about which direction to look. Were there other messages when the problem occurred? Can you post the output of show version (that might have some helpful information about the problem)?

HTH

Rick

HTH

Rick

fredgraham19 · ‎01-02-2007

Hi Rick, Thanks. Here is the output:

System was restarted by error - Program Exception (0x0700)!, PC 0x32A6F0

C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(25)SEC2, RELEASE SOFTWARE (f

c1)

Compiled Wed 31-Aug-05 10:26 by antonino (current version)

Image text-base: 0x00003000, data-base: 0x010207E0

Stack trace from system failure:

FP: 0x261FDA0, RA: 0x32A6F0

FP: 0x261FDB8, RA: 0x3E05F4

FP: 0x261FDD0, RA: 0x3D72C8

FP: 0x261FDF0, RA: 0x541DE8

FP: 0x1BDCD4C, RA: 0x4D6E90

FP: 0x1BDCD5C, RA: 0x4D6E38

FP: 0x1BDCD7C, RA: 0x4D919C

FP: 0x1BDCD84, RA: 0x4D9660

***************************************************

******* Information of Last System Crash **********

***************************************************

Using flash:/crashinfo/crashinfo_1.

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(25)SEC2,

RELEASE SOFTWARE (fc1)

Compiled Wed 31-Aug-05 10:26 by antonino

Program Exception (0x0700)!

SRR0 = 0x0032A6F0 SRR1 = 0x00029210 SRR2 = 0x004D7A0C SRR3 = 0x00021200

ESR = 0x08000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x00000000

Thanks

amit-singh · ‎01-02-2007

Hi Fred,

I have tried seraching some bug or other software issues, that can be related to our problem. I havenot come across any bug as such. This could be some hardware or software related issues. I would suggest to upgrade the switch to the latest IOS in case of eliminating any possible software issues. If this problem happens again, please collect the output from " show stacks" crash info file and contact TAC with the outputs.

HTH,

-amit singh

Richard Burts · ‎01-02-2007

Fred

I ran what you posted through the output interpreter. Since I did not have the complete show version it did not interpret the info in the show version. But it did interpret the stack trace and it indicates that it is a software bug. So likely not hardware and the advice to update software is probably good advice.

Here is the output from the output interpreter:

The failure was caused by a software defect.

The stack trace decoded symbols are:

sasq_asic_rx_detect_and_fix_rxportfifo_lockup

asic_rx_detect_and_fix_rxportfifo_lockup

hulc_stack_corrupt_header_workaround

hulc_net_interrupt

ISR_wrapper

hulc_pit_isr

hulc_intr_dispatch

Possible bug matches are listed below. Bugs with a score of .90 or more

are the most likely candidates:

Score Bugid Status Fixed In Duplicate Title

0.85 CSCed03214 V 12.2(20)SE 12.1(14)AX1 12.1(19)EA1d None SUPQ-4-CPUHB_RECV_STARVE: still seeing receive Q stuck message

HTH

Rick

HTH

Rick

fredgraham19 · ‎01-02-2007

Hi Rick, Just so we on the right page, here is the full output on sho version:

Minimum process stacks:

Free/Size Name

5048/6000 hulc_flash init

5768/6000 CDP BLOB

8536/9000 IP Background

3456/6000 SaveCrashBuffer

5464/6000 MDFS LC IPC Init Process

10400/12000 master cfg mgr init process

5728/6000 vqpc_shim_create_addr_tbl

5712/6000 SPAN Subsystem

5712/6000 bulk config process

5584/6000 hpm bulk_vlan_state process

8744/12000 Init

8760/9000 cdp init process

5720/6000 MDFS Reload

5560/6000 RADIUS INITCONFIG

5704/6000 Hulc Backup Interface Process

4464/6000 HULC CrashBuffer

2600/3000 Rom Random Update Process

35208/36000 TCP Command

7744/12000 Virtual Exec

17096/24000 hulc_techsupport_process

Interrupt level stacks:

Level Called Unused/Size Name

4 1439173378 8472/9000 NETWORK INTERFACE CHIP

5 0 9000/9000 SUPERVISOR EXCEPTIONS

6 6186 8936/9000 NS16550 VECTOR

System was restarted by error - Program Exception (0x0700)!, PC 0x32A6F0

C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(25)SEC2, RELEASE SOFTWARE (f

c1)

Compiled Wed 31-Aug-05 10:26 by antonino (current version)

Image text-base: 0x00003000, data-base: 0x010207E0

Stack trace from system failure:

FP: 0x261FDA0, RA: 0x32A6F0

FP: 0x261FDB8, RA: 0x3E05F4

FP: 0x261FDD0, RA: 0x3D72C8

FP: 0x261FDF0, RA: 0x541DE8

FP: 0x1BDCD4C, RA: 0x4D6E90

FP: 0x1BDCD5C, RA: 0x4D6E38

FP: 0x1BDCD7C, RA: 0x4D919C

FP: 0x1BDCD84, RA: 0x4D9660

***************************************************

******* Information of Last System Crash **********

***************************************************

Using flash:/crashinfo/crashinfo_1.

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(25)SEC2,

RELEASE SOFTWARE (fc1)

Compiled Wed 31-Aug-05 10:26 by antonino

Program Exception (0x0700)!

SRR0 = 0x0032A6F0 SRR1 = 0x00029210 SRR2 = 0x004D7A0C SRR3 = 0x00021200

ESR = 0x08000000 DEAR = 0x00000000 TSR = 0x84000000 DBSR = 0x00000000

CPU Register Context:

Vector = 0x00000700 PC = 0x0032A6F0 MSR = 0x00029210 CR = 0x22444028

LR = 0x003E05F4 CTR = 0x0032A5E4 XER = 0x40000044

R0 = 0x003E05F4 R1 = 0x0261FDA0 R2 = 0x00000000 R3 = 0x00000000

R4 = 0x0261FDF8 R5 = 0x01770000 R6 = 0x00029210 R7 = 0x00000000

R8 = 0x40000000 R9 = 0x01B6D384 R10 = 0x00000060 R11 = 0x00000000

R12 = 0x00F9CAA5 R13 = 0x00010000 R14 = 0x01FD4408 R15 = 0x01600000

R16 = 0x01370000 R17 = 0x01600000 R18 = 0x00000000 R19 = 0x00000000

R20 = 0x00000000 R21 = 0x01CD0000 R22 = 0x01AD0000 R23 = 0x01AD0000

R24 = 0x00000000 R25 = 0x01CD0000 R26 = 0x00000000 R27 = 0x00000000

R28 = 0x00000060 R29 = 0x01AD0000 R30 = 0x00000000 R31 = 0x00000000

Stack trace:

PC = 0x0032A6F0, SP = 0x0261FDA0

Frame 00: SP = 0x0261FDB8 PC = 0x003E05F4

Frame 01: SP = 0x0261FDD0 PC = 0x003D72C8

Frame 02: SP = 0x0261FDF0 PC = 0x00541DE8

Frame 03: SP = 0x01BDCD4C PC = 0x004D6E90

Frame 04: SP = 0x01BDCD5C PC = 0x004D6E38

Frame 05: SP = 0x01BDCD7C PC = 0x004D919C

Frame 06: SP = 0x01BDCD84 PC = 0x004D9660

Frame 07: SP = 0x01BDCDD0 PC = 0x004D6F40

Frame 08: SP = 0x01FD4400 PC = 0x0079D1AC

Frame 09: SP = 0x01FD4580 PC = 0x0079D1D0

Frame 10: SP = 0xFFFFFFFF PC = 0x00797EA0

Once again, Thanks. So far I have some GREAT information to tackle possible issues such as this.

Richard Burts · ‎01-02-2007

Fred

This is still not the complete show version. The complete show version will have information identifying the plarform, the software running, resources on the platform, etc.

I ran the information that you posted through the output interpreter (more info than the first time) and it came up with the same results, pointing at a software failure rather than a hardware failure and identifying a potential bug match (the same one as before).

I feel pretty confident that you are dealing with a software bug. If you continue to run the same software it may or may not crash again. I would suggest an upgrade of the software. If you want to do something other than upgrade the software I would suggest gathering the information that you have and opening a case with Cisco TAC.

HTH

Rick

HTH

Rick

fredgraham19 · ‎01-02-2007

Hi Rick, I think I will go with upgrading the IOS to the switch. It makes more sense, since I has not been updated in awhile. Thanks, for your assistance. Have a GREAT DAY!