Solved: Cisco 3750X Platform Crash

mohankumarm · ‎05-05-2013

Hello,

We have three Cisco 3750X-24T-S 12.2(58) SE2 switches in a Stackwise Plus configuration with priorities hardcoded as follows:

Master 15

Member 2 14

Member 3 13

Due to hardware issues suspected with Member 2 switch, this switch was replaced with a new switch. After a couple of days, we observed that Member 2 has promoted itself to Master and the following logs was displayed:

2 12:13:24.475: %PLATFORM-1-CRASHED: Vector = 0x00001100 PC = 0x00458F6C MSR = 0x00029230 CR = 0x53000099 (SW1) May 2 12:13:24.475: %PLATFORM-1-CRASHED: LR = 0x00436B78 CTR = 0x026CC0CC XER = 0xE0000012 (SW1) May 2 12:13:24.475: %PLAT May 2 12:13:24.475: %PLATFORM-1-CRASHED: R4 = 0x0750ACA9 R5 = 0x0000000F

R6 = 0x04660B90 R7 = 0x0726D7A0 (SW1) May 2 12:13:24.484: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0xC93C0952

R10 = 0x00000001 R11 = 0x03E4C170 (SW1) May 2 12:13:24.484: %PLATFORM-1-CRASHED: R12 = 0x8E63FF8A R13 = 0x0B98279C R14 = 0x02006834 R15 = 0x00000000 (SW1) May 2 12:13:24.484: %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 =

0x00000000 R18 = 0x00000000 R19 = 0x00000000 (SW1) May 2 12:13:24.484: %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 =

0x00000000 R22 = 0x00000000 R23 = 0x03917FB4 (SW1) May 2 12

The show switch shows the following:

Switch/Stack Mac Address : 442b.0363.e880

H/W Current

Switch# Role Mac Address Priority Version State

----------------------------------------------------------

1 Member 6c20.56b8.0f80 15 1 Ready ----------Previous Master, currently Member

*2 Master 442b.0363.e880 14 1 Ready -------- Previous Member, currently Master

3 Member fc99.47c0.2880 13 1 Ready

Please advise..

Just wondering if there is a hardware issue with SW1 above and SW1 seems to be running ok for the last three days.

Thanks and Regards,

MOhan

InayathUlla Sharieff · ‎05-05-2013

Hi Mohan,

I have decoded the tracebacks and the decodes are pointing to the following bug id: CSCtr86594.

HTH

Regards

Inayath

*Please rate useful posts and remember to mark any solved questions as answered. Thank you.

View solution in original post

Leo Laohoo · ‎05-05-2013

3750X-24T-S 12.2(58) SE2

Unless you see an operational need to run this IOS, I suggest you downgrade to 12.2(55)SE7 and observe.

mohankumarm · ‎05-05-2013

Thanks very much leolaohoo..well, we have a number of switch stacks running 12.2(58) SE2 and there are no problems with it, this is happening in only one stack. So is there any workaround or is this a real bug with 12.2(58) SE2?

InayathUlla Sharieff · ‎05-05-2013

Hi Mohan,

Let me see what has crashed the cisco 3750 switch. Could you please share the following information:

1) show version

2) show log

3) crash info file( will be under show flash)

Regards

Inayath

mohankumarm · ‎05-05-2013

Hello,

Thanks very much for your response and here is the attached output from the switch.

Thanks and Regards,

Mohan

InayathUlla Sharieff · ‎05-05-2013

Hi Mohan,

The crash file is not captured?

Could you send me the crash info file( 459 drwx 512 May 2 2013 22:08:58 +10:00 crashinfo_ext) and complete output of show log.

Regards

Inayath

mohankumarm · ‎05-05-2013

Sorry the show log output is as follows:

2 12:13:24.475: %PLATFORM-1-CRASHED: Vector = 0x00001100 PC = 0x00458F6C MSR = 0x00029230 CR = 0x53000099 (Sw-1)

May 2 12:13:24.475: %PLATFORM-1-CRASHED: LR = 0x00436B78 CTR = 0x026CC0CC XER = 0xE0000012 (Sw-1)

May 2 12:13:24.475: %PLAT

May 2 12:13:24.475: %PLATFORM-1-CRASHED: R4 = 0x0750ACA9 R5 = 0x0000000F R6 = 0x04660B90 R7 = 0x0726D7A0 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: R8 = 0x00000000 R9 = 0xC93C0952 R10 = 0x00000001 R11 = 0x03E4C170 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: R12 = 0x8E63FF8A R13 = 0x0B98279C R14 = 0x02006834 R15 = 0x00000000 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: R20 = 0x00000000 R21 = 0x00000000 R22 = 0x00000000 R23 = 0x03917FB4 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-

May 2 12:13:24.484: %PLATFORM-1-CRASHED: R28 = 0x0000000F R29 = 0x0750ACA9 R30 = 0x07509EDC R31 = 0x0726D7A0 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Stack trace: (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: PC = 0x00458F6C, SP = 0x070B33E8 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 00: SP = 0x070B3418 PC = 0x03E20000 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 01: SP = 0x070B3468 PC = 0x00436B78 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 02: SP = 0x070B3480 PC = 0x01D3C8DC (Sw-1)

May 2 1

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 04: SP = 0x070B34A8 PC = 0x0048D1FC (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 05: SP = 0x070B3528 PC = 0x00483D88 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 06: SP = 0x070B3558 PC = 0x004836E0 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 07: SP = 0x070B35B0 PC = 0x00509E7C (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 08: SP = 0x070B35D0 PC = 0x01D38A34 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 09: SP = 0x070B35F0 PC = 0x01D38ED0 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 10:

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 11: SP = 0x070B3678 PC = 0x00483D88 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 12: SP = 0x070B36A8 PC = 0x004836E0 (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 13: SP = 0x070B3748 PC = 0x01D3A1DC (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 14: SP = 0x070B3750 PC = 0x0048D1FC (Sw-1)

May 2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 15: SP = 0x070B37D0 PC = 0x00483D88 (Sw-1)

I cant remember having seen the extended crash file info in flash, so had the "show stacks" output

GS-PCR-SFDS01# show flash

Directory of flash:/

460 -rwx 1216 May 1 2013 17:46:02 +10:00 vlan.dat

3 -rwx 32480 May 5 2013 18:48:16 +10:00 config.text

4 -rwx 5481 May 5 2013 18:48:16 +10:00 private-config.text

5 drwx 512 Mar 1 1993 11:09:48 +11:00 c3750e-universalk9-mz.122-58.SE2

461 -rwx 1943 Mar 1 1993 11:00:58 +11:00 config.text.backup

462 -rwx 1936 Mar 1 1993 11:00:59 +11:00 private-config.text.backup

463 -rwx 7192 May 5 2013 18:48:16 +10:00 multiple-fs

57671680 bytes total (36256256 bytes free)

GS-PCR-SFDS01#

GS-PCR-SFDS01#sh stacks

Minimum process stacks:

Free/Size Name

4880/6000 hulc_flash init

5608/6000 CDP BLOB

8524/9000 IP Background

8560/9000 IP Connected Route Background

8576/9000 EEM ED RF

4192/6000 EEM Shell Director

5596/6000 vqpc_shim_create_addr_tbl

5580/6000 SPAN Subsystem

19336/24000 Init

58800/60000 EEM Auto Registration Proc

5556/6000 SASL MAIN

5296/6000 LICENSE AGENT DEFAULT

8592/9000 cdp init process

5500/6000 IPC Zone Manager

5584/6000 RTTY Slave Registration Process

5184/6000 MDFS LC IPC Init Process

5600/6000 IP SLAs Deferred Schedule Processor

4724/6000 IPC Split Init

5180/6000 HUDLD fiber table process

5392/6000 RADIUS INITCONFIG

5468/6000 bulk config process

5600/6000 MDFS LC Download Ack Process

5548/6000 URPF stats

2484/3000 Rom Random Update Process

4812/6000 platform OBFL

9892/12000 OBFL VOLT obfl0

5208/6000 HVLAN Mapped Vlan Id Update Process

10700/12000 Router Init

6836/12000 cfg mgr new master process

5572/6000 HDF Stack-Mac

5592/6000 HCMP redirect process

5200/6000 XDR LC PRIO Critcal Back

5588/6000 CEF Post-Switchover Tidy

5516/6000 HCMP new mac process

5164/6000 XDR LC PRIO Background

11480/12000 CFC LC Test Stats

11572/12000 CFC stats broker process

5104/6000 CEF LC Stats

9524/12000 XDR LC Background

5032/6000 CEF LC IPC Background

5020/6000 RTTY Client Registration

5076/6000 IPC LC Port Opener

5088/6000 IP ACL RP Background

5496/6000 MDFS Reload

4716/6000 hpm bulk_vlan_state process

4452/6000 HL3U ARP Refresh Proc

4992/12000 SSH Process

18628/24000 hulc_techsupport_process

Interrupt level stacks:

Level Called Unused/Size Name

1 0 9000/9000 Maxwell MACsec ISR

4 3252150534 7400/9000 NETWORK INTERFACE CHIP

5 0 9000/9000 SUPERVISOR EXCEPTIONS

6 23544 8936/9000 NS16550 VECTOR

InayathUlla Sharieff · ‎05-05-2013

Hi mohan,

Runn the following command:

#more flash1:/crashinfo_ext/crashinfo_ext

#more flash1:/crashinfo_ext

and update the same on this.

Regards

Inayath

mohankumarm · ‎05-05-2013

Hi Inayath,

I can see the following file

57671680 bytes total (36256256 bytes free)

GS-PCR-SFDS01#more info

version_suffix: universalk9-122-58.SE2

version_directory: c3750e-universalk9-mz.122-58.SE2

image_system_type_id: 0x00000002

image_name: c3750e-universalk9-mz.122-58.SE2.bin

ios_image_file_size: 17879552

total_image_file_size: 20982272

image_family: C3750

stacking_number: 1.46

board_ids: 0x00000068 0x00000069 0x0000006a 0x0000006b 0x000000b1 0x000000b2 0x000000b3 0x000000b4 0x000001e8 0x000001f8 0x000001e7 0x000001f7

info_end:

^@

InayathUlla Sharieff · ‎05-05-2013

Hi Mohan,

I have decoded the tracebacks and the decodes are pointing to the following bug id: CSCtr86594.

HTH

Regards

Inayath

*Please rate useful posts and remember to mark any solved questions as answered. Thank you.

mohankumarm · ‎05-05-2013

Hi Inayath,

THanks very much indeed and looks like this is a bug for the Cisco 2960S Flex stack platform, so likely on 3750X as well (IOS 12.2 58SE2) I am not sure if i configured the line password from the member console though!.Currently the Master is the switch with priority 14, so if we set the password from member console this may not trigger the issue..and wondering how to go about doing this..

Symptom:

when configure line 0 password from member switch's console. it seems that the same password applies to vty 0 15 as well, and as soon as we remove the password from vty 0 4 and try to do write memory, the master switch get crashed.

Conditions:

This issue has not been seen in case we do the same operation from master switch's console.

Workaround:

To do the same operation from master switch's console.