cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2317
Views
0
Helpful
10
Replies

Cisco 3750X Platform Crash

mohankumarm
Level 1
Level 1

Hello,

We have three Cisco 3750X-24T-S 12.2(58) SE2 switches in a Stackwise Plus configuration with priorities hardcoded as follows:

Master 15

Member 2 14

Member 3 13

Due to hardware issues suspected with Member 2 switch, this switch was replaced with a new switch. After a couple of days, we observed that Member 2 has promoted itself to Master and the following logs was displayed:

2 12:13:24.475: %PLATFORM-1-CRASHED: Vector = 0x00001100  PC = 0x00458F6C MSR = 0x00029230  CR = 0x53000099 (SW1) May  2 12:13:24.475: %PLATFORM-1-CRASHED: LR = 0x00436B78  CTR = 0x026CC0CC XER = 0xE0000012 (SW1) May  2 12:13:24.475: %PLAT May  2 12:13:24.475: %PLATFORM-1-CRASHED: R4 = 0x0750ACA9  R5 = 0x0000000F

R6 = 0x04660B90  R7 = 0x0726D7A0 (SW1) May  2 12:13:24.484: %PLATFORM-1-CRASHED: R8 = 0x00000000  R9 = 0xC93C0952

R10 = 0x00000001  R11 = 0x03E4C170 (SW1) May  2 12:13:24.484: %PLATFORM-1-CRASHED: R12 = 0x8E63FF8A  R13 = 0x0B98279C  R14 = 0x02006834  R15 = 0x00000000 (SW1) May  2 12:13:24.484: %PLATFORM-1-CRASHED: R16 = 0x00000000  R17 =

0x00000000  R18 = 0x00000000  R19 = 0x00000000 (SW1) May  2 12:13:24.484: %PLATFORM-1-CRASHED: R20 = 0x00000000  R21 =

0x00000000  R22 = 0x00000000  R23 = 0x03917FB4 (SW1) May  2 12

The show switch shows the following:

Switch/Stack Mac Address : 442b.0363.e880

                                           H/W   Current

Switch#  Role   Mac Address     Priority Version  State

----------------------------------------------------------

1       Member 6c20.56b8.0f80     15     1       Ready ----------Previous Master, currently Member

*2       Master 442b.0363.e880     14     1       Ready -------- Previous Member, currently Master

3       Member fc99.47c0.2880     13     1       Ready

Please advise..

Just wondering if there is a hardware issue with SW1 above and SW1 seems to be running ok for the last three days.

Thanks and Regards,

MOhan

1 Accepted Solution

Accepted Solutions

Hi Mohan,


I have decoded the tracebacks and the decodes are pointing to the following bug id: CSCtr86594.

HTH

Regards

Inayath

*Please rate useful posts and remember to mark any solved questions as answered. Thank you.

View solution in original post

10 Replies 10

Leo Laohoo
Hall of Fame
Hall of Fame
3750X-24T-S 12.2(58) SE2

Unless you see an operational need to run this IOS, I suggest you downgrade to 12.2(55)SE7 and observe.

Thanks very much leolaohoo..well, we have a number of switch stacks running 12.2(58) SE2 and there are no problems with it, this is happening in only one stack. So is there any workaround or is this a real bug with 12.2(58) SE2?

Hi Mohan,

Let me see what has crashed the cisco 3750 switch. Could you please share the following information:

1) show version

2) show log

3) crash info file( will be under show flash)

Regards

Inayath

Hello,

 

Thanks very much for your response and here is the attached output from the switch.

 

Thanks and Regards,

Mohan

Hi Mohan,

The crash file is not captured?

Could you send me the crash info file(   459  drwx         512   May 2 2013 22:08:58 +10:00  crashinfo_ext) and complete output of show log.

Regards

Inayath

Sorry the show log output is as follows:

2 12:13:24.475: %PLATFORM-1-CRASHED: Vector = 0x00001100  PC = 0x00458F6C  MSR = 0x00029230  CR = 0x53000099 (Sw-1)

May  2 12:13:24.475: %PLATFORM-1-CRASHED: LR = 0x00436B78  CTR = 0x026CC0CC  XER = 0xE0000012 (Sw-1)

May  2 12:13:24.475: %PLAT

May  2 12:13:24.475: %PLATFORM-1-CRASHED: R4 = 0x0750ACA9  R5 = 0x0000000F  R6 = 0x04660B90  R7 = 0x0726D7A0 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: R8 = 0x00000000  R9 = 0xC93C0952  R10 = 0x00000001  R11 = 0x03E4C170 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: R12 = 0x8E63FF8A  R13 = 0x0B98279C  R14 = 0x02006834  R15 = 0x00000000 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: R16 = 0x00000000  R17 = 0x00000000  R18 = 0x00000000  R19 = 0x00000000 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: R20 = 0x00000000  R21 = 0x00000000  R22 = 0x00000000  R23 = 0x03917FB4 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-

May  2 12:13:24.484: %PLATFORM-1-CRASHED: R28 = 0x0000000F  R29 = 0x0750ACA9  R30 = 0x07509EDC  R31 = 0x0726D7A0 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED:   (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Stack trace: (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: PC = 0x00458F6C, SP = 0x070B33E8 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 00: SP = 0x070B3418    PC = 0x03E20000 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 01: SP = 0x070B3468    PC = 0x00436B78 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 02: SP = 0x070B3480    PC = 0x01D3C8DC (Sw-1)

May  2 1

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 04: SP = 0x070B34A8    PC = 0x0048D1FC (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 05: SP = 0x070B3528    PC = 0x00483D88 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 06: SP = 0x070B3558    PC = 0x004836E0 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 07: SP = 0x070B35B0    PC = 0x00509E7C (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 08: SP = 0x070B35D0    PC = 0x01D38A34 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 09: SP = 0x070B35F0    PC = 0x01D38ED0 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 10:

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 11: SP = 0x070B3678    PC = 0x00483D88 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 12: SP = 0x070B36A8    PC = 0x004836E0 (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 13: SP = 0x070B3748    PC = 0x01D3A1DC (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 14: SP = 0x070B3750    PC = 0x0048D1FC (Sw-1)

May  2 12:13:24.484: %PLATFORM-1-CRASHED: Frame 15: SP = 0x070B37D0    PC = 0x00483D88 (Sw-1)

I cant remember having seen the extended crash file info in flash, so had the "show stacks" output

GS-PCR-SFDS01# show flash

Directory of flash:/

  460  -rwx        1216   May 1 2013 17:46:02 +10:00  vlan.dat

    3  -rwx       32480   May 5 2013 18:48:16 +10:00  config.text

    4  -rwx        5481   May 5 2013 18:48:16 +10:00  private-config.text

    5  drwx         512   Mar 1 1993 11:09:48 +11:00  c3750e-universalk9-mz.122-58.SE2

  461  -rwx        1943   Mar 1 1993 11:00:58 +11:00  config.text.backup

  462  -rwx        1936   Mar 1 1993 11:00:59 +11:00  private-config.text.backup

  463  -rwx        7192   May 5 2013 18:48:16 +10:00  multiple-fs

57671680 bytes total (36256256 bytes free)

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#

GS-PCR-SFDS01#sh stacks 

Minimum process stacks:

Free/Size   Name

4880/6000   hulc_flash init

5608/6000   CDP BLOB

8524/9000   IP Background

8560/9000   IP Connected Route Background

8576/9000   EEM ED RF

4192/6000   EEM Shell Director

5596/6000   vqpc_shim_create_addr_tbl

5580/6000   SPAN Subsystem

19336/24000  Init

58800/60000  EEM Auto Registration Proc

5556/6000   SASL MAIN

5296/6000   LICENSE AGENT DEFAULT

8592/9000   cdp init process

5500/6000   IPC Zone Manager

5584/6000   RTTY Slave Registration Process

5184/6000   MDFS LC IPC Init Process

5600/6000   IP SLAs Deferred Schedule Processor

4724/6000   IPC Split Init

5180/6000   HUDLD fiber table process

5392/6000   RADIUS INITCONFIG

5468/6000   bulk config process

5600/6000   MDFS LC Download Ack Process

5548/6000   URPF stats

2484/3000   Rom Random Update Process

4812/6000   platform OBFL

9892/12000  OBFL VOLT obfl0

5208/6000   HVLAN Mapped Vlan Id Update Process

10700/12000  Router Init

6836/12000  cfg mgr new master process

5572/6000   HDF Stack-Mac

5592/6000   HCMP redirect process

5200/6000   XDR LC PRIO Critcal Back

5588/6000   CEF Post-Switchover Tidy

5516/6000   HCMP new mac process

5164/6000   XDR LC PRIO Background

11480/12000  CFC LC Test Stats

11572/12000  CFC stats broker process

5104/6000   CEF LC Stats

9524/12000  XDR LC Background

5032/6000   CEF LC IPC Background

5020/6000   RTTY Client Registration

5076/6000   IPC LC Port Opener

5088/6000   IP ACL RP Background

5496/6000   MDFS Reload

4716/6000   hpm bulk_vlan_state process

4452/6000   HL3U ARP Refresh Proc

4992/12000  SSH Process

18628/24000  hulc_techsupport_process

Interrupt level stacks:

Level    Called Unused/Size  Name

  1           0   9000/9000  Maxwell MACsec ISR

  4  3252150534   7400/9000  NETWORK INTERFACE CHIP

  5           0   9000/9000  SUPERVISOR EXCEPTIONS

  6       23544   8936/9000  NS16550 VECTOR

Hi mohan,

Runn the following command:

#more flash1:/crashinfo_ext/crashinfo_ext

#more flash1:/crashinfo_ext

and update the same on this.

Regards

Inayath

Hi Inayath,

I can see the following file

57671680 bytes total (36256256 bytes free)

GS-PCR-SFDS01#more info

version_suffix: universalk9-122-58.SE2

version_directory: c3750e-universalk9-mz.122-58.SE2

image_system_type_id: 0x00000002

image_name: c3750e-universalk9-mz.122-58.SE2.bin

ios_image_file_size: 17879552

total_image_file_size: 20982272

image_feature: IP|LAYER_3|PLUS|SSH|3DES|MIN_DRAM_MEG=128

image_family: C3750

stacking_number: 1.46

board_ids: 0x00000068 0x00000069 0x0000006a 0x0000006b 0x000000b1 0x000000b2 0x000000b3 0x000000b4 0x000001e8 0x000001f8 0x000001e7 0x000001f7

info_end:

^@

Hi Mohan,


I have decoded the tracebacks and the decodes are pointing to the following bug id: CSCtr86594.

HTH

Regards

Inayath

*Please rate useful posts and remember to mark any solved questions as answered. Thank you.

Hi Inayath,

THanks very much indeed and looks like this is a bug for the Cisco 2960S Flex stack platform, so likely on 3750X as well (IOS 12.2 58SE2) I am not sure if i configured the line password from the member console though!.Currently the Master is the switch with priority 14, so if we set the password from member console this may not trigger the issue..and wondering how to go about doing this..

Symptom:

when configure line 0  password from member switch's  console. it seems that  the same password applies to vty 0 15 as well,  and as soon as we remove the password from vty 0 4 and try to do write  memory, the master switch get crashed.

Conditions:

This issue has not been seen in case we do the same operation from master switch's console.

Workaround:

To do the same operation from master switch's console.

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community:

Innovations in Cisco Full Stack Observability - A new webinar from Cisco