Catalyst 6513 Unexpected Reloads !!!

[radzuweit]

Hi,

we got a phenomenon here. One of our Catalyst 6513 Access-Switches reboots every

2 or 3 days by itself. There are no indications what caused the reloads in Syslog or Counters !

Supervisor-Engine is WS-X6K-SUP2-2GE with

CatOS 6.3(5) HW:3.2 FW:6.1(4).

Syslog severity is 7 (debugging), but before the reload occurs,there are just normal syslog-messages, no errors, no cpu or mem overloads.

We got 8 6513 in this HW/SW constallation in our access layer, but only one shows this behavior???

Any ideas or similar experiences here ?

Thanks for your efforts !

[ankurbhasin]

Hi Friend,

Please paste the sh version from the switch that may help in looking the reload reason. May be we can see some information regarding the same.

Regards,

Ankur

[radzuweit]

ok, here is the output of sh ver !

Thx

WS-C6513 Software, Version NmpSW: 6.3(5)

Copyright (c) 1995-2002 by Cisco Systems

NMP S/W compiled on Feb 7 2002, 19:33:49

System Bootstrap Version: 6.1(4)

Hardware Version: 1.0 Model: WS-C6513 Serial #: TBM05501556

PS1 Module: WS-CAC-2500W Serial #: ART05490014

PS2 Module: WS-CAC-2500W Serial #: ART05450070

Mod Port Model Serial # Versions

--- ---- ------------------- ----------- --------------------------------

1 2 WS-X6K-SUP2-2GE SAD055002N6 Hw : 3.2

Fw : 6.1(4)

Fw1: 6.1(3)

Sw : 6.3(5)

Sw1: 6.3(5)

WS-F6K-PFC2 SAD055000PX Hw : 3.0

2 8 WS-X6408A-GBIC SAL0601G69X Hw : 2.0

Fw : 5.4(2)

Sw : 6.3(5)

3 24 WS-X6324-100FX-MM SAD055005T8 Hw : 3.0

Fw : 5.4(2)

Sw : 6.3(5)

.

.

. ## mod 4-12 are WS-X6324-100FX-MM ; Slot 13 is empty ##

.

.

DRAM FLASH NVRAM

Module Total Used Free Total Used Free Total Used Free

------ ------- ------- ------- ------- ------- ------- ----- ----- -----

1 130944K 63102K 67842K 32768K 15970K 16798K 512K 369K 143K

Uptime is 2 days, 10 hours, 10 minutes

[ankurbhasin]

Hi Friend,

I wa snot aware you are running CATOS on yoru switch will not show the reaload reason like the IOS switch shows.

Can you check the sh log and dir bootflash: and confirm if you see any crash file?

Regards,

Ankur

[radzuweit]

looking for a crashfile was one of my first steps too,to analyse these prob, but there is no crashfile :(

let's have a look on the sh log cmd :

s3c1-1> (enable) sh log

Network Management Processor (ACTIVE NMP) Log:

Reset count: 466

Re-boot History: Feb 27 2006 02:43:25 0, Feb 25 2006 19:27:38 0

Feb 24 2006 05:33:13 0, Feb 23 2006 08:27:26 0

Feb 22 2006 11:45:51 0, Feb 19 2006 07:33:48 0

Feb 13 2006 14:21:00 0, Feb 12 2006 20:08:41 0

Feb 11 2006 15:40:56 0, Feb 11 2006 15:34:42 0

Bootrom Checksum Failures: 0 UART Failures: 0

Flash Checksum Failures: 0 Flash Program Failures: 0

Power Supply 1 Failures: 4 Power Supply 2 Failures: 1

Swapped to CLKA: 0 Swapped to CLKB: 0

Swapped to Processor 1: 0 Swapped to Processor 2: 0

DRAM Failures: 0

Exceptions: 0 <------ well ...

Loaded NMP version: 6.3(5)

Reload same NMP version count: 193 <----!!!aarrgghh !!! that was not me !!!

Last software reset by user: 2/27/2002,07:54:21

EOBC Exceptions/Hang: 0 <----- seems OK ...

Heap Memory Log:

Corrupted Block = none

_____________________________________________________

the flash ...

s3c1-1> (enable) dir

-#- -length- -----date/time------ name

1 9046328 Feb 27 2002 07:31:10 cat6000-sup2cv.6-3-3.bin

2 5733300 May 24 2002 05:23:07 cat6000-sup2.6-3-5.bin

17201684 bytes available (14779884 bytes used)

best regards

Maik

[Bobby Thekkekandam]

Hi Maik,

You are likely experiencing a MISTRAL(system controller ASIC) exception, which only logs debug output to the console port. In versions after 6.3(7), these crashes are saved to NVRAM via CSCdx26043. I would recommend connecting a PC with a console session open and logging to catch the crash output.

Or, you may want to go ahead and schedule a move to the latest 6.x rebuild (currenty 6.4(21)), which is GD code, as it will most likely fix whatever is causing the exception in the first place.

HTH,

Bobby

*Please rate helpful posts

[radzuweit]

Ok thats an idea,i'll connect a Notebook via Console (in addition to the CiscoWorks Logging) to see what happend there.

Thank you so far , i'll report the result !

PS: a move to 6.4 is already planed but i realy want to understand these error ;)

[Bobby Thekkekandam]

I agree that this is the best way to troubleshoot and identify a problem. Once you get a console capture of the output, post it here, and I'll see if I can help you identify a specific bug.

-Bobby

[gmaccisco1]

Hi,

Ihave seen bad Power Supplies causing problems like this one in the past.

Thx,

Masood

[radzuweit]

reHi,

i already had the same suspecion some days ago.

4 weeks ago a bad power supply demaged a standby supervisor engine in one of our core-switches.

But bad power supplys should leave significant logging entrys, and we're using two power supplys per switch on redundant power circuits. Show Log and Show Enviroment

cmd displaying good power conditions, but i'll keep that in mind and schedule a replacement of the power supplys (we got some 2500W's cold-standby ;)

Thx and regards

Maik

[gmaccisco1]

Hi,

You know, I had two wrong Power Supplies (wattage wise) on a Cisco CAT 4500 (Cisco IOS) but the commands; "sh env", etc, showed as all were right and those Power Supplies damaged two 48 cards.

what I mean is that when it comes to Power Supplies, these commands aren't much of any help or atleast per my experience.

i would appreciate it if you notify me on your final finding on this issue.

Thx,

Masood

[radzuweit]

I'll post the results !

Thanks

Maik