3750 randomly rebooting

[Lsimancek_2]

Hi all,

I have a 3750 stack of 2 switches. Switch-2 has now rebooted twice in the past week without any error or any apparent reason. No power issues, enviroment, memory and processor indicate normal levels so I don't think any certain process killed the thing.

Any ideas? All commands to try or suggestions of issues are welcome.

Thanks,

Lindsay

[bs6825]

My experience leads me to suspect a power issue or hardware failure. In the show version command you should see a line like this:

System returned to ROM by power cycle at 09:37:31 UTC Tue Jan 8 2008 (SP by power on)

If this says you had a reboot due to a power cycle, check your power connections to make sure they are secure. If it says anything else, I would recommend doing a show tech and sending it to TAC. You may have more problems than can be solved here.

[Lsimancek_2]

This stack has had issues and rebooted once back in the fall so it does have an error like that but its from the last reboot. clear as mud right? The cmd is below, with the unrelated parts deleted out

#sh ver

Compiled Mon 22-May-06 08:52 by yenanh

Image text-base: 0x00003000, data-base: 0x01255B58

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SE1, RELEASE SOFTWAR

E (fc)

Roch1-AC-SW06 uptime is 24 weeks, 32 minutes

System returned to ROM by error - Debug Exception (Could be NULL pointer derefer

ence) Exception (0x2000)!, PC 0x90A5D8

System restarted at 07:13:52 EST Tue Nov 18 2008

System image file is "flash:c3750-advipservicesk9-mz.122-25.SEE1.bin"

cisco WS-C3750G-24TS-1U (PowerPC405) processor (revision C0) with 118784K/12280K bytes of memory.

Processor board ID FOC1003Y1WH

Last reset from power-on

3 Virtual Ethernet interfaces

56 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:16:C8:65:A8:00

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 28 WS-C3750G-24TS-1U 12.2(25)SEE1 C3750-ADVIPSERVICESK

2 28 WS-C3750G-24TS-1U 12.2(25)SEE1 C3750-ADVIPSERVICESK

Switch 02

---------

Switch Uptime : 19 hours, 15 minutes

Base ethernet MAC Address : 00:16:C8:72:83:80

[seghettia]

I had a similar issue with a pair of 3750's and I upgraded to 12.2.25 SEE4 and haven't had it happen since.

[Leo Laohoo]

Hi Lindsay,

Does switch 2 have any crashinfo file?

[Lsimancek_2]

The only crashinfo files are from the reboot back in Nov, dealing with a mem alloc issue. It rebooted and had since been fine.

[Lsimancek_2]

Anyone have anymore ideas? This is so random but when its happening we're taking a hit, so I don't want to waste time replacing things if this is a software issue.

Any wild guesses are welcome ...

[davy.timmermans]

My first idea is a memory problem or problem with IOS (or did you add the "null" part.

I found this usefull document: with explanation about output interpreter and how to troubleshoot

http://www.cisco.com/en/US/products/sw/iosswrel/ps1831/products_tech_note09186a008010876d.shtml

I suggest to upgrade to check the link above and to upgrade to 12.2.46-SE(ED)or newer to exlude software as reason.

[Lsimancek_2]

Hi,

I didn't add the "null" comments, those appeared after it rebooted itself last fall due to mem alloc issues.

But if you look at that closely, that line is referencing the whole switch stack, not the individual switches, and any reason why those would reboot. Any ideas on how I can get that info? Something like "show ver switch-2" I wish it works but it doesn't any ideas?

[Leo Laohoo]

Hi Lindsay,

You sure switch 2 doesn't have any crashinfo file? Can you do "dir flash2:" and see if it's there?

[glen.grant]

Check the power cables and the stackwise cables , i know the stack cables can look like they are tight when they are not .

[Lsimancek_2]

Hi leolaohoo,

Thats it! I have been looking at dir: or dirflash: thinking that it would show me all but the second i looked at flash2 I found some of the older crash file indicating a memory allo issue. And a bunch of other stuff in there. Time to investigate! Thanks for the reminder to look at the individual flash

-Lindsay

[eborcher]

Hi Lindsay,

If you look through the old crashinfo file you should see a message about "Debug Exception (Could be NULL pointer der

eference) Exception". If that is the case then that crashinfo file would be helpful for us to look atto track this problem down

[Leo Laohoo]

Hi Lindsay,

Thanks for the rating.

Made the same mistake the first time I handled one of those too!

Can you post the lastest crashinfo so we can verify?

[Lsimancek_2]

First off, thanks guys for all the help.

Now the upsetting news, the crash info is all from previous crashes in the past year to due to mem allocation issues, and in one case a null pointer exception.

Like I said before, 'sh ver' seems to only be telling me info about switch-1 not switch-2 so if anyone knows how to look at that it would be helpful.

Otherwise, given the time (I'm right in the middle of cutting over about 120 switches from our old 6509/cat/sup1 to 6509/IOS/720) I'm going to be replacing this switch and working on this off online since I have found out that they are running on a mismatched SDM template, and the software should be upgraded, like previously suggested, and this stack didn't have the files loaded correctly to do the auto update.