Switch stack reboots every 24 hours

[changjoe]

Hi,

I've got a really strange problem here. I have four-switch Cat 3750 stack that has been working reliably for a couple years. Suddenly the other morning it reboots itself, and has been doing so every 24 hours since. Could someone please take a look at the attached log? This should really be a Smartnet thing, but as it turns out the warranty we have on this is not Smartnet .

[nkarpysh]

Hi Joe,

Switches don't look to be crashing - seems to be powered down (as we would see crashdump messages in log in case of crash).

Can you please provide more logs to confirm:

remote command 1 sh ver   (do it for all switches changing 1 to 2,3 Nand 4)

show stack

check for crashinfo files in flash and do "more crashinfo_1" if any.

Do you also have syslog messages from these swithces prior to reload?

Do you know if anyone has access to server room by that time?

Nik

[changjoe]

Hi Nik,

Thank you so much for taking interest !

No one should have access to the server room at that time ...

No sign of crashinfo files:

-sw01#show flash

Directory of flash:/

    2  -rwx         796   Jun 7 2011 17:02:52 +08:00  vlan.dat

    3  -rwx    12700373   Mar 1 1993 08:22:40 +08:00  c3750-ipservicesk9-mz.122-53.SE2.bin

    4  -rwx       44668   Aug 6 2011 11:18:29 +08:00  config.text

    5  -rwx        1934   Aug 6 2011 11:18:29 +08:00  private-config.text

    7  -rwx       38015   Mar 1 1993 08:04:31 +08:00  config.text.backup

    8  -rwx          24   Mar 1 1993 08:04:31 +08:00  private-config.text.backup

    9  -rwx        2072  Nov 11 2011 06:41:59 +08:00  multiple-fs

15998976 bytes total (3207680 bytes free)

Here are the show version commands:

-sw01#remote comamand 1 sh ver

Switch : 1 :

------------

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Wed 21-Apr-10 04:49 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

-sw01-1 uptime is 9 hours, 54 minutes

System returned to ROM by power-on

System restarted at 06:49:44 HKT Mon Nov 14 2011

System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750G-48TS (PowerPC405) processor (revision F0) with 131072K bytes of memory.

Processor board ID

Last reset from power-on

4 Virtual Ethernet interfaces

72 FastEthernet interfaces

58 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : 08:1F:F3:84:03:00

Motherboard assembly number     : 73-10218-08

Power supply part number        : 341-0107-01

Motherboard serial number       :

Power supply serial number      : AZS140905NR

Model revision number           : F0

Motherboard revision number     : G0

Model number                    : WS-C3750G-48TS-S

System serial number            :

Top Assembly Part Number        : 800-26857-02

Top Assembly Revision Number    : D0

Version ID                      : V04

CLEI Code Number                : COM7X10ARA

Hardware Board Revision Number  : 0x09

Configuration register is 0xF

-sw01#remote command 2 sh ver

This command is for remote switches only

-sw01#show ver

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Wed 21-Apr-10 04:49 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

-sw01 uptime is 1 year, 23 weeks, 1 day, 18 hours, 42 minutes

System returned to ROM by power-on

System restarted at 22:01:54 HKT Fri Jun 4 2010

System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750-24TS (PowerPC405) processor (revision M0) with 131072K bytes of memory.

Processor board ID CAT1115NJ66

Last reset from power-on

4 Virtual Ethernet interfaces

72 FastEthernet interfaces

58 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : 00:1B:90:65:F5:00

Motherboard assembly number     : 73-9677-10

Power supply part number        : 341-0034-01

Motherboard serial number       :

Power supply serial number      : DTH111210NE

Model revision number           : M0

Motherboard revision number     : B0

Model number                    : WS-C3750-24TS-S

System serial number            : CAT1115NJ66

Top Assembly Part Number        : 800-25857-02

Top Assembly Revision Number    : F0

Version ID                      : V05

CLEI Code Number                : CNMV100CRE

Hardware Board Revision Number  : 0x01

Switch Ports Model              SW Version            SW Image                

------ ----- -----              ----------            ----------              

     1 52    WS-C3750G-48TS     12.2(53)SE2           C3750-IPSERVICESK9-M    

*    2 26    WS-C3750-24TS      12.2(53)SE2           C3750-IPSERVICESK9-M    

     3 26    WS-C3750-24P       12.2(53)SE2           C3750-IPSERVICESK9-M    

     4 26    WS-C3750-24P       12.2(53)SE2           C3750-IPSERVICESK9-M    

Switch 01

---------

Switch Uptime                   : 9 hours, 54 minutes

Base ethernet MAC Address       : 08:1F:F3:84:03:00

Motherboard assembly number     : 73-10218-08

Power supply part number        : 341-0107-01

Motherboard serial number       :

Power supply serial number      : AZS140905NR

Model revision number           : F0

Motherboard revision number     : G0

Model number                    : WS-C3750G-48TS-S

System serial number            :

Top assembly part number        : 800-26857-02

Top assembly revision number    : D0

Version ID                      : V04

CLEI Code Number                : COM7X10ARA

Switch 03

---------

Switch Uptime                   : 9 hours, 54 minutes

Base ethernet MAC Address       : 00:11:BB:C2:77:80

Motherboard assembly number     : 73-8312-04

Power supply part number        : 341-0029-03

Motherboard serial number       :

Power supply serial number      : LIT082802A9

Model revision number           : C0

Motherboard revision number     : B0

Model number                    : WS-C3750-24PS-S

System serial number            : CAT0833R119

Top assembly part number        : 800-21982-01

Top assembly revision number    : F0

Version ID                      : N/A

Switch 04

---------

Switch Uptime                   : 9 hours, 52 minutes

Base ethernet MAC Address       : 00:11:BB:97:BB:80

Motherboard assembly number     : 73-8312-04

Power supply part number        : 341-0029-03

Motherboard serial number       :

Power supply serial number      : LIT082802Q1

Model revision number           : C0

Motherboard revision number     : B0

Model number                    : WS-C3750-24PS-S

System serial number            :

Top assembly part number        : 800-21982-01

Top assembly revision number    : F0

Version ID                      : N/A

Configuration register is 0xF

-sw01#remote command 3 sh ver

Switch : 3 :

------------

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Wed 21-Apr-10 04:49 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

-sw01-3 uptime is 9 hours, 54 minutes

System returned to ROM by power-on

System restarted at 06:50:00 HKT Mon Nov 14 2011

System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750-24P (PowerPC405) processor (revision C0) with 131072K bytes of memory.

Processor board ID CAT0833R119

Last reset from power-on

4 Virtual Ethernet interfaces

72 FastEthernet interfaces

58 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : 00:11:BB:C2:77:80

Motherboard assembly number     : 73-8312-04

Power supply part number        : 341-0029-03

Motherboard serial number       :

Power supply serial number      : LIT082802A9

Model revision number           : C0

Motherboard revision number     : B0

Model number                    : WS-C3750-24PS-S

System serial number            :

Top Assembly Part Number        : 800-21982-01

Top Assembly Revision Number    : F0

Version ID                      : N/A

Hardware Board Revision Number  : 0x09

Configuration register is 0xF

-sw01#remote command 4 sh ver

Switch : 4 :

------------

Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)

Technical Support: http://www.cisco.com/techsupport

Copyright (c) 1986-2010 by Cisco Systems, Inc.

Compiled Wed 21-Apr-10 04:49 by prod_rel_team

Image text-base: 0x01000000, data-base: 0x02F00000

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)

-sw01-4 uptime is 9 hours, 52 minutes

System returned to ROM by power-on

System restarted at 06:51:28 HKT Mon Nov 14 2011

System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750-24P (PowerPC405) processor (revision C0) with 131072K bytes of memory.

Processor board ID

Last reset from power-on

4 Virtual Ethernet interfaces

72 FastEthernet interfaces

58 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address       : 00:11:BB:97:BB:80

Motherboard assembly number     : 73-8312-04

Power supply part number        : 341-0029-03

Motherboard serial number       :

Power supply serial number      : LIT082802Q1

Model revision number           : C0

Motherboard revision number     : B0

Model number                    : WS-C3750-24PS-S

System serial number            :

Top Assembly Part Number        : 800-21982-01

Top Assembly Revision Number    : F0

Version ID                      : N/A

Hardware Board Revision Number  : 0x09

Configuration register is 0xF

[nkarpysh]

Hi

All switches indeed showing reloade due to power-on. SO they all lost power and then got it back. I would consider problem with power source. I would doubt all 4 switches in stack got HW porblem with power controller. So I would check the UPS/power/sources/outlets used to see if any problems there. If any other devices connected to same source have similar problems.

Nik

[Leo Laohoo]

I agree with Nik here.  3 of your 4 switches had a reboot due to power-related reason.  1 of your switch has been running for > 1 year.

If there was a crash, you'd see a crashinfo directory.  Can you please post the output to the following command:

1.  dir flash1:

2.  dir flash2:

3.  dir flash3:

4.  dir flash4:

[changjoe]

You're right, there is no chrashinfo anywhere.

Thanks so much, everyone. I'll go check out the UPS and let you know what I find

ttinthk-sw01#dir flash1:

Directory of flash1:/

    2  -rwx         796   Mar 1 1993 08:03:01 +08:00  vlan.dat

    3  -rwx        2072  Nov 11 2011 06:46:12 +08:00  multiple-fs

    4  -rwx        1934   Aug 6 2011 11:18:27 +08:00  private-config.text

    5  -rwx    12700373   Mar 1 1993 10:07:55 +08:00  c3750-ipservicesk9-mz.122-53.SE2.bin

    6  -rwx       44668   Aug 6 2011 11:18:27 +08:00  config.text

32514048 bytes total (19761664 bytes free)

ttinthk-sw01#dir flash2:

Directory of flash:/

    2  -rwx         796   Jun 7 2011 17:02:52 +08:00  vlan.dat

    3  -rwx    12700373   Mar 1 1993 08:22:40 +08:00  c3750-ipservicesk9-mz.122-53.SE2.bin

    4  -rwx       44668   Aug 6 2011 11:18:29 +08:00  config.text

    5  -rwx        1934   Aug 6 2011 11:18:29 +08:00  private-config.text

    6  -rwx       46367  Nov 14 2011 18:42:02 +08:00  config-14-Nov-2011.txt

    7  -rwx       38015   Mar 1 1993 08:04:31 +08:00  config.text.backup

    8  -rwx          24   Mar 1 1993 08:04:31 +08:00  private-config.text.backup

    9  -rwx        2072  Nov 11 2011 06:41:59 +08:00  multiple-fs

15998976 bytes total (3160576 bytes free)

ttinthk-sw01#dir flash3:

Directory of flash3:/

    2  -rwx         796   Mar 1 1993 08:02:30 +08:00  vlan.dat

    3  -rwx        1934   Aug 6 2011 11:18:30 +08:00  private-config.text

    4  -rwx    12700373   Mar 1 1993 08:51:23 +08:00  c3750-ipservicesk9-mz.122-53.SE2.bin

    5  -rwx       44668   Aug 6 2011 11:18:30 +08:00  config.text

    6  -rwx        2072  Nov 11 2011 06:46:22 +08:00  multiple-fs

15998976 bytes total (3246592 bytes free)

ttinthk-sw01#dir flash4:

Directory of flash4:/

    2  -rwx         796   Mar 1 1993 08:01:10 +08:00  vlan.dat

    3  -rwx    12700373   Mar 1 1993 08:11:03 +08:00  c3750-ipservicesk9-mz.122-53.SE2.bin

    4  -rwx        1934   Aug 6 2011 11:18:29 +08:00  private-config.text

    5  -rwx       44668   Aug 6 2011 11:18:29 +08:00  config.text

    7  -rwx        2072  Nov 15 2011 06:45:16 +08:00  multiple-fs

15998976 bytes total (3246592 bytes free)

[darren.g]

JOE CHANG wrote:
You're right, there is no chrashinfo anywhere.
Thanks so much, everyone. I'll go check out the UPS and let you know what I find  

Look for stupid things like a cleaner unplugging the rack power to plug in his vacuum cleaner.

(Don't laugh. I've had it happen to me! The cleaner was *very* rapidly an ex-cleaner)

Cheers.