11-13-2011 10:26 PM - edited 03-07-2019 03:22 AM
Hi,
I've got a really strange problem here. I have four-switch Cat 3750 stack that has been working reliably for a couple years. Suddenly the other morning it reboots itself, and has been doing so every 24 hours since. Could someone please take a look at the attached log? This should really be a Smartnet thing, but as it turns out the warranty we have on this is not Smartnet .
11-13-2011 11:06 PM
Hi Joe,
Switches don't look to be crashing - seems to be powered down (as we would see crashdump messages in log in case of crash).
Can you please provide more logs to confirm:
remote command 1 sh ver (do it for all switches changing 1 to 2,3 Nand 4)
show stack
check for crashinfo files in flash and do "more crashinfo_1" if any.
Do you also have syslog messages from these swithces prior to reload?
Do you know if anyone has access to server room by that time?
Nik
11-14-2011 01:04 AM
Hi Nik,
Thank you so much for taking interest !
No one should have access to the server room at that time ...
No sign of crashinfo files:
-sw01#show flash
Directory of flash:/
2 -rwx 796 Jun 7 2011 17:02:52 +08:00 vlan.dat
3 -rwx 12700373 Mar 1 1993 08:22:40 +08:00 c3750-ipservicesk9-mz.122-53.SE2.bin
4 -rwx 44668 Aug 6 2011 11:18:29 +08:00 config.text
5 -rwx 1934 Aug 6 2011 11:18:29 +08:00 private-config.text
7 -rwx 38015 Mar 1 1993 08:04:31 +08:00 config.text.backup
8 -rwx 24 Mar 1 1993 08:04:31 +08:00 private-config.text.backup
9 -rwx 2072 Nov 11 2011 06:41:59 +08:00 multiple-fs
15998976 bytes total (3207680 bytes free)
Here are the show version commands:
-sw01#remote comamand 1 sh ver
Switch : 1 :
------------
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 04:49 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)
-sw01-1 uptime is 9 hours, 54 minutes
System returned to ROM by power-on
System restarted at 06:49:44 HKT Mon Nov 14 2011
System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
cisco WS-C3750G-48TS (PowerPC405) processor (revision F0) with 131072K bytes of memory.
Processor board ID
Last reset from power-on
4 Virtual Ethernet interfaces
72 FastEthernet interfaces
58 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 08:1F:F3:84:03:00
Motherboard assembly number : 73-10218-08
Power supply part number : 341-0107-01
Motherboard serial number :
Power supply serial number : AZS140905NR
Model revision number : F0
Motherboard revision number : G0
Model number : WS-C3750G-48TS-S
System serial number :
Top Assembly Part Number : 800-26857-02
Top Assembly Revision Number : D0
Version ID : V04
CLEI Code Number : COM7X10ARA
Hardware Board Revision Number : 0x09
Configuration register is 0xF
-sw01#remote command 2 sh ver
This command is for remote switches only
-sw01#show ver
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 04:49 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)
-sw01 uptime is 1 year, 23 weeks, 1 day, 18 hours, 42 minutes
System returned to ROM by power-on
System restarted at 22:01:54 HKT Fri Jun 4 2010
System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
cisco WS-C3750-24TS (PowerPC405) processor (revision M0) with 131072K bytes of memory.
Processor board ID CAT1115NJ66
Last reset from power-on
4 Virtual Ethernet interfaces
72 FastEthernet interfaces
58 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:1B:90:65:F5:00
Motherboard assembly number : 73-9677-10
Power supply part number : 341-0034-01
Motherboard serial number :
Power supply serial number : DTH111210NE
Model revision number : M0
Motherboard revision number : B0
Model number : WS-C3750-24TS-S
System serial number : CAT1115NJ66
Top Assembly Part Number : 800-25857-02
Top Assembly Revision Number : F0
Version ID : V05
CLEI Code Number : CNMV100CRE
Hardware Board Revision Number : 0x01
Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
1 52 WS-C3750G-48TS 12.2(53)SE2 C3750-IPSERVICESK9-M
* 2 26 WS-C3750-24TS 12.2(53)SE2 C3750-IPSERVICESK9-M
3 26 WS-C3750-24P 12.2(53)SE2 C3750-IPSERVICESK9-M
4 26 WS-C3750-24P 12.2(53)SE2 C3750-IPSERVICESK9-M
Switch 01
---------
Switch Uptime : 9 hours, 54 minutes
Base ethernet MAC Address : 08:1F:F3:84:03:00
Motherboard assembly number : 73-10218-08
Power supply part number : 341-0107-01
Motherboard serial number :
Power supply serial number : AZS140905NR
Model revision number : F0
Motherboard revision number : G0
Model number : WS-C3750G-48TS-S
System serial number :
Top assembly part number : 800-26857-02
Top assembly revision number : D0
Version ID : V04
CLEI Code Number : COM7X10ARA
Switch 03
---------
Switch Uptime : 9 hours, 54 minutes
Base ethernet MAC Address : 00:11:BB:C2:77:80
Motherboard assembly number : 73-8312-04
Power supply part number : 341-0029-03
Motherboard serial number :
Power supply serial number : LIT082802A9
Model revision number : C0
Motherboard revision number : B0
Model number : WS-C3750-24PS-S
System serial number : CAT0833R119
Top assembly part number : 800-21982-01
Top assembly revision number : F0
Version ID : N/A
Switch 04
---------
Switch Uptime : 9 hours, 52 minutes
Base ethernet MAC Address : 00:11:BB:97:BB:80
Motherboard assembly number : 73-8312-04
Power supply part number : 341-0029-03
Motherboard serial number :
Power supply serial number : LIT082802Q1
Model revision number : C0
Motherboard revision number : B0
Model number : WS-C3750-24PS-S
System serial number :
Top assembly part number : 800-21982-01
Top assembly revision number : F0
Version ID : N/A
Configuration register is 0xF
-sw01#remote command 3 sh ver
Switch : 3 :
------------
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 04:49 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)
-sw01-3 uptime is 9 hours, 54 minutes
System returned to ROM by power-on
System restarted at 06:50:00 HKT Mon Nov 14 2011
System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
cisco WS-C3750-24P (PowerPC405) processor (revision C0) with 131072K bytes of memory.
Processor board ID CAT0833R119
Last reset from power-on
4 Virtual Ethernet interfaces
72 FastEthernet interfaces
58 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:11:BB:C2:77:80
Motherboard assembly number : 73-8312-04
Power supply part number : 341-0029-03
Motherboard serial number :
Power supply serial number : LIT082802A9
Model revision number : C0
Motherboard revision number : B0
Model number : WS-C3750-24PS-S
System serial number :
Top Assembly Part Number : 800-21982-01
Top Assembly Revision Number : F0
Version ID : N/A
Hardware Board Revision Number : 0x09
Configuration register is 0xF
-sw01#remote command 4 sh ver
Switch : 4 :
------------
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(53)SE2, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Wed 21-Apr-10 04:49 by prod_rel_team
Image text-base: 0x01000000, data-base: 0x02F00000
ROM: Bootstrap program is C3750 boot loader
BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(44)SE5, RELEASE SOFTWARE (fc1)
-sw01-4 uptime is 9 hours, 52 minutes
System returned to ROM by power-on
System restarted at 06:51:28 HKT Mon Nov 14 2011
System image file is "flash:/c3750-ipservicesk9-mz.122-53.SE2.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
cisco WS-C3750-24P (PowerPC405) processor (revision C0) with 131072K bytes of memory.
Processor board ID
Last reset from power-on
4 Virtual Ethernet interfaces
72 FastEthernet interfaces
58 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.
512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : 00:11:BB:97:BB:80
Motherboard assembly number : 73-8312-04
Power supply part number : 341-0029-03
Motherboard serial number :
Power supply serial number : LIT082802Q1
Model revision number : C0
Motherboard revision number : B0
Model number : WS-C3750-24PS-S
System serial number :
Top Assembly Part Number : 800-21982-01
Top Assembly Revision Number : F0
Version ID : N/A
Hardware Board Revision Number : 0x09
Configuration register is 0xF
11-14-2011 04:40 AM
Hi
All switches indeed showing reloade due to power-on. SO they all lost power and then got it back. I would consider problem with power source. I would doubt all 4 switches in stack got HW porblem with power controller. So I would check the UPS/power/sources/outlets used to see if any problems there. If any other devices connected to same source have similar problems.
Nik
11-14-2011 02:03 PM
I agree with Nik here. 3 of your 4 switches had a reboot due to power-related reason. 1 of your switch has been running for > 1 year.
If there was a crash, you'd see a crashinfo directory. Can you please post the output to the following command:
1. dir flash1:
2. dir flash2:
3. dir flash3:
4. dir flash4:
11-14-2011 05:07 PM
You're right, there is no chrashinfo anywhere.
Thanks so much, everyone. I'll go check out the UPS and let you know what I find
ttinthk-sw01#dir flash1:
Directory of flash1:/
2 -rwx 796 Mar 1 1993 08:03:01 +08:00 vlan.dat
3 -rwx 2072 Nov 11 2011 06:46:12 +08:00 multiple-fs
4 -rwx 1934 Aug 6 2011 11:18:27 +08:00 private-config.text
5 -rwx 12700373 Mar 1 1993 10:07:55 +08:00 c3750-ipservicesk9-mz.122-53.SE2.bin
6 -rwx 44668 Aug 6 2011 11:18:27 +08:00 config.text
32514048 bytes total (19761664 bytes free)
ttinthk-sw01#dir flash2:
Directory of flash:/
2 -rwx 796 Jun 7 2011 17:02:52 +08:00 vlan.dat
3 -rwx 12700373 Mar 1 1993 08:22:40 +08:00 c3750-ipservicesk9-mz.122-53.SE2.bin
4 -rwx 44668 Aug 6 2011 11:18:29 +08:00 config.text
5 -rwx 1934 Aug 6 2011 11:18:29 +08:00 private-config.text
6 -rwx 46367 Nov 14 2011 18:42:02 +08:00 config-14-Nov-2011.txt
7 -rwx 38015 Mar 1 1993 08:04:31 +08:00 config.text.backup
8 -rwx 24 Mar 1 1993 08:04:31 +08:00 private-config.text.backup
9 -rwx 2072 Nov 11 2011 06:41:59 +08:00 multiple-fs
15998976 bytes total (3160576 bytes free)
ttinthk-sw01#dir flash3:
Directory of flash3:/
2 -rwx 796 Mar 1 1993 08:02:30 +08:00 vlan.dat
3 -rwx 1934 Aug 6 2011 11:18:30 +08:00 private-config.text
4 -rwx 12700373 Mar 1 1993 08:51:23 +08:00 c3750-ipservicesk9-mz.122-53.SE2.bin
5 -rwx 44668 Aug 6 2011 11:18:30 +08:00 config.text
6 -rwx 2072 Nov 11 2011 06:46:22 +08:00 multiple-fs
15998976 bytes total (3246592 bytes free)
ttinthk-sw01#dir flash4:
Directory of flash4:/
2 -rwx 796 Mar 1 1993 08:01:10 +08:00 vlan.dat
3 -rwx 12700373 Mar 1 1993 08:11:03 +08:00 c3750-ipservicesk9-mz.122-53.SE2.bin
4 -rwx 1934 Aug 6 2011 11:18:29 +08:00 private-config.text
5 -rwx 44668 Aug 6 2011 11:18:29 +08:00 config.text
7 -rwx 2072 Nov 15 2011 06:45:16 +08:00 multiple-fs
15998976 bytes total (3246592 bytes free)
11-14-2011 06:17 PM
JOE CHANG wrote:
You're right, there is no chrashinfo anywhere.
Thanks so much, everyone. I'll go check out the UPS and let you know what I find
Look for stupid things like a cleaner unplugging the rack power to plug in his vacuum cleaner.
(Don't laugh. I've had it happen to me! The cleaner was *very* rapidly an ex-cleaner)
Cheers.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide