3750X, Code 12.2-25.see2

MikeCarter1 · ‎12-14-2011

Good Day Everyone.....

Recently, we had a 3750X stack of 9 switches reload with no information as to why. Our syslog server received no spurious errors regarding the cause nor did the location lose power. The UPS to which it is attached logged no glitches relating to power. Other cisco devices (router, other switches not in the stack) did not reload. No one was in the MDF during this time.

We self insure these line of switches so a TAC case is not an option at this time. I was curious if anyone knew of any forensic tools that I could try in an attempt to narrow down the cause. BugSearch returned no help.

Thank you for your time.

Leo Laohoo · ‎12-14-2011

Hi Mike,

What do you mean by the title of this thread? 3750X will not boot with IOS version 12.2(25). The lowest IOS for the 3750X is the 12.2(53)SE2 and nothing earlier than that.

Regarding the cause of the reboot, can you please post the complete output for the following command:

1. sh version;

2. sh switch detail;

3. dir flashNNN: < --- NNN = the number of switches in your stack. If you have, say 4, then you need to do "dir flash1:", "dir flash2:", "dir flash3:" and "dir flash4:".

4. sh log

MikeCarter1 · ‎12-15-2011

Interesting results from dir flash7: and dir flash9:

Also, I believe these to be regular 3750's and not X's. My bad.

Cisco IOS Software, C3750 Software (C3750-IPBASEK9-M), Version 12.2(25)SEE2, RELEASE SOFTWARE (fc1)

Compiled Fri 28-Jul-06 12:55 by yenanh

Image text-base: 0x00003000, data-base: 0x010272D8

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.2(25r)SEC, RELEASE SOFTWARE (fc4)

XXXXmstack uptime is 3 days, 21 hours, 8 minutes

System returned to ROM by power-on

System restarted at 12:03:47 est Sun Dec 11 2011

System image file is "flash:c3750-ipbasek9-mz.122-25.SEE2/c3750-ipbasek9-mz.122-25.SEE2.bin"

This product contains cryptographic features and is subject to United

States and local country laws governing import, export, transfer and

use. Delivery of Cisco cryptographic products does not imply

third-party authority to import, export, distribute or use encryption.

Importers, exporters, distributors and users are responsible for

compliance with U.S. and local country laws. By using this product you

agree to comply with applicable laws and regulations. If you are unable

to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:

http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to

export@cisco.com.

cisco WS-C3750-24TS (PowerPC405) processor (revision M0) with 118784K/12280K bytes of memory.

Processor board ID CAT1114ZJZ1

Last reset from power-on

2 Virtual Ethernet interfaces

168 FastEthernet interfaces

62 Gigabit Ethernet interfaces

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:1A:E3:96:39:00

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT11145774

Power supply serial number : PHI1022B4K1

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1114ZJZ1

Top Assembly Part Number : 800-25857-02

Top Assembly Revision Number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Hardware Board Revision Number : 0x01

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

1 24 WS-C3750G-24T 12.2(25)SEE2 C3750-IPBASEK9-M

2 24 WS-C3750G-24T 12.2(25)SEE2 C3750-IPBASEK9-M

3 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

4 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

5 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

6 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

7 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

* 8 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

9 26 WS-C3750-24TS 12.2(25)SEE2 C3750-IPBASEK9-M

Switch 01

---------

Switch Uptime : 3 days, 21 hours, 0 minutes

Base ethernet MAC Address : 00:1B:0D:BF:1E:00

Motherboard assembly number : 73-9679-10

Power supply part number : 341-0048-03

Motherboard serial number : CAT11235C2F

Power supply serial number : DTN110931TA

Model revision number : M0

Motherboard revision number : C0

Model number : WS-C3750G-24T-S

System serial number : FHK1134Z01G

Top assembly part number : 800-25855-01

Top assembly revision number : G0

Version ID : V05

CLEI Code Number : COMR100BRA

Switch 02

---------

Switch Uptime : 3 days, 21 hours, 0 minutes

Base ethernet MAC Address : 00:18:BA:F2:F0:00

Motherboard assembly number : 73-9679-10

Power supply part number : 341-0048-03

Motherboard serial number : CAT10325U7L

Power supply serial number : DTH10284AY4

Model revision number : L0

Motherboard revision number : B0

Model number : WS-C3750G-24T-S

System serial number : CAT1033ZJJ9

Top assembly part number : 800-25855-01

Top assembly revision number : E0

Version ID : V05

CLEI Code Number : COMR100BRA

Switch 03

---------

Switch Uptime : 3 days, 21 hours, 0 minutes

Base ethernet MAC Address : 00:1B:90:6B:74:00

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT11155109

Power supply serial number : DTH111210UH

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1115NHL5

Top assembly part number : 800-25857-02

Top assembly revision number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Switch 04

---------

Switch Uptime : 3 days, 21 hours, 0 minutes

Base ethernet MAC Address : 00:1B:90:6B:9C:00

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT111550VW

Power supply serial number : DTH111210FB

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1115NHDY

Top assembly part number : 800-25857-02

Top assembly revision number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Switch 05

---------

Switch Uptime : 3 days, 21 hours, 0 minutes

Base ethernet MAC Address : 00:1B:90:6B:C5:00

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT111550Z3

Power supply serial number : DTH111210KC

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1115RHBS

Top assembly part number : 800-25857-02

Top assembly revision number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Switch 06

---------

Switch Uptime : 3 days, 21 hours, 0 minutes

Base ethernet MAC Address : 00:1B:90:6B:7E:80

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT1115510K

Power supply serial number : DTH111210UV

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1115NHK6

Top assembly part number : 800-25857-02

Top assembly revision number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Switch 07

---------

Switch Uptime : 3 days, 21 hours, 9 minutes

Base ethernet MAC Address : 00:1B:90:6B:54:00

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT111550XT

Power supply serial number : DTH111210J5

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1115NHG9

Top assembly part number : 800-25857-02

Top assembly revision number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Switch 09

---------

Switch Uptime : 3 days, 21 hours, 9 minutes

Base ethernet MAC Address : 00:1B:90:6B:B4:80

Motherboard assembly number : 73-9677-10

Power supply part number : 341-0034-01

Motherboard serial number : CAT11155119

Power supply serial number : DTH111210SL

Model revision number : M0

Motherboard revision number : B0

Model number : WS-C3750-24TS-S

System serial number : CAT1115RHAL

Top assembly part number : 800-25857-02

Top assembly revision number : F0

Version ID : V05

CLEI Code Number : CNMV100CRE

Configuration register is 0xF

Sh switch detail:

Current

Switch# Role Mac Address Priority State

--------------------------------------------------------

1 Member 001b.0dbf.1e00 15 Ready

2 Member 0018.baf2.f000 10 Ready

3 Member 001b.906b.7400 1 Ready

4 Member 001b.906b.9c00 1 Ready

5 Member 001b.906b.c500 1 Ready

6 Member 001b.906b.7e80 1 Ready

7 Member 001b.906b.5400 1 Ready

*8 Master 001a.e396.3900 1 Ready

9 Member 001b.906b.b480 1 Ready

Stack Port Status Neighbors

Switch# Port 1 Port 2 Port 1 Port 2

--------------------------------------------------------

1 Ok Ok 9 2

2 Ok Ok 1 3

3 Ok Ok 2 4

4 Ok Ok 3 5

5 Ok Ok 4 6

6 Ok Ok 5 7

7 Ok Ok 6 8

8 Ok Ok 7 9

9 Ok Ok 8 1

Dir flash1:

Directory of flash1:/

2 -rwx 7205024 Feb 28 1993 21:18:10 -05:00 c3750-ipbasek9-mz.122-25.SEE2.bin

3 -rwx 616 Dec 11 2011 12:13:29 -05:00 vlan.dat

5 drwx 128 Feb 28 1993 21:15:35 -05:00 c3750-ipbase-mz.122-35.SE5

6 -rwx 38838 Aug 30 2011 13:42:36 -04:00 config.text

509 -rwx 46 Aug 30 2011 13:42:37 -04:00 private-config.text

Dir flash2:

2 -rwx 7205024 Feb 28 1993 21:24:53 -05:00 c3750-ipbasek9-mz.122-25.SEE2.bin

3 -rwx 46 Aug 30 2011 13:42:38 -04:00 private-config.text

4 -rwx 38838 Aug 30 2011 13:42:38 -04:00 config.text

5 drwx 192 Feb 28 1993 19:08:17 -05:00 c3750-ipbase-mz.122-25.SEB4

363 -rwx 616 Dec 11 2011 12:13:29 -05:00 vlan.dat

Dir flash3:

Directory of flash3:/

2 -rwx 38838 Aug 30 2011 13:42:37 -04:00 config.text

3 -rwx 46 Aug 30 2011 13:42:37 -04:00 private-config.text

4 -rwx 616 Dec 11 2011 12:13:18 -05:00 vlan.dat

5 drwx 192 Feb 28 1993 20:56:02 -05:00 c3750-ipbasek9-mz.122-25.SEE2

Dir flash4:

Directory of flash4:/

2 -rwx 46 Aug 30 2011 13:42:37 -04:00 private-config.text

3 -rwx 616 Dec 11 2011 12:13:47 -05:00 vlan.dat

4 -rwx 38838 Aug 30 2011 13:42:37 -04:00 config.text

458 drwx 192 Feb 28 1993 20:59:57 -05:00 c3750-ipbasek9-mz.122-25.SEE2

Dir flash5:

Directory of flash5:/

2 -rwx 38838 Aug 30 2011 13:42:37 -04:00 config.text

3 -rwx 46 Aug 30 2011 13:42:37 -04:00 private-config.text

4 -rwx 616 Dec 11 2011 12:13:41 -05:00 vlan.dat

5 drwx 192 Feb 28 1993 21:04:20 -05:00 c3750-ipbasek9-mz.122-25.SEE2

Dir flash6:

Directory of flash6:/

2 -rwx 38838 Aug 30 2011 13:42:37 -04:00 config.text

3 -rwx 46 Aug 30 2011 13:42:37 -04:00 private-config.text

4 -rwx 616 Dec 11 2011 12:13:35 -05:00 vlan.dat

5 drwx 192 Feb 28 1993 21:08:38 -05:00 c3750-ipbasek9-mz.122-25.SEE2

Dir flash7:

XXXXmstack#dir flash7:

% Incomplete command.

Dir flash8:

Directory of flash:/

2 -rwx 46 Aug 30 2011 13:42:37 -04:00 private-config.text

3 -rwx 616 Feb 28 1993 19:01:43 -05:00 vlan.dat

5 drwx 192 Feb 28 1993 21:17:16 -05:00 c3750-ipbasek9-mz.122-25.SEE2

458 -rwx 38838 Aug 30 2011 13:42:37 -04:00 config.text

Dir of flash9:

XXXXmstack#dir flash9:

% Incomplete command.

Sh log:

The show log buffer is long since been overwritten:

Log Buffer (4096 bytes):

DOWN: Line protocol on Interface FastEthernet9/0/10, changed state to down

Dec 15 06:31:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/0/9, changed state to down

Dec 15 06:31:34: %LINK-3-UPDOWN: Interface FastEthernet9/0/12, changed state to up

Dec 15 06:31:34: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/13, changed state to up

Dec 15 06:31:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/10, changed state to up

Dec 15 06:31:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/0/9, changed state to up

Dec 15 06:31:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/12, changed state to up

Dec 15 06:31:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/0/9, changed state to down

Dec 15 06:31:43: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet5/0/9, changed state to up

Dec 15 07:28:49: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to down

Dec 15 07:28:50: %LINK-3-UPDOWN: Interface FastEthernet6/0/13, changed state to down

Dec 15 07:28:53: %LINK-3-UPDOWN: Interface FastEthernet6/0/13, changed state to up

Dec 15 07:28:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to up

Dec 15 07:29:14: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to down

Dec 15 07:29:16: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to up

Dec 15 07:29:29: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to down

Dec 15 07:29:31: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to up

Dec 15 07:29:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to down

Dec 15 07:29:40: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet6/0/13, changed state to up

Dec 15 07:32:22: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to down

Dec 15 07:32:23: %LINK-3-UPDOWN: Interface FastEthernet7/0/20, changed state to down

Dec 15 07:32:26: %LINK-3-UPDOWN: Interface FastEthernet7/0/20, changed state to up

Dec 15 07:32:27: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to up

Dec 15 07:32:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to down

Dec 15 07:32:35: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to up

Dec 15 07:32:48: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to down

Dec 15 07:32:50: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to up

Dec 15 07:32:57: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to down

Dec 15 07:32:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet7/0/20, changed state to up

Dec 15 08:45:33: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to down

Dec 15 08:45:34: %LINK-3-UPDOWN: Interface FastEthernet9/0/8, changed state to down

Dec 15 08:45:37: %LINK-3-UPDOWN: Interface FastEthernet9/0/8, changed state to up

Dec 15 08:45:38: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to up

Dec 15 08:45:39: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to down

Dec 15 08:45:41: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to up

Dec 15 08:45:54: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to down

Dec 15 08:45:56: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to up

Dec 15 08:46:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to down

Dec 15 08:46:06: %LINK-3-UPDOWN: Interface FastEthernet9/0/8, changed state to down

Dec 15 08:46:09: %LINK-3-UPDOWN: Interface FastEthernet9/0/8, changed state to up

Dec 15 08:46:10: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet9/0/8, changed state to up

Leo Laohoo · ‎12-15-2011

Errr ... You have NINE (9) switch members in a stack???? Gee whiz!

1. The "sh version" has shown that the entire stack rebooted due to power: System returned to ROM by power-on

This could mean that your power source may have been hit with a power "spike" or the power board(s) where the stack is connected to is faulty. Are the stack members connected to a single power board or power source?

2. Regarding switch member 7 and 9, can you console into them and do the command "dir flash7:" and "dir flash9:" please? I am interested to know if there are any crashinfo file(s)/directory.

3. Wow. That's an OLD IOS. Can you try to upgrade it to something more recent? Because your entire switch stack members have only 16mb of flash, the highest IOS you can go to is the 12.2(55)SE and later. 12.2(58) or 15.0(1)SE aren't supported. IF you are going to the 12.2(55)SE-range, choose the 12.2(55)SE4 as this sub-version is more mature.

4. You may need to reconsider "breaking" your stack. When dealing with 3750/3750G/3750E/3750X, my rule of thumb for the amount of stack members in a stack should not exceed 6. If you go 7 and more, some commands will cause the switch to "lag" and you'll think your switch stack has crashed. If you don't believe me, I've got a number of soiled pants to show you.

MikeCarter1 · ‎12-16-2011

leo......

Did I say thank you for your response?

The switches are not close by. We are attempting to gather the information you suggested but must wait for a tech to visit the site in order to retrieve information via console.

We have also taken to heart the suggestion of breaking the stacks up. More information to follow regarding that.

Mike

Leo Laohoo · ‎12-16-2011

Thanks Mike. Looking forward to the result of the last two switches and the inspection of the power board.