Re: Catalyst 3750 stack member reboots randomly

dmitrysev · ‎05-21-2009

We have a stack of 3 Catalyst 3750 (WS-C3750G-24TS) switches. The second switch in the stack started to reboot randomly several times a day, causing complete chaos on the network. There is no indication on what is going wrong. I see no crashlog file on any of the switches. The log files contain no useful information, other than the system restart and the interfaces coming back up, for example:

00:01:18: %SYS-5-RESTART: System restarted -- (CAT3750-01-2)

2w2d: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/23, changed state to up

2w2d: %STACKMGR-6-SWITCH_REMOVED: Switch 2 has been REMOVED from the stack

13:49:36: %STACKMGR-6-SWITCH_REMOVED: Switch 2 has been REMOVED from the stack

I have checked and replaced the power cables, does not seem to be a problem.

Please help.

The IOS version is 12.1(19) EA1c.

The results of sh ver:

CAT3750-01#sh ver

Cisco Internetwork Operating System Software

IOS (tm) C3750 Software (C3750-I5-M), Version 12.1(19)EA1c, RELEASE SOFTWARE (fc2)

Compiled Tue 03-Feb-04 08:18 by yenanh

Image text-base: 0x00003000, data-base: 0x0091D318

ROM: Bootstrap program is C3750 boot loader

BOOTLDR: C3750 Boot Loader (C3750-HBOOT-M) Version 12.1(14r)EA1a, RELEASE SOFTWA

RE (fc1)

CAT3750-01 uptime is 2 weeks, 2 days, 16 hours, 9 minutes

System returned to ROM by power-on

System image file is "flash:c3750-i5-mz.121-19.EA1c/c3750-i5-mz.121-19.EA1c.bin"

cisco WS-C3750G-24TS (PowerPC405) processor (revision D0) with 118776K/12288K by

tes of memory.

Processor board ID CAT0812N2GG

Last reset from power-on

Bridging software.

22 Virtual Ethernet/IEEE 802.3 interface(s)

84 Gigabit Ethernet/IEEE 802.3 interface(s)

The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.

Base ethernet MAC Address : 00:0F:8F:EA:4A:80

Motherboard assembly number : 73-7058-10

Power supply part number : 341-0045-01

Motherboard serial number : CAT0812071F

Power supply serial number : PHI074601EC

Model revision number : D0

Motherboard revision number : A0

Model number : WS-C3750G-24TS-E

System serial number : CAT0812N2GG

Hardware Board Revision Number : 0x09

Switch Ports Model SW Version SW Image

------ ----- ----- ---------- ----------

* 1 28 WS-C3750G-24TS 12.1(19)EA1c C3750-I5-M

2 28 WS-C3750G-24TS 12.1(19)EA1c C3750-I5-M

3 28 WS-C3750G-24TS 12.1(19)EA1c C3750-I5-M

Switch 02

---------

Switch Uptime : 2 hours, 53 minutes

Base ethernet MAC Address : 00:0F:8F:EA:B8:80

Motherboard assembly number : 73-7058-10

Power supply part number : 341-0045-01

Motherboard serial number : CAT08120747

Power supply serial number : PHI074601G6

Model revision number : D0

Motherboard revision number : A0

Model number : WS-C3750G-24TS-E

System serial number : CAT0812N2H0

Switch 03

---------

Switch Uptime : 11 hours, 43 minutes

Base ethernet MAC Address : 00:13:C3:35:CF:80

Motherboard assembly number : 73-7058-12

Power supply part number : 341-0045-01

Motherboard serial number : CAT0913093W

Power supply serial number : LIT0909078B

Model revision number : H0

Motherboard revision number : A0

Model number : WS-C3750G-24TS-E

System serial number : CAT0913X09X

Configuration register is 0xF

Leo Laohoo · ‎05-21-2009

So you can't find a crashinfo file even if you type the command dir flash2:?

Just as a side-note but IOS version 12.1(19)EA1c is kind of pretty ollllllllllld. You sure you still want to use this version?

dmitrysev · ‎05-22-2009

I have upgraded the stack to the latest firmware 12.2(50)SE1, the switch 2 still reboots every 2 hours. After the upgrade it lasted for almost 12 hours without the reboot. No crash files on flash1, 2 or 3.

Two interesting side effects after the firmware upgrade:

1. On the GBIC ports 25-28, the LEDs are mixed-up. For example, port 28 is connected, but the LED 28 is dark and the LED 26 is blinking. On the second switch the port 27 is connected, LED 27 is dark but the LED 25 is blinking. Traffic passes normally. â€œshow ip interface brief â€œ displays the status of the interfaces correctly. The web interface also shows the status of the ports correctly.

2. I have notice a lot of messages similar to â€œ00:06:40: %SW_MATM-4-MACFLAP_NOTIF: Host 0030.4880.6166 in vlan 10 is flapping between port Gi2/0/13 and port Gi1/0/13â€ The MAC address belongs to the server with teamed network cards. The teaming protocol is â€œStatic Link Aggregationâ€. Changing it to â€œAdaptive Load Balancingâ€ or â€œIEEE 802.3ad Dynamic Link Aggregationâ€ eliminates the message. Silly question - which one is better?

Tonight I will be replacing the switch 2 with the spare one. I have set the existing stack master to the higher priority, so it stays master with the addition of the new switch. The new switch has exactly the same firmware. Anything else I need to take care of before replacing the switch?

Thanks

glen.grant · ‎05-26-2009

Make sure the stack cables are tight , they can seem tight by hand but you may have to snug them by screwdriver to get them as tight as they should be . Also the stack cables themselves have been known to be faulty from the factory , I think there was even a TSB on this at one point.

hobbe · ‎05-26-2009

if the ports are physically mixed up I would seriously be thinking of opening a tac case.

if nothing else just to check if they are real cisco 3750s and not a bad copies.

if they are real there should not be a problem exchanging them for properly working ones since that just have to be a manufacturing issue.

thats what I would start with.

dmitrysev · ‎05-26-2009

The ports are not mixed up, only LEDs are. The ports status of the interfaces shows correctly both from CLI and web-interface. Before the firmware update LEDs were working properly. Is it the firmware itself? I was using the latest at the time version 12.2(50)SE1. I see that the newer, 12.2(50)SE2 is now available. Should I update the firmware again?

dmitrysev · ‎05-26-2009

An update. After replacing the switch in question with the spare one the reboots stopped. Must be the hardware failure. The only problem left - LED are mixed up on GBIC ports, as described in my other posting.