STACKMGR-4-SWITCH_REMOVED: Switch 2 has been REMOVED from the stack

Mark Reynolds · ‎08-24-2016

This is on a 2960x flex stack.

I have checked with our TACACS server. No one was logged in at the time of reboot. Have checked the security cameras and no one was in the IT closet. Both units have separate UPS. Other devices on #2's UPS did not bounce. Googled these forums and found a few similar events from years ago that never went anywhere on ver 12.0. Hoping I will have some more luck. I'd like some suggested commands to run or any suggestions on tracking down this problem.

Basically as the subject says, switch2 is getting randomly removed from the stack. It reboots and rejoins the stack. This has been randomly happening for 3 days so I would like to get a direction to move in. Hopefully one that doesn't include replacing all the elements of this switch stack.

Show ver:

hostname-1d uptime is 2 days, 23 hours, 36 minutes
System returned to ROM by power-on
System restarted at 13:11:46 PDT Sun Aug 21 2016

sess 2
show ver:

hostname-1d-2 uptime is 4 hours, 35 minutes
System returned to ROM by power-on
System restarted at 08:14:38 PDT Wed Aug 24 2016

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 28 WS-C2960X-24PS-L 15.0(2a)EX5 C2960X-UNIVERSALK9-M
2 28 WS-C2960X-24PS-L 15.0(2a)EX5 C2960X-UNIVERSALK9-M

8/24/2016 6:51:07 AM hostname-1d Error 208: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/28, changed state to up
8/24/2016 6:51:06 AM hostname-1d Informational 207: %USB_CONSOLE-6-MEDIA_RJ45: Console media-type is RJ45. (hostname-1d-2)
8/24/2016 6:51:02 AM hostname-1d Informational 206: %PLATFORM-6-FLEXSTACK_INSERTED: FlexStack module inserted in Switch 2. (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Notice 205: %STACKMGR-5-SWITCH_READY: Switch 2 is READ
8/24/2016 6:51:01 AM hostname-1d Notice 204: %STACKMGR-5-MASTER_READY: Master Switch 1 is READY (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Notice 203: %SYS-5-RESTART: System restarted -- (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Notice 202: %STACKMGR-5-SWITCH_READY: Switch 1 is READY (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Informational 200: %SYS-6-CLOCKUPDATE: System clock has been updated from 05:50:59 PST Wed Aug 24 2016 to 06:50:59 PDT Wed Aug 24 2016, configured from console by console. (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Notice 201: %SYS-5-CONFIG_I: Configured from memory by console (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Informational 199: %SYS-6-CLOCKUPDATE: System clock has been updated from 13:50:59 UTC Wed Aug 24 2016 to 05:50:59 PST Wed Aug 24 2016, configured from console by console. (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Notice 198: %SPANTREE-5-EXTENDED_SYSID: Extended SysId enabled for type vlan (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Notice 197: %LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0, changed state to down (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Warning 196: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been ADDED to the stack (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Warning 195: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Informational 194: %IOS_LICENSE_IMAGE_APPLICATION-6-LICENSE_LEVEL: Module name = c2960x_lanbase Next reboot level = lanbase and License = lanbase (hostname-1d-2)
8/24/2016 6:51:01 AM hostname-1d Warning 193: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 2 has changed to state UP
8/24/2016 6:51:01 AM hostname-1d Warning 192: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 2 has changed to state UP
8/24/2016 6:51:00 AM hostname-1d Notice 191: %STACKMGR-5-SWITCH_READY: Switch 2 is READY
8/24/2016 6:50:53 AM hostname-1d Warning 190: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been ADDED to the stack
8/24/2016 6:50:07 AM hostname-1d Warning 189: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state UP
8/24/2016 6:50:07 AM hostname-1d Warning 188: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 1 has changed to state UP
8/24/2016 6:46:47 AM hostname-1d Notice 187: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/28, changed state to down
8/24/2016 6:46:47 AM hostname-1d Notice 186: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to down
8/24/2016 6:46:47 AM hostname-1d Notice 185: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/18, changed state to down
8/24/2016 6:46:47 AM hostname-1d Notice 184: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/17, changed state to down
8/24/2016 6:46:46 AM hostname-1d Error 181: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/19, changed state to down
8/24/2016 6:46:46 AM hostname-1d Error 179: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/17, changed state to down
8/24/2016 6:46:46 AM hostname-1d Notice 183: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/1, changed state to down
8/24/2016 6:46:46 AM hostname-1d Error 178: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to down
8/24/2016 6:46:46 AM hostname-1d Error 182: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/28, changed state to down
8/24/2016 6:46:46 AM hostname-1d Error 180: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/18, changed state to down
8/24/2016 6:46:44 AM hostname-1d Warning 177: %STACKMGR-4-SWITCH_REMOVED: Switch 2 has been REMOVED from the stack

Jose Solano · ‎08-24-2016

Hi,

This is normally an issue with either a bad stack cable or a bad stack port, check the link below on how to address this:

Resolving StackWise Connection Problems

http://www.cisco.com/c/en/us/td/docs/switches/lan/catalyst3750/software/troubleshooting/switch_stacks.html#wp39183

HTH,

Jose

Mark Reynolds · ‎08-30-2016

So we have gone ahead and replaced both the cable and flexstack modules on both switches. The problem just showed up again today. Any advice beyond swapping the units?

Jose Solano · ‎08-30-2016

Hi,

Can you post a show version ?

Regards,

Jose

Mark Reynolds · ‎08-30-2016

cs697-1d#show ver
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2a)EX5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 16-Feb-15 08:16 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(2r)E1, RELEASE SOFTWARE (fc1)

cs697-1d uptime is 1 week, 2 days, 3 hours, 3 minutes
System returned to ROM by power-on
System restarted at 13:11:46 PDT Sun Aug 21 2016
System image file is "flash:/c2960x-universalk9-mz.150-2a.EX5/c2960x-universalk9-mz.150-2a.EX5.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-24PS-L (APM86XXX) processor (revision F0) with 524288K bytes of memory.
Processor board ID FOC1919W2D0
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
56 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : B0:7D:47:D7:8D:80
Motherboard assembly number : 73-15972-02
Power supply part number : 341-0528-01
Motherboard serial number : FOC19183NJQ
Power supply serial number : LIT19130R2H
Model revision number : F0
Motherboard revision number : C0
Model number : WS-C2960X-24PS-L
Daughterboard assembly number : 73-14200-03
Daughterboard serial number : FOC19156KJ8
System serial number : FOC1919W2D0
Top Assembly Part Number : 800-41469-01
Top Assembly Revision Number : C0
Version ID : V02
CLEI Code Number : CMMLR00ARB
Daughterboard revision number : A0
Hardware Board Revision Number : 0x12

Switch Ports Model SW Version SW Image
------ ----- ----- ---------- ----------
* 1 28 WS-C2960X-24PS-L 15.0(2a)EX5 C2960X-UNIVERSALK9-M
2 28 WS-C2960X-24PS-L 15.0(2a)EX5 C2960X-UNIVERSALK9-M

Switch 02
---------
Switch Uptime : 6 hours, 59 minutes
Base ethernet MAC Address : B0:7D:47:D7:9E:00
Motherboard assembly number : 73-15972-02
Power supply part number : 341-0528-01
Motherboard serial number : FOC19183PF0
Power supply serial number : LIT19130SPB
Model revision number : F0
Motherboard revision number : C0
Model number : WS-C2960X-24PS-L
Daughterboard assembly number : 73-14200-03
Daughterboard serial number : FOC19192B0G
System serial number : FOC1919W2D7
Top assembly part number : 800-41469-01
Top assembly revision number : C0
Version ID : V02
CLEI Code Number : CMMLR00ARB
Daughterboard revision number : A0

Configuration register is 0xF

cs697-1d-2#show ver
Cisco IOS Software, C2960X Software (C2960X-UNIVERSALK9-M), Version 15.0(2a)EX5, RELEASE SOFTWARE (fc3)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Mon 16-Feb-15 08:16 by prod_rel_team

ROM: Bootstrap program is C2960X boot loader
BOOTLDR: C2960X Boot Loader (C2960X-HBOOT-M) Version 15.2(2r)E1, RELEASE SOFTWARE (fc1)

cs697-1d-2 uptime is 7 hours, 0 minutes
System returned to ROM by power-on
System restarted at 09:16:17 PDT Tue Aug 30 2016
System image file is "flash:/c2960x-universalk9-mz.150-2a.EX5/c2960x-universalk9-mz.150-2a.EX5.bin"

This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

cisco WS-C2960X-24PS-L (APM86XXX) processor (revision F0) with 524288K bytes of memory.
Processor board ID FOC1919W2D7
Last reset from power-on
1 Virtual Ethernet interface
1 FastEthernet interface
56 Gigabit Ethernet interfaces
The password-recovery mechanism is enabled.

512K bytes of flash-simulated non-volatile configuration memory.
Base ethernet MAC Address : B0:7D:47:D7:9E:00
Motherboard assembly number : 73-15972-02
Power supply part number : 341-0528-01
Motherboard serial number : FOC19183PF0
Power supply serial number : LIT19130SPB
Model revision number : F0
Motherboard revision number : C0
Model number : WS-C2960X-24PS-L
Daughterboard assembly number : 73-14200-03
Daughterboard serial number : FOC19192B0G
System serial number : FOC1919W2D7
Top Assembly Part Number : 800-41469-01
Top Assembly Revision Number : C0
Version ID : V02
CLEI Code Number : CMMLR00ARB
Daughterboard revision number : A0
Hardware Board Revision Number : 0x12

Configuration register is 0xF

Jose Solano · ‎08-30-2016

Hi,

Thanks, there is a bug that causes silent reloads on 2960X POE units and on the last reload reason you will see "power-on".

Based on the show version you may be hitting that issue. The bug is internal thus you wont be able to see the details, ID CSCus47446. I suggest that if you have a service contract, go ahead and open a TAC case to get assistance with that and a confirmation of the bug.

HTH,

Jose

Leo Laohoo · ‎08-24-2016

System returned to ROM by power-on

Says here switch member 2 had a power reload. So it's either the following culprit:

1. System crash;

2. Power supply about to fail;

3. Human intervention.

In regards to #1, post the complete output to the command "dir flash2:".

In regards to #2, explain how power is applied to the stack.

Mark Reynolds · ‎08-30-2016

1. No crash file...
cs697-1d#dir flash2:
Directory of flash2:/

2 -rwx 1329 Jul 22 2015 13:02:21 -07:00 express_setup.debug
3 -rwx 15858 Jun 27 2016 11:38:19 -07:00 config.text
4 -rwx 1456 Aug 30 2016 02:17:37 -07:00 vlan.dat
5 -rwx 14579 Jun 27 2016 11:38:19 -07:00 config.text.backup
6 -rwx 2008 Jun 27 2016 11:38:19 -07:00 private-config.text
7 drwx 512 May 9 2015 07:50:39 -07:00 c2960x-universalk9-mz.150-2a.EX5
646 drwx 512 May 9 2015 07:50:40 -07:00 dc_profile_dir
648 -rwx 2008 Jun 27 2016 11:38:19 -07:00 private-config.text.backup
649 -rwx 1048 Nov 16 2015 22:36:37 -08:00 multiple-fs

2: Power supply about to fail :
Power supplied via UPS. UPS is running many additional elements in the network rack and does not appear overutilized as far as power goes. Power cables and power ports were swapped with flexstack cable/modules. (We have 4 seperate UPS)

3. Unit is in a locked rack in a locked room and video camera shows no human bodies at reboot time.

So probably the power supply?
I guess it is time to swap out units. Any thoughts on if I should swap the secondary first or them both at the same time?

Leo Laohoo · ‎08-30-2016

No crashinfo sub-directories so this is not a crash.

Sounds like a UPS issue, a power rail or the power supply of the switch.