Catalyst 3850 - Stack lost

raul.rico — Fri, 21 Feb 2020 15:41:40 GMT

Hello,

we have seen a strange behaviour in the stack created with two Catalyst 3850. The only evidences we could take were logs.

The problem started at 6:03. After losing stack, switch-2 had no working ports, until 10:39 when it was rebooted (AC cable unplugged from switch-2).

According to logs, stack fail cause was "stack port 1" down, but physical cable was connected all time. Is there any other reason for this port to go down?

Apr 15 06:03:02.093 CET: %STACKMGR-1-STACK_LINK_CHANGE: STANDBY:1 stack-mgr: Stack port 1 on switch 1 is down
Apr 15 06:03:02.096 CET: %STACKMGR-1-SWITCH_REMOVED: STANDBY:1 stack-mgr: Switch 2 has been removed from the stack.
Apr 15 06:03:02.262 CET: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_NOT_PRESENT)
Apr 15 06:03:02.263 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby removal (raw-event=PEER_NOT_PRESENT(3))

Apr 15 06:03:02.263 CET: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_DOWN)
Apr 15 06:03:02.263 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected standby down or crashed (raw-event=PEER_DOWN(2))

Apr 15 06:03:02.263 CET: %REDUNDANCY-3-STANDBY_LOST: Standby processor fault (PEER_REDUNDANCY_STATE_CHANGE)
Apr 15 06:03:02.263 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby removal (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
Apr 15 06:03:04.036 CET: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 2 inserted
Apr 15 06:03:04.036 CET: %PLATFORM_STACKPOWER-6-LINK_EVENT: Switch 1 stack power protocol is up on cable 2
Apr 15 06:03:05.865 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to down
Apr 15 06:03:05.873 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to down
Apr 15 06:03:05.873 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to down
Apr 15 06:03:05.873 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/5, changed state to down
Apr 15 06:03:05.873 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/6, changed state to down
Apr 15 06:03:05.943 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to down
Apr 15 06:03:05.953 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/24, changed state to down
Apr 15 06:03:06.323 CET: %LINK-3-UPDOWN: Interface Null0, changed state to up
Apr 15 06:03:06.324 CET: %LINK-3-UPDOWN: Interface Vlan124, changed state to up
Apr 15 06:03:06.325 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/1, changed state to up
Apr 15 06:03:06.326 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/3, changed state to up
Apr 15 06:03:06.328 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/4, changed state to up
Apr 15 06:03:06.329 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/5, changed state to up
Apr 15 06:03:06.330 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/6, changed state to up
Apr 15 06:03:06.331 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/7, changed state to up
Apr 15 06:03:06.332 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/8, changed state to up
Apr 15 06:03:06.333 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/9, changed state to up
Apr 15 06:03:06.334 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/10, changed state to up
Apr 15 06:03:06.336 CET: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/24, changed state to up
Apr 15 06:03:06.337 CET: %LINK-3-UPDOWN: Interface Port-channel15, changed state to up
Apr 15 06:03:06.452 CET: %LINK-5-CHANGED: Interface Vlan1, changed state to administratively down
Apr 15 06:03:06.452 CET: %LINK-5-CHANGED: Interface Vlan426, changed state to administratively down
Apr 15 06:03:07.322 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface Null0, changed state to up
Apr 15 06:03:07.323 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan124, changed state to up
Apr 15 06:03:07.323 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Apr 15 06:03:07.323 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/3, changed state to up
Apr 15 06:03:07.324 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/4, changed state to up
Apr 15 06:03:07.324 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/5, changed state to up
Apr 15 06:03:07.324 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/6, changed state to up
Apr 15 06:03:07.325 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/7, changed state to up
Apr 15 06:03:07.325 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/8, changed state to up
Apr 15 06:03:07.332 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/9, changed state to up
Apr 15 06:03:07.332 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/10, changed state to up
Apr 15 06:03:07.333 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/24, changed state to up
Apr 15 06:03:07.334 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel15, changed state to up
Apr 15 06:12:00.510 CET: %SYS-6-CLOCKUPDATE: System clock has been updated from 06:16:58 CET Sun Apr 15 2018 to 06:12:00 CET Sun Apr 15 2018, configured from NTP by 192.168.1.10.
Apr 15 10:39:02.072 CET: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 2 inserted
Apr 15 10:39:11.668 CET: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 2 inserted
Apr 15 10:39:20.749 CET: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 2 inserted
Apr 15 10:39:35.233 CET: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 1 stack power cable 1 inserted
Apr 15 10:39:35.233 CET: %PLATFORM_STACKPOWER-6-LINK_EVENT: Switch 1 stack power protocol is up on cable 1
Apr 15 10:42:17.677 CET: %PARSER-5-CFGLOG_LOGGEDCMD: User: XXX logged command:!exec: enable
Apr 15 10:43:14.380 CET: %STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr: Stack port 1 on switch 1 is up
Apr 15 10:43:17.690 CET: %STACKMGR-1-SWITCH_ADDED: 1 stack-mgr: Switch 2 has been added to the stack.
Apr 15 10:43:31.963 CET: %STACKMGR-1-SWITCH_READY: 1 stack-mgr: Switch 2 is ready.
Apr 15 10:43:52.459 CET: %PLATFORM_PM-5-UNKNOWN_REQ: Ack received for unknown request from driver process (FED). msg_type:FEC_MSG (36), Request id:2147483648
Apr 15 10:43:56.581 CET: %PLATFORM_PM-6-FRULINK_INSERTED: 4x1G uplink module inserted in the switch 2 slot 1
Apr 15 10:43:58.919 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to up
Apr 15 10:43:59.920 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/1, changed state to up
Apr 15 10:44:00.597 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Apr 15 10:44:00.757 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/7, changed state to up
Apr 15 10:44:00.908 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/5, changed state to up
Apr 15 10:44:01.071 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to up
Apr 15 10:44:01.481 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/6, changed state to up
Apr 15 10:44:01.601 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to up
Apr 15 10:44:01.757 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/7, changed state to up
Apr 15 10:44:01.868 CET: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/24, changed state to up
Apr 15 10:44:01.908 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/5, changed state to up
Apr 15 10:44:02.069 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/4, changed state to up
Apr 15 10:44:02.872 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/24, changed state to up
Apr 15 10:44:03.488 CET: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/6, changed state to up
Apr 15 10:44:32.437 CET: %NGWC_USB_CONSOLE-6-CONFIG_ENABLE: Switch 2: Console media-type changed to default
Apr 15 10:44:32.438 CET: %NGWC_PLATFORM_FEP-6-FRU_PS_OIR: Switch 2: FRU power supply A inserted
Apr 15 10:44:32.451 CET: %PLATFORM_STACKPOWER-6-CABLE_EVENT: Switch 2 stack power cable 2 inserted
Apr 15 10:44:32.451 CET: %PLATFORM_STACKPOWER-6-LINK_EVENT: Switch 2 stack power protocol is up on cable 2
Apr 15 10:45:31.971 CET: %STACKMGR-1-STANDBY_ELECTED: 1 stack-mgr: Switch 2 has been elected STANDBY.
Apr 15 10:45:32.061 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
Apr 15 10:45:32.061 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_FOUND(4))
Apr 15 10:45:32.061 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))
Apr 15 10:45:32.062 CET: %REDUNDANCY-5-PEER_MONITOR_EVENT: Active detected a standby insertion (raw-event=PEER_REDUNDANCY_STATE_CHANGE(5))

Re: Catalyst 3850 - Stack lost

johnd2310 — Wed, 02 May 2018 08:09:32 GMT

Hi,

What software version is running on the stack?

Thanks

John

Re: Catalyst 3850 - Stack lost

raul.rico — Wed, 02 May 2018 08:53:14 GMT

Hi John,

the stack is running the following version:

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
     1 32    WS-C3850-24T       03.02.03.SE       cat3k_caa-universalk9 INSTALL
     2 32    WS-C3850-24T       03.02.03.SE       cat3k_caa-universalk9 INSTALL

Re: Catalyst 3850 - Stack lost

Leo Laohoo — Wed, 02 May 2018 08:57:24 GMT

I'd upgrade the firmware first.

Re: Catalyst 3850 - Stack lost

raul.rico — Wed, 02 May 2018 09:23:31 GMT

Yes, it seems the best thing we can do right now.

One last question; is it safe to update directly from 3.2.3 SE to 3.6.8E MD? Or an intermediate version is required/recommended?

Thank you,

Re: Catalyst 3850 - Stack lost

Leo Laohoo — Wed, 02 May 2018 09:45:07 GMT

You can upgrade straight.
And don't forget to read the Release Notes.

topic Re: Catalyst 3850 - Stack lost in Network Security

Catalyst 3850 - Stack lost

Re: Catalyst 3850 - Stack lost

Re: Catalyst 3850 - Stack lost

Re: Catalyst 3850 - Stack lost

Re: Catalyst 3850 - Stack lost

Re: Catalyst 3850 - Stack lost