A stack with 3 units SG500X-24P-K9 crashes occasionally since a few weeks

Arturo Bianchi · ‎02-15-2021

Hi ,

the stack is running in a ring topology, and there appears to be some difficulty between unit 1 and unit 2; in particular, a restart of the second unit almost always occurs, which destabilizes everything; I checked the environment and the system status, I can't focus on what event triggers this sporadic reboot ... now capturing the console of unit 1 (master)...

15-Feb-2021 10:14:56 %LINK-W-Down:  gi3/1/21, aggregated (13)
15-Feb-2021 10:14:56 %STP-W-PORTSTATUS: gi2/1/11: STP status Forwarding, aggregated (4)
15-Feb-2021 10:17:34 %DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped: vlan - 1, port - gi2/1/12, mac source address - 6c:71:d9:XX:XX:XX, mac dest address - 00:15:5d:XX:XX:X, hw client address - 78:24:af:XX:XX:XX, error - DHCPSnP_handle_dhcp_snooping
15-Feb-2021 10:18:08 %LINK-I-Up:  gi3/1/21, aggregated (1)
15-Feb-2021 10:18:08 %LINK-W-Down:  gi2/1/11, aggregated (1)
15-Feb-2021 10:19:05 %MLDP-W-ABORT: Connection to Unit 2 is aborted due to Timeout.
15-Feb-2021 10:19:05 %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .
15-Feb-2021 10:19:05 %LINK-I-Up:  gi2/1/11, aggregated (1)
15-Feb-2021 10:19:05 %LINK-W-Not Present:  gi2/1/10
15-Feb-2021 10:19:05 %STP-W-PORTSTATUS: gi3/1/21: STP status Forwarding
15-Feb-2021 10:19:05 %LINK-W-Down:  gi2/1/11, aggregated (1)
15-Feb-2021 10:19:17 %CSCDLAG-W-DOWN: Stack port S1 operational status is Down
15-Feb-2021 10:19:17 %FFO-W-FFO-ACTIVE: Fast Failover is triggered
15-Feb-2021 10:19:17 %STCK SYSL-W-UNITMSG: UNIT ID 3,Msg:%CSCDLAG-W-DOWN: Stack port S2 operational status is Down

15-Feb-2021 10:19:17 %STCK SYSL-W-UNITMSG: UNIT ID 3,Msg:%FFO-W-FFO-ACTIVE: Fast Failover is triggered

15-Feb-2021 10:20:17 %2SWDMAIN-F-MSTRTMREXPR: SW2P_dist_tp_change_timer_expiry : TIMER expired on MASTER ENABLE unit

***** FATAL ERROR *****
Reporting Task: BOXS.
Software Version: 1.4.11.5 (date  08-Apr-2020 time  13:46:26)
0x16bd2c
0x16c290
0x167e50
0x801fd8
0x55ecc0
0x55f5b8
0x55f7d4
0xa867b8
0xbbf9c0
0xbbfb90
0x1223f0

**************************************************
*****************  SYSTEM RESET  *****************
**************************************************

Now I try to investigate a couple of events that happened a while before the last reboot:

%DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped
%LINK-W-Down: gi2/1/11 (but only because I see it goes up and down a little too much)

I try to investigate, but in the meantime any suggestions?

PS: in the previous reboots (about 5) there were other events in the moments before the crash, and in one case unit 3 had problems... maybe this happens to the unit that is "slave" at that moment?

Thanks.

73,
Arturo.

marce1000 · ‎02-15-2021

- Disconnect all stack-cables during a moment of downtime and reconnect. Check if that helps,

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

Arturo Bianchi · ‎02-15-2021

Thanks,

I could try but keep in mind that:

I have already cold restarted the whole stack and the problem occurred with an old firmware version, I also updated the entire stack to the latest firmware version!

I also had a difficult moment where it seemed to have a case of double brain after disconnecting the stack cables... for a moment I found a stack consisting of only unit 2 and 3 stable and I could not bring all three units back to the stack for several tens of minutes

73,
Arturo.

Martin Aleksandrov · ‎02-15-2021

Hello arturo,

1. What is your PHY connection between the stack members i.e what type of cables do you use?

2. Have you made any changes on the stack configuration prior the second unit restarts appeared?

3. Is it only the second unit restarts? How often?

4. What is the role of each unit (which unit is the master, backup and the slave)?

5. Have you enabled remote logging? If not, please set the remote loggind so you can gather all logs during the event and share.

6. It seems your stacking ports are set to LAG. Is it possible in your environment to rebuild the stack?

7. What link is the gi2/1/11? Which ports participate in the stack?

Regards,

Martin

Martin Aleksandrov · ‎02-15-2021

@Arturo Bianchi

if you are able to rebuild the stack please follow the following guide:

https://www.cisco.com/c/en/us/support/docs/smb/switches/cisco-small-business-500-series-stackable-managed-switches/smb2553-stack-settings-on-sx500-series-stackable-switches.html

Regards,

Martin

Arturo Bianchi · ‎02-15-2021

@Martin Aleksandrov wrote:

1. What is your PHY connection between the stack members i.e what type of cables do you use?

sfp+ copper cables

2. Have you made any changes on the stack configuration prior the second unit restarts appeared?

not earlier than the initial 4 crashes! Later I updated the firmware, and also found a case of double brain with unit two playing as master with a configuration that had a subtle difference...

3. Is it only the second unit restarts? How often?

nine times out of 10 is unit 2, unit 3 only once! Random, at the moment not more that1-2 times a day but enough to be uncomfortable, especially with phones!

4. What is the role of each unit (which unit is the master, backup and the slave)?

Before now, every time I checked they were like that:

1 - Master

2 - Slave

3 - Backup

I don't know if on that one occasion of the crash on unit 3, this was slave at that time ... because now after the last crash unit 2 is no longer slave!

5. Have you enabled remote logging? If not, please set the remote loggind so you can gather all logs during the event and share.

Sure ... but to be honest I didn't go to look at the remote one since I also captured the console... I'm going to see if I have anything in the syslog.... I update the thread later if I find anything useful

6. It seems your stacking ports are set to LAG. Is it possible in your environment to rebuild the stack?

Forgive me, what makes you think I'm in LAG on those restricted ports? I don't think I've ever put these ports in LAG... the only ones in LAG are g1/1/24 and g2/1/24 (LAG 1)... then I'm going to look at this too, also because I found a strange piece of configuration ... a LAG, number 2, empty and I don't understand why there is:

interface Port-channel1
 flowcontrol auto
 description "Trunk vs ASA"
 ip dhcp snooping trust
 spanning-tree link-type point-to-point
 switchport trunk allowed vlan add 100
 macro description router
 macro auto smartport type router $native_vlan 1
!
interface Port-channel2
 spanning-tree link-type point-to-point
 switchport trunk allowed vlan add 100
 macro description switch
 !next command is internal.
 macro auto smartport dynamic_type switch
!

...the second port-channel it is present but it is not indicated in any port; it also jumped at my eyes because when unit 2 passed master while I disconnected the cables, it had the same configuration without this LAG number 2 ... so I didn't understand the misalignment...

7. What link is the gi2/1/11? Which ports participate in the stack?

that port goes up and down but we can't figure out what's connected to it ... so now we've just removed the cable !!! But there shouldn't be any active PCs or peripherals in that port... only the leds flashing occasionally... no IP, no MAC from that port.

I answered long, I hope that the added things do not create too much entropy but maybe they highlight something out of place ... thank you very much!

73,
Arturo.

Arturo Bianchi · ‎02-15-2021

Hi,
this is what the syslog recorded, considering that the sending of messages was set from error up:

2021-01-24T05:41:22.500052+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:BB:CC, ip - X.Y.9.114, port - gi3/1/15, type - 1, error - lease time expired  
2021-01-25T20:56:33.606557+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:CC:DD, ip - X.Y.9.127, port - gi3/1/7, type - 1, error - lease time expired  
2021-01-26T14:04:44.877598+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-01-26T15:08:49.417311+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:DD:EE, ip - X.Y.9.136, port - gi2/1/9, type - 1, error - lease time expired  
2021-01-26T17:08:17.383353+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:CC:DD, ip - X.Y.9.127, port - gi3/1/7, type - 1, error - lease time expired  
2021-01-26T17:08:17.383353+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:EE:FF, ip - X.Y.9.117, port - gi3/1/20, type - 1, error - lease time expired  
2021-01-26T17:08:17.383399+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:FF:00, ip - X.Y.9.129, port - gi3/1/16, type - 1, error - lease time expired  
2021-01-26T17:42:25.362143+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 7c:2f:80:11:22:33, ip - X.Y.3.52, port - gi2/1/12, type - 1, error - lease time expired  
2021-01-27T15:53:36.472615+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - c8:ff:28:BB:DD:66, ip - X.Y.3.63, port - gi2/1/12, type - 1, error - lease time expired  
2021-01-27T17:18:56.461455+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 10:e9:53:22:22:88, ip - X.Y.3.57, port - gi2/1/12, type - 1, error - lease time expired  
2021-01-27T18:27:12.398291+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - ac:92:32:77:99:99, ip - X.Y.3.61, port - gi2/1/12, type - 1, error - lease time expired  
2021-01-28T10:39:59.720348+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - 4c:ed:fb:15:f1:27, ip - X.Y.9.115, port - gi1/1/14, type - 1, error - lease time expired  
2021-01-29T14:04:23.875088+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-01-29T14:12:48.040191+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-01-29T14:19:14.536306+01:00 %DHCPSNOOP-E-HDRMAC - - - %DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped: vlan - 1, port - gi3/1/13, mac source address - 70:4d:7b:BB:CC:EE, mac dest address - ff:ff:ff:ff:ff:ff, hw client address - 00:50:56:c0:00:01, error - DHCPSnP_handle_dhcp_snooping  
2021-01-29T14:19:24.917567+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-01-29T18:24:36.250446+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:BB:CC, ip - X.Y.9.114, port - gi3/1/15, type - 1, error - lease time expired  
2021-01-31T15:38:11.235115+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 70:26:05:33:44:55, ip - X.Y.3.62, port - gi3/1/24, type - 1, error - lease time expired  
2021-02-03T22:25:38.541937+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:CC:DD, ip - X.Y.9.127, port - gi3/1/7, type - 1, error - lease time expired  
2021-02-03T22:25:38.541937+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:EE:FF, ip - X.Y.9.116, port - gi3/1/20, type - 1, error - lease time expired  
2021-02-03T22:25:38.541985+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:FF:00, ip - X.Y.9.129, port - gi3/1/16, type - 1, error - lease time expired  
2021-02-05T08:50:41.362688+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 84:89:ad:63:db:ce, ip - X.Y.3.53, port - gi2/1/12, type - 1, error - lease time expired  
2021-02-05T09:07:45.378424+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 14:4f:8a:c6:46:75, ip - X.Y.3.82, port - gi2/1/12, type - 1, error - lease time expired  
2021-02-05T09:43:04.641182+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 3 reason : connection timeout. Unit will be rebooted .  
2021-02-05T09:58:57.415396+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 4c:ed:fb:15:f1:27, ip - X.Y.3.55, port - gi1/1/11, type - 1, error - lease time expired  
2021-02-05T11:41:21.274005+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - ec:8c:9a:b1:8e:1b, ip - X.Y.3.54, port - gi2/1/12, type - 1, error - lease time expired  
2021-02-06T14:08:33.380302+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 2c:fd:a1:24:1f:b9, ip - X.Y.3.80, port - gi3/1/8, type - 1, error - lease time expired  
2021-02-06T14:25:37.388914+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - d4:5d:64:63:22:5d, ip - X.Y.3.60, port - gi2/1/17, type - 1, error - lease time expired  
2021-02-06T14:42:41.361135+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 70:4d:7b:BB:CC:EE, ip - X.Y.3.69, port - gi3/1/13, type - 1, error - lease time expired  
2021-02-06T18:24:33.350364+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 5a:1f:7a:84:75:eb, ip - X.Y.3.51, port - gi2/1/12, type - 1, error - lease time expired  
2021-02-07T09:29:04.440487+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:FF:00, ip - X.Y.9.129, port - gi3/1/16, type - 1, error - lease time expired  
2021-02-09T11:32:47.261218+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 1e:d9:59:dc:a5:95, ip - X.Y.3.50, port - gi1/1/12, type - 1, error - lease time expired  
2021-02-10T10:11:35.405628+01:00 %DHCPSNOOP-E-HDRMAC - - - %DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped: vlan - 1, port - gi2/1/9, mac source address - 78:24:af:EE:DD:FF, mac dest address - ff:ff:ff:ff:ff:ff, hw client address - 6c:71:d9:DD:AA:00, error - DHCPSnP_handle_dhcp_snooping  
2021-02-10T16:05:03.341883+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-02-10T16:50:38.178160+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:CC:DD, ip - X.Y.9.127, port - gi3/1/7, type - 1, error - lease time expired  
2021-02-10T16:50:38.178160+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:EE:FF, ip - X.Y.9.113, port - gi3/1/20, type - 1, error - lease time expired  
2021-02-10T17:07:42.143024+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:BB:CC, ip - X.Y.9.114, port - gi3/1/15, type - 1, error - lease time expired  
2021-02-11T17:59:07.939947+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-02-11T18:05:39.162702+01:00 %DHCPSNOOP-E-HDRMAC - - - %DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped: vlan - 1, port - gi2/1/9, mac source address - 78:24:af:EE:DD:FF, mac dest address - ff:ff:ff:ff:ff:ff, hw client address - 6c:71:d9:DD:AA:00, error - DHCPSnP_handle_dhcp_snooping  
2021-02-11T18:05:39.162702+01:00 %DHCPSNOOP-E-HDRMAC - - - %DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped: vlan - 1, port - gi1/1/12, mac source address - 6c:71:d9:DD:AA:00, mac dest address - 00:15:5d:03:12:04, hw client address - 78:24:af:EE:DD:FF, error - DHCPSnP_handle_dhcp_snooping  
2021-02-11T19:04:59.979137+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - 7c:2f:80:88:99:00, ip - X.Y.9.120, port - gi2/1/14, type - 1, error - lease time expired  
2021-02-11T19:04:59.979137+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - 7c:2f:80:88:99:BB, ip - X.Y.9.119, port - gi2/1/13, type - 1, error - lease time expired  
2021-02-11T19:04:59.979186+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:BB:CC, ip - X.Y.9.114, port - gi3/1/15, type - 1, error - lease time expired  
2021-02-11T19:04:59.979186+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:DD:EE, ip - X.Y.9.136, port - gi2/1/9, type - 1, error - lease time expired  
2021-02-11T20:04:09.358563+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:CC:DD, ip - X.Y.9.127, port - gi3/1/7, type - 1, error - lease time expired  
2021-02-11T20:04:09.358563+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:EE:FF, ip - X.Y.9.113, port - gi3/1/20, type - 1, error - lease time expired  
2021-02-11T20:04:09.358609+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:FF:00, ip - X.Y.9.129, port - gi3/1/16, type - 1, error - lease time expired  
2021-02-11T21:24:32.490572+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-02-11T21:49:38.721929+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - 7c:2f:80:88:99:00, ip - X.Y.9.120, port - gi2/1/14, type - 1, error - lease time expired  
2021-02-11T21:49:38.721929+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - 7c:2f:80:88:99:BB, ip - X.Y.9.119, port - gi2/1/13, type - 1, error - lease time expired  
2021-02-11T21:49:38.721976+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:DD:EE, ip - X.Y.9.136, port - gi2/1/9, type - 1, error - lease time expired  
2021-02-12T09:46:25.766111+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 24:0a:64:1b:fd:ab, ip - X.Y.3.73, port - gi2/1/12, type - 1, error - lease time expired  
2021-02-12T11:28:49.746693+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 60:a4:4c:d6:ce:e2, ip - X.Y.3.72, port - gi3/1/13, type - 1, error - lease time expired  
2021-02-12T14:03:14.483293+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 70:4d:7b:46:46:68, ip - X.Y.3.74, port - gi3/1/9, type - 1, error - lease time expired  
2021-02-12T14:07:13.974348+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .  
2021-02-12T14:44:20.149879+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 100, mac - bc:cd:45:AA:CC:DD, ip - X.Y.9.127, port - gi3/1/7, type - 1, error - lease time expired  
2021-02-13T09:30:43.529971+01:00 %DHCPSNOOP-E-FILEENTRYREMD - - - %DHCPSNOOP-E-FILEENTRYREMD: DHCP Snooping entry will be removed from file: vlan - 1, mac - 78:24:af:c9:d9:0a, ip - X.Y.3.67, port - gi2/1/23, type - 1, error - lease time expired  
2021-02-15T10:12:22.579124+01:00 %DHCPSNOOP-E-HDRMAC - - - %DHCPSNOOP-E-HDRMAC: DHCP packet mac addresses verification problem - packet dropped: vlan - 1, port - gi2/1/12, mac source address - 6c:71:d9:DD:AA:00, mac dest address - 00:15:5d:03:12:04, hw client address - 78:24:af:EE:DD:FF, error - DHCPSnP_handle_dhcp_snooping  
2021-02-15T10:13:53.710638+01:00 %Stack-E-STCK-EXCEP-HNDLR - - - %Stack-E-STCK-EXCEP-HNDLR: Lost connection with unit 2 reason : connection timeout. Unit will be rebooted .

73,
Arturo.

Arturo Bianchi · ‎02-18-2021

I thought I would find some extra evidence on the console ... but despite the ring configuration connection, only a drop of the stack ports ... and why!?!?!?

Here's what the console of unit two has recorded after 3 days:

18-Feb-2021 16:10:26 %CSCDLAG-W-DOWN: Stack port S2 operational status is Down
18-Feb-2021 16:10:26 %FFO-W-FFO-ACTIVE: Fast Failover is triggered
18-Feb-2021 16:10:26 %FFO-I-FFO-COMPLETE: Fast Failover is disabled
18-Feb-2021 16:10:26 %CSCDLAG-W-DOWN: Stack port S1 operational status is Down

-----------------------------------
Unit rebooted: Stack reload request
-----------------------------------

18-Feb-2021 16:10:26 %FFO-W-FFO-ACTIVE: Fast Failover is triggered, aggregated (1)
18-Feb-2021 16:10:28 %CSCDLAG-I-UP: Stack port S1 operational status is UP
18-Feb-2021 16:10:28 %CSCDLAG-I-ACTIVE: Stack port S1 is active in stack LAG 2
18-Feb-2021 16:10:32 %Stack-W-UNIT-REBOOT: Unit rebooted: Stack reload request
18-Feb-2021 16:10:33 %MLDP-W-ABORT: Connection to Unit 1 is aborted due to Timeout.
Resetting local unit

**************************************************
*****************  SYSTEM RESET  *****************
**************************************************
Boot1 Checksum Test...............................PASS

Boot2 Checksum Test...............................PASS

Flash Image Validation Test.......................PASS



BOOT Software Version 1.4.0.02 Built  24-Jun-2014  12:01:31

                    #                               #
                   ###                             ###
                   ###                             ###
                   ###                             ###
            #      ###      #               #      ###      #
           ###     ###     ###             ###     ###     ###
    #      ###     ###     ###      #      ###     ###     ###      #
   ###     ###     ###     ###     ###     ###     ###     ###     ###
   ###     ###     ###     ###     ###     ###     ###     ###     ###
   ###     ###     ###     ###     ###     ###     ###     ###     ###
    #       #      ###      #       #       #      ###      #       #
                   ###                             ###
                   ###                             ###
                    #                               #


            #######   ###    #######       #######      #####
          #########   ###   ###    ##    #########    #########
         ###          ###    ####       ###          ###     ###
         ###          ###      ###      ###          ###     ###
         ###          ###       ####    ###          ###     ###
          #########   ###   ##    ###    #########    #########
            #######   ###    #######       #######      #####



Networking device with Marvell ARM CPU core. 256 MByte SDRAM.
I-Cache 16 KB. D-Cache 16 KB. L2 Cache 256 KB. Cache Enabled.


MAC Address   :  e0:d1:73:xx:xx:xx.

Autoboot in 2 seconds - press RETURN or Esc. to abort and enter prom.
Preparing to decompress...
 100%
Decompressing SW from image-2
 100%

OK
Running from RAM...
GPIO ID is 9
Module type is 40
Device ID 0xdd7411ab

*********************************************************************
*** Running  SW  Ver. 1.4.11.5  Date  08-Apr-2020  Time  13:46:26 ***
*********************************************************************

HW version is V06
Base Mac address is: e0:d1:73:xx:xx:xx
Dram size is  : 256M bytes
Dram first block size is  : 211968K bytes
Dram first PTR is  : 0x2D00000
Dram second block size is  : 4096K bytes
Dram second PTR is  : 0xFC00000
Flash size is: 32M
....................................................................................................................................................................................................................................................................................................................................................................................................08-Apr-2020 13:46:40 %CDB-I-LOADCONFIG: Loading running configuration.
08-Apr-2020 13:46:44 %CDB-I-LOADCONFIG: Loading startup configuration.
Device configuration:
Slot 1 - SG500X-24P
Device 0: GT_98DX4122 (BobCat)

------------------------------------
-- Unit Number 2                  --
------------------------------------

08-Apr-2020 13:46:57 %Entity-I-SEND-ENT-CONF-CHANGE-TRAP: entity configuration change trap.
08-Apr-2020 13:47:18 %INIT-I-InitCompleted: Initialization task is completed


>08-Apr-2020 13:47:27 %CSCDLAG-I-UP: Stack port S1 operational status is UP
08-Apr-2020 13:47:27 %CSCDLAG-I-ACTIVE: Stack port S1 is active in stack LAG 1
08-Apr-2020 13:47:27 %CSCDLAG-I-UP: Stack port S2 operational status is UP
08-Apr-2020 13:47:27 %CSCDLAG-I-ACTIVE: Stack port S2 is active in stack LAG 2

-----------------------------------
-- Unit Number 2  Master Enabled --
-----------------------------------

08-Apr-2020 13:47:38 %MLDP-I-SLAVE: Switching to the Slave Mode.
08-Apr-2020 13:47:38 %MLDP-I-CONNECT: Connection to Unit 1 is established.
18-Feb-2021 16:12:21 %Entity-I-SEND-ENT-CONF-CHANGE-TRAP: entity configuration change trap.
18-Feb-2021 16:12:34 %Environment-I-FAN-STAT-CHNG: FAN# 1 status changed - operational.
18-Feb-2021 16:12:34 %Environment-I-FAN-STAT-CHNG: FAN# 2 status changed - operational.
18-Feb-2021 16:12:34 %Environment-I-FAN-STAT-CHNG: FAN# 3 status changed - operational.
18-Feb-2021 17:12:44 %SYSLOG-N-LOGGING: Logging started.
18-Feb-2021 17:13:14 %FFO-I-FFO-OPER: Fast Failover is enabled

------------------ show clock ------------------
 18:40:04    Feb 18 2021
Time source is sntp
Time from Browser is disabled
------------------ show system ------------------

Unit          Type          
---- ---------------------- 
 1         SG500X-24P       
 2         SG500X-24P       
 3         SG500X-24P       

Unit   FAN       
       Status    
----   --------- 
 1        OK       
 2        OK       
 3        OK       

Unit   Temperature (Celsius)   Temperature Sensor Status 
---- ------------------------- ------------------------- 
 1              39                        OK             
 2              40                        OK             
 3              40                        OK             


Unit     Up time     
---- --------------- 
 1     03,08:19:25   
 2     00,02:29:12   
 3     06,23:24:58   

------------------ show version ------------------

       Unit             SW version         Boot version         HW version      
------------------- ------------------- ------------------- ------------------- 
         1               1.4.11.5            1.4.0.02               V06         
         2               1.4.11.5            1.4.0.02               V06         
         3               1.4.11.5            1.4.0.02               V06         

------------------ show system mode ------------------

Feature                 State
-------------------     ---------
Queues Configuration:   4 queues


------------------ show stack ------------------

Current stack mode is  : native
Topology is Ring
Stack image auto synchronization is enabled


Unit Model Name     MAC Address       Software  Master   Status
---- -------------- ----------------- --------- -------- ------
1    SG500X-24P     e0:d1:73:xx:xx:xx 1.4.11.5  Forced   master
                                                         
2    SG500X-24P     e0:d1:73:xx:xx:xx 1.4.11.5  Enabled  backup
                                                         
3    SG500X-24P     e0:d1:73:xx:xx:xx 1.4.11.5  Disabled slave 
                                                         


Unit Port  Connected  Port    Administrative Operational 
No.        Unit       Status  Speed          Speed       
---- ----- --------- -------- -------------- ----------- 
1    S1-XG 2         active   Auto           10G         
     S2-XG 3         active   Auto           10G         
2    S1-XG 3         active   Auto           10G         
     S2-XG 1         active   Auto           10G         
3    S1-XG 1         active   Auto           10G         
     S2-XG 2         active   Auto           10G         


After Reboot Configuration:

Current  New      Stack Mode         Stack Ports                        Speed
Unit No. Unit No.     
-------- -------- ------------------ ---------------------------------- -----
1        1        native             S1-S2-Xg                           Auto
2        2        native             S1-S2-Xg                           Auto
3        Auto     native             S1-S2-Xg                           Auto

73,
Arturo.

A stack with 3 units SG500X-24P-K9 crashes occasionally since a few weeks

Cisco Business Product Family

Cisco Switching Product Family