Re: 9300L Auto-LAGG Failure

a1c-smelrose · ‎03-26-2023

Hello guys,

At midnight UTC, our core switches (a stack of 2 x C9300L-24T-4G) experienced a failure of the Auto-LAGG process, causing all the port using it for LACP to go offline.

Mar 26 23:59:56.781: %ETC-5-L3DONTBNDL2: Gi1/0/1 suspended: LACP currently not enabled on the remote port.
Mar 26 23:59:56.800: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 26 23:59:57.423: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/2, changed state to down
Mar 26 23:59:57.796: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/1, changed state to up
Mar 26 23:59:59.058: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/1, changed state to down
Mar 26 23:59:59.059: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel3, changed state to down
Mar 26 23:59:59.474: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/13, changed state to down
Mar 27 00:00:00.061: %LINK-3-UPDOWN: Interface Port-channel3, changed state to down
Mar 27 00:00:00.266: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/14, changed state to down
Mar 27 00:00:04.387: %ETC-5-L3DONTBNDL2: Gi2/0/2 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:04.403: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:05.136: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/4, changed state to down
Mar 27 00:00:05.399: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/2, changed state to up
Mar 27 00:00:06.455: %ETC-5-L3DONTBNDL2: Gi2/0/1 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:06.997: %ETC-5-L3DONTBNDL2: Gi2/0/13 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:07.014: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:07.143: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/3, changed state to down
Mar 27 00:00:07.576: %ETC-5-L3DONTBNDL2: Gi1/0/14 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:07.595: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:07.627: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to down
Mar 27 00:00:07.632: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel4, changed state to down
Mar 27 00:00:08.009: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/13, changed state to up
Mar 27 00:00:08.590: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/14, changed state to up
Mar 27 00:00:08.640: %LINK-3-UPDOWN: Interface Port-channel4, changed state to down
Mar 27 00:00:09.607: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/3, changed state to down
Mar 27 00:00:09.608: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel7, changed state to down
Mar 27 00:00:10.611: %LINK-3-UPDOWN: Interface Port-channel7, changed state to down
Mar 27 00:00:12.028: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to down
Mar 27 00:00:12.312: %ETC-5-L3DONTBNDL2: Gi2/1/4 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:12.330: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:12.699: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/11, changed state to down
Mar 27 00:00:13.129: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to down
Mar 27 00:00:13.324: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/4, changed state to up
Mar 27 00:00:13.983: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to down
Mar 27 00:00:13.988: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel2, changed state to down
Mar 27 00:00:14.186: %ETC-5-L3DONTBNDL2: Gi2/1/3 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:14.215: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:14.939: %ETC-5-L3DONTBNDL2: Gi1/0/13 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:14.959: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:14.977: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:14.996: %LINK-3-UPDOWN: Interface Port-channel2, changed state to down
Mar 27 00:00:15.203: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/3, changed state to up
Mar 27 00:00:15.955: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/13, changed state to up
Mar 27 00:00:16.852: %ETC-5-L3DONTBNDL2: Gi1/1/3 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:16.872: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:16.894: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:17.866: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/3, changed state to up
Mar 27 00:00:19.094: %ETC-5-L3DONTBNDL2: Gi2/0/12 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:19.115: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:19.882: %ETC-5-L3DONTBNDL2: Gi1/0/11 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:19.901: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:20.111: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/12, changed state to up
Mar 27 00:00:20.414: %ETC-5-L3DONTBNDL2: Gi1/0/12 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:20.429: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:20.894: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/11, changed state to up
Mar 27 00:00:21.139: %ETC-5-L3DONTBNDL2: Gi1/0/2 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:21.160: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:21.182: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:21.426: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/12, changed state to up
Mar 27 00:00:22.155: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/0/2, changed state to up
Mar 27 00:00:23.134: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/1, changed state to down
Mar 27 00:00:23.500: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/1, changed state to down
Mar 27 00:00:23.503: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel6, changed state to down
Mar 27 00:00:23.999: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/4, changed state to down
Mar 27 00:00:24.001: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel8, changed state to down
Mar 27 00:00:24.063: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/11, changed state to down
Mar 27 00:00:24.065: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel1, changed state to down
Mar 27 00:00:24.503: %LINK-3-UPDOWN: Interface Port-channel6, changed state to down
Mar 27 00:00:25.003: %LINK-3-UPDOWN: Interface Port-channel8, changed state to down
Mar 27 00:00:25.078: %LINK-3-UPDOWN: Interface Port-channel1, changed state to down
Mar 27 00:00:25.642: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to down
Mar 27 00:00:25.644: %LINEPROTO-5-UPDOWN: Line protocol on Interface Port-channel5, changed state to down
Mar 27 00:00:26.065: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan140, changed state to down
Mar 27 00:00:26.073: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan141, changed state to down
Mar 27 00:00:26.077: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan143, changed state to down
Mar 27 00:00:26.077: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1421, changed state to down
Mar 27 00:00:26.646: %LINK-3-UPDOWN: Interface Port-channel5, changed state to down
Mar 27 00:00:29.797: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan140, changed state to up
Mar 27 00:00:29.799: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan141, changed state to up
Mar 27 00:00:29.800: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan143, changed state to up
Mar 27 00:00:29.800: %LINEPROTO-5-UPDOWN: Line protocol on Interface Vlan1421, changed state to up
Mar 27 00:00:29.901: %ETC-5-L3DONTBNDL2: Gi2/1/1 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:29.925: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:30.922: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/1/1, changed state to up
Mar 27 00:00:31.078: %ETC-5-L3DONTBNDL2: Gi2/0/11 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:31.098: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:31.120: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:31.211: %ETC-5-L3DONTBNDL2: Gi1/1/1 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:31.229: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:31.250: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:31.757: %ETC-5-L3DONTBNDL2: Gi1/1/4 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:32.091: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/11, changed state to up
Mar 27 00:00:32.223: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/1, changed state to up
Mar 27 00:00:32.853: %ETC-5-L3DONTBNDL2: Gi2/0/14 suspended: LACP currently not enabled on the remote port.
Mar 27 00:00:36.576: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:36.598: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:36.620: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:36.643: %SYS-5-CONFIG_P: Configured programmatically by process AUTO LAG Protocol from console as vty0
Mar 27 00:00:36.650: %SCHED-3-STUCKMTMR: Sleep with expired managed timer 80007F1688711AC0, time 0x22162CBC (00:00:04 ago). -Process= "AUTO LAG Protocol", ipl= 2, pid= 316
-Traceback= 1#02a3fb72f4410ca043bb895c9eef5b99 :55CFF47FD000+B31E175 :55CFF47FD000+B31DF91 :55CFF47FD000+44E96F5
Mar 27 00:00:37.572: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet1/1/4, changed state to up
Mar 27 00:00:37.600: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/14, changed state to up

Has anyone experienced anything similar before?

This is running software version 17.06.04.

marce1000 · ‎03-26-2023

- Since you are already using a (the) latest advisory release contact TAC

M.

-- ' 'Good body every evening' ' this sentence was once spotted on a logo at the entrance of a Weight Watchers Club !

a1c-smelrose · ‎03-27-2023

Thanks M, wish I could - we're a small business, so no support contract for TAC eligibility.

It seems fine again after a reload, so will just have to wait and see if it happens again.

a1c-smelrose · ‎03-28-2023

Just had the same problem again, not in the middle of the night this time, but after a couple of days online.

Without being able to contact TAC, I'm going to try a nightly reload using kron, to see if it'll help (assuming it's a bug that is only triggered after a couple of days of uptime).

kron occurrence RELOAD at 2:30 recurring
policy-list RELOAD
!
kron policy-list RELOAD
cli reload at 03:45

a1c-smelrose · ‎04-05-2023

In the end, we managed to log this with TAC, as we found out we'd got support via the new product warranty.

The `kron` stuff didn't work, as the issue occurred a few times within 24 hours of a reload.

We discovered a stack link with loads of CRC errors & on re-seating the stack cable, the CRC errors went away, as well as this issue.

It's been about 5 days since we last saw any problems, so hopefully the faulty stack link was the problem.