c9300-48p stack, all ports bounce at 6am and 7pm for 1-2 seconds

[JimFranklin6775]

Hello...we're having a very strange issue at one of our sites. We have the below stack of 9300s, and at 6am and 7pm every day, many ports in switches 2-5 bounce. It's very short, and the timing points to an HVAC system issue maybe? I am lost as to what would cause this. Sometimes each burst is accompanied by a LINK-3-UPDOWN port, but it's never the same port during each burst. I would greatly appreciate any ideas on this, it is too wacky. I haven't received any user complains, so I don't think it's "real".

thanks,

jim 

Switch Ports Model SW Version SW Image Mode
------ ----- ----- ---------- ---------- ----
* 1 64 C9300-48UXM 16.8.1a CAT9K_IOSXE INSTALL
2 64 C9300-48P 16.8.1a CAT9K_IOSXE INSTALL
3 64 C9300-48P 16.8.1a CAT9K_IOSXE INSTALL
4 64 C9300-48P 16.8.1a CAT9K_IOSXE INSTALL
5 64 C9300-48P 16.8.1a CAT9K_IOSXE INSTALL

Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/30, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/5, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/23, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/24, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/38, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/18, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/46, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/1, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/7, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/40, changed state to down
Sep 21 18:59:58: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/36, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/36, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/13, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/34, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/35, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/15, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/12, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/20, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/12, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/29, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/43, changed state to down
Sep 21 18:59:59: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/2, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/27, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/47, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/30, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/34, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/35, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/5, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/41, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/4, changed state to down
Sep 21 19:00:00: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/15, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/23, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/24, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/38, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/42, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/18, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/46, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/1, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/8, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/7, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/11, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/40, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/6, changed state to down
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/36, changed state to up
Sep 21 19:00:00: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/24, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/40, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/36, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/10, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/9, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/13, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/34, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/35, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/1, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/25, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/5, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/21, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/39, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/12, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/20, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/39, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/4, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/12, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/47, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to down
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/29, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/43, changed state to up
Sep 21 19:00:01: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/3, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/2, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/27, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/47, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/34, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/35, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/41, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/4, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/42, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/8, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/11, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/6, changed state to up
Sep 21 19:00:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/24, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/10, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/9, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/19, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/1, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/25, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/21, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet3/0/39, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/4, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet4/0/47, changed state to up
Sep 21 19:00:03: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet2/0/33, changed state to up
Sep 21 19:00:04: %LINK-3-UPDOWN: Interface GigabitEthernet5/0/15, changed state to up
Sep 21 19:00:05: %LINEPROTO-5-UPDOWN: Line protocol on Interface GigabitEthernet5/0/15, changed state to up

[balaji.bandi]

have not seen this issue, what is this ports, what end device it connected?

can you post-show version, show power, possible post the show run.

are you sure there no power issue around the site that time may end device rebooted all at a time due to power issue?

not sure jut giving my thoughts.

[Reza Sharifi]

Jim,

This type of issue is not usually HVAC system related. This is more like a bug in the IOS causing the ports to bounce. You may also want to open a ticket with Cisco and send them the logs for further analysis.

HTH

[Leo Laohoo]

The stack is running a re-build and a single-release-only version, 16.8.1a. 
I would recommend upgrading the firmware to something more recent (and less "funny").

[JimFranklin6775 · Accepted solution]

Hi all...thanks for the replies. During our team meeting this morning I mentioned this to our VOIP phone person and he said the phones (which are plugged into all ports) go to sleep at 7pm and wake up at 6am. Clearly they are doing something funny and the switch is reacting. And it's happening at many sites, we just didn't see it. Hopefully an upgrade from 16.8.1 will fix this. It's not affecting anything, but it does fill the logs. 

jim