Switch port flaps up and down endlessly when PC connected go to sleep

[ahmed136846]

Dear Community,

Hope everything is great,

I am a network administrator and recently i faced a very strange behavior. In one of our customer's switches (C9300-48P-A), when the PC connected to a port goes to sleep mode, we can see in the logging that the port is flapping up and down every 1-3 minutes.

we did multiple troubleshooting steps like adjusting speed and duplex, changing the cables and changed the ISE configuration on the port with no luck.

we concluded that there is a strange behavior which happens when the PC go to sleep, eliminating any network issue. the port doesn't just goes down when the PC falls asleep, but keeps flapping up and down every few seconds, i will share the logging below.

my question here is what might be the cause of this issue, what really happens when the PC go to sleep and what should we investigate further to solve this issue.

please help. TYIA.

Logs:

Line 7286: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:57:42.371+00:00 Sep 5 2022 01:57:41.359 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7291: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:57:37.927+00:00 Sep 5 2022 01:57:36.920 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7307: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:56:05.064+00:00 Sep 5 2022 01:56:04.052 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7310: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:55:59.315+00:00 Sep 5 2022 01:55:59.303 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7340: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:53:33.070+00:00 Sep 5 2022 01:53:32.058 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7341: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:53:28.625+00:00 Sep 5 2022 01:53:27.617 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7347: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:52:23.227+00:00 Sep 5 2022 01:52:22.216 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7349: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:52:18.394+00:00 Sep 5 2022 01:52:17.385 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7384: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:49:51.318+00:00 Sep 5 2022 01:49:50.307 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7385: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:49:46.875+00:00 Sep 5 2022 01:49:45.867 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7393: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:49:16.990+00:00 Sep 5 2022 01:49:15.978 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7397: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:49:12.081+00:00 Sep 5 2022 01:49:11.073 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7432: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:46:44.795+00:00 Sep 5 2022 01:46:43.783 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7433: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:46:35.204+00:00 Sep 5 2022 01:46:34.195 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7454: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:44:08.090+00:00 Sep 5 2022 01:44:07.079 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7457: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:43:57.699+00:00 Sep 5 2022 01:43:57.688 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7483: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:41:31.353+00:00 Sep 5 2022 01:41:30.341 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7485: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:41:20.909+00:00 Sep 5 2022 01:41:19.902 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7512: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:38:53.758+00:00 Sep 5 2022 01:38:52.744 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7513: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:38:49.392+00:00 Sep 5 2022 01:38:48.383 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7519: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:38:43.642+00:00 Sep 5 2022 01:38:42.629 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7521: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:38:38.887+00:00 Sep 5 2022 01:38:37.878 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7548: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:36:11.744+00:00 Sep 5 2022 01:36:10.729 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7549: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:36:06.381+00:00 Sep 5 2022 01:36:06.367 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7551: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:36:04.180+00:00 Sep 5 2022 01:36:03.167 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7553: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:35:59.427+00:00 Sep 5 2022 01:35:58.418 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7584: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:33:32.228+00:00 Sep 5 2022 01:33:31.215 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7585: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:33:27.786+00:00 Sep 5 2022 01:33:26.777 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7586: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:33:25.368+00:00 Sep 5 2022 01:33:24.357 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7587: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:33:20.693+00:00 Sep 5 2022 01:33:19.685 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7616: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:30:53.392+00:00 Sep 5 2022 01:30:52.381 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7617: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:30:48.950+00:00 Sep 5 2022 01:30:47.941 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7621: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:30:36.570+00:00 Sep 5 2022 01:30:35.557 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7623: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:30:31.895+00:00 Sep 5 2022 01:30:30.886 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7654: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:28:04.771+00:00 Sep 5 2022 01:28:03.759 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7655: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:28:00.321+00:00 Sep 5 2022 01:27:59.312 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7657: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:27:56.970+00:00 Sep 5 2022 01:27:55.958 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7659: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:27:52.059+00:00 Sep 5 2022 01:27:51.050 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7686: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:25:24.832+00:00 Sep 5 2022 01:25:23.821 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7691: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:25:14.340+00:00 Sep 5 2022 01:25:13.770 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7715: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:22:47.547+00:00 Sep 5 2022 01:22:46.534 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7717: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:22:43.186+00:00 Sep 5 2022 01:22:42.178 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7726: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:22:10.664+00:00 Sep 5 2022 01:22:09.623 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7727: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:22:05.880+00:00 Sep 5 2022 01:22:04.872 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7751: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:19:38.737+00:00 Sep 5 2022 01:19:37.724 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7753: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:19:34.292+00:00 Sep 5 2022 01:19:33.284 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7754: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:19:32.260+00:00 Sep 5 2022 01:19:31.251 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7755: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:19:26.812+00:00 Sep 5 2022 01:19:25.803 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7786: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:16:59.762+00:00 Sep 5 2022 01:16:58.749 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7787: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:16:55.319+00:00 Sep 5 2022 01:16:54.310 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7792: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:16:23.165+00:00 Sep 5 2022 01:16:22.154 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7795: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:16:17.090+00:00 Sep 5 2022 01:16:16.702 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7820: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:13:50.483+00:00 Sep 5 2022 01:13:49.471 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7821: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:13:46.041+00:00 Sep 5 2022 01:13:45.034 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7823: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:13:42.615+00:00 Sep 5 2022 01:13:41.603 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7825: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:13:37.042+00:00 Sep 5 2022 01:13:36.864 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down
Line 7859: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:11:10.741+00:00 Sep 5 2022 01:11:09.728 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7863: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:11:06.296+00:00 Sep 5 2022 01:11:05.287 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down

[balaji.bandi]

Cisco cat 9300 what Version of IOS XE ? what ISE version you running ?

can you post-show run Interface GigabitEthernet1/0/2  

what device is this ? Windows ? or any other devices ?

[ahmed136846]

Hello,

please find the answers to your questions, i pasted the logs for port Gi2/0/10 as it's one of the flapping ports.

Cisco cat 9300 what Version of IOS XE:

Cisco IOS XE Software, Version 16.09.03
Cisco IOS Software [Fuji], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 16.9.3, RELEASE SOFTWARE (fc2)

what ISE version you running ? :

Version: 2.6.0.156

can you post-show run Interface GigabitEthernet1/0/2:

kzda30244#sh int gi2/0/10
GigabitEthernet2/0/10 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is
Description:
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 2y24w, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 292749
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 5000 bits/sec, 2 packets/sec
5 minute output rate 29000 bits/sec, 10 packets/sec
37601920 packets input, 10088105386 bytes, 0 no buffer
Received 558654 broadcasts (522147 multicasts)
0 runts, 0 giants, 0 throttles
21 input errors, 10 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 522147 multicast, 0 pause input
0 input packets with dribble condition detected
208537998 packets output, 68202190620 bytes, 0 underruns
0 output errors, 0 collisions, 7 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

what device is this ? Windows ? or any other devices ?:

windows device

Thanks

[MHM Cisco World]

21 input errors, 10 CRC, 0 frame, 0 overrun, 0 ignored
this CRC meaning there is problem in HW of NIC or cable 

[Leo Laohoo]

---

@MHM Cisco World wrote:
37601920 packets input
21 input errors, 10 CRC

---

21 input errors is insignificant compared to "37601920 packets input".  

[balaji.bandi]

is this only occurring on Cat 9300? how is your switch port config (not show interface X/x) - post-show run interface x/x

[ahmed136846]

Hello,

thanks for your reply, please find below the running configuration over the interface.

kzda30244# sh run int GigabitEthernet2/0/10
Building configuration...

Current configuration : 165 bytes
!
interface GigabitEthernet2/0/10
description DCC-B4252
switchport access vlan 210
switchport mode access
logging event link-status
spanning-tree portfast
end

[Leo Laohoo]

Update the wired NIC drivers and see if it makes any difference.

[Leo Laohoo]

@ahmed136846, 

What firmware version is the stack on and what is the uptime of the stack? 

[ahmed136846]

Hello,

please find the sh version for the switch attached.

[Leo Laohoo]

---

@ahmed136846 wrote: 

uptime is 2 years, 44 weeks, 4 days, 2 hours, 19 minutes

---

I know exactly what this is.  

There is a bug in IOS-XE, from 16.9.X up to 16.12.5 (was not fixed until 16.12.6), that we calll "ghosting". 

The flapping of an interface, alone, is more than enough to trigger this.  It first starts by silently crashing some process in the background, like SNMP and PoE, and will finally stop passing traffic to a cluster of ports. 

We call it "ghosting" because if the wired client is moved to a different "cluster" of port (or even different member of the stack), the link will go up/up.   While the wired client has been moved to a different port or switch member, the link will continuously flap like nothing has happened.  This is already a sign of ghosting.  

If a switch/stack has an uptime of >6 months on Cat 9 (>1 month for 3650/3850) this can happen very easily on affected IOS-XE version.  I have never seen this sort of behaviour in classic IOS. 

The only fix is to stay away from 16.9.X or 16.12.X.  Workaround is to reboot the entire stack.  Cold reboot preferred. 

What is more important is the use of 16.9.X.  Check the control-plane of the stack to make sure it is not affected by FN - 72323 - Cisco IOS XE Software: QuoVadis Root CA 2 Decommission Might Affect Smart Licensing, Smart Call Home, and Other Functionality.

Use the command "sh platform software status con brief" and make sure the memory is <49%.  If the memory utilization is >50%, either upgrade the software of apply the workaround.  Do not wait for the switch/stack to hit >80% or things will go nasty very quickly!

[MHM Cisco World]

@Leo Laohoo  mention that high number of input packet compare to CRC error counter is nothing, 
and he is totally right 
so I Have idea here 
normally the PC when go to sleep SW will give log error (interface go from UP to Down) but the repeat is make me think that there is wake on Lan Server or widow is config as wake on Lan, both case make PC wake then go to sleep and this make SW repeat show log message. 

Line 7286: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:57:42.371+00:00 Sep 5 2022 01:57:41.359 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to up
Line 7291: xxxxxx logs_cisco xxxxxxxxxxxx 2022-09-05T01:57:37.927+00:00 Sep 5 2022 01:57:36.920 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/2, changed state to down

[ahmed136846]

Hello,

Many thanks for your observation. Can you help with where to find such
option in the windows settings so we can advise the user to check.

[MHM Cisco World · Accepted solution]

make him disable Wake on Lan 

[ahmed136846]

Thanks, i will try this with the user and update the thread if this worked.