Catalyst 9200 switch port flapping (up/down, not MAC flaps)

[benweber]

I'm wondering if anyone else has seen this. I have a customer that has a number of Cayalyst 9200 switches (C9200-24T, though some are 48-ports).  I'm seeing frequent flaps where the port goes down, stays down for a few seconds, and then comes back up.  The log files look like this:

Oct 17 21:20:08.857: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Oct 17 21:20:14.242: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up
Oct 17 21:20:19.551: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Oct 17 21:20:22.577: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up

Most of the switches are on 17.9.5, though one that's causing particular problems is on 17.12.4 (which is having the same issue).  None of them are stacked as they are branch LAN switches, all running layer-2 only.

Has anyone seen this?  It's super annoying.  At one branch in particular port 1 keeps flapping, which is the port connecting to the local WAN router, so it takes the whole branch offline.

Any suggestions would be much appreciated.

Ben

[Leo Laohoo]

---

@benweber wrote:

No.of LinkDownEvents : 6
Time Stamp Last Link Flapped(U) : Oct 21 20:49:34.284
LastLinkDownDuration(sec) 3
LastLinkUpDuration(sec): 18825

---

I am not seeing any signs of Gi1/0/16 "flapping".  Let me explain this output: 

First line says the number of times the link has gone down (and up) base on the uptime of the switch.  Time is UTC.

Next line says, the last time the switch saw the port go down. 

And how long did that port stay down before going up again. 

And before that, how long was the port up (seconds).

[benweber]

Isn't that going up and down flapping?  This is from yesterday evening:

Oct 23 17:33:06.042: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Oct 23 17:33:11.386: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up
Oct 23 17:33:17.262: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Oct 23 17:33:20.361: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up

It's that going up and down inexplicably that's the problem.  I've always referred to that as flapping.

[Leo Laohoo]

Is that all it does?  Two up and two down?  After 17:33 in the afternoon? 

If that is all it does, then it is a normal behaviour when the someone logs in/logs out.  

[benweber]

You think switchports change state at layer 1 when users log in or out?

[Leo Laohoo]

Yes, it does.

[nguyenbmt106]

Probably you're hitting this bug : Cisco Bug ID: CSCus35889

You can try to apply the work around ("no power-efficient ethernet" on flapping interfaces)

[benweber]

That mentions 2960s.  Would it carry over to 9200s?

[nguyenbmt106]

This bug also happened with 9k series switches

[benweber]

gotcha.  I didn't see that.  For now (fingers crossed) it looks like the portfast command is fixing it, but if that doesn't work out I'll give this a shot.  Thanks.

[paul driver]

Hello
not sure why you wouldn’t enable portfast on alll switches edge ports  be it small or large 
if you do not you could cause a lot of topology changes occurring in the whole stp estate as/when a port transitions into a up/down state - so that means every time end devices logs onto the network - not so much in rstp but it’s required anyway.
Additionally looking again at your cfg:
you have IPDT enabled even though your using port security sticky mac addressing so do you really need to track hosts that are hard binded to specific ports?

You also have disabled fast switching which means process switching now being performed - which is a much slower switching process and higher demand on cpu resources.

[benweber]

The port security thing is really just that I'm old school.  I cut my teeth back in the late 90s when portfast could cause problems.  I know that's not really the case anymore but old habits . . .  And the switch that I flipped over to portfast last night hasn't had a single port flap since so that may well be the issue.  If it goes another 24 hours without a flap I'll be confident that's it and add it to the rest.

I'll look into the IPDT and fast switching, but since these are small branch switches that sit at the other end of 50mbps WAN connections speed isn't really an issue.  The ports really just terminate a handful of workstations, printers, and ATMs, so not much need for performance.

Thanks for the input.

[benweber]

And what's odd is that for the most problematic site my temporary workaround has been to put an old 2960 back in place.  That's been working fine despite also not having portfast configured.  So whatever this is it's specific to the 9200s.  But if portfast fixes it the 9200 then that's all I need.  I'll be happy with just making the problem go away.

[david-corral]

Hello,

Noticed the 757 unknown protocol drops.

Maybe LLDP packets not being accepted?

https://bst.cisco.com/bugsearch/bug/CSCvt35452?rfs=qvred

[benweber · Accepted solution]

Just to wrap this up it was a code bug. The switches were all on 17.9.5 and were all experiencing the problem. I have since installed 17.12.4 on about half of them. The port flapping persists on all switches I have not yet upgraded but has completely stopped on all upgraded switches.

So it's definitely not normal behavior, nor was it a configuration issue.  Purely a code bug.