Catalyst 9200 switch port flapping (up/down, not MAC flaps)

[benweber]

I'm wondering if anyone else has seen this. I have a customer that has a number of Cayalyst 9200 switches (C9200-24T, though some are 48-ports).  I'm seeing frequent flaps where the port goes down, stays down for a few seconds, and then comes back up.  The log files look like this:

Oct 17 21:20:08.857: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Oct 17 21:20:14.242: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up
Oct 17 21:20:19.551: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to down
Oct 17 21:20:22.577: %LINK-3-UPDOWN: Interface GigabitEthernet1/0/16, changed state to up

Most of the switches are on 17.9.5, though one that's causing particular problems is on 17.12.4 (which is having the same issue).  None of them are stacked as they are branch LAN switches, all running layer-2 only.

Has anyone seen this?  It's super annoying.  At one branch in particular port 1 keeps flapping, which is the port connecting to the local WAN router, so it takes the whole branch offline.

Any suggestions would be much appreciated.

Ben

[marce1000]

 - Check port counters for that port (look at the error encounters , have they stayed on 0 or not) ?

 M.

[benweber]

Yeah, I've checked that.  They're all at zero.  And it's happening at other customer sites that have the same switches, as well as in all five locations this particular customer has.  So it's not EM interference or anything like that.  We've tried different cables.  No joy there either.

[marce1000]

  - You may find this useful : https://www.cisco.com/c/en/us/support/docs/switches/catalyst-9500-series-switches/218397-troubleshoot-port-flaps-on-catalyst-9000.html

  M.

[benweber]

Thanks.  The problem is that doc seems focused mostly on SFPs.  These switches are all just 1gig copper with all ports being native to the switch.

Here's a redacted copy of the config in case you see anything.  But they are all pretty straightforward.  These are mostly single switches in small branch offices so in all cases except one there aren't even uplink ports beyond the ones pointing to the local router.

version 17.9
service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
no platform punt-keepalive disable-kernel-core
!
hostname <redacted>
!
!
vrf definition Mgmt-vrf
!
address-family ipv4
exit-address-family
!
address-family ipv6
exit-address-family
!
logging buffered warnings
aaa new-model
!
!
aaa group server radius Radius-Server-Group
server name netmgmt
ip radius source-interface Vlan10
!
aaa authentication login default group Radius-Server-Group local
!
!
aaa session-id common
!
!
!
clock timezone EST -5 0
clock summer-time EST recurring
boot system switch all flash:cat9k_lite_iosxe.17.09.05.SPA.bin
switch 1 provision c9200-24t
!
!
!
!
!
no ip domain lookup
ip domain name <redacted>
!
!
!
login on-failure log
login on-success log
vtp version 1
!
!
!
!
!
device-tracking logging theft
!
crypto pki trustpoint SLA-TrustPoint
enrollment pkcs12
revocation-check crl
!
!
crypto pki certificate chain SLA-TrustPoint
certificate ca 01 nvram:CiscoLicensi#1CA.cer
!
license boot level network-essentials addon dna-essentials
license smart reservation
license smart transport off
memory free low-watermark processor 22870
!
diagnostic bootup level minimal
!
spanning-tree mode rapid-pvst
spanning-tree extend system-id
!
!
!
enable secret 9 $14$C2Ss$d7j3M/rr1ZEl3E$MIePZ/ple2rynLIZAY1vaP36SmS7b0IkWxedo.R074k
!
username <redacted> privilege 15 password 7 14122218050825257D7137
username <redacted> privilege 15 secret 9 $9$9jCaqwvRbl5QyE$m2vcnGIF4xXFVNaipHcwoM7tiREbRztdK0aofmtvrY.
!
redundancy
mode sso
crypto engine compliance shield disable
!
!
!
!
!
transceiver type all
monitoring
!
!
class-map match-any system-cpp-police-ewlc-control
description EWLC Control
class-map match-any system-cpp-police-topology-control
description Topology control
class-map match-any system-cpp-police-sw-forward
description Sw forwarding, L2 LVX data packets, LOGGING, Transit Traffic
class-map match-any system-cpp-default
description EWLC data, Inter FED Traffic
class-map match-any system-cpp-police-sys-data
description Openflow, Exception, EGR Exception, NFL Sampled Data, RPF Failed
class-map match-any system-cpp-police-punt-webauth
description Punt Webauth
class-map match-any system-cpp-police-l2lvx-control
description L2 LVX control packets
class-map match-any system-cpp-police-forus
description Forus Address resolution and Forus traffic
class-map match-any system-cpp-police-multicast-end-station
description MCAST END STATION
class-map match-any system-cpp-police-high-rate-app
description High Rate Applications
class-map match-any system-cpp-police-multicast
description MCAST Data
class-map match-any system-cpp-police-l2-control
description L2 control
class-map match-any system-cpp-police-dot1x-auth
description DOT1X Auth
class-map match-any system-cpp-police-data
description ICMP redirect, ICMP_GEN and BROADCAST
class-map match-any system-cpp-police-stackwise-virt-control
description Stackwise Virtual OOB
class-map match-any non-client-nrt-class
class-map match-any system-cpp-police-routing-control
description Routing control and Low Latency
class-map match-any system-cpp-police-protocol-snooping
description Protocol snooping
class-map match-any system-cpp-police-dhcp-snooping
description DHCP snooping
class-map match-any system-cpp-police-ios-routing
description L2 control, Topology control, Routing control, Low Latency
class-map match-any system-cpp-police-system-critical
description System Critical and Gold Pkt
class-map match-any system-cpp-police-ios-feature
description ICMPGEN,BROADCAST,ICMP,L2LVXCntrl,ProtoSnoop,PuntWebauth,MCASTData,Transit,DOT1XAuth,Swfwd,LOGGING,L2LVXData,ForusTraffic,ForusARP,McastEndStn,Openflow,Exception,EGRExcption,NflSampled,RpfFailed
!
policy-map system-cpp-policy
!
!
!
!
!
!
!
!
!
!
!
interface GigabitEthernet0/0
vrf forwarding Mgmt-vrf
no ip address
shutdown
!
interface GigabitEthernet1/0/1
switchport trunk allowed vlan 2-998,1000-4094
switchport mode trunk
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/2
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 00e0.0707.0860
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/3
switchport access vlan 999
switchport mode access
switchport port-security mac-address sticky
switchport port-security
shutdown
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/4
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 9c7b.efa6.a5ec
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/5
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 9c7b.efb1.1b1b
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/6
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky f439.0902.047d
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/7
switchport access vlan 20
switchport mode access
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/8
switchport access vlan 20
switchport mode access
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/9
switchport access vlan 10
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 00a3.8ec4.f2b3
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/10
switchport access vlan 999
switchport mode access
switchport port-security mac-address sticky
shutdown
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/11
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 0017.c80d.54b6
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/12
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 0020.6bf9.f372
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/13
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 7085.c21f.96ee
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/14
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 84a9.3e64.15a0
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/15
switchport access vlan 999
switchport mode access
switchport port-security mac-address sticky
switchport port-security
shutdown
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/16
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 9c7b.efb1.12f2
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/17
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 0001.2e8b.14b1
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/18
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 9c7b.efa6.a68d
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/19
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/20
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky f439.0902.02a2
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/21
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 1860.24ed.12d0
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/22
switchport access vlan 20
switchport mode access
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/23
switchport access vlan 20
switchport mode access
spanning-tree bpduguard enable
!
interface GigabitEthernet1/0/24
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 0007.9a53.3580
switchport port-security
spanning-tree bpduguard enable
!
interface GigabitEthernet1/1/1
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface GigabitEthernet1/1/2
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface GigabitEthernet1/1/3
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface GigabitEthernet1/1/4
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface TenGigabitEthernet1/1/1
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface TenGigabitEthernet1/1/2
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface TenGigabitEthernet1/1/3
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface TenGigabitEthernet1/1/4
switchport access vlan 999
shutdown
spanning-tree bpduguard enable
!
interface Vlan1
description Do Not Use!!!
no ip address
no ip route-cache
shutdown
!
interface Vlan10
description Admin Vlan
ip address <redacted> 255.255.255.0
no ip route-cache
!
ip default-gateway <redacted>
no ip http server
ip http authentication local
no ip http secure-server
ip forward-protocol nd
ip ssh version 2
!
ip access-list standard acl-SNMP-Access
10 permit <redacted>
ip access-list standard acl-VTY-Access
10 permit <redacted> log
20 permit <redacted> log
30 permit <redacted> log
!
!
logging host <redacted>
logging host <redacted>
logging host <redacted>
logging host <redacted>
!
snmp-server community <redacted> RO acl-SNMP-Access
!
radius server netmgmt
address ipv4 <redacted> auth-port 1645 acct-port 1646
key 7 000F47565D0358520B3645414A
!
!
!
control-plane
service-policy input system-cpp-policy
!
banner motd ^CCC
*************************************************************************
* Warning! This is a Private System! *
* Use by unauthorized persons is prohibited! *
* *
* Authorization from management is required to use this system. *
* *
* Your login attempt has been recorded. *
* *
*************************************************************************
^C
!
line con 0
exec-timeout 5 0
timeout login response 300
password <redacted>
login authentication CONSOLE
stopbits 1
line vty 0 4
access-class acl-VTY-Access in
exec-timeout 5 0
timeout login response 300
transport input ssh
line vty 5 15
access-class acl-VTY-Access in
exec-timeout 5 0
timeout login response 300
transport input ssh
!
!
monitor session 1 source interface Gi1/0/17
monitor session 1 destination interface Gi1/0/10
ntp server <redacted>
!
!
!
!
!
!
end

[paul driver]

Hello
Looking at your timestamps, it suggest your ports are missing their stp bpdu hellos thus losing connection.
I also do not see stp portfast enabled either globally or on any interface, which is recommended for ALL switch edge ports and especially when using RPVST  due to its reliance on portfast for rapid transition/forwarding.

[benweber]

Thanks.  I can try that.  I don't usually bother with portfast on these smaller standalone switches but no harm in putting it there.  I'll activate it on one of the switches and see if that helps.

[Leo Laohoo]

Post the complete output to the command "sh interface Gi1/0/16". 

I also want to see the complete output to the "sh interface <ROUTER PORT>" from both the router and switch interface.  

[benweber]

Sure.  Here's the sho run for the interface:

interface GigabitEthernet1/0/16
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 9c7b.efb1.12f2
switchport port-security
spanning-tree bpduguard enable
end

And the show int from the switch side going to the router:

GigabitEthernet1/0/1 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 802d.bf03.6681 (bia 802d.bf03.6681)
Description: Uplink to Wan Router
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:15, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 63000 bits/sec, 68 packets/sec
5 minute output rate 141000 bits/sec, 44 packets/sec
141873633 packets input, 58044789835 bytes, 0 no buffer
Received 2790925 broadcasts (911589 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 911589 multicast, 0 pause input
0 input packets with dribble condition detected
98611228 packets output, 37332722810 bytes, 0 underruns
Output 3659763 broadcasts (2329920 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

And on the router pointing to the switch:

GigabitEthernet0/0/1 is up, line protocol is up
Hardware is ISR4331-3x1GE, address is 98a2.c08a.3f41 (bia 98a2.c08a.3f41)
Description: to Enosburg Inside Interface
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set
Keepalive not supported
Full Duplex, 1000Mbps, link type is auto, media type is RJ45
output flow-control is off, input flow-control is off
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/375/0/0 (size/max/drops/flushes); Total output drops: 296106
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 123000 bits/sec, 37 packets/sec
30 second output rate 50000 bits/sec, 30 packets/sec
1745380758 packets input, 544133281353 bytes, 0 no buffer
Received 17652907 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 24789843 multicast, 0 pause input
2005394832 packets output, 718571085823 bytes, 0 underruns
Output 19931466 broadcasts (0 IP multicasts)
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
2 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

[Leo Laohoo]

Please provide the complete output (from the switch) to the following command: 

sh platform pm interface-flaps Gi1/0/1 | begin Sw LinkNeg State

[benweber]

Sure:

SW1#sh platform pm interface-flaps Gi1/0/1 | begin Sw LinkNeg State
Sw LinkNeg State : LinkStateUp
No.of LinkDownEvents : 0
XgxsResetOnLinkDown(10GE):
LastLinkDownDuration(sec) 1728726375
LastLinkUpDuration(sec): 836192

SW1#sh platform pm interface-flaps Gi1/0/16 | begin Sw LinkNeg State
Sw LinkNeg State : LinkStateUp
No.of LinkDownEvents : 6
XgxsResetOnLinkDown(10GE):
Time Stamp Last Link Flapped(U) : Oct 21 20:49:34.284
LastLinkDownDuration(sec) 3
LastLinkUpDuration(sec): 18825

I did it for port 16 too since that particular switch hasn't flapped on port 1.  Unfortunately I can't easily show the stats on the switch where port 1 was flapping as I worked around it by turning the uplink into a port channel on the theory that both ports would be unlikely to flap at the same time.

[Leo Laohoo]

Can we see the output of "sh interface Gi1/0/16"?

[benweber]

Sure.  That's:

interface GigabitEthernet1/0/16
switchport access vlan 20
switchport mode access
switchport port-security mac-address sticky
switchport port-security mac-address sticky 9c7b.efb1.12f2
switchport port-security
spanning-tree bpduguard enable
end

[Leo Laohoo]

That is not the output to the command I requested. 

Please post the complete output to the command "sh interface Gi1/0/16".

[benweber]

Right, sorry:

GigabitEthernet1/0/16 is up, line protocol is up (connected)
Hardware is Gigabit Ethernet, address is 802d.bf03.6690 (bia 802d.bf03.6690)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:12:49, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 690
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 77000 bits/sec, 16 packets/sec
5 minute output rate 326000 bits/sec, 37 packets/sec
11808728 packets input, 5365991754 bytes, 0 no buffer
Received 182685 broadcasts (1419 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 1419 multicast, 0 pause input
0 input packets with dribble condition detected
22763239 packets output, 9711520142 bytes, 0 underruns
Output 4365839 broadcasts (752819 multicasts)
0 output errors, 0 collisions, 2 interface resets
757 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

The switch has been up for ten days.