9200L Cisco Switch flapping up and down

[KBuck]

We are seeing a lot of Down, Down, Up, Up status on our switches.  At times the Up status does no recover and we are forced to either reseat the fiber cable or power cycle a switch to get us back online.  Could you help determine what is our cause based on the below?

Errors on the switch

2023-08-01 02:23:52 Local7.Notice FMG-678Main-Cluster 55135: FMG-678Main-SW01: *Aug 1 06:10:48.391: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/3, changed state to down
2023-08-01 02:23:52 Local7.Notice FMG-678Main-Cluster 20505: FMG-678Main-SW04: *Aug 1 06:02:04.413: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/3, changed state to down
2023-08-01 02:23:54 Local7.Notice FMG-678Main-Cluster 55136: FMG-678Main-SW01: *Aug 1 06:10:50.391: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/1/3, changed state to up
2023-08-01 02:23:54 Local7.Notice FMG-678Main-Cluster 20506: FMG-678Main-SW04: *Aug 1 06:02:06.417: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet2/1/3, changed state to up

The ports are connected via fiber patch using SFP1G-LX-31 modules, we have replaced the modules and patch cables with no change.  These Down, Up notices happen throughout the day and not on regular intervals.

Attached has been included the sh run, sh int, sh int counters and sh int counter error.  All switches are at version 17.06.03.

Any help or assistance would be greatly appreciated. 

[Leo Laohoo]

Post the complete output to the following commands: 

sh interface Ten 1/1/3
sh interface Ten 2/1/3

NOTE:  Remove the IP address &/or interface description but leave everything else alone.

[paul driver]

Hello
In addition to what @Leo Laohoo  suggested:

sh int tenGigabitEthernet x/1/3 history 60min
sh controllers ethernet-controller tenGigabitEthernet x/1/3
sh platform pm interface-flaps tenGigabitEthernet x/1/3

[KBuck]

Leo, data has been pasted below, as well as attached.

Paul, some of the date you have requested has also been included.  Please note the history command ran but no output file was available.  If this is stored in another location please post how to pull for you.  The 'sh platform..' command is invalid on the 9200L switch so that one is missing from below.

Thank you both for your assistance.
FMG-678Main-SW01>
FMG-678Main-SW01>sh interface ten 1/1/3
TenGigabitEthernet1/1/3 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 98a2.c090.e633 (bia 98a2.c090.e633)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 24/255, rxload 40/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:19, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1728018
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 160330000 bits/sec, 15802 packets/sec
5 minute output rate 95562000 bits/sec, 17026 packets/sec
83979608878 packets input, 99641608240909 bytes, 0 no buffer
Received 206705436 broadcasts (106184977 multicasts)
0 runts, 0 giants, 0 throttles
95462 input errors, 53 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 106184977 multicast, 0 pause input
0 input packets with dribble condition detected
58935692647 packets output, 42710844706105 bytes, 0 underruns
Output 393792158 broadcasts (1152838289 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
FMG-678Main-SW01>

FMG-678Main-SW01>sh controllers ethernet-controller ten 1/1/3
Transmit TenGigabitEthernet1/1/3 Receive
42712952475563 Total bytes 99644952021456 Total bytes
57391925372 Unicast frames 83775583166 Unicast frames
42427732869935 Unicast bytes 99618260317950 Unicast bytes
1152857770 Multicast frames 106186167 Multicast frames
246607971655 Multicast bytes 13827644438 Multicast bytes
393799368 Broadcast frames 100522472 Broadcast frames
38611634293 Broadcast bytes 12864059068 Broadcast bytes
3 System FCS error frames 1022 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
12958548899 Minimum size frames 4376362774 Minimum size frames
11176764367 65 to 127 byte frames 5636543392 65 to 127 byte frames
5088112177 128 to 255 byte frames 6493131681 128 to 255 byte frames
2276186753 256 to 511 byte frames 2865448782 256 to 511 byte frames
2189392234 512 to 1023 byte frames 1870930303 512 to 1023 byte frames
25249578080 1024 to 1518 byte frames 62739956581 1024 to 1518 byte frames
0 1519 to 2047 byte frames 714 1519 to 2047 byte frames
0 2048 to 4095 byte frames 2 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 1022 > 32768 byte frames
0 Late collision frames 95409 SymbolErr frames
1728018 Excess Defer frames 227 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 53 FcsErr frames
0 Gold frames truncated
1 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 4724 msecs AGO

FMG-678Main-SW01>sh interface ten 2/1/3
TenGigabitEthernet2/1/3 is up, line protocol is up (connected)
Hardware is Ten Gigabit Ethernet, address is 3c26.e4b4.51b3 (bia 3c26.e4b4.51b3)
MTU 1500 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 41/255, rxload 25/255
Encapsulation ARPA, loopback not set
Keepalive not set
Full-duplex, 1000Mb/s, link type is auto, media type is 1000BaseLX SFP
input flow-control is on, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:06, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 376464
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 98777000 bits/sec, 17470 packets/sec
5 minute output rate 161438000 bits/sec, 16091 packets/sec
20196360399 packets input, 14915028466696 bytes, 0 no buffer
Received 487011822 broadcasts (362621534 multicasts)
0 runts, 0 giants, 0 throttles
6 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 362621534 multicast, 0 pause input
0 input packets with dribble condition detected
31379432682 packets output, 38967991648263 bytes, 0 underruns
Output 35018409 broadcasts (32469081 multicasts)
0 output errors, 0 collisions, 2 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

FMG-678Main-SW01>sh controllers ethernet-controller ten 2/1/3
Transmit TenGigabitEthernet2/1/3 Receive
38969098426207 Total bytes 14915681251731 Total bytes
31312812662 Unicast frames 19710280707 Unicast frames
38960816784772 Unicast bytes 14826070650021 Unicast bytes
32469861 Multicast frames 362628099 Multicast frames
4151435951 Multicast bytes 76145961346 Multicast bytes
35019212 Broadcast frames 124392699 Broadcast frames
4130210038 Broadcast bytes 13464640364 Broadcast bytes
7 System FCS error frames 0 IpgViolation frames
0 MacUnderrun frames 0 MacOverrun frames
0 Pause frames 0 Pause frames
0 Cos 0 Pause frames 0 Cos 0 Pause frames
0 Cos 1 Pause frames 0 Cos 1 Pause frames
0 Cos 2 Pause frames 0 Cos 2 Pause frames
0 Cos 3 Pause frames 0 Cos 3 Pause frames
0 Cos 4 Pause frames 0 Cos 4 Pause frames
0 Cos 5 Pause frames 0 Cos 5 Pause frames
0 Cos 6 Pause frames 0 Cos 6 Pause frames
0 Cos 7 Pause frames 0 Cos 7 Pause frames
0 Oam frames 0 OamProcessed frames
0 Oam frames 0 OamDropped frames
1289473062 Minimum size frames 4414020600 Minimum size frames
1877758707 65 to 127 byte frames 3940865956 65 to 127 byte frames
1997615089 128 to 255 byte frames 1545537445 128 to 255 byte frames
865223402 256 to 511 byte frames 715194148 256 to 511 byte frames
571311992 512 to 1023 byte frames 693539117 512 to 1023 byte frames
24778919486 1024 to 1518 byte frames 8888144246 1024 to 1518 byte frames
0 1519 to 2047 byte frames 0 1519 to 2047 byte frames
0 2048 to 4095 byte frames 0 2048 to 4095 byte frames
0 4096 to 8191 byte frames 0 4096 to 8191 byte frames
0 8192 to 16383 byte frames 0 8192 to 16383 byte frames
0 16384 to 32767 byte frame 0 16384 to 32767 byte frame
0 > 32768 byte frames 0 > 32768 byte frames
0 Late collision frames 6 SymbolErr frames
376464 Excess Defer frames 0 Collision fragments
0 Good (1 coll) frames 0 ValidUnderSize frames
0 Good (>1 coll) frames 0 InvalidOverSize frames
0 Deferred frames 0 ValidOverSize frames
0 Gold frames dropped 0 FcsErr frames
0 Gold frames truncated
1 Gold frames successful
0 1 collision frames
0 2 collision frames
0 3 collision frames
0 4 collision frames
0 5 collision frames
0 6 collision frames
0 7 collision frames
0 8 collision frames
0 9 collision frames
0 10 collision frames
0 11 collision frames
0 12 collision frames
0 13 collision frames
0 14 collision frames
0 15 collision frames
0 Excess collision frames

LAST UPDATE 3500 msecs AGO

[paul driver]

Hello
Looks like you may have a physical issue but you say you have already changed the modules, cabling etc...?
You have LX SM SFP module installed so the question is do you need SM fibre, is there large distances between interconnections on these ports requiring SM?

[KBuck]

Yes, we have replaced the modules and the fiber patch cables.  This is not a great distance, these are all in the same rack, 3m cable is long enough based on our placement.

[paul driver]

Hello
So the SM Fibre (OS cabling) & LX/LR SFPs are designed for long distances and can be problematic if  used over short distances usually preferred to MM fibre (OM) cabling and SX/SR SFPs

Can you try swapping the SFP modules  and cabling to run over MM instead?

[Leo Laohoo]

Post the complete output to the following commands: 

sh platform pm interface-flaps Te1/1/3 | begin Sw Link
sh platform pm interface-flaps Te2/1/3 | begin Sw Link