Cisco C9300 Link Flapping

[cedric1104]

Hi there,

im configuring a Stack of 4 C9300-48P and now i have the problem that my uplink starts flapping after around 15-20min.
im really confused because of this behaviour and im trying to fix it since 3 weeks but it wont work.

Im using Flexoptix GBICs (SFP-25GBase-SR) 
i already did:
"no errdisable detect cause gbic-invalid"
"service unsupported-transceiver"
STP is deactivated

i hope someone of you could help me.

[M02@rt37]

Hello @cedric1104 

Please provide show inventory command output please.

Do show interface x/x/x transceiver detail whhere x/x/x is the interface where this SFP is inserted to.

Find here matrix compatibility for you platform vs SFPs: https://tmgmatrix.cisco.com/?npid=50&npid=183

[cedric1104]

Hi M02@rt37 

show inventory:
NAME: "c93xx Stack", DESCR: "c93xx Stack"
PID: C9300-48P , VID: V05 , SN:XXX

NAME: "Switch 1", DESCR: "C9300-48P"
PID: C9300-48P , VID: V05 , SN: XXX

NAME: "Switch 1 - Power Supply A", DESCR: "Switch 1 - Power Supply A"
PID: PWR-C1-715WAC-P , VID: V01 , SN: XXX

NAME: "Switch 1 FRU Uplink Module 1", DESCR: "2x25G Uplink Module"
PID: C9300-NM-2Y , VID: V01 , SN: XXX

NAME: "Twex/x/x", DESCR: "SFP-25GBase-SR"
PID: SFP-25G-SR-S , VID: V01 , SN: XXX

show interface x/x/x transceiver detail:

ITU Channel not available (Wavelength not available),
Transceiver is internally calibrated.
mA: milliamperes, dBm: decibels (milliwatts), NA or N/A: not applicable.
++ : high alarm, + : high warning, - : low warning, -- : low alarm.
A2D readouts (if they differ), are reported in parentheses.
The threshold values are calibrated.

High Alarm High Warn Low Warn Low Alarm
Temperature Threshold Threshold Threshold Threshold
Port (Celsius) (Celsius) (Celsius) (Celsius) (Celsius)
--------- ----------------- ---------- --------- --------- ---------
Twex/x/x 26.3 75.0 70.0 -5.0 -10.0

High Alarm High Warn Low Warn Low Alarm
Voltage Threshold Threshold Threshold Threshold
Port (Volts) (Volts) (Volts) (Volts) (Volts)
--------- ----------------- ---------- --------- --------- ---------
Twex/x/x 3.29 3.60 3.50 3.00 2.90

High Alarm High Warn Low Warn Low Alarm
Current Threshold Threshold Threshold Threshold
Port Lane (milliamperes) (mA) (mA) (mA) (mA)
--------- ---- --------------- ---------- --------- --------- ---------
Twex/x/x N/A 6.4 14.0 13.0 3.0 2.0

Optical High Alarm High Warn Low Warn Low Alarm
Transmit Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twex/x/x N/A 1.1 3.40 2.40 -7.60 -8.61

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twex/x/x N/A -40.0 3.40 2.40 -10.30 -11.30

[M02@rt37]

Optical High Alarm High Warn Low Warn Low Alarm
Receive Power Threshold Threshold Threshold Threshold
Port Lane (dBm) (dBm) (dBm) (dBm) (dBm)
--------- ---- --------------- ---------- --------- --------- ---------
Twex/x/x N/A -40.0 3.40 2.40 -10.30 -11.30

---

Rec signal to low... -40dbm

Check Fiber optical _ Swap with an other SFP-25G to test.

[Stefan Mihajlov]

@cedric1104 

The link flapping at 15-20 minutes in a C9300 environment is very likely to be optics/PHY issues or thermal/power issues and not an issue with STP. The switch will continue with DOM monitoring even with the service unsupported-transceiver command. I'd like to check show interface transceiver detail error, CRC/FCS, show logging for messages related to optics issues, validate Flexoptix module is specifically validated for C9300 + IOS-XE version, and reproduce with a Cisco-coded SFP module as a control. I'd also like to validate the type/length of fiber used (OM3/OM4), check connectors for cleanliness and that members show the same version of IOS-XE and there is no mixed speed and/or auto-negotiation...

[cedric1104]

@Stefan Mihajlov

i´ve already swapped out the fibre optic cable (OM4), we don´t have any Cisco branded SFPs. We only have Flexoptix. On the Flexoptix Website the SFP is listed "Compatible" We got a other stack with the same Hardware (Switches and SFPs) thats already running. 
Our IOS-XE is 17.12.05
What is the best way to check the connectors for cleanliness?
Speed is set to auto-negotiation.

[pieterh]

1) when using autonegotiation, make sure this is set at both sides!
but in my opinion for this type of links that do not change ( as a client link does) you should not use autonegotiate 
for a fibre port you also do not use MDI/MDI-X detection that depends on autonegotiation
maybe this autonegotiation is performed periodically -> set the link fixed to the maximum speed possible

2) make sure you configured the switchport as mode trunk on both sides

3) you call it a stack, but your inventory only reports switch-1 ?

about checking the fibre connectors -> the only way is with a microscope meant for this purpose
you can find many check/cleaning procedures on the internet,
this is one of my search results
https://www.chemtronics.com/how-to-clean-and-how-not-to-clean-fiber-optic-connectors?srsltid=AfmBOooO6k-lmeqNltBO9IWpQJLnoKXwQgetJWLi6iBAlo62Bpye5l4E

 

[Dustin Anderson]

Fiber can be a pain to diagnose sometimes, but if another stack is working, then it's probably a bad SFP or fiber.

From above someone pointed out low received value, so it could be an issue with the sending SFP on the other end. 

Assuming possible dirty fiber, you can use a cleaner like below for LC connectors, but options for others. There are cheaper, but depending on quality could be abrasive and scratch your fiber, so I tend to stick to name brands.

https://www.amazon.com/Jonard-Tools-FCC-125-Connector-Cleaner/dp/B0164696SU 

If you have a spare SFP, I would change the upstream one and see if that corrects it. You should be able to redo the "show interface x/x/x transceiver detail" command and hopefully not see the receive of -40 any more.

[Stefan Mihajlov]

@cedric1104 

When it comes to cleanliness, a visible inspection is not sufficient for the connector. You have to use a fiber inspection scope and one-click LC cleaner (dry cleaning) for both the patch cord and the SFP connector. New OM4 cables can also be dirty.

As you're on IOS-XE 17.12.5 and you see Flexoptix is compatible, you want to watch the values on the DOM (show interface transceiver detail), specifically Temperature and Tx Rx Power, prior to the failure. When troubleshooting, disable auto negotiation and set speed on both sides.

[cedric1104 · Accepted solution]

Hi,

sorry for letting you wait.
I solved the problem meanwhile.
It was the VLAN configuration.

On the Backbone Switch it was set to:
switchport trunk allowed vlan all
On the Access Switch it was set to:
switchport trunk allowed vlan 1-4094

i changed the vlan on the access switch to all and it works since then.
Thank you for your help!