Ok, after hours of testing, I think this sums up the issues I'm having with fiber SFPs on my 9300L (both 24P-4G and 48P-4Gs).
It does not seem to do this with GLC-T copper SFPs just all fiber SFPS (Cisco 1000BaseLX, 1000BaseSX either GLC-LH-SMD++= or GLC-LH-SM= or GLC-SX-MMD++=).
If you power up 2 new 9300L switches disconnected, and then connect them together, they work.
The issue is that if you power up the 9300L switches connected together and they start talking before their startup configs hit, it puts their connected fiber SFP ports in a sort of limbo. Neither will talk to each other, worse yet, they will each hold this limbo until a port state change from both sides (shut/no-shut). Unplugging and re-plugging the fiber from either side also works as a state change for both.
It will do this with brand new switches with the factory config or on fully configured and baselined switches under 17.3.x or 17.6.x.
Adding "speed nonegotiate" in mix only confuses the issue, I think this is because it firsts acts like a state change if the switch is running, so it can make it work, but following a reboot it behaves differently with the bug. If all the switches have the command and they all start back up at the same time, it still doesn't work, but now even shut/no-shutting won't work until you remove the command off each side.
It can look like a solution, because if you apply it to each side and the state change makes it work, subsequent reboots will work if you only test it by rebooting one of the switches at a time since it looks like the issue happens before the configs hit, but after the ports come up, this doesn't work if they are both power cycled at the same time.
The only thing that works consistently are shutting/no-shutting both sides (which is hard to do when you can only reach one through the other). And making sure you don’t have "speed nonegotiate" on either side of the devices with the bug.
Unplugging and re-plugging one of the ends from the SFPs will always bring the link back up regardless.
I came up with this EEM script until I can find a fix for this issue. It checks for a neighbor on the configured link every 5 minutes and if none exists, it bounces the port. Applying this to both sides solves the problem when the devices in question are hours away and connected to each other. It shouldn't do anything if there is already a neighbor (other than an annoying log entry about an unknown user running enable).
no event manager applet check_neighbor_1/1/1
event manager applet check_neighbor_1/1/1
event timer watchdog time 300
action 1.0 cli command "enable"
action 2.0 cli command "show cdp neighbors GigabitEthernet1/1/1 detail | include Device ID"
action 3.0 regexp "Device ID: .*" "$_cli_result"
action 4.0 if $_regexp_result eq "0"
action 4.5 syslog priority errors msg "No neighbor found, bouncing port 1/1/1"
action 5.0 cli command "config t"
action 6.0 cli command "interface GigabitEthernet1/1/1"
action 7.0 cli command "shutdown"
action 8.0 cli command "no shutdown"
action 9.0 cli command "end"
action 9.5 end
It works for me, please don’t blame me if it doesn’t work or screws up everything in your network, you have been warned. Obviously use the correct interface name and not just “GigabitEthernet1/1/1” for your setup.
Am I the only person to experience this with the C9300L-48P-4G / C9300L-24P-4Gs?
BOOTLDR: System Bootstrap, Version 17.8.1r[FC2], RELEASE SOFTWARE (P)
17.03.06, 17.06.04, 17.06.05 - CAT9K_IOSXE - INSTALL
