Wavelength 10G fiber speed issues - Cisco SG350XG-24T

[pbnjAdmin]

Hi All,

Looking for a bit of help with a networking issue I’ve been battling with for a few months now.

I have been trying to connect 2 different sites over a Wavelength unmanaged 10G single mode fiber connection. The fiber connects on both sides to a transceiver (https://www.amazon.ca/gp/product/B01D45RYL0/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1) that is plugged into one of the SFP+ port in a Cisco SG350XG-24T.SG350XG-24T, Fiber, SFP

The connection is up and stable but the speeds we experience are no where near 10G. When using a 10G Nic on a workstation connected to a 10G ethernet port on the switch using a CAT6.a cable, I can send traffic at 10G speed from Site 1 to Site 2, but Site 2 can only send traffic to Site 1 at 60MB/s. Also, 10G devices in Site 1 can communicate on full 10G with each other and 10G devices in Site 2 can communicate on full 10G with each other. This would lead me to believe that the NICs on the workstations/servers are not the issue. It also proved to me that the ethernet ports on each switch is capable of full 10G speed.

Other things that I have tried is turning off auto-negotiate on the Wavelength SFP+ connection port but it doesn’t seem to make a difference. I have also tried using an SFP-to-ethernet converter on both sides and found that when Site 1 have a workstation plugged directly into the converter and Site 2 had the fiber plugged into the switch at Site 2, I could get 10G traffic in both directions. With the setup reversed, the converter now plugged directly into the workstation at Site 2 and the fiber plugged into the Switch at Site 1, I could only get 10G speed from Site 1 to Site 2. Site 1 seems to only accept speed of about 60MB/s when traffic flows through the switch at Site 1.

We have also tried different transceivers (non-Cisco ones), but they do not improve the situation. I have compared the SFP+ port config on the switch in Site 1 and 2 and they seem identical to me. Not sure where else to go from here so any help would be appreciated!

Another note is that the Site 1 and Site 2 are in a stacking config. Site 1 has 2 SG350XG-24T switches stacked and Site 2 has 2 SG350XG-24T switches. These are stacked using the other remaining SFP+ port on the switch of unit 1

Thanks in advance

[Joseph W. Doherty]

Are you familiar with LFN issues?  Maybe an issue?

[pbnjAdmin]

Thanks for your reply.

Yes it looks like it could be LFN. UDP speeds are hitting 10G over the Network no problem.

From my research I have to increase the TCP window scaling on the switch but having a hell of a time trying to do that (Cisco phone support have been pretty poor)

[Joseph W. Doherty]

You want to increase RWIN on receiving host(s), not the switch (which, BTW, should be possible, but it won't help you unless the switch is the receiving host, i.e. not a transit device).

[pbnjAdmin]

So my testing is between 2 Windows 10 devices. Are you recommending increasing the RWIN on the Windows PCs? 

[Joseph W. Doherty]

Ah, between Windows hosts, much depends on the actual Windows version and variant being used and what options are enabled.

I'm not current in different Windows and other OS host technologies, but for TCP, the only network transit device settings that usually matter are interface buffer settings.

Anyway, here's a couple of links to get you "into" Windows TCP options and considerations, Speedguide Windows 8/10 TCP  and MS tech community blog.

BTW, unless there's been some new breakthrough in TCP for LFNs, even the "best" often have throughput issues if there are ANY in-flight drops.  One method to deal with LFNs, I've seen work well in the past, for high volume data transfers, is some app that slices and dices a data transfer using multiple concurrent TCP streams.