Low TCP in one direction

[nzChris]

Hi there,

I need some advise, we have a new 1Gb carrier ethernet circuit between Madrid and Spain.
When performing iPerf (v3.6) TCP testing we get 950mbps from Barcelona towards Madrid but Madrid to Barcelona we get no higher than exactly 201mbps.
I can run this test a thousand times and the results will be the same, it never goes past 201mbps for the inbound direction.

Traffic flow from Barcelona to Madrid (Reverse test)	Traffic flow Madrid to Barcelona
C:\Users\iPerf\Downloads>iperf3 -c [Site B IP] -R warning: Ignoring nonsense TCP MSS 0 Connecting to host [Site B IP], port 5201 Reverse mode, remote host [Site B IP] is sending [ 5] local [Site A IP] port 62129 connected to [Site B IP] port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 107 MBytes 895 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 937 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec [ 5] 3.00-4.00 sec 112 MBytes 938 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 938 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 938 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 938 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 938 Mbits/sec [ 5] 8.00-9.00 sec 112 MBytes 937 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 938 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.04 sec 1.10 GBytes 943 Mbits/sec sender [ 5] 0.00-10.00 sec 1.09 GBytes 934 Mbits/sec receiver	C:\Users\iPerf\Downloads>iperf3 -c [Site B IP] warning: Ignoring nonsense TCP MSS 0 Connecting to host [Site B IP], port 5201 [ 5] local [Site A IP] port 62137 connected to [Site B IP] port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 22.6 MBytes 189 Mbits/sec [ 5] 1.01-2.00 sec 23.8 MBytes 200 Mbits/sec [ 5] 2.00-3.00 sec 23.6 MBytes 198 Mbits/sec [ 5] 3.00-4.00 sec 24.0 MBytes 201 Mbits/sec [ 5] 4.00-5.01 sec 23.9 MBytes 200 Mbits/sec [ 5] 5.01-6.00 sec 23.8 MBytes 201 Mbits/sec [ 5] 6.00-7.01 sec 23.8 MBytes 198 Mbits/sec [ 5] 7.01-8.00 sec 23.8 MBytes 200 Mbits/sec [ 5] 8.00-9.00 sec 23.6 MBytes 198 Mbits/sec [ 5] 9.00-10.00 sec 24.0 MBytes 201 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 237 MBytes 199 Mbits/sec sender [ 5] 0.00-10.05 sec 237 MBytes 198 Mbits/sec receiver
	It seems the TCP window is not able to negotiate higher, as above the sending iPerf server is only sending traffic at 198mbps.

We have isolated and replaced every piece of hardware/software we can at both ends but can not find a resolution.
We had our service provider put a test machine into an intermediate exchange in Madrid before the circuit goes to Barcelona and we could get 950mbps in both directions from there, so this would say our Madrid server is capable of sending the traffic.

Next we had the same test machine taken to Barcelona and placed into the exchange directly before the local tail to our Barcelona office, and here we also see asymmetric speeds 950mbps/200mbps.

So from our perspective it seems the long line is the issue.
However when we send a larger window or do parallel streams, then we can max out the test toward Barcelona.

C:\Users\iPerf\Downloads>iperf3 -c [Site B IP] -w 2m

warning: Ignoring nonsense TCP MSS 0
Connecting to host [Site B IP], port 5201
[ 5] local [Site A IP] port 62137 connected to [Site B IP] port 5201
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 98.8 MBytes 828 Mbits/sec
[ 5] 1.00-2.00 sec 113 MBytes 948 Mbits/sec
[ 5] 2.00-3.00 sec 113 MBytes 949 Mbits/sec
[ 5] 3.00-4.00 sec 113 MBytes 948 Mbits/sec
[ 5] 4.00-5.00 sec 113 MBytes 948 Mbits/sec
[ 5] 5.00-6.00 sec 113 MBytes 949 Mbits/sec
[ 5] 6.00-7.00 sec 113 MBytes 948 Mbits/sec
[ 5] 7.00-8.00 sec 113 MBytes 949 Mbits/sec
[ 5] 8.00-9.00 sec 113 MBytes 948 Mbits/sec
[ 5] 9.00-10.00 sec 113 MBytes 948 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.00 sec 1.09 GBytes 936 Mbits/sec sender
[ 5] 0.00-10.02 sec 1.09 GBytes 935 Mbits/sec receiver

So our service provider say the circuit is ok because they can see the full load of traffic when the window is increased.
It doesn't make sense to me that one direction can negotiate a larger window and utilise the link, but the other direction won't go higher than 200mbps without manipulation. 

Solarwinds shows the Barcelona office has low inbound traffic (peaks at 150mbps) and higher outbound traffic (peaks at 400-600mbps). Our users are complaining of slow speed and when desktop copying files between both ends we see one direction fast and the other is roughly four times slower no matter what time of day.

Anyone ideas? is our logic correct? should we expect to see traffic the same in both directions without having to increase the window size or send parallel streams?


I have posted this question before but I have been able to narrow the scope since then so I wanted to start the thread fresh.

[balaji.bandi]

What device the links connected  between these location

Device1 ------NTU---------(MPLS / Carrier Network)----NTU ---Device 2

Device 1 and Device 2 ( what model and what IOS code they running)

Please also provide connected interface config.

is the Iperf you testing like below or any other network device in the path ?

PC--Device1 ------NTU---------(MPLS / Carrier Network)----NTU ---Device 2--PC

This could be a MTU issue, But only confirmed when we see all the necessary information provided to us.,

[nzChris]

Okay here is the circuit path:

iPerf Server -> Nexus 9k -> 10Gb interface -> Carrier Network -> Test machine in the POP

Both ends are running Windows 10 and iPerf version 3.6
The 10gb interface terminates multiple circuits from our ISP on different VLANs, they strip of the tag before sending to the far end device.

The iPerf server is plugged into a port with only switchport enabled and a vlan set.

WAN interface:
speed 10000
bandwidth 10000000

.sub interface
bandwidth 1000000
encapsulation dot1q x
ip address x.x.x.x/30

The far end is terminated on our service providers switch in an exchange so I don't have the config.

But we have the same result when we have the far end terminating at our other site in Barcelona.

From the iPerf servers at each end I can successfully ping the opposite end with the DF bit set and a length of 1472, any higher and there is need to fragment.

[balaji.bandi]

WAN interface:
speed 10000   <---- ???
bandwidth 10000000

what is this device ?

Carrier Network ->(you get Ethernet Delivery) which directly connent to PC ? Test machine in the POP

Test Machine what Speed you see when it negotiated with Interface ? 10GB ? 1GB ? 100MB ?

what is the Service provider Bert Test results ? end to end ?

[nzChris]

At our Madrid end the circuit terminated in a 10gb LR SFP in a Nexus 9k.
The remote end I don't have the port configuration as this was setup for a test by the service provider.

Ive been doing some iPerf testing with packet capturing and have noticed the following:

BCN to MAD = receive window advised by MAD builds up to 16,000,000
MAD to BCN = receive window advised by BCN office stays at 4,374

And at the Sending end in Madrid, every 7th message is a "TCP Previous segment not captured", see attachment.

[nzChris]

To add to my previous post, we have a 10ms ping to the remote office.
And here is what the advertised window size looks like when received.

Window size value: 4374
[Calculated window size: 4374]
[Window size scaling factor: -1 (unknown)] ?