File Transfer Latency on Network

[scaudle]

We have WAN transport that runs from Hawaii to SC (distance is approx 4700 miles) and rides two DISA MSPP circuits for the entire path between the sites.  We connect to a SONNET RING (15454 ONS) in SC to the final end point in the network.  The network has several Cisco 7600 series routers running in a full mesh and utilizing  BGP, IS-IS and MPLS.  The dedicated BW on the links is 100 Mbps. We are seeing network latency when we perform JPerf testing from Client\Server at Hawaii to SC (TCP Window Size set to 64).  The file transfer rates are between 400-900 Kbps which is very slow for a link with this much BW.  We are seeing file transfer rates at around 80 Mbps when performing layer 3 testing with JDSU MTS-3000 test set.

• We normally run QoS, but currently don't have any Service Policies applied anywhere
• MTU settings on all of our interfaces is set to 9000 
• ICMP\PING RTT from Hawaii to SC is 146ms

We have tried various things and have not been able to improve the file transfer through put.  Any assistance would be appreciated.   

V/r

Scottie Caudle

[Walter Astori]

Why the MTU is set to 9000 ? Do you have set jumbo frame ? The file transfer between two computer does support TCP offload ? The MTU size 9000 are set on switch ?

[scaudle]

Yes, the MTU size on the switch is also set to 9000 MTU.

Keep in mind is that all the network devices along the communication support jumbo frames. We have Jumbo frames configured to work on the ingress and egress interfaces of each device along the end-to-end transmission path.  Furthermore, all devices in the topology must also agree on the maximum jumbo frame size. If there are devices along the transmission path that have varying frame sizes, then you can also end up with fragmentation problems. Also, if a device along the path does not support jumbo frames and it receives one, it will drop it.  With that being said, we have checked with our Service Provider DISA and they say their equipment is set for Jumbo Frames.  Based on my experience, I would say that we have an issue with the SP equipment but I have to assume that that their Nodes are set up correct.

[milan.kulik]

Hi,

I was facing similar problem years ago with huge file transfers via FTP from India to Germany.

The root problem cause was the RTD (over 200 ms) of the ACK packets.

And the solution was using an advanced FTP client transferring a file within several parallel sessions running at the same time.

Possibly your problem is a similar one?

Best regards,

Milan

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

For TCP to utilize all path bandwidth, the receiver's RWIN needs to support the path's BDP (bandwidth delay product).  For 100 Mbps with 146ms RTT, you need an RWIN of about 1.74 MB.  So your 64 KB is too small.

Second, TCP slow start and (especially) packet loss recovery slows as RTT increases.  For the former, network nodes might have insufficient buffers to support the BDP.  For the latter, transient congestion, from other concurrent flows, might cause packet loss.  Can you determine if there's any packet loss during your tests?

Is your jumbo Ethernet supported end-to-end?  I.e. there's no fragmentation happening?

PS:

Milan's "solution" is an excellent one, i.e. a special FTP file utility that uses multiple concurrent FTP flows to transfer a file.

[scaudle]

Joseph,

1. We are not seeing any packet loss during our tests.

2. We have verified Jumbo Frames are set End to End.  Had DISA verify all of their router       connections at their POP.  

3. All of our WAN interface look clean, i.e. no CRCs, input errors, output errors or interface resets.

4. Based on BDP we should be seeing the following transfer rate:

File Transfer from Hawaii to Charleston

BW: 100\mbps

RTT:146 ms

WIN:64

Maximum throughput should be <= 3.51 Mbit/sec

Actual throughput <= 500-900 Kbit\sec

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Agreed that with your noted stats, maximum transfer rate should be about 3.5 Mbps.

You're file transfer is also large enough to get beyond slow start?  If so, something isn't working as it should.  You might also try a MTU settings of 1500 and 576 and see if you obtain about the same performance.