Slow FTP transfer

[ITSS Network]

Hi,

We have a network between two locations. WAN is 100 Mbps MPLS provided by service provider. There is a FWSM module on core switches.

Problem we are facing is FTP file transfer between two servers across WAN link never crosses 20 Mbps. Link is 100 Mbps. However if we do parallel file tranfers each transfer is 20 Mbps . But we dont get transfer rate above 20 Mbps in a single FTP session. Is there any bottle neck where traffic is getting restricted.

Traffic between two mservers passes through 100Mbps MPLS WAn link , 6500 core switch and FWSM module in switch.

Thanks

KP

[Joseph W. Doherty]

Disclaimer

The   Author of this posting offers the information contained within this   posting without consideration and with the reader's understanding that   there's no implied or expressed suitability or fitness for any purpose.   Information provided is for informational purposes only and should not   be construed as rendering professional advice of any kind. Usage of  this  posting's information is solely at reader's own risk.

Liability Disclaimer

In   no event shall Author be liable for any damages whatsoever (including,   without limitation, damages for loss of use, data or profit) arising  out  of the use or inability to use the posting's information even if  Author  has been advised of the possibility of such damage.

Posting

TCP will self limit its transmission rate if the receiver's RWIN is not at least sized for the BDP (bandwidth delay product).

If for example, your latency was 100 ms, for 100 Mbps you would need 100,0000,000 * .1 / 8 bytes (1.25 MB).

[ITSS Network]

Thanks Joseph !

I like to understand further on this. By receiver do you mean the server or the router/firewall. We are using FTP on AIX servers.

Regards

KP

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I like to understand further on this. By receiver do you mean the server or the router/firewall. We are using FTP on AIX servers.

Actual host system that is receiving the FTP.

If you google BDP, you'll find more info about this, and you can also google for information on setting various host OSs.

BTW, suspect what Milan described was the same issue.  Even with a properly set RWIN, multiple FTP streams will likely transfer the data faster because parallel stream will each ramp-up (TCP slow start).  This is especially noticeable if a particular file is small as even with ideal settings the flow might not ramp up to full speed before the transfer is completed.

[milan.kulik]

Hi Joseph,

I believe multiple FTP streams might also be faster in a case of line error recovery under some conditions (like SACK not supported by the FTP client).

BR,

Milan

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

I believe multiple FTP streams might also be faster in a case of line error recovery under some conditions (like SACK not supported by the FTP client).

SACK should be a function of the TCP stack, not the application, but yes multiple FTP stream would recover faster, with or without SACK, again because each one does recovery in parallel.  Further, with multiple streams not all may be in recovery from packet lost at the same time or at the same flow rate state.

PS:

There are a couple of file transfer packages, I believe, that will automatically spawn off concurrent flows for a file transfer.  There are others that will try to optimize the transfer, in various other ways such as using compression and/or data block deltas.  Some might use multiple techniques.

In other words, although adjusting the receiving host's RWIN might allow a single TCP flow to utilize all the bandwidth, on an LFN, other techniques for file transfer can be much more optimal.

[milan.kulik]

Hi,

you possibly reached an FTP rate limit caused by the ACK rount trip delay.

I remember similar problem some years ago when we were transferring large files from Germany to India via FTP.

A solution that time was using a multi-thread FTP client.

HTH,

Milan

[David Santos]

My advice just for testing purposes is to generate UDP traffic among sites and use a QoS scheme. You can see if there is any congestion in the network.

Beside this I must leave here a link for a perfect explanation of BDP:

http://bradhedlund.com/2008/12/19/how-to-calculate-tcp-throughput-for-long-distance-links/

[MaciejWlazlinski]

Hello ITSS Network,

Recently I have encountered the same error, making long story short everyone was saying that this is MTU issue or the problem with my MPLS link (dropping packets do to QoS policies) or firewall issue... and all that kind of nonsense.

In the end the problem is super slow flash card which Cisco is putting to 2960X or 3850 switches what is absolutely sad(especially that it is just 2 bucks more), so you can do nothing about it.

(how it was tested:

-switch and host where in the same subnet, downloading from the same FTP FileZilla server, over the same MPLS line with no extra qos, linux client was cable to download with the speed couple of tens time faster the the switch)

Cheers!

[jirochaguerra]

Hello Guys!

I have the same issue, but I am waiting for the ISP response, because when I do a ping 172.16.1.1 size 1497 it's present issues but It's only in one direction.

On the other way, It's possible use ping 172.16.1.2 size 17999 and ping is succesfully.

Don't down the arms!

See you later with the solution!

Regards!

[Joseph W. Doherty]

Cannot say for sure, but I once dealt with a WAN vendor, who had MTU screwed up with MPLS.  Had packet drops when packet sized 1497 to 1500 (MPLS label is 4 bytes).

[Joseph W. Doherty]

Yes, the flash, on many Cisco devices is "slow" when writing to it, but it's not purported to be SSD.

Also, depending on how recent your Linux client is, they support some of the latest/best TCP stacks.  Likely it's more "efficient" or better performing stack then found on Cisco network devices, which are also not designed to be high performance network hosts.

Lastly, I recall many Cisco network devices have a rather small default RWIN (4K ?) and/or have PMTUD off, again by default.  Both might be changed via device configuration.

Again, although the Cisco flash is likely to be a bottleneck, when doing something like obtaining a full Internet route table (BGP using TCP), how long the initial transfer takes may drop much if you increase RWIN (e.g. 64K) and enable PMTUD (MTU jumps from 576, often to 1500).  (As routes go to RAM, it avoids the flash bottleneck.)

[MaciejWlazlinski]

I have HP Procurve 2920 in the same location and transfer over ftp is lightning fast, I do not expect switch to have build in SSD but putting the cheapest the worst flash to business dedicated hardware is really a wrong way to go

[Joseph W. Doherty]

No doubt your HP Procurve has better FTP to (or from) it, but when you look at performance/benchmark studies of switches, data transfer rates to or from the switch are generally not considered.  Data transfer rates through the switch are.  Additionally, when "shopping" switches, other features, are often considered important, whether something like MACSEC, AAA, and so forth.

I.e. I'm sure Cisco could provide faster FTP transfer rate to/from their switches, but that doesn't appear to be what they market.

If FTP transfer rates to/from your switch are most important to you, then you make a good case why you might want to buy another vendor's switch.

BTW, although I believe you're correct there might be little cost difference in flash cards that are faster, I'm not so sure that's also true of the supporting circuitry.  Of course, for what a Cisco asks for in pricing, you would expect to get the best, but again, I think their focus on "what's the best" does not include data rates to/from the switch.

[MaciejWlazlinski]

Joseph,

With all due respect, are you part of presales department? - this is not the point of this discussion, the problem is slow FTP transfer, and it is not doubt that Cisco switches are light years ahead of the competition (99% or our infrastructure is based on Cisco),

This forum is to point out what could be done better, and this particular part of the switch (regardless if it is software or hardware fault or a mix of both) literally sucks

Cisco above all has always known, that devil is in the details and that they should keep every part of their devices to be a masterpiece in the class.