Transferring Large Data Volume Advice

[Mokhalil82]

Hi

We want to send 500gb worth of data to a 3rd party that currently sits on our SFTP server. From what I can see this will exhaust our network as well as take a long time as the smallest link between point A & B is 100meg. 

Also I don't want the transfer over the network to affect any other traffic. What would you advise is the best way to get this data over to them? USB??

Thanks

[Seb Rupik · Accepted solution]

Hi there,

Why don't you use QoS? Mark the traffic as it leaves the server and restrict its bandwidth as it traverses your 100Mb link.

What switches/ routers do you have at either end of the 100Mb link?

cheers,

Seb.

[Mokhalil82]

Hi


The 100meg over 1G is the internet pipe that is terminated on a cisco 2960 switch which connects into a 5585 firewall.


If the 3rd party was to directly fetch the data from the sftp server this is the bottleneck from our end. Even if we did QOS I take it would take some considerable time for them to get all the data?

[Seb Rupik]

Do you know what your utilization of the 100Mb link is? if it is fully saturated then you will have a hard time allocating a reasonable amount of bandwidth for your transfer. Any continuous spare capacity in the link (bursty traffic withstanding) could be assumed safe to allocate to the SFTP transfer in the form of QoS policing.

The amount time the transfer will take will be dependent on how much you can allocate, but surely that is better than a box of USB sticks? :)

cheers,

Seb.

[Mokhalil82]

The link is actually under utilized at 40% max with the odd burst now and again. So based on that QOS policing is the better option?

[Mokhalil82]

Is it simpler to shape the traffic at the interface where the server is connected so that it shapes down to say 30mbps?

[Seb Rupik · Accepted solution]

Normally you would apply your marking as close to the source as possible and the shaping/ policing at ingress to the congested links.

There is no harm in doing the marking and shaping/ policing of packets as soon as they hit the network.

cheers,

Seb.

[Mokhalil82]

Thnakyou appreciate your advice

[Joseph W. Doherty]

Other posters have mentioned QoS in context of policing and/or shaping, but ideally, what you might want to do is use QoS to deprioritize the bulk data transfer so if only uses "left over" bandwidth and yields to other traffic.  I.e. placing such bulk traffic in a "scavenger" class.

Such an approach insures this traffic transfer has little impact against your other traffic.  The only question would be whether your usual "left over" bandwidth is enough to allow the data transfer to take place in a "reasonable" amount of time.

You mention you have about 60% of 100 Mbps commonly unused so you could calculate what the expected transfer time might be.