Long Fat Network

[#1]

I am trying to figure out how to utilize more of our available bandwidth on a 200Mb LFN pipe between Houston and San Diego for the migration of a 250 +Gig database.

When the DB is copied, the data is stale before it's finished because it's taking nearly 24 hours. Right now, we're only transferring at about 7Mbps. The delay is averaging 38ms and the send/recv window size is set to 2MB on client and server.

I think the server guys need to increase their send and recv window sizes, but I don't know how to calculate the optimal settings. What do you think?

[DouglasScott99]

This is quite a hard problem to work with.

Assuming that you "delay" is the Round Trip Time,

say as displayed by ping. This would be about right

for the 1300 miles you mention.

From your figures you are sending 32k every RTT.

First you need to eliminate TCP receive window

as being the issue. Make sure that you are setting

the TCP receive Window to 64k ON BOTH ENDS. TCP uses

the smaller of the two offered window sizes.

Another likely explanation is that you are using

an application that is limiting the transfer rate.

In either case you need to check with a packet

capture tool like say ethereal.

This is a skilled job and if you want to identify

the issue quickly you maybe need to hire someone.

What database are you using?

[#3]

Thanks for the response. I actually already knew the problem had to do with the TCP window being too small, but I was trying to find a mathematical formula that would predict what my expected max possible throughput would be with the given RTT and send/recv window size.

Recently we figured out that for whatever reason, the servers aren't staying at the 2MB TCP window that we set. We wound up getting our performance increase by specifying the window size from within the FTP application (ftp -w). Once that was set, we saw a huge improvement.

[scottmac]

Shop aroud a little; there are some FTP clients that will open multiple sessions for a single file (it's a user configuarble thing) to help negate the ack-response bottlenecks (multiple parallel sessions, each getting their own ACK/NAK).

I can't remember any name specifically, but I know they're out there.

It might end up being better / easier to set up the database as a publisher / subscriber pair, so that changes made at one end are automaticaly updated at the other .... some flavor of replication where the DB is done in smaller chunks.

It may also work to do a horizontal partition and send each partition as a separate session, or do a horizontal partition at some time interval and send the smaller chunk.

Or maybe some combination ...

Good Luck

FWIW

Scott