LAN to WAN circuit sizing

[fgleeson]

Hi all,

Just a query on how a WAN router/circuit works with LAN connections.

Most LAN's are GB Ethernet. WAN circuits can be E1/T3/10M Ethernet - lots of options here.

Question.

If you have a user copying a 10GB file across the WAN to another system

- how come this user doesn't consume all available WAN bandwidth?

Assume that there is no prioritization, no QoS - everything just best effort.

Why doesn't everyone's WAN circuit burst up to 100% usage all of the time during the day, when users at both ends of the link are working away and copying files, running applications and so on?

Is it that the LAN interface on the WAN router is GB Ethernet - the WAN interface is a fraction of this (T1/E1/T3) and the router acts as the arbitor? I don't think so - the router can route the traffic at wire speeds in my head and could fill the WAN pipe if it was let.

Or is it something else?

It's probably a basic question but a really interesting one at the same time.

I've had a good browse around Google itself, but haven't got a good answer as yet.

Any feedback would be appreciated.

Thanks, Fergus

[lgijssel]

There are two main causes that contribute to the fact that a wan link is not saturated by a single session.

1: Latency. The latency on the WAN causes the transmitting end-node to wait for ack's after sending a block of data. The wait-time equals the round trip time of the link and no new data is sent during this time. The router can utilize this time to send other data.

2: The default queueing mechanism on a WAN-link is weigthed fair queuing which is more or less session-oriented. Each session gets an equal share of the bandwidth.

Regards,

Leo

[fgleeson]

Thanks Leo.

I did a crude test. We have a 45Mbps link to the US. Latency is about 100msec on a ping.

I started a copy of a 1GB file.

I got about 140Kbps throughput. See the attached jpeg from the NAM module.

I also had a look via Ethereal at the copy. The TCP window size was 17520 bytes.

LAN Test - same file but to a LAN server.

I got throughput of 8MB/sec.

Ethereal looks the same to me, with the only difference being the TCP window size is maximum 65535 bytes.

So, my laptop can send out traffic at up to 8MB/sec on LAN conversations.

It is limited on this WAN circuit to 140Kbps.

Just an observation really Leo.

Any comment on my crude test. Does it point more towards latency in this case?

[tdrais]

140 k bytes/second is very consistant with a ping time of a 100ms and a 17520 byte window.

With a 100ms you get 10 windows/second which give 175200 -- 170k bytes/second.

I suspect the 17k window size is from the remote end. If you can get this change to 64K you should be able to in theory get 640K bytes/second. You are still less than 10% utilization of your 45m line and this cannot be improved upon using a single tcp transfer stream.

[mmorris11]

What is the average packet size during the wan copy versus the lan copy?

[fgleeson]

The average packet size on both the LAN and WAN copy is 1514 bytes - the full size Ethernet packet. This is what i would expect - it's a 1GB file so its going to be full size packets.

I am curious about what Tim had to say regarding the windows/sec.

With the 100msec latency, we will indeed get 10 windows per second.

Does this equate to 10 packets per second?

If so, this equals 10 x 1460 bytes of actual data = 14600bytes/sec.

I'm not too sure if that adds up.

Did i take your point up correctly Tim?

We have another link to a closer site (about 60 miles away) over 10Mbps Ethernet with a 6msec latency.

That would give me 166 windows/second.

That equals 166 x 1460 = 250KB/sec (approx).

We actually get 1.1MB/sec in reality.

I'm just trying to marry the logic and i'm falling down somewhere.

Thanks for the replies and keep the coming if you can.

[tdrais]

Nope you get many packets per window.

Think of a window as big packet that is sent all at the same time. Of course when you look at the sniffer you can see the packets but if you look very closely you can count the number of unacknoledged packets in the data stream. The window represents the number of packets that can be in transit on the wire at a time.

This is actually much more complex since you have to worry about packet retransmission but for capacity planning purposed just think of the window as a single packet with a MTU of the window size.

Your one that is at 166 windows/second should give you 166xwindowsize

So in a max case of 64k you would get about 10M byte which of couse will not fit in a 10M bit ethernet. A window size of 7.5k at 6ms (166 windows) will max out a 10mbit ethernet

[lgijssel]

OK this is a typical LFN, a long Fat Network.

There is quite some theoretic information available about this. Please check the attached link:

http://www.cisco.com/en/US/products/ps6350/products_configuration_guide_chapter09186a00804550e6.html

A search on Cisco for "LFN" will give many more hits. Perhaps you can find some useful info there.

Regards,

Leo