The following will give you an indepth knowledge on configuring CEF and load sharing and retransmission
1. The maximum theoretical TCP throughput is for any path is
T-RW = TCP-Receive-Window
T-SW = TCP-Send-Window
RTT = Round-Trip-Time
L-BW = minimum Bandwidth link in path
T-BW = maximum effective TCP bandwidth
T-PS = TCP MSS packet size.
T-BW = min(min(T-RW, T-SW)/RTT, L-BW)
So if you increase the mBW link it may not increase your throughput if the first factor is still
smaller.
If we look at this for a T1 link (1.5 Mbs) and pick a reasonable window size (16KB = 128Kb), and
RTT (50ms - enough to get across US).
T-BW = min(128Kb/50ms, 1500Kbs)
= min(2560Kbs, 1500Kbs)
So this would mean that increasing the T1 to 2 T1s would help but 3 T1s wouldn't gain anything
more to speed up a single transfer. 3 T1s and would help by allowing more transfers to take place
simultaneously.
In the above if you either increase the TCP window sizes or decrease the RTT then the first
factor becomes larger and you will get higher TCP
effective-BW as longs as the minimum Bandwidth link in path is larger.
It doesn't matter whether you are doing per-packet load-balancing over multiple links or Layer-2
MUXing of slower links together to increase the L-BW.
Per-destination (flow) load-balancing will not increase the throughput for a single TCP transfer
since you are only using one of the multiple slow path links.
2. There are other factors that can come into play when you do per-packet load-balancing over
multiple links and this caused by the chance of out-of-order packets. It does depend on how out of
order your packets are.
In general if your packets are never more than 2 packets out of order then the TCP sender should
ramp up its congestion window to be able to "fill the pipe" (L-BW*RTT)/T-PS. This will mean that the
TCP sender will be sending at a rate that matches the L-BW. The L-BW may be an aggregation of links
(per-packet-LB)or not.
If your packets are often more than 3 packets out of order, perhaps because of per-packet
load-balancing over 4 links, then TCP can erroneously trigger fast-retransmission.
Fast-retransmission is triggered by 3 duplicate-ACKs coming back from the receiver. Even if the
sender doesn't actually retransmit a packet it will still reduce its congestion window, and slow
down sending. This can result in slower throughput than if the transfer had been going through a
single link (per-destination load-balancing).
In general per-packet load-balancing may or may not improve performance of a single TCP transfer. If
the throughput for a single TCP transfer needs to be increased then it is better to get a single
higher speed line or MUX together multiple lower speed lines at layer-2 (for example MLPPP). Usually
for multiple links it is better to use per-destination (per-flow) load-balancing.
