There are three modes for CEF load balancing. Per Destination, Per Packet, Tunnel mode.
Per Packet alternates each packet across a different link. This gives almost 100% bw load sharing, but very easily packets can arrive out of order which would lead to retransmission. This method is only recommended in networks that have a very few far end devices that need load balancing paths. Per Destination is Default and most commonly used.
View from the Sending Side
Seq 1001 : 10.1.1.1 ---> sends packet 1 over path 1
Seq 1002 : 10.1.1.1 ---> sends packet 2 over path 1
Seq 1003 : 10.1.1.1 ---> sends packet 3 over path 1
View From the Receiving Side
Receives Packet with the Seq of 1002
Receives Packet with the Seq of 1003
Receives Packet with the Seq of 1001
Problem: since the packets arrived out of order this transmission seq. must retransmit.
Per Destination uses the per Source - Destination pairings to distribute packets across multiple equal/non equal paths. All packets that belong to a specific Source-Destination pair are guaranteed to be sent over that same path. When more Source-Destination pair's arrive on this router the paths are then alternated. Keeping the packets that belong to a Source-Destination session if you will prevents packets from arriving out of order across multiple paths.
IE. 10.1.1.1 ---> 20.1.1.1 is sending data. All data packets in this pairing will go over path A. The next pairing of 10.1.1.2 ----> 20.1.1.2 will have all of it's associated packets over path B. This method guarantees the orderly and timely delivery of tcp/ip packets.
Tunnel Mode
When the number of sessions is small, per destination load balancing load balances not as well. Per-destination load balancing assigns sessions to paths independently of other sessions. This means that for a small number of sessions, statistically the paths may not be used evenly. Only when the number of sessions is large, will traffic share be assigned to each path as required by the routing protocols. This is important when considering an IP tunnel environment, which by its nature, runs traffic over a small number of sessions (although traffic within a tunnel may represent many sessions). Per-destination load balancing may not give the desired traffic sharing in such an environment.
