David,
Your results are perfectly valid and as expected.
The link aggregation technique (or EtherChannel, as Cisco calls it) works by choosing a particular link for each passing Ethernet frame and sending the whole frame over that single link. The particular link selection mechanism on Catalyst switches is always based on the addressing information in the frame and its contents, and can usually be one of the following:
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr
This output is taken directly from a Catalyst 3560 (the options for the port-channel load-balance global configuration command). Some higher switches can also take Layer4 ports into account. These options cause the switch to perform a hashing function over the configured address field (or fields) and use the result of the hash to point to a particular link in the bundle.
Note that for a single data flow between two fixed hosts in a particular direction, all the addressing information is the same - source and destination MACs, source and destination IPs. Consequently, performing the hash on any of the available address field choices will produce the same value, and as a result, the entire flow in one direction will be carried by a single link in an EtherChannel bundle. For the data flow in the opposite direction, the situation is just the same - it will be carried by just a single link (perhaps the same as the original flow, or some other, it does not make a difference). The key takeaway from this is that with an EtherChannel, you will not experience any increase in the throughput for one singular flow because that flow will still be carried by a single link in the bundle. Only when you have multiple flows between various sources and/or destinations, these flows will be distributed across multiple links in the bundle, and the aggregate throughput will be higher.
Some people argue that a simple round-robin technique would solve this issue: Have a switch send frames over all links in a cyclic sequence, i.e. with two links in a bundle, send frame 1 over link 1, frame 2 over link 2, frame 3 over link 1, frame 4 over link 2, etc. Cisco does not implement this method because of an important drawback: This mechanism can cause frames to arrive at the destination in a different order than in which they have been sent out, that is, a frame reordering may occur. Because in a basic simple Ethernet, frame reordering never occurs, and EtherChannel is supposed to be a transparent technique, it must not introduce any new phenomena that were not present with basic Ethernet, and so the frame reordering is a big no-no. Some would argue that TCP will handle the reordering. That is true, however, in general, TCP aggressively slows down if reordering is detected, causing poor performance, and moreover, not all traffic is TCP-based, or for that matter, not even IP-based. An application that uses direct Layer2 encapsulation into Ethernet frames, relying on the fact that Ethernet preserves ordering, would be badly broken.
LACP has nothing to do with how the EtherChannel works and distributes the load across multiple links. LACP is only a signalling/negotiation protocol that allows two directly connected devices to negotiate whether the links are to be aggregated, and performs certain sanity checks to avoid creating an EtherChannel from links that either are not bundled together at the opposite device, do not belong to the same aggregation group at the opposite device, or are not connected to the same opposite device at all. That is its only purpose - to negotiate the creating of the bundle and to make sure that the links can be validly bound without causing issues. Once the EtherChannel has been negotiated and brought up, LACP merely serves a monitoring function but has not impact on how the frames are actually carried over the bundle. Using LACP is highly recommended but the fact you're not seeing a fourfold increase in the throughput is not LACP's fault - it's just the way EtherChannel works, and neither LACP nor its proprietary predecessor PAgP can do anything about it.
What I can recommend is that you make sure on your Catalyst that you base the load-balancing choice in your EtherChannel on the source and destination IP, or even on a source and destination L4 port, as that choice gives you the maximal likelihood that different flows will be spread across different links in the bundle:
SW-Dist1(config)# port-channel load-balance ?
dst-ip Dst IP Addr
dst-mac Dst Mac Addr
src-dst-ip Src XOR Dst IP Addr
src-dst-mac Src XOR Dst Mac Addr
src-ip Src IP Addr
src-mac Src Mac Addr
However, the added value of an EtherChannel starts to show only when there are multiple flows carried by it. A single flow will not experience any improvement.
Best regards,
Peter
