Etherchannel Load balancing

[6rlopez_2]

I have three Gigabit aggregated with LACP to a IBM server. When I ask one ftp from this server I saw the performance is around 80 MB/s, if I ask two ftp to different equipmpents the performance is 160 MB/s (it uses 2 of the links of the bundle). However if I ask for three ftp the performance keeps in 160 MB/s although it uses the three links of the bundle. Is there a limit for Cisco? Anyone knows the reason? It seems like we can transmit as maximum at 160 MB/s

[glen.grant]

Something to keep in mind when using etherchannel is a particular conversation is always going to use the same 1 link to get to the server whether using mac addresses or ip addresses as the load balancers , one conversation does not get load balanced across all 3 links . Someone can correct if I am wrong . Here is a link explaining this http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

[dan_track]

Hi

The first line in the article states:

Fast EtherChannel allows multiple physical Fast Ethernet links to combine into one logical channel. This allows load sharing of traffic among the links in the channel as well as redundancy in the event that one or more links in the channel fail.

Seems like it does load share.

Dan

[amit-singh]

Glen is absolutely correct. on layer 2 switches EtherChannel load balancing can use either source-MAC or destination-MAC address forwarding.

With source-MAC address forwarding, when packets are forwarded to an EtherChannel, the packets are distributed across the ports in the channel based on the source-MAC address of the incoming packet. Therefore, to provide load balancing, packets from different hosts use different ports in the channel, but packets from the same host use the same port in the channel. With destination-MAC address forwarding, when packets are forwarded to an EtherChannel, the packets are distributed across the ports in the channel based on the destination host MAC address of the incoming packet. Therefore, packets to the same destination are forwarded over the same port, and packets to a different destination are sent on a different port in the channel.

So with every different source or destination mac it will use a diff link. If there is a session started on a aprticular link for a source to a destination it will keep on using the same link until the session termiantes or link goes down.

HTH, Please rate if it helps.

-amit singh

[6rlopez_2]

Thank you for your comments but it is not my case. I am using as load balancing criteria: IP address dest and source. And the three ftp sessions have the same IP source but different IP destinations so it should use the three links of the bundle. In fact I can see in the server that it is receiving for the three interfaces but the performance of the links goes down to 50MB/s for each link.

Another question is :the criteria is to use the two last bits of the source and dest IP address and use a XOR function. This gives me 4 possible combinations: 0 0, 01,10, 11 but I have only three links. To which link goes the packet if the function has as result 11 as I do not have a 4th link?

Thanks!

[Bobby Thekkekandam]

The way the traffic is distributed among the channel ports is that the earl computes a hash value in the range 0 - 7 for every flow based on the config (mac address or ip or port numbers, destination only or source only or src xor dst). The ports are set up with a mask which indicates which values it would accept for transmission.

So if you had 8 ports, each port would accept one value.

If you had 4 ports, each port would accept two values. The mask would be something like:

11, 22, 33, 44

If you had 2 ports, each port would accept four values

AA, 55

When you have odd number of ports like 3, 5 or 7, some ports will have to take up more masks than other ports.

3 ports : the distribution would be

port 1 - 3 values,

port 2 - 3 values,

port 3 - 2 values

5 ports : the distribution would be 2, 2, 2, 1, 1

HTH,

Bobby

*Please rate helpful posts

[6rlopez_2]

Very helpful your explanation to know how it load balances!! But that does not explain the poor performance of the links. The theory is the performance should be around 80% of the bandwidht, I think.

Is there any reason why the link in an Etherchannel is not offering the whole bandwidth?

[Bobby Thekkekandam]

How do you know that the etherchannel is "slowing down" the transfer?

[6rlopez_2]

Because I can see the transfer rate in the server that the ftp session has : one ftp -> 80 MBps

two ftp sessions -> 80 MBps each

three ftp sessions -> around 50 MBps each

So the maximum we get is 160 MBps of transfer while we have in theory 3 Gbps to use.

[Bobby Thekkekandam]

You mentioned that there is one IP source and 3 destinations. So, The source is sending streams for up to 3 TCP sessions.

How fast is the source sending the data? Do you have a way of measuring that? The channel link can only send the data as fast as it can receive it from the source.

Also keep in mind that net throughput is going to be a function of the TCP windowing. If you want a true measure of throughput over an etherchannel, you'll need to measure that using a protocol that is not limited by upper layer protocols, in this case TCP.

-Bobby

[6rlopez_2]

The server with the etherchsannel is which asks the ftps to 3 different ftp servers. The connections of these ftp servers are Gigabit links.

What method do you recommend to measure the throughput instead of using TCP?

[glen.grant]

I would think if you have one server doing all the ftp's it is going to naturally slow down , your weakest link will always be your harddrive transfer rate and the more ftp's you have going the less the thruput will be .

[6rlopez_2]

I know the reason of the poor performance.If it is helpful to someone...

The 10/100/1000 module that we have does not offer 1 Gigabit to each port,it oversubscribed so the switch only offers 1 Gigabit to a group of 8 ports (in my case)