BLOG (No Title)

[Akula Jyoti Lakshmi]

In this section I would like to highlight on very basic topic "How etherchannel hash algorithm works". In day-to-day networking life some common issues that we face with etherchannel is,

1. Traffic is not distributed properly through each link in etherchannel across the network.
   
2. One port mostly, like an example, gi8/11 is being selected mostly for every traffic during peak hours. Why Gi8/11 is highly utilized and others have normal utilization?
   
3. How to change algorithm so that it select ports based on traffic pattern?
   

Let me start with basic fundamental: How etherchannel algorithm actually works ?

On most of the CISCO switches platform, default ether-channel load-balance algorithm is "XOR source and destination IP " to do the load-balance. The load-balancing is purely based on hashing provided by the hardware, which is in turn based on flows. You can essentially end up with a load balance setup that never changes links depending on your configured flows. You can choose whether to hash on Source or Destination Layer2 (MAC), Layer3(IP Address), or Layer4(Ports) flow information, but hash algorithm cannot be configured or changed to load balance the traffic among the ports in an Etherchannel.

Typically, you will need to play with your Source and Destination settings to find which settings used with your traffic flows creates the best load balance.

The Cisco-proprietary hash algorithm computes a value in the 0-7 range. With this value as a basis, a particular port in the Etherchannel is chosen. The port setup includes a mask which indicates which values the port accepts for transmission. With the maximum number of eight ports in a single Etherchannel, each port accepts only one value. If the Etherchannel has four ports, each port accepts two values, and so on.

Note: The hash algorithm cannot be configured or changed to load balance the traffic among the ports in an Etherchannel.

Assuming the default configuration, a flow is the source IP destination IP pair. Regardless of what the traffic is, that source destination pair will always use the same physical path. As an example,

If you have “http and ftp to/from the same source/destination IP, it will always use the same physical link. If a source destination pair is idle for a time, it will still use the same physical path since it always hashes with the same result. For instance, there is a very high value of replication traffic between several servers. You have 3 flows generating 50 mbps and 10 flows generating 1mbps. From what, it's entirely possible that the 3 largeflows could end up on the same physical link which means that you will have one circuit running at 50mbps and the other at 1mpbs.

So if in traffic capture, if it shows the traffic patterns are mainly in UDP with some of identical source and destination IP address pairs. Since most of traffic patterns are UDP with different port number during the peak hour, the better way to load balance the traffic is either to use source-port or destination-port hash algorithm depends on the traffic path. Again, this will not guarantee the traffic load balance to 50/50.

EXAMPLE :

----------------

Let's take a live example,

I have 4 port etherchannel configured on my switch,

Gi7/11

Gi7/12

Gi8/11

Gi8/12

Sw1#sh ether summ

Flags: D - down P - bundled in port-channel

------+-------------+-----------+-----------------------------------------------

1 Po1(SU) LACP Gi7/11(P) Gi7/12(P) Gi8/11(P) Gi8/12(P)

Let's pick up some traffic flow for particular Source and Destination IP and have a look which link it selects ,

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.16 203.116.42.149

Computed RBH: 0x6

Would select Gi8/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.90 203.116.42.149

Computed RBH: 0x6

Would select Gi8/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.17 203.116.42.149

Computed RBH: 0x7

Would select Gi8/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.91 203.116.42.149

Computed RBH: 0x7

Would select Gi8/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.18 203.116.42.149

Computed RBH: 0x4

Would select Gi7/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.19 203.116.42.149

Computed RBH: 0x5

Would select Gi7/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.92 203.116.42.149

Computed RBH: 0x0

Would select Gi7/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.93 203.116.42.149

Computed RBH: 0x1

Would select Gi7/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.20 203.116.42.149

Computed RBH: 0x2

Would select Gi8/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.94 203.116.42.149

Computed RBH: 0x2

Would select Gi8/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.21 203.116.42.149

Computed RBH: 0x3

Would select Gi8/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.95 203.116.42.149

Computed RBH: 0x3

Would select Gi8/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.22 203.116.42.149

Computed RBH: 0x0

Would select Gi7/11 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.96 203.116.42.149

Computed RBH: 0x5

Would select Gi7/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.23 203.116.42.149

Computed RBH: 0x1

Would select Gi7/12 of Po1

SW1#test etherchannel load-balance interface port-channel 1 ip 203.116.22.97 203.116.42.149

Computed RBH: 0x4

Would select Gi7/11 of Po1

Here we do see the traffic is going with round robin which was expected. However, we do see the links of G7/11 and G8/11 were utilized more than the other 2 links. This is highly likely caused by same pair of IP address, source and destination. Same pair of source and destination IP will always use the same link since the load balance hash algorithm calculation will always be the same.

Computed RBH: 0x0 Would select G7/11 of Po 1

Computed RBH: 0x1 Would select G7/12 of Po 1

Computed RBH: 0x2 Would select G8/11 of Po 1

Computed RBH: 0x3 Would select G8/12 of Po 1

Computed RBH: 0x4 Would select G7/11 of Po 1

Computed RBH: 0x5 Would select G7/12 of Po 1

Computed RBH: 0x6 Would select G8/11 of Po 1

Computed RBH: 0x7 Would select G8/12 of Po 1

BEST PRACTICES :

---------------------------

We should follow below steps to ensure that etherchannel works best in all conditions,

1. Try to put even number of links in ether-channel and monitor the throughput of link.

Explanation:

If the number of physical port is not equal to power of 2 (not 2, 4 or 8), you will have unequal load-balance as well. For example, pick a case,

Under port-channel X, we have 3 interfaces being bundled gi8/8, gi9/8 and gi9/12. The load balance algorithm will need to use two bits from the hash.

Two bit will give 4 different patterns (00, 01, 10, and 11) and the patterns are assigned to the physical port. Therefore, the overall traffic is not 33% on each channel. It will be 50% on one and 25% on the other two links.

2. Please monitor the link utility during peak hours and note down the data. Based on traffic pattern in your network , you can change ether-channel load-balance algorithm to "src-dst mac" or "srd-dst ip" or “src-dst port” etc. "src-dst mac" works best in most of the cases in network

I hope it could help a bit in your understanding of operation of ether-channel.

Enjoy browsing CSC blogs folks

[the_ios_inquisition]

Thanks for this write up. I had a question about the math you used to determine the percentage the load-balancing alogithim would select a member interface. Early in you post you made the statement that IOS uses the values of 0-7 and assigned those values to the member interfaces. You then mentioned further down in the post that with 3 interfaces the percentage each member would be hashed is 50%, 25%, 25%. However, if values 0-7 are assigned to each member interface, and if they are assigned in a round-robin fashion, wouldn't that percentage be 37.5%, 37.5%, 25%? If each value 0-7 represents a selection for the load-balancing algorithm, and let's say for arguments sake we have interfaces numbered 1, 2, & 3, then two interfaces would have 2 values associated to them and one interface would have one value assigned to it. That should equal out to 37.5/37.5/25. I feel like I'm obviously missing something here but it's not clear to me. Again, thanks so much for the write up, and if anyone could help me out of my confusuion, I would greatly appreciate it.

[Akula Jyoti Lakshmi]

Under port-channel  X, we have 3 interfaces being bundled gi8/8, gi9/8 and gi9/12. The load  balance algorithm will need to use two bits from the hash.

Two  bit will give 4 different patterns (00, 01, 10, and 11) and the  patterns are assigned to the physical port. Therefore, the overall  traffic is not 33% on each channel. It will be 50% on one and 25% on the  other two links.

RBH  0x0--  Gi8/8

RBH 0x1   G9/8

RBH 0X2 - gi9/12

RBh  0x3 - gi8/8

RBH  0x0-  gi8/8

RBH  0X1 --  Gi 9/8

BRH  0x2  --  Gi9/12

RBH  0x3 -- gi8/8

Here  you  see -  gi8/8  is used  50%  times  an  Gi9/8  and  gi9/12  used 25%  25%  times.

[the_ios_inquisition]

I think I'm starting to get what you're saying. Please correct me if I'm wrong:

The load-balancing algorithm will select the least number of bits from the hash that will give unique values to each member interface, with a maximum of 3 bits that would cover up to 8 interfaces. Each unique value will need to be assigned to an interface, and in many situations, more than one value is assigned to them.

Two(2) member interfaces would only need 1 bit selected from the hash and values 0 & 1 will be assigned to the interfaces

Three(3) and four(4) member interfaces would need 2 bits and assign values 0 - 3

Five(5) through eight(8) member interfaces will use 3 bits and assign values 0 - 7

If someone could let me know if I'm wrong, I would greatly appreciate it. Thanks for you help!

[Akula Jyoti Lakshmi]

Yes  ,  your  understanding is correct.  Rate  the  post  if  it  helps

[tommyboay]

Thanks for the interesting note.

I'd like to add that people really should look at the output of the "show etherchannel load-balance" to check their current algorithm. I haven't seen any catalyst series under the 4500/6500 using the src-dst-ip as default value. They will rather be set on source mac. This includes the blade switches that usually offers connectivity to server farm where the src-mac parameter is particularly inadequate.

In the end, I have also seen some switches where changing to a more elaborate algorithm would cause overruns and complete loss of the Po interfaces so be careful on testing new settings.

Tom

[gnsc_ip.ttsl]

I am facing some problem with ether-channel loadbalancing. I have two etherchannel configured on 7606 switch with SUP720, one is L2  (2 port) and the other is L3 (4 port) and the loadbalancing configured is "src-dest-ip vlan include". Now the problem is that one of the interface in the L3 etherchannel is highly utilised. I have checked by " test etherchannel load-balance interface port-channel" command and found one interface is mostly utilised, but how can I select the load-balancing method in this case as I have L2 and L3 both type of ether-channel running on my router. Please give me a suggestion.

[Kattubava Edaisal Shahul Hameed]

Hi Jyothi,

I am not sure about the number of bits used by the load  balance algorithm for hashing, Could you pls explain it furher.For example in what sense the the 2memeber to 8member bundle chosses the bits for hashing

Thanks in adv,

Bava

[fcmartinez]

Thanks for this good information i only want to add this link:

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml#catalyst

use this command:

show etherchannel load-balance

to see your SW load balance alg.

[HummusChipsSalat]

Hi all !

does the RBH value of the physical port change when i reload the machine ?

can i rely on port Gi1/1 to stay RBH 0x1 when the machine boots up again [ and all others as well ] ?

or does it happens randomly "first port to botup would get RBH 0x1 "?

if i take two switches and run a portchannel between them .

i would like to see if i can cause downstream traffic that goes through switch A to switch B to return via the same Link .

[gsidhu]

Hi Jyothi

Thank you for this very good informative link.

I have layer 2 etherchannel between Catalyst 4500 VSS in two buildings. The 4500 VSS have server farm VLANs than span bothe 4500 VSS. Both VSS will also provide layer 2 MEC to access layer 3560 switches.

I want to implement layer 4 hash on the 4500 VSS (port-channel load-balance src-dst-port) as this appears to be the best choice for load balancing traffic between the servers (for example for replication traffic)

The 3560 access switches support layer 3 hash but NOT layer 4 hash so will use layer 3 hash for the 3560 switches (port-channel load-balance src-dst-ip)

So basically two ends of the MEC will have different hash algorithms.

Is this a valid and/or supported design?

Thanks

G

[expertadvisor20151]

Cisco has a new solution called ITD:

http://blogs.cisco.com/datacenter/itd-load-balancing-traffic-steering-clustering-using-nexus-5k6k7k

ITD (Intelligent Traffic Director) is a hardware based multi-Tbps Layer 4 load-balancing, traffic steering, redirection, and clustering solution on Nexus 5K/6K/7K series of switches. It supports IP-stickiness, resiliency, NAT (EFT), VIP, health monitoring, sophisticated failure handling policies, N+M redundancy, IPv4, IPv6, VRF, weighted load-balancing, bi-directional flow-coherency, and IPSLA probes including DNS. There is no service module or external appliance needed. ITD is much superior than legacy solutions like PBR, WCCP, ECMP, etc.

[Dumitru Otel]

in other words you want say that:

2 links = 50%/50% balanced traffic

3 links = 50%/25%/25% balanced  traffic

4 links = 25%/25%/25%/25% balanced traffic

....................................................

8 links = 12.5% /12.5% /12.5% /12.5% /12.5% /12.5% /12.5% /12.5% balanced traffic

:D I think that now all is clear for me !

-----------------------------------------------------------------------------------------------------------

|Number of Ports in the EtherChannel  |           Load Balancing     |

-------------------------------------------------------+--------------------------------------------------

|             8                               |         1:1:1:1:1:1:1:1              | 12.5:12.5:12.5:12.5:12.5:12.5:12.5  -> 3 bits (000|001|011|111|100|110|010|101)

-------------------------------------------------------+--------------------------------------------------

|             7                                         2:1:1:1:1:1:1               | 25:12.5:12.5:12.5:12.5:12.5:12.5  -> 3 bits (000|001|011|111|100|110|010|101)

-------------------------------------------------------+--------------------------------------------------

|             6                                |          2:2:1:1:1:1                  | 25:25:12.5:12.5:12.5:12.5% --> 3 bits (000|001|011|111|100|110|010|101)

-----------------------------------------------------------------------------------------------------------

|             5                                |          2:2:2:1:1                     | 25:25:25:12.5:12.5% --> 3 bits (000|001|011|111|100|110|010|101)

-----------------------------------------------------------------------------------------------------------

|             4                                |          2:2:2:2                        | 25:25:25:25% --> 2 bits (00|01|10|11)

-----------------------------------------------------------------------------------------------------------

|             3                                |         2:1:1                            |  50:25:25% --> 2 bits (00|01|10|11)

-----------------------------------------------------------------------------------------------------------

|             2                                |         1:1                               |  50:50%    --> 1 bit (0|1)

-----------------------------------------------------------------------------------------------------------

Thanks!

[Nuno Lobao]

Thanks for your post,

After trying to find without success I must ask, do you have a picture of the frame level view (802.3) which depicts etherchannel (802.3ad)?

I would like to have a frame level view of etherchannel but cant find it online. Would appreciate it if you can share if you have it.

thanks

[xine xine]

Hi !

I'd like to understand how you compute the hash value it self, because base on what I remember my calculation ends with different RBH value...

in your example the hash between : 203.116.22.94 & 203.116.42.149

I calculate the hash like that :

94 in binary is 1011110

149 in binary is 10010101

by taking the last 3 digits of both because we have 4 physicals interfaces in the channel so : 110 XOR 101 = 11 which is 0x3 but your answer was 0x2   

which end by selecting Gi8/12 instead of Gi8/11 of Po1 

What I'm doing wrong ?

Thanks !

[#16]

"Three(3) and four(4) member interfaces would need 2 bits and assign values 0 - 3"

Let us say I am using a four link interface, and the fourth link goes down, so the distribution would be 25/25/50/0%

What happens if the 2nd or 3rd link goes down (and 4th is up).  if 2nd link down is the distribution 50/0/25/25% or is the distribution 25/0/50/25%?

Would the distribution favor the lower values or the higher values??

Thank you, 

Bob