Introduction:
In this document you will learn about Ether-channel load balancing on Cat 6K, 7600, 4500, 3750.
Catalyst 6k and 7600:
How it is implemented on this platform?
The way EtherChannel load balancing works is that the switch assigns a hash result from 0-7 based on the configured hash method ( load balancing algorithm ) for the type of traffic. This hash result is commonly called as Result Bundle Hash (RBH).
As for an example, let us consider that the port-channel algorithm configured on the box is src-dst ip then the source and destination IP of the packet will be sending to the hash algorithm (a complicated 17th degree polynomial) to calculate the RBH. Now each RBH is mapped to one unique physical port in the port-channel, whereas one physical port can be mapped to multiple RBHs (please look at following example for further clarification).
Let us consider the configured LB algorithm is src-mac and the switch is trying to send packets from 3 different src macs a, b & c over the ether channel( 5/1-2).
Now for packets from "a" the hash algorithm computes a RBH of 6, 5 for "b" and 4 for "c".
It is possible that RBH of 5/1 is mapped to RBH of 6 & 4, 5 for 5/2 but one RBH can be mapped to one physical port only. It is not possible that a RBH ( say 3) is mapped to both 5/1 and 5/2.
Things to check/how to check
------------------------------
1. What is the configured load balancing algorithm?
From the SP "show etherchannel load-balance"
e.g.
6500-20-sp#show etherchannel load-balance
EtherChannel Load-Balancing Configuration:
src-dst-ip <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
mpls label-ip
EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
IPv4: Source XOR Destination IP address
IPv6: Source XOR Destination IP address
MPLS: Label or IP
6500-20-sp#
2. What is the RBH value chosen for the packet from 1.1.1.1 to 2.2.2.2?
From the SP " test etherchannel load-balance interface port-channel <port-channel #><mac/ip> <source address> <destination address>"
Choose the attributes of the above command based on the output of step 1. If the configured load balancing algorithm is src_ip then just give the src-IP of the packet 1.1.1.1. Since in our example the configured load-balancing algorithm is src-dst IP we will feed both 1.1.1.1 and 2.2.2.2 in the above command.
e.g.
6500-20-sp#test etherchannel load-balance int port-channel 1 ip 1.1.1.1 2.2.2.2
Computed RBH: 0x5
3. How to check which physical port is mapped to what RBH value?
In certain version of IOS the output of test etherchannel command will give you the physical interface chosen; this step (3) is when you do not get the egress interface information from the test etherchannel command output.
For the IOS where the command gives the egress interface the o/p will look like this.
6500-20-sp#test etherchannel load-balance int port-channel 1 ip 1.1.1.1 2.2.2.2
Computed RBH: 0x5
Would select Gi3/2 of Po1
For other versions go to the RP and type the command "show interface port-channel <num> etherchannel". Look at the "Load" column o/p corresponding to a physical interface.
Convert the "Load" value into binary (look at the below example)
6500-20#show interface port-channel 1 etherchannel
Port-channel1 (Primary aggregator)
Age of the Port-channel = 0d:01h:05m:54s
Logical slot/port = 14/1 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Fast-switchover = disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 55 Gi3/1 Active 4
1 AA Gi3/2 Active 4
Here the load value for gi3/1 is "55" and for gi3/2 is "AA".
7654 3210
gig3/2 - AA -1010 1010
| |
A A
gi3/1 - 55 - 0101 0101
------ -----
| |
5 5
for gig3/2 bits 1,3,5 and 7 are set. So RBH value of 1,3,5,and 7 will choose gi3/1.
for gi3/1 bits 0,2,4 and 6 are set. So RBH value of 0,2,4,and 6 will choose gi3/2
From the outputs you can observe that 4bits are set for each of the two interfaces. Hence when there are 2 links in the ether channel then each link has equal probability of getting utilized.
However for 3 links in etherchannel the test etherchannel output will look similar to this.
6500-20#show interface port-channel 1 etherchannel
Port-channel1 (Primary aggregator)
Age of the Port-channel = 0d:01h:05m:54s
Logical slot/port = 14/1 Number of ports = 2
HotStandBy port = null
Port state = Port-channel Ag-Inuse
Protocol = LACP
Fast-switchover = disabled
Ports in the Port-channel:
Index Load Port EC state No of bits
------+------+------+------------------+-----------
0 49 Gi3/1 Active 3
1 92 Gi3/2 Active 3
2 24 Gi3/3 Active 2
Here the bit sharing ratio is 3:3:2 hence two links has higher probability of getting utilized as compared to the third link (more in the additional section at the end).
This table lists the ratios of the values that each port accepts, which depends on the number of ports in the EtherChannel:
8 | 1:1:1:1:1:1:1:1 |
7 | 2:1:1:1:1:1:1 |
6 | 2:2:1:1:1:1 |
5 | 2:2:2:1:1 |
4 | 2:2:2:2 |
3 | 3:3:2 |
2 | 4:4 |
Note: This table only lists the number of values, which the hash algorithm calculates, that a particular port accepts. You cannot control the port that a particular flow uses. You can only influence the load balance with a frame distribution method that results in the greatest variety.
We also support “per module load balancing” for DFC LC, where you can define LB algorithm per module basis.
http://www.cisco.com/en/US/docs/switches/lan/catalyst6500/ios/12.2SX/configuration/guide/channel.html#wp1020804
For this implementation we would need to keep in mind that the hash decision is taken on the INGRESS line card. If you have configured ether channel LB for the DFC where the actual physical link in the ether channel exist then your Load balancing might not work as you have desired as ingress LC will decide the egress physical port. By default any LC ( with or without DFC ) will load balance traffic based on the algorithm configured on the PFC. To check the “test etherchannel” command session to the DFC module and then issue the command.
For Cat 4500:
How it is implemented on this platform?
On this platform we use the concept of Agg2PhyPort mapping table. Agg2PhyPort table as the array of 8 elements, each can contain a port number, say for 2 ports a and b then {a,b}: .
Hash function will calculate the index into that array based on the input information: so it will be either 0 or 1 (index is 0 based).
Here’s an example:
Imagine you use 3 links in a bundle (say port 5, 10 and 20), then agg2phyport table would look like:
0: 5
1: 10
2: 20
max-length=3 (number of ports in a bundle)
Now, hash algorithm produces, say 7 (for configured input parameters), then index will be calculated as 7%3=1 and port 10 will be selected.
What to check/how to check?
1. How to check Agg2PhyPort mapping table?
"show platform mapping port" is the command however it is not worth doing it as the output of the command provided in step 4 gives you the egress port all the time.
2. How to check the o/p of hashing algorithm?
Not worth checking because of the above reason. The hash value for the 4500 is calculated via a 'rolling XOR' which is Cisco Confidential.
3. Check the configured load balancing algorithm by using the command "show etherchannel load-balance".
4. Use the command "show platform software etherchannel port-channel 1 map " to find the egress interface.
BGL-4500-12#show platform software etherchannel port-channel 1 map ip 1.1.1.1 2.2.2.2
Map port for Ip 1.1.1.1, 2.2.2.2 is Gi2/1(Po1)
NOTE: Software forwarded traffic will use Gi2/1(Po1)
While using the above command please keep in mind CSCtf75400(registered customer only)
If you hit this bug then unfortunately you have to rely on the sniffer capture to get the actual egress interface,
In K5 based architecture we have actually got rid of unequal load balancing problem when the number of links are 3,5,6 or 7 . As mentioned in the doc that we use 8 bits of hash result to determine the load balancing, in a scenario where we have 3 physical links in the ether channel 3 bits will be chosen for link 1, 3 for link 2 and 2 for link 3. So the ether channel load balancing probability is ( 3:3:2). However in K5 we use only last 3 bits of the hash result ( for 3 links in EC , 5 bits for 5 links in EC , 6 bits for 6 links in EC and so on ) for 3 links in EC. This way we ensured that all the links in the EC has equal probability of taking the traffic. In K5, in order to improve load-balancing determination and flow distribution we stepped away from the “modulo” approach and load-balancing is based on the pre-programmed hardware mapping table.
For Cisco 3750:
On 3750 we use a similar 8 bit hashing algorithm and hence traffic distribution is more even when number of links in the ether channel is 2 4 or 8 ( please look at the common scenario section for details).
Command to check the egress interface in the port-channel is
" test etherchannel load-balance interface port-channel <port-channel #><mac/ip> <source address> <destination address>"
Common Scenarios:
-----------------------------
Ether channel not load-balancing properly?
To understand the scenario it is important for us to determine all the flows which the etherchannel is handling. Number of flows will depend on the configured load balancing algorithm. Let us take an example.
Source 10.0.0.1 (mac a.a.a ) sending a tcp stream to 15.0.0.1 ( mac b.b.b ) with a source tcp port of 50 and destination 2000
Source 10.0.0.1 (mac a.a.a ) sending a tcp stream to 15.0.0.2 ( mac c.c.c ) with a source tcp port of 60 and destination 2000.
If configured load balancing algorithm is SRC_MAC
Then no of flows = 1
If configured load balancing algorithm is DST_MAC
Then no of flows = 2
If configured load balancing algorithm is DST_PORT
Then no of flows= 1
and so on.
The ways you can capture the flows are:
-> Sniffer - difficult and hectic.
-> Netflow - relatively easier.
-> External monitoring tool.
Once you have a good idea of the flows then check which flow will take which physical interface. Use the tools discussed above to determine the physical interface.
This step will help you to explain why we see unequal distribution of traffic over the physical interfaces.
Here are the few scenarios which can cause unequal distribution?
1. Let us consider we have two flows and two physical interfaces in the etherchannel. It might be possible that one flow is more talkative than the other.
Now consider i have 5 flows out of which one is super talkative, this flow can overwhelm others. Whichever physical interface this flow is choosing will have relatively higher utilization than the others.
Resolution- flow control the super talker, need to look at from the host side.
2. One very common problem is that you do not have enough number of flows and out of whatever small number of flows which you have most of them are hashing to the same physical interface.
Resolution- Increase the number of flows. Try changing the hashing algorithm to the one most appropriate to the traffic.
3. When you have 3, 5, 6 or 7 physical links in the ether channel then few links will have higher probability of taking the traffic than the other ( based on number of hashing bit assigned to each physical interface )and hence there is an inherent chance the traffic will be unequally distributed.
Resolution - use 2, 4 or 8 numbers of links in the ether channel.
Related Information:
Understanding EtherChannel Load Balancing and Redundancy on Catalyst Switches
