Etherchannel, two links, but no increase in bandwidth

[j-larsson]

Hi,

We have a problem with an EtherChannel that we have between us and a content provider.
We have two 1Gb links in the Etherchannel but we can't seem to get the total bandwidth above 1Gb. 
I have created the same situation in a lab environment.  Same thing  there, I cannot get the total bandwidth to more than 10Mbps ( I  statically set the speed of the physical interfaces/members to 10Mbps  individually).
Does anyone have any idea if there is some configuration parameter that  we missed that is limiting the total bandwith of the Etherchannel to the  same as the speed of the individual interfaces, rather than the  combined bandwidth?  A big benefit about implementing an Etherchannel is  the increased bandwidth, and we kind of miss out on that.

Below is some of the configuration of the lab.

Two Catalyst 3560 running 12.2(50)SE3

Switch A, and respectively on B

interface Port-channel47
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 1,28,900
  switchport mode trunk

interface GigabitEthernet0/15
  description eth-cha1
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 1,28,900
  switchport mode trunk
  speed 10
  channel-group 47 mode on
!
interface GigabitEthernet0/16
  description eth-cha2
  switchport trunk encapsulation dot1q
  switchport trunk allowed vlan 1,28,900
  switchport mode trunk
  speed 10
  channel-group 47 mode on

port-channel load-balance src-dst-ip

I have tried different load balancing parameters, and different channel-group x modes but it does not make a difference.

Thanks

Johan Larsson

Message was edited by: Johan Larsson Update, We have not been able to solve the issue as of yet. We have done some serious testing the last few days to solve the problem.  We have noticed that in an etherchannel where we use trunks we may increase the bandwidth by changing it to port-channel load-balancing src-dst-mac However, on the link that we have to the content provider we use a routed interface for the port-channel and on this connection we cannot go above the total of 1Gb (limit for one of the physical links). The config is: interface Port-channel10 no switchport ip address x.x.x.x x.x.x.x ip pim sparse-mode interface GigabitEthernet1/0/1 no switchport no ip address no cdp enable channel-group 10 mode active ! interface GigabitEthernet1/0/2 no switchport no ip address no cdp enable channel-group 10 mode active port-channel load-balance src-dst-mac Any ideas to what is going wrong? Greatful for any tips / ideas JL

[Latchum Naidu]

Hi Johan,

Please configure the port channels like below as per best practices and check.
You should get the improved bandwidth than the single link.

interface Port-channel90
switchport
switchport trunk encapsulation dot1q
switchport trunk native vlan 190
switchport mode trunk

interface GigabitEthernet3/30
description *********************
switchport trunk encapsulation dot1q
switchport trunk native vlan 190
switchport mode trunk
channel-group 90 mode on

Hope this helps you...

Please rate the all helpfull posts.
Regards,
Naidu.

[hobbe]

Hi

If you only test with one machine or to one machine then it is very likely that you will have max 1 gig due to the traffic all are going through one link in the channel.

max you will get in one session regardles will be 1 gig.

in multi sessions from different sources or destinations you should be able to set up loadbalancing to use both links.

There is a command to pre test wich link the packets will take when it hits the switch.

something like

test etherchannel load-balance ...... and add the information needed..

Good luck

HTH

[Jon Marshall]

Johann

In addition to Hobbe's post -

A common misconception with etherchannel is that it will increase bandwidth for single connections. It won't. So if you have a client talking to a server and it sends 1.5Gbps of traffic and you create a 2Gbps etherchannel this 1.5Gbps will not be spread across both links ie. the client to server traffic for that particular connection will still only use on of the physical 1Gbps links.

The purpose of using etherchannel is -

1) redundancy of links

2) for multiple connections you can indeed increase the bandwidth because different connections can spread across different physical links within the etherchannel.

Note that when referring to connections above a connection could be -

1) src/dst mac-address pair

2) src/dst IP address pair

3) src/dst L4 port (eg TCP or UDP port)

not all switches support all methods listed above.

If you actually need to increase the bandwidth for a particular connection between 2 end devices then etherchannel won't do this. You could look at load-balancing or simply using higher bandwidth NICs if this is possible.

Jon

[glen.grant]

  Jon is correct you are never going to get more than 1 gig out of a single conversation across the etherchannel .  For a given conversation say between 2 devices the cisco etherchannel algorithim  hashes  it out and it determines which "single " link in the etherchannel to send it .  While etherchannel will give a bigger total pipe to run conversations across , all conversations will go across a single given link  so the total bandwidth for a given ip conversation will never be more than  1 gig  or 10 gig if its a ten gig etherchannel.

[j-larsson]

Allright guys,

I think I finally get the concept   Now it's just a matter of choosing the best load balancing option for the "production" etherchannel.

port-channel load-balance src-dst-ip

interface Port-channel10

no switchport

ip address xxxx yyyy

ip pim sparse-mode

We have a BGP link to our content provider, where all the traffic is routed to and from us.

In your opinion is src-dst-ip the best way to load balance this etherchannel?

Thank you very much for your help.

Have a great day guys.

[hobbe]

Hi

I would think that you should check out what information you have about the connections that will pass over the etherchannel.

If they all are between thesame two ip addresses ie a proxy and a server fx then this is not a good way since that would all be translated to one link in the etherchannel and you might want to us src ip/dst ip protocol port.

If however it is from a group of DNS round robin designated servers out to any number of users on the internet directly then src-dst ip would be good.

If it is just a local LAN then mac address might be a good alternative.

so it all depends on what the traffic looks like that you want to have going over the etherchannel.

if you are still unsure, use the test command to find out wich one your traffic will be spread most by.

Good luck

HTH

[j-larsson]

Just for kicks, and since the lab was in place I just had to verify what you guys stated above.

I connected yet another computer behind the etherchannel and with two different sessions in place I reached a total of 2x10Mbps (app. 20Mbps) over the etherchannel.

[j-larsson]

Update:

We have not been able to solve the issue as of yet.

We have done some serious testing the last few days to solve the  problem.  We have noticed that in an etherchannel where we use trunks we  may increase the bandwidth by changing it to   port-channel load-balancing src-dst-mac. However, on the link that we have to the content provider we use a  routed interface for the port-channel and on this connection we cannot  go above the total of 1Gb (limit for one of the physical links).

The config is:

interface Port-channel10

no switchport

ip address x.x.x.x x.x.x.x

ip pim sparse-mode

interface GigabitEthernet1/0/1

no switchport

no ip address

no cdp enable

channel-group 10 mode active 

interface GigabitEthernet1/0/2

no switchport

no ip address

no cdp enable

channel-group 10 mode active

port-channel load-balance src-dst-mac

Any ideas to what is going wrong?

Greatful for any tips / ideas    JL

[cadet alain]

Hi,

did you do the test etherchannel load-balance command while changing the load-balancing method to see which method will use different physical links if any?

Note that you can also have traffic on one link and return traffic on another depending on the load-balance algorithm at the other end of the etherchannel which may be different that  on the other side.

Regards.

Alain.

[j-larsson]

Hi,

Yes, I have tried all kinds of combinations, and I can clearly see how the loadbalance is changing, bur I cannot reach above 1Gb combined.

The weired thing is that it is only on the etherchannel where the logical interface is L3 that it's not working properly, one an etherchannel with a L2 interface it's working like a charm.

Thanks

Johan

[Joseph W. Doherty]

Disclaimer

The  Author of this posting offers the information contained within this  posting without consideration and with the reader's understanding that  there's no implied or expressed suitability or fitness for any purpose.  Information provided is for informational purposes only and should not  be construed as rendering professional advice of any kind. Usage of this  posting's information is solely at reader's own risk.

Liability Disclaimer

In  no event shall Author be liable for any damages whatsoever (including,  without limitation, damages for loss of use, data or profit) arising out  of the use or inability to use the posting's information even if Author  has been advised of the possibility of such damage.

Posting

As the other posters have already noted, single flows only use one link of an Etherchannel.  However, even when using multiple flows and even the "ideal" hashing algorithm, it's also possible two flows will hash to the same link.  I.e. it's not a given that multiple flows will use all the possible aggregate bandwidth.

On dual Etherchannel, for example, since the odds are 50/50 which link a flow will hash to, I assume my effective aggregate bandwidth at about 150% not 200%.  For three link Etherchannel, I expect about 200% not 300%, etc.  (NB: also note for odd [and non-binary?] number of links, hashing isn't even across links.)

[j-larsson]

Hi,

I understand the basic concept I think.  It's just really strange that when we set up a lab environment with two 3750s one etherchannel with two physical links, and use L2 for the Port-channel interfaces we are able to easily reach 1.5 Gbps, but when we change the Port-channel interfaces to L3 we hit the roof at 1Gb.

We use test equipment to generate about 986Mbps and use laptops to generate FTP traffic to generate traffic.

The "production" etherchannel is using L3 since we use BGP to a provider.

Any ideas please feel free

Thanks this far...

JL

[Stedly1982]

Johan,

I believe the problem you have is you have redundant links on the individual physical interfaces itself, which can create a loop of STP traffic trunking in circles.  If you just set the physical ports to dynamic, and just configure the Port-Channel alone, this should correct the issue by simply running 'no switchport trunk encapsulation dot1q'.  If a link fails, the logical interface will still communicate over the degraded logical interface.

conf t

int range gig0/15-16

no switchport mode trunk

no switchport trunk encapsulation dot1q

speed 1000

channel-group 47 mode active

exit