Teamed interfaces only performing at single speed

[vancampdax]

We've installed new servers, HP DL380 G8, and configured HP teaming ( windows 2008R2 SP1 ) to bundle 2 NIC's.

The switch is also configured with an etherchannel and both intrfaces are bundled and active in the port-channel.

However.

Every time copy a file from 1 server to another server we only are using 1 of both interfaces while HP teaming is set to use 802.1ad with fault tollerance.

What could be the issue ?

[jonathanaxford · Accepted solution]

Hi,

This sounds like the correct behaviour. With an etherchannel/port channel, all traffic going from the same source to the same destination will always take the same link. In your example, the traffic is part of the same conversation - always sourced from the same MAC Address/IP and destined for the same MAC Address/IP so the load balancing algorithm in use will always determine the same path for the traffic to take.

You will still have fault tolerance as if the connection being used fails, the traffic will switch over to the other interface.

I am not sure that it is possible to balance the traffic evenly over the interfaces in this situation.

Many thanks

Jonathan

[glen.grant]

Jonathan is correct , a given conversation will flow down a single interface in the etherchannel bundle so even if you have a dual  nic port channel any given conversation will not transmit over 1 gig .  Etherchannel gives you a bigger total pipe to balance your flows but it does not increase bandwidth , if you need higher than gig speed transfer rates  then you will have to invest in 10 gig interfaces for all your equipment.

[vancampdax]

Glen,

If I read the technology paper on the CISCO site, it clearly states that you can increase your bandwidth up to 8 times the original speed ( for servers and switch connections ).

So, I'm a bit confused here.

[hobbe · Accepted solution]

Hi all

a session will not split over 2 different physical links. (exeption= one physical link breaks and another other takes over that session)

if you do not want the explanation but only the answer then that answer is in the bottom after the last  ----- line.

ok I think this is best described with a couple of examples.

setup

we have 2 (up to 8) 1 gig links connected as a etherchannel.

---------------

example 1

we are using L2 load balancing for the etherchannel

1 local server talks to 1 other local server and transfer 2 files 1 over http and one over ftp

The load balancing will be calculated on the mac addresses of the participants  and this means that they will always arrive at the same answer between the two servers

ie only one physical link will ever be used. = max 1 gig since that is the max we can transfer over 1 physical link.

--------------

example 2

we are using L3 load balancing for the etherchannel

1 local server talks to 1 other local server and transfer 2 files 1 over http and one over ftp

The load balancing will be calculated on the ip address of the participants and this means that they will always arrive at the same answer between the two servers

ie only one physical link will ever be used.= max 1 gig since that is the max we can transfer over 1 physical link.

---------------

example 3

we are using L3+port (L4) load balancing for the etherchannel

1 local server talks to 1 other local server and transfer 2 files 1 over http and one over ftp

The load balancing will be calculated on the ip address and the ports of the participants and this means that depending on what port they might come to a different result as to wich channel should handle the session.

ie 1 OR 2 links depending on port will be used = max 2 gigs since we hopefully will arrive at 2 different links = 1 gig per session.

--------------

example 4

we are using l2 load balancing for the etherchannel and we have 8 links in that etherchannel.

we have 10 servers talking to 1 server transfering 1 file each.

we are using mac addresses to calculate the load balancing. Each server will have a 1 in 8 chance (since there are only 8 physical links to share the load) to be in a specific link. This most likely means that 6 of them will have 1 gig each and 4 of them will share bandwith over two physical links.

But in worst case scenario that will translate to all 8 of them sharing a single link.

--------------

so etherchannels do not loadbalance on the load it has in the channels but ut balances on mac addresses, ip addresses or ip addresses and ports.

of the traffic it recieves.

This means that in the end it will sort of balance all the traffic in the etherchannel on all the links due to the spread of the formula of doing that.

but it also means that you have to have several different loadbalancing formulas for different scenarios. none is best all the time.

-----------------------

SHORT ANSWER:

So in short, yes the etherchannel will increase the total link capacity to max 8 times, BUT each session will max be the size of one link.

the load balancing algorithm just divides the different sessions on different links, it does not spread the session over several links.

Good luck

HTH

[vancampdax]

Hobbe,

Is there documentation available on which algorithm CISCO uses for each of the load balancing options available ?

src-mac

src-ip

src-port

dst-mac

dst-ip

dst-port

src-dst-mac

src-dst-ip

src-dst-port

I would like to know this to investigate which algorithm suits our needs

[vancampdax]

Jonathan,

If I look at the MAC ADDRESSES on the port-channel and gigabit interfaces, they are all the same ( both for the source and destination servers )

So I would assume that the load would be distributed between all the interfaces, thus increasing my bandwidth to 2Gb.

I always thought that an etherchannel would increase my total bandwidth, or is this only valid if you link CISCO switches with etherchannels ?

[jonathanaxford]

Hi,

Hobbe's explanation of this is excellent. Using his example and taking your MAC addresses into account, you would be doing Layer 2 load balancing.

As the MAC addresses (both source and destination) are the same each time, the algorithm will force the traffic to always use the same path.

If you add another server, this will have a different MAC Address, so the algorithm will produce a different result and may go over the other link.

The bottom line, as Hobbe says, is that overall, you can increase your link speed by up to 8GB, but any SINGLE conversation between the same source/destination pairs (or applications) will ALWAYS go down the same channel, thereby effectively limiting it to 1GB.

If you need more than 1gb of bandwidth to go between the servers in question you will need to look into some sort of 10GBe link.

Hope this helps.

Jonathan

[vancampdax]

Correct,

I was hoping that the HP teaming software would be more intelligent and utilize both NIC's and thus increase the speed of the connection.

My colleague investigated this more deeply and found some information on how the teaming is achieved.

HP doesn't allow utilization over both NIC's for a single connection because of the packet reordering that needs to be processed afterwards to put back together the TCP stream ( if it would be using both NIC's ).

We' ve done tests with the settings of the HP teaming software and the load-balancing options and were able to get the most performance if we use the following settings ( both transmit and receive connections are load balanced over all interfaces for multiple servers )

HP teaming software:

Team Type settings: automatic ( switch is configured with LACP )

Transmit load balancing method: destination IP

Cisco switch:

src-dst-port

However...

We also noticed that instead of 1Gb full duplex we only have 1Gb distributed among both TX and RX on the HP interfaces.

As soon as I started an upload and a download the link speed dropped to around 50% for TX and RX...

Is this normal behavior ? CISCO documentation states that u have full duplex posibilities....

[hobbe]

Hi

This link is quite good at explaining it, I think. its old but good.

http://www.cisco.com/en/US/tech/tk389/tk213/technologies_tech_note09186a0080094714.shtml

there is a test you can do to se what link a packet will take

test etherchannel load-balance ......

Thanx for the ratings.

Good luck

HTH