cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
10380
Views
7
Helpful
3
Replies

network bonding between Catalyst 3560 and two ubuntu 12.04 servers

stephan.hermann
Level 1
Level 1

Hi everybody,

I try to transfer data with more than 1GBit/s between two servers, but I just get the performance of one NIC (about 1GBit/s). Here's my setting:

srvnettest1 and srvnettest2 are two ubuntu 12.04 servers with three NICs each. eth0 is for management, eth1 and eth2 are the NICs which should work as a team. ;-) Here are the relevant parts of the /etc/network/interfaces:

root@srvnettest1:~# less /etc/network/interfaces

...

auto bond0

iface bond0 inet static

        address 172.16.200.100

        netmask 255.255.255.0

        bond-mode 4

        bond-miimon 100

        bond-slaves none

        bond-lacp-rate 1

        bond-primary eth1 eth2

auto eth1

allow-bond0 eth1

iface eth1 inet manual

        bond-master bond0

auto eth2

allow-bond0 eth2

iface eth2 inet manual

        bond-master bond0

...

root@srvnettest2:~# less /etc/network/interfaces

...

auto bond0

iface bond0 inet static

        address 172.16.200.200

        netmask 255.255.255.0

        bond-mode 4

        bond-miimon 100

        bond-slaves none

        bond-lacp-rate 1

        bond-primary eth1 eth2

auto eth1

allow-bond0 eth1

iface eth1 inet manual

        bond-master bond0

auto eth2

allow-bond0 eth2

iface eth2 inet manual

        bond-master bond0

...

This is the configuration of the Switch (btw it is a WS-C3560G-48TS running IOS version 12.2(55)SE)

Switch#show running-config

...

interface Port-channel10

switchport access vlan 200

switchport mode access

!

interface Port-channel20

switchport access vlan 200

switchport mode access

!

interface GigabitEthernet0/1

switchport access vlan 200

switchport mode access

channel-group 10 mode active

!

interface GigabitEthernet0/2

switchport access vlan 200

switchport mode access

channel-group 10 mode active

!

interface GigabitEthernet0/3

switchport access vlan 200

switchport mode access

channel-group 20 mode active

!

interface GigabitEthernet0/4

switchport access vlan 200

switchport mode access

channel-group 20 mode active

...

This is my etherchannel summary:

Switch#show etherchannel summary

Flags:  D - down        P - bundled in port-channel

        I - stand-alone s - suspended

        H - Hot-standby (LACP only)

        R - Layer3      S - Layer2

        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met

        u - unsuitable for bundling

        w - waiting to be aggregated

        d - default port

Number of channel-groups in use: 2

Number of aggregators:           2

Group  Port-channel  Protocol    Ports

------+-------------+-----------+-----------------------------------------------

10     Po10(SU)        LACP      Gi0/1(P)    Gi0/2(P)

20     Po20(SU)        LACP      Gi0/3(P)    Gi0/4(P)

My testing tools are nuttcp (for tranferring) and bmon (for watching what's going on during the transfer). Unfortunately I'm not able to transfer with more than about 1GBit/s:

root@srvnettest2:~# nuttcp -i1 172.16.200.100

   97.1875 MB /   1.00 sec =  815.2409 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.4763 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.7321 Mbps     0 retrans

   98.1250 MB /   1.00 sec =  823.1001 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.5306 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.7560 Mbps     0 retrans

   98.1250 MB /   1.00 sec =  822.9890 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.6753 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.5528 Mbps     0 retrans

   98.0625 MB /   1.00 sec =  822.7058 Mbps     0 retrans

  982.5000 MB /  10.03 sec =  821.9606 Mbps 21 %TX 37 %RX 0 retrans 0.32 msRTT

In bmon I can see that one NIC (eth1) of bond0 is doing the uplink, and the other one (eth2) is doing the downlink:

  #   Interface                RX Rate         RX #     TX Rate         TX #

qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq

srvnettest2 (source: local)

  0   lo                         0.00B            0       0.00B            0

  1   eth2                     269.26KiB       4175       0.00B            0

  2   eth1                     123.00B            0     102.56MiB      71030

  3   eth0                     179.00B            2     491.00B            1

  4   bond0                    269.38KiB       4176     102.56MiB      71030


I tried a lot but now I haven't got any clue what to do or what to try next. Admittedly I have no deep understanding of Cisco etherchannels yet,so I guess my mistake is somewhere in the IOS configuration.

Thanks a lot for your support and many greets from Germany

Stephan

2 Accepted Solutions

Accepted Solutions

Peter Paluch
Cisco Employee
Cisco Employee

Hello Stephan,

With EtherChannels, a single stream (the flow of frames/packets having the same source and destination) is always carried through a single link only. Cisco implementation does not implement per-packet load balancing across links in an EtherChannel, and it avoids it for a good reason: the frames could get reordered, something that plain Ethernet never should do. That means that you will not see a bandwidth improvement over the speed of any single link in your EtherChannel for a single flow. It is only the aggregate bandwidth for multiple flow that will increase. The advantage of EtherChannel therefore becomes obvious if your server starts handling multiple conversations and multiple flows.

Best regards,

Peter

View solution in original post

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello Stephan,

despite the high traffic volume your test traffic is classified as a single IP flow with specific IP source address, IP destination address, MAC source address, MAC destination address.

Unless using a load balancing algorythm at OSI layer 4 ( TCP or UDP ports) the traffic is classified as a single flow and it does use a single member link per direction.

So I would say that what you see is normal and it is expected with your setup.

Note:

according to C3560 configuration guide for 12.2(55)SE load balancing at OSI layer 4 is not avaiable

see

http://www.cisco.com/en/US/docs/switches/lan/catalyst3560/software/release/12.2_55_se/configuration/guide/swethchl.html#wp1276203

Configure an EtherChannel load-balancing method.

The default is src-mac.

Select one of these load-distribution methods:

dst-ip—Load distribution is based on the destination-host IP address.

dst-mac—Load distribution is based on the destination-host MAC address of the incoming packet.

src-dst-ip—Load distribution is based on the source-and-destination host-IP address.

src-dst-mac—Load distribution is based on the source-and-destination host-MAC address.

src-ip—Load distribution is based on the source-host IP address.

src-mac—Load distribution is based on the source-MAC address of the incoming packet.

Hope to help

Giuseppe

View solution in original post

3 Replies 3

Peter Paluch
Cisco Employee
Cisco Employee

Hello Stephan,

With EtherChannels, a single stream (the flow of frames/packets having the same source and destination) is always carried through a single link only. Cisco implementation does not implement per-packet load balancing across links in an EtherChannel, and it avoids it for a good reason: the frames could get reordered, something that plain Ethernet never should do. That means that you will not see a bandwidth improvement over the speed of any single link in your EtherChannel for a single flow. It is only the aggregate bandwidth for multiple flow that will increase. The advantage of EtherChannel therefore becomes obvious if your server starts handling multiple conversations and multiple flows.

Best regards,

Peter

Giuseppe Larosa
Hall of Fame
Hall of Fame

Hello Stephan,

despite the high traffic volume your test traffic is classified as a single IP flow with specific IP source address, IP destination address, MAC source address, MAC destination address.

Unless using a load balancing algorythm at OSI layer 4 ( TCP or UDP ports) the traffic is classified as a single flow and it does use a single member link per direction.

So I would say that what you see is normal and it is expected with your setup.

Note:

according to C3560 configuration guide for 12.2(55)SE load balancing at OSI layer 4 is not avaiable

see

http://www.cisco.com/en/US/docs/switches/lan/catalyst3560/software/release/12.2_55_se/configuration/guide/swethchl.html#wp1276203

Configure an EtherChannel load-balancing method.

The default is src-mac.

Select one of these load-distribution methods:

dst-ip—Load distribution is based on the destination-host IP address.

dst-mac—Load distribution is based on the destination-host MAC address of the incoming packet.

src-dst-ip—Load distribution is based on the source-and-destination host-IP address.

src-dst-mac—Load distribution is based on the source-and-destination host-MAC address.

src-ip—Load distribution is based on the source-host IP address.

src-mac—Load distribution is based on the source-MAC address of the incoming packet.

Hope to help

Giuseppe

Hello Peter, hello Guiseppe,

thank you so much for these information!

Good to know that there's nothing terribly wrong in my setting - and bad to know that I spent much time with this "problem"! ;-)

Many greets

Stephan

Review Cisco Networking for a $25 gift card