network bonding between Catalyst 3560 and two ubuntu 12.04 servers

[stephan.hermann]

Hi everybody,

I try to transfer data with more than 1GBit/s between two servers, but I just get the performance of one NIC (about 1GBit/s). Here's my setting:

srvnettest1 and srvnettest2 are two ubuntu 12.04 servers with three NICs each. eth0 is for management, eth1 and eth2 are the NICs which should work as a team. ;-) Here are the relevant parts of the /etc/network/interfaces:

root@srvnettest1:~# less /etc/network/interfaces
...
auto bond0
iface bond0 inet static
        address 172.16.200.100
        netmask 255.255.255.0
        bond-mode 4
        bond-miimon 100
        bond-slaves none
        bond-lacp-rate 1
        bond-primary eth1 eth2
auto eth1
allow-bond0 eth1
iface eth1 inet manual
        bond-master bond0
auto eth2
allow-bond0 eth2
iface eth2 inet manual
        bond-master bond0
...

root@srvnettest2:~# less /etc/network/interfaces
...
auto bond0
iface bond0 inet static
        address 172.16.200.200
        netmask 255.255.255.0
        bond-mode 4
        bond-miimon 100
        bond-slaves none
        bond-lacp-rate 1
        bond-primary eth1 eth2
auto eth1
allow-bond0 eth1
iface eth1 inet manual
        bond-master bond0
auto eth2
allow-bond0 eth2
iface eth2 inet manual
        bond-master bond0
...

This is the configuration of the Switch (btw it is a WS-C3560G-48TS running IOS version 12.2(55)SE)

Switch#show running-config
...
interface Port-channel10
 switchport access vlan 200
 switchport mode access
!
interface Port-channel20
 switchport access vlan 200
 switchport mode access
!
interface GigabitEthernet0/1
 switchport access vlan 200
 switchport mode access
 channel-group 10 mode active
!
interface GigabitEthernet0/2
 switchport access vlan 200
 switchport mode access
 channel-group 10 mode active
!
interface GigabitEthernet0/3
 switchport access vlan 200
 switchport mode access
 channel-group 20 mode active
!
interface GigabitEthernet0/4
 switchport access vlan 200
 switchport mode access
 channel-group 20 mode active
...

This is my etherchannel summary:

Switch#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port
Number of channel-groups in use: 2
Number of aggregators:           2
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
10     Po10(SU)        LACP      Gi0/1(P)    Gi0/2(P)
20     Po20(SU)        LACP      Gi0/3(P)    Gi0/4(P)

My testing tools are nuttcp (for tranferring) and bmon (for watching what's going on during the transfer). Unfortunately I'm not able to transfer with more than about 1GBit/s:

root@srvnettest2:~# nuttcp -i1 172.16.200.100
   97.1875 MB /   1.00 sec =  815.2409 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.4763 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.7321 Mbps     0 retrans
   98.1250 MB /   1.00 sec =  823.1001 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.5306 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.7560 Mbps     0 retrans
   98.1250 MB /   1.00 sec =  822.9890 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.6753 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.5528 Mbps     0 retrans
   98.0625 MB /   1.00 sec =  822.7058 Mbps     0 retrans
  982.5000 MB /  10.03 sec =  821.9606 Mbps 21 %TX 37 %RX 0 retrans 0.32 msRTT

In bmon I can see that one NIC (eth1) of bond0 is doing the uplink, and the other one (eth2) is doing the downlink:

  #   Interface                RX Rate         RX #     TX Rate         TX #
qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq
srvnettest2 (source: local)
  0   lo                         0.00B            0       0.00B            0
  1   eth2                     269.26KiB       4175       0.00B            0
  2   eth1                     123.00B            0     102.56MiB      71030
  3   eth0                     179.00B            2     491.00B            1
  4   bond0                    269.38KiB       4176     102.56MiB      71030

I tried a lot but now I haven't got any clue what to do or what to try next. Admittedly I have no deep understanding of Cisco etherchannels yet,so I guess my mistake is somewhere in the IOS configuration.

Thanks a lot for your support and many greets from Germany

Stephan

[Peter Paluch · Accepted solution]

Hello Stephan,

With EtherChannels, a single stream (the flow of frames/packets having the same source and destination) is always carried through a single link only. Cisco implementation does not implement per-packet load balancing across links in an EtherChannel, and it avoids it for a good reason: the frames could get reordered, something that plain Ethernet never should do. That means that you will not see a bandwidth improvement over the speed of any single link in your EtherChannel for a single flow. It is only the aggregate bandwidth for multiple flow that will increase. The advantage of EtherChannel therefore becomes obvious if your server starts handling multiple conversations and multiple flows.

Best regards,

Peter

[Giuseppe Larosa · Accepted solution]

Hello Stephan,

despite the high traffic volume your test traffic is classified as a single IP flow with specific IP source address, IP destination address, MAC source address, MAC destination address.

Unless using a load balancing algorythm at OSI layer 4 ( TCP or UDP ports) the traffic is classified as a single flow and it does use a single member link per direction.

So I would say that what you see is normal and it is expected with your setup.

Note:

according to C3560 configuration guide for 12.2(55)SE load balancing at OSI layer 4 is not avaiable

see

http://www.cisco.com/en/US/docs/switches/lan/catalyst3560/software/release/12.2_55_se/configuration/guide/swethchl.html#wp1276203

Configure an EtherChannel load-balancing method.

The default is src-mac.

Select one of these load-distribution methods:

•dst-ip—Load distribution is based on the destination-host IP address.

•dst-mac—Load distribution is based on the destination-host MAC address of the incoming packet.

•src-dst-ip—Load distribution is based on the source-and-destination host-IP address.

•src-dst-mac—Load distribution is based on the source-and-destination host-MAC address.

•src-ip—Load distribution is based on the source-host IP address.

•src-mac—Load distribution is based on the source-MAC address of the incoming packet.

Hope to help

Giuseppe

[stephan.hermann]

Hello Peter, hello Guiseppe,

thank you so much for these information!

Good to know that there's nothing terribly wrong in my setting - and bad to know that I spent much time with this "problem"! ;-)

Many greets

Stephan