LACP between Linux host and 2 3560Es

[Lsimancek_2]

Hi,

Looking for someone that might know what we are missing in this config.

We have a Linux server with 2 ports bonded together using 802.3ad LACP (config below)

Bonding Mode: IEEE 802.3ad Dynamic link aggregation

Transmit Hash Policy: layer3+4 (1)

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

These to 2 ports go to 2 separate switches (Catalyst 3560E-48td ver 12.2(4)6se) with an etherchannel set up on the ports (config below - exact same on both switches).

interface Port-channel4
switchport access vlan 3xx
switchport mode access

interface GigabitEthernet0/4
description Linux box

switchport access vlan 3xx

switchport mode access
no logging event link-status
no cdp enable
channel-group 4 mode active
spanning-tree portfast
spanning-tree bpduguard enable
spanning-tree guard loop

These 2 switches are 802.1q trunked to ARs ( 6509s) with a 802.1q trunk inbetween the two 6509s.

The issue is that with this setup when we shut down g0/4 (eth0 on linux box) on sw1 our pings drop (does not fail over to the other link). The only time the failover works is 1) if we ping the linx box(eth1) from sw2  OR  2) if we wait 30 minutes we saw the pings start again. If we shut down g0/4 on sw2 we experience a long and inconsistent delay before the server responds to pings.

It also appears that:

eth0/sw1 g0/4 is only receving packets

eth1/sw2 g0/4 is transmit/receiving packets

Our assumption was that with LACP on both the switch and server, the server would failover and not loose its connection (we have this current setup with our windows boxes)

Their is a drawing attached to help visualize the setup.

Thank you in advance for the help.

Lindsay

[Lsimancek_2]

Revised drawing

We don't have the dual links connected

[Aaron Harrison]

Hi Lindsay

The problem here is that your two switches are seperate devices, and aren't aware of the port states on the other switch of the pair; let alone what is actually connected to them.

For LACP to work, the two ports of the server must be connected to the same physical switch.

The exceptions to this rule are (for example) if you have a stack of 3750 switches using StackWise, as these then become a single managed device but with the advantage of resilience in the hardware.

With 3560s you will need to connect the two ports to one switch (if you need the throughput), or connect the server to the two switches as standard access ports, and configure the server for Adapter Fault Tolerance (i.e. simple failover) if the resilience is the important thing.

Regards

Aaron

Please rate helpful posts..    

[Jon Marshall]

Lindsay

Are you sure you have it working for other devices because etherchannel is not supported across switches ie. your linux server would have to have both it's connections to the same 3560 switch for it to work.

If you had 3750 switches or 6500 VSS switches then you can support etherchannel across switches but not with 3560s.

Jon

[Lsimancek_2]

Yes I am certain that we have this currently functioning. The only difference is that each switch has 2 links - 1 to each AR. In our test enviroment we have this. The switches have uplinks trunked, and the 4 gig trunk between the 2 ARs.

Although the more I'm looking into this it appears that this maybe a way the HP Proliant teaming software works. We are selecting "auto/auto" which implies that it uses 802.3ad but maybe this is additional functionality ontop of it ....

[Jon Marshall]

Lindsay

3560 switches do not support an etherchannel that is spread between 2 of the 3560 switches. Which IOS version are you running on your 3560 switches ?

Also for one of the windows servers that works can you do a "sh etherchannel summary" on both switches that the etherchannel is split across and make it clear which etherchannel is for the server and then post the results.

Jon

[Lsimancek_2]

In the the test enviroment we have 3560E

Switch Ports Model              SW Version            SW Image
------ ----- -----              ----------            ----------
*    1 54    WS-C3560E-48TD     12.2(46)SE            C3560E-UNIVERSALK9-M

For our prod enviroment; we have 3750Es that ARE NOT stacked (and stacking is not an opition)

Switch 1

Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   30     WS-C3750E-24TD     12.2(35)SE5             C3750E-UNIVERSAL-M

switch 1#sh etherchannel summary
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 5
Number of aggregators:           5

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/1(P)  Gi1/0/24(D)
2      Po2(SU)         LACP      Gi1/0/2(P)
3      Po3(SU)         LACP      Gi1/0/3(P)
4      Po4(SU)         LACP      Gi1/0/4(P)
5      Po5(SU)         LACP      Gi1/0/5(P)

Switch 2

Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   30     WS-C3750E-24TD     12.2(35)SE5             C3750E-UNIVERSAL-M

switch2#  sh etherchannel sum
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 5
Number of aggregators:           5

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/1(P)  Gi1/0/24(D)
2      Po2(SU)         LACP      Gi1/0/2(P)
3      Po3(SU)         LACP      Gi1/0/3(P)
4      Po4(SU)         LACP      Gi1/0/4(P)
5      Po5(SU)         LACP      Gi1/0/5(P)

[Jon Marshall]

For our prod enviroment; we have 3750Es that ARE NOT stacked (and stacking is not an opition)
Switch 1

Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   30     WS-C3750E-24TD     12.2(35)SE5             C3750E-UNIVERSAL-M
switch 1#sh etherchannel summary
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 5
Number of aggregators:           5
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/1(P)  Gi1/0/24(D)
2      Po2(SU)         LACP      Gi1/0/2(P)
3      Po3(SU)         LACP      Gi1/0/3(P)
4      Po4(SU)         LACP      Gi1/0/4(P)
5      Po5(SU)         LACP      Gi1/0/5(P)
Switch 2
Switch   Ports  Model              SW Version              SW Image
------   -----  -----              ----------              ----------
*    1   30     WS-C3750E-24TD     12.2(35)SE5             C3750E-UNIVERSAL-M
switch2#  sh etherchannel sum
Flags:  D - down        P - in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator
        u - unsuitable for bundling
        w - waiting to be aggregated
        d - default port

Number of channel-groups in use: 5
Number of aggregators:           5
Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
1      Po1(SU)         LACP      Gi1/0/1(P)  Gi1/0/24(D)
2      Po2(SU)         LACP      Gi1/0/2(P)
3      Po3(SU)         LACP      Gi1/0/3(P)
4      Po4(SU)         LACP      Gi1/0/4(P)
5      Po5(SU)         LACP      Gi1/0/5(P)

Lindsay

These etherchannels are just seen as 2 separate links ie. if the 3750 is not stacked but simply interconnected with a L2 trunk then what you are seeing is simply an etherchannel on one switch with a single port eg. Po2 - gi1/0/2 and an etherchannel on the other switch with a single port but the 2 switches are not aware of the others etherchannel ie. it does not become an etherchannel with 2 ports just because it is referenced as Po2 on each switch.

if they were seen as one etherchannel then Po2 on the switch would report seeing 2 ports in the etherchannel.

Jon

[Lsimancek_2]

it is understood that the 2 switches will not see the etherchannel. But that does not change the fact that this is working with the HP servers, which we did a lot of testing and development with their docs as well as Cisco docs on how to get the 802.3ad teaming working for single ping lost when a failover occurs. It was our belief that the trunking both between the access switches and aggregate switches and in between the aggregate switches would extend the the LACP packets between the 2 NICs. This has been proven that it is not the networks functionality but rather the server functionality of this that makes it work. It does appear, after looking at it again, that the etherchannel is used just to tag the packet so the server accepts rather then for the full functionality of the LACP protocol.

It appears that it may be the HP software that is adding in that functionaly.

[Aaron Harrison]

Hi

Just to throw in my penny worth again - you aren't using LACP. Or at least, your server may be running an LACP mode, but the two seperate switches it is connected to will not negotiate an etherchannel with it.

What you may be doing is running the in-between option between etherchannel/LACP and adapter fault tolerance, where the server load balances traffic in the transmit direction, but traffic to the server just goes to one NIC.

Do a 'show mac-address-table int x/x' on each of the switch ports attached to the server, and a 'show arp' to see what the L3 IP of the server is mapped to. If you are using transmit load balancing, then you should see different MAC addresses on each port, and the IP of the server linked to either one of those addresses (or a virtual one, can't recall).

The various modes are talked about in this HP white paper:

http://bizsupport.austin.hp.com/bc/docs/support/SupportManual/c01415139/c01415139.pdf

My advice would be to use auto/auto for your speed and duplex. Use a specific configured type of resilience on the NIC team, and understand the selection you are making.

Regards

Aaron