Nexus, LACP, Graceful Convergence

[mfarrenkopf]

I have a pair of Nexus 6001 switches running NX-OS 6.0(2)N2(7).  They are configured in a vPC configuration.  vPC is stable.

My server team and I connected a standalone server to the pair of switches as a vPC port-channel.  It is a VMWare host for our UC environment.  Cisco's recommendation is to set "no lacp graceful-convergence" for everything except Cisco devices.  However, with that command configured an interface will usually be marked as Hot Standby instead of being up in the port channel.

I don't need assistance with the configuration.  But I want to understand, in detail, what is going on with LACP -- what exactly changes in the LACP exchange -- when "no lacp graceful-convergence" is configured.

I grabbed two captures.  The first was of an interface that did not have "no lacp graceful-convergence" configured.  What I saw was that after several exchanges (the switch was sending Activity/Timeout/Aggregation, the server was sending Activity/Aggregation), I began sending Activity/Timeout/Aggregation/Synchronization.  After that, the server added the Synchronization bit as well.  The dialog transitioned to each side turning on its Collecting and Distributing bits, then the link was fully aggregated and ready for use.

The second capture, however, was from an interface that DID have "no lacp graceful-convergence" configured. The exchange was similar, except that neither side set the Synchronization bit.  I believe this was the cause of the interface going into Hot Standby.

So how do end devices misbehave that "no lacp graceful-convergence" fixes?  And how does having that setting fix the problem?  I'm very interested in the nuts and bolts of what's going on in the protocol.

Thanks,

Matt

[Tyler Jeffords · Accepted solution]

Matt,

Cisco recommends disabling lacp graceful-convergence for connecting with non-LACP compliant devices (for example server, FW, loadbalancer). This is recommended to reduce packet loss during port bring up or failure scenarios.

The LACP flags that are exchanged between the peer ports include:

Sync (S): Set to indicate that the system is ready and willing to use this link in the bundle to carry traffic. Cleared to indicate the link is not usable or is in standby mode.

Collecting (C): Set to indicate that traffic received on this interface will be processed by the device. Cleared otherwise.

Distributing (D): Set to indicate that the device is using this link transmit traffic. Cleared otherwise.

Essentially, when the ports come up, the operation is as follows:

Source Peer
S:0 C:0 D:0 S:0 C:0 D:0
S:1 C:0 D:0 ->
<- S:1 C:0 D:0 ?> At this time, peer LACP members have been chosen
S:1 C:1 D:0 -> ?> Source port indicates that it is ready to accept traffic
<- S:1 C:1 D:0 ?> Destination port indicates that it is ready to accept traffic
S:1 C:1 D:1 -> ?> Source port indicates that it is ready to send traffic
<- S:1 C:1 D:1 ?> Destination port indicates that it is ready to send traffic

At this time, both ports are up and ready to exchange traffic. However, there are non-LACP compliant devices which begin transmitting as soon as the S flag is cleared on either side. Since the peer has not reached a C state, all these packets will be essentially dropped. To avoid losing packets, we advice to turn off graceful convergence, in which case the S packet will wait for a longer time before being sent out and thereby prevent the peer port from sending traffic before the Source port can accept it. You will want to confirm with the vendor if the peer device is LACP compliant to confirm if disabling lacp graceful-convergence is needed.

HTH,

Tyler Jeffords