Nexus 7000 - lacp suspend-individual

[d.elefsiniotis]

Hello all community members,

I would like to discuss here a couple of NX-OS enhancements that Nexus 7000 support regarding the LACP protocol. The two mentioned enhancements are :

lacp suspend-individual

lacp graceful-convergence

Cisco's documentation is ambiguous from NX-OS version to NX-OS version. What i would like to know is the meaning of each command in order to utilize it or not (default is to be enabled) when connecting to a non-nexus peer. The concept is that servers are connected to Nexus 7000 via vPC and that i would like to use the PXE boot/build process on them.

According to Cisco's latest documentation "Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide (Release 6.x)" :

By default, LACP sets a port to the suspended state if it does not receive an LACP PDU from the peer. In some cases, although this feature helps in preventing loops created due to misconfigurations, it can cause servers to fail to boot up because they require LACP to logically bring up the port. You can put a port into an individual state by using the lacp suspend-individual command.

According to my understanding, when PXE boot is selected for server, it only uses one  the two 10GE NICs i.e., the first physical NIC on the host. The  second  interface is operational i.e., "link" is established between the  host  and the Nexus switch, but the PXE client on the host does nothing  with  the NIC. All  traffic from the server is received by the Nexus switch on the physical  port that connects to the first physical NIC, but the MAC associated  with that NIC will be learnt by the Nexus 7000 via its port-channel  interface (vPC). When the switch sends traffic to the server it is just as  likely to utilise the physical port that connects to the servers second  physical NIC, which during the PXE boot / build process, is not  operational. This traffic will be dropped and the build process will  fail.

If the above statement from Cisco's documentation is correct, with lacp suspend-individual (default) enabled, PXE boot/build process should work. Nexus will not receive LACP PDU from 2nd non-operational NIC and will suspend this interface during the time of PXE boot. The remaining link, connected to 1st server NIC (operational) acts as if it were a single switch port.

Regarding the second command I attach the relevant explanations from two different NX-OS versions configuration guides :

According to Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide (Release 6.x) :

When the Cisco Nexus 7000 Series device is connected to a non-Nexus peer, its graceful failover defaults may delay the time taken for a disabled port to be brought down or cause traffic from the peer to be lost. To address these conditions, the lacp graceful-convergence command was added.

According to Cisco Nexus 7000 Series NX-OS Interfaces Configuration Guide, Release 4.x (December 2011) :

By default LACP graceful convergence is enabled. In situations where you need to support LACP interoperability with devices where the graceful failover defaults may delay the time taken for a disabled port to be brought down or cause traffic from the peer to be lost, you can disable convergence.

From the above it is obvous that two statements are mutually exclusive!!! I believe the latter is the correct. Please help me clarify the above.

In my opinion when a Nexus 7000 is connected via LACP with a non-nexus peer the configuration should be :

lacp suspend-individual (PXE boot/build will work this way)

no lacp graceful-convergence

Thanks in advance for any help provided.

Best Regards

Dimitris

[Kevin Dorrell]

That is really strange. As far as I know, the function of lacp suspend-individual is exactly the opposite to what is written there.  I thought the default was no lacp suspend-individual, meaning that if the host does not talk LACP then the switch would treat the links as individual and would not suspend them.  And that if you configure lacp suspend-individual, then if the host does not talk LACP then the switch would suspend the ports.

But then I only know about 5500 at 5.1(3)N3, and not the 7000. Here is my evidence on the 5000:

    http://www.cisco.com/en/US/docs/switches/datacenter/nexus5000/sw/command/reference/layer2/n5k-l2_cmds_l.html#wp2679029

Kevin Dorrell

Luxembourg

[d.elefsiniotis]

Hello Kevin,

thanks for your response. According to common sense "suspend-individual" means to me suspend any individual port within the port-channel that do not talk LACP with the peer.

Unfortunately, Cisco's documentation is ambiguous and misleading from HW to HW and from software version to sofware version.

Has anyone really test this feature and can comment on what exactly it means. I have previously clearly written my thoughts and what I'm trying to accomplish by using (or not using it). PXE boot/build of a server is my main concern. Server is connected with LACP and vPC with two Nexus 7000.

BR/Dimitris

[Kevin Dorrell · Accepted solution]

I pretty sure that on the 5000 it behaves as I have described - I have used links both individually and collectively, and it works OK.  I had a look at the CLI help for the command, and it says:

suspend-individual     Configure lacp port-channel state. Enabling this will cause lacp to put the port to suspend instead of inidividual state in case it does not get LACP BDPU from the peer ports in the port-channel.

Kevin Dorrell

Luxembourg

[d.elefsiniotis]

In Nexux 7K we have the exact opposite behaviour.

switch(config-if)# lacp ?

  graceful-convergence  Configure port-channel lacp graceful convergence. Disable this only with lacp ports connected to

                        Non-Nexus peer. Disabling this with Nexus peer can lead to port suspension

  suspend-individual    Configure lacp port-channel state. Disabling this will cause lacp to put the port to individual

                        state and not suspend the port in case it does not get LACP BPDU from the peer ports in the

                        port-channel

Thanks for your answers Kevin.

BR/Dimitris

[r.heitmann]

11 years later clients are still confused about this.

I'll reformat the original Cisco-Help-Message to clear up any ambiguities once and for all:

• Disabling "lacp suspend-individual" will cause lacp to put the port to individual state

which sounds absolutely logical

• no suspend? => Port's not gonna suspended => operated as "individual"

and...

• enable "lacp suspend-individual" => suspend the port in case it does not get LACP BPDU from the peer ports in the port-channel

___

Note: @N9K-Switches "lacp suspend-individual" is the default, which imho seems to align perfect for switch-to-switch connections or normal servers who don't want to PXE which is just a cornercase, isn't it?

Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

[BenBen]

Verified on the Nexus 7000. With following configurations:

!

interface port-channel5

  lacp suspend-individual

!

interface Ethernet101/1/7

  channel-group 5 mode active

!

I see the following:

STC-N7004-1-SHR-PROD-ETH# sh int Eth101/1/7
Ethernet101/1/7 is down (suspended(no LACP PDUs))
admin state is up

!

N7706-A(config-if)# lacp suspend-individu?
suspend-individual Configure lacp port-channel state. Disabling this will cause lacp to put the port to individual state and not suspend the port in case it does not get LACP BPDU
from the peer ports in the port-channel

!

So on Nexus 7000, suspend-individual means suspend if no LACP PDU is heard. So Nexus 7000 and Nexus 5000 behave the same. But the description of the command is confusing or opposite on Nexus 7000.

This is a port-channel level configuration, not a physical port level configuration. If we apply "no lacp suspend-individual", all the interfaces in this port channel should be in "individual" mode and up (if the host physical ports are up) on the switch side.

If the PXE server only uses the first physical port, the switch will only see MAC address coming from the first port. The switch to PXE host communication should use the first port only and not use other ports. This is just my assumption.

Dimitris, please let me know if you have the real answer.

Thanks.

-Frank