Nexus 2k to HP Server LACP issue

[jphwilliams]

Hi,

We are having problems connecting our Nexus 2k to HP server ports in LACP mode (channel-group mode active).

The below is snippet from logs, before the port goes into suspended mode.

2013 Jun  7 17:14:21 PBR-core-1-PoC %LACP-FEX104-3-SYN_COLL_DIS_EN: WARNING: Potential Interop issue on  [Ethernet104/1/7(0x1f670180)]: SYNC, COLLECT and DISTRIBUTE flags enabled too early by partner

ERROR: Cannot set/reset lacp suspend-individual for port-channel305 that is admin up

PBR-core-1-PoC(config-if)# 2013 Jun  7 17:14:41 PBR-core-1-PoC %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet104/1/7: Ethernet104/1/7 is suspended

2013 Jun  7 17:14:40 PBR-core-1-PoC %LACP-FEX104-5-LACP_SUSPEND_INDIVIDUAL: LACP port Ethernet104/1/7(0x1f670180) of port-channel port-channel305(0x16000130) not receiving any LACP BPDUs  suspending (individual) port

Cisco: Nexus 7000 6.1.4 with N2K-C2248TP-1GE 6.1.4

->

HP Ethernet 1Gb 4-port 331FLR Adapter running Windows 2008 R2 sp1, Broadcom  Driver Version : 15.4.0.19

Has anyone had success with LACP and this setup?

Cheers,

J-Dogg

HP Ethernet 1Gb 4-port 331FLR Adapter

[Steve Fuller]

Hi,

I've not come across the first message before, but the second message (%LACP-FEX104-5-LACP_SUSPEND_INDIVIDUAL) is because the switch is not seeing any LACPDU (Link Aggregation Control Protocol Data Units) on the link from the server.

What teaming mode options do you have on the server? You need something like 802.3ad Dynamic, which means it will be running an IEEE 802.3ad Link Aggregate with LACP as the control protocol. Note that you might see references to 802.1ax instead of 802.3ad as that's the correct name for the link aggregation standard these days.

Regards

[jphwilliams]

Thanks Steve,

Yes, the server teaming is set to Dynamic. We systematically tried every single combination, the only one which works is static LAG (channel-group mode on for Cisco, switch-load balancing for HP).

Cheers,

[Steve Fuller]

Hi,

I'm not sure if you already found this, but the first error is documented, for the Nexus 5000 series at least in the System Messages and Recovery Procedures for the Cisco Nexus 5000 Family:

Error Message: LACP-3-SYN_COLL_DIS_EN: [chars] [[chars]]: SYNC, COLLECT and DISTRIBUTE flags enabled too early by partner

Explanation: Potential interop issue. Partner system seems to have enabled sync as well collecting or distributing flags too early even before actor has selected an aggregator

Recommended Action: No action is required.

I just love the Recommended Action

The message is indicating that the LACPDU from the server has the Synchronise, Collecting and Distributing flags all set on the first LACPDU it receives. When the link is first established the LACPDU is only supposed to have the Synchronise flag set, with the Collecting and Distributing flags set once the switch and host are sync'd. There's a prety good diagram of this at Networking Bodges in the Bringing Links Up section.

On the basis of this it would seem this is possibly a bug in the Broadcom driver. I see on the Broadcom website that version 15.6.0.10 of the driver is available. Are you able to try that driver on this server?

Regards

[jphwilliams]

Thanks Steve, I did see that error message, however didn't understand the full implications. You have given me some good leads to follow. Will try some different drivers.

Cheers,

[Imran Moulvi]

Hi Julian,

Were you able to resolve this issue?

Thanks.

Imran.

[l33d5un1t3d]

Hi there,

Even with Broadcom driver 16.0.0.17 we still see these interop messages.The vPC comes up ok and this is only an interop warning messge so I am not sure if it actually affects any end user experience?

[etaylormcp]

I have this same issue with a DL360 G8. I put two servers in at the same time same server same nic etc and only have this issue on one of the two servers. Will keep combing the forums on this and respond if I get anything on it,

[jphwilliams]

Ended up using static LAG with no LACP as did not have time to continue troubleshooting the problem.

Best of luck.

[David Davids]

The LACP Timeout on the switch and the server needs to be configured with the same value. Short Timeout (1s) or Long Timeout (30s).

You can do show lacp interface ethernet1xx/y/z

The results will show the switch (Local) LACP_Timeout and the server (Neighbour)

[l33d5un1t3d]

Cisco only supports ISSU when LACP neighbours are using long timeouts, so would recommend all neighbours are using Long Timeouts only for that reason.

[Cisco.Support1]

This issue is VERY common. Setting the LACP Timeout does not fix this issue:

N5548# sh lacp interface Ethernet112/1/10
Local Port: Eth112/1/10 MAC Address= 0-2a
System Identifier=0x8000,0-2a
Port Identifier=0x8000,0x2d0a
Operational key=33978
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Actor Admin State=(Ac-1:To-1:Ag-1:Sy-0:Co-0:Di-0:De-0:Ex-0)
Actor Oper State=(Ac-1:To-0:Ag-1:Sy-1:Co-1:Di-1:De-0:Ex-0)
Neighbor: 1/0
MAC Address= 2c
System Identifier=0x8000,2c
Port Identifier=0x0,0x100
Operational key=768
LACP_Activity=active
LACP_Timeout=Long Timeout (30s)
Synchronization=IN_SYNC
Collecting=true
Distributing=true
Partner Admin State=(Ac-0:To-1:Ag-0:Sy-0:Co-0:Di-0:De-0:Ex-0)
Partner Oper State=(Ac-1:To-0:Ag-1:Sy-1:Co-1:Di-1:De-0:Ex-0)
N5548# sh logg | i SYNC
2018 Jun 12 10:49:13 N5K %LACP-FEX112-3-SYN_COLL_DIS_EN: WARNING: Potential Interop issue on [Ethernet112/1/10(0x1f6f0240)]: SYNC, COLLECT and DISTRIBUTE flags enabled too e
arly by partner

Using 7.1(4)N1(1).