Cisco UCS C3260 M4 and LACP/Bonding/802.3ad with Linux

[Vasyl]

I have Cisco UCS C3260 chassis with M4 nodes. I believe they have Cisco VIC Ethernet NIC 2x 40G QSFP+ adapters.

Do they support 802.3ad/LACP configuration?

I am trying to configure CentOS Linux to bond the interfaces together on a single node. But it keeps failing.

The interfaces connected with 40G fiber (QSFP-40G-SR) modules to the same Cisco Nexus sw-9396PX switch on 40G ports.

1. Once I put these 2 ports into port-channel group, it becomes “suspended” almost instantly for both ports.
2. The Linux machine complains “bond0: Warning: No 802.3ad response from the link partner for any adapters in the bond”

I tried both active and passive configuration, with or without mtu, with speed and no speed defined. No luck.

Switch configuration:

<cut>
feature lacp
<cut>
interface port-channel31
  description "Linux-C3260-n2"
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10, 200,210,220,300-399
<cut>
interface Ethernet2/4
  description " Linux-C3260-n2.enp9s0"
  lacp rate fast
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10, 200,210,220,300-399
  speed 40000
  mtu 9216
  channel-group 31 mode active
  no shutdown
interface Ethernet2/6
  description " Linux-C3260-n2.enp10s0"
  lacp rate fast
  switchport mode trunk
  switchport trunk native vlan 10
  switchport trunk allowed vlan 10, 200,210,220,300-399
  speed 40000
  mtu 9216
  channel-group 31 mode active
  no shutdown

Linux configuration.

• Interface bond0
• Mode: 802.3ad
• Link monitoring: MII
• Monitor freq: 100ms
• Slaves: enp9s0, enp10s0

[marce1000]

 - Check this thread : https://community.cisco.com/t5/unified-computing-system-discussions/cisco-ucs-link-aggregation/td-p/4425937

 M.

[Vasyl]

Still no luck with the bonding/port-channel

I have tried of course mode “4” on the Linux side. I tired both modules from Linux – bonding and teaming.

The Linux configs (NetworkManager-way), bonding:

# cat /etc/NetworkManager/system-connections/bond0.nmconnection:

[connection]
id=bond0
uuid=ef908b70-bec8-4f2e-a0a2-db3146fee8e3
type=bond
interface-name=bond0

[bond]
miimon=1000
mode=802.3ad

[ipv4]
address1=192.168.222.232/24,192.168.222.1
dns=192.168.222.1;
method=manual

[ipv6]
addr-gen-mode=default
method=disabled

[proxy]

# cat /etc/NetworkManager/system-connections/bond0-port1.nmconnection
[connection]
id=bond0-port1
uuid=093c6bf7-f306-498e-a0eb-2de52da4002a
type=ethernet
interface-name=enp9s0
master=bond0
slave-type=bond

[ethernet]

[bond-port]

# cat /etc/NetworkManager/system-connections/bond0-port2.nmconnection

[connection]
id=bond0-port2
uuid=71a6a6f4-0379-4f65-85a7-61bcb2336c84
type=ethernet
interface-name=enp10s0
master=bond0
slave-type=bond

[ethernet]

[bond-port]

The Linux configs (NetworkManager-way), teaming:
# team0.nmconnection
[connection]
id=team0
uuid=c38e1cd5-aa0e-46a1-9c97-45f37c220532
type=team
interface-name=team0

[team]
config={ "runner": { "name": "lacp" }, "link_watch": { "name": "ethtool" } }

[ipv4]
address1=192.168.222.232/24,192.168.222.1
dns=192.168.222.1;
method=manual

[ipv6]
addr-gen-mode=default
method=disabled

[proxy]

# team0-port1.nmconnection
[connection]
id=team0-port1
uuid=dc6b695b-a386-4e1c-a534-ac9c691d2ff5
type=ethernet
interface-name=enp9s0
master=team0
slave-type=team

[ethernet]

[team-port]

# team0-port2.nmconnection
[connection]
id=team0-port2
uuid=fc822203-31ae-4a80-a8ee-783a75a74a01
type=ethernet
interface-name=enp10s0
master=team0
slave-type=team

[ethernet]

[team-port]

--- Cisco Nexus config:
sw-n9k-c9396px-01# show running-config
version 6.1(2)I3(3a)
<cut>
feature lacp
<cut>
vlan dot1Q tag native
<cut>
vlan 1,10,200,210,220,300-399
<cut>
interface port-channel31
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 1,10,200,210,220,300-399
<cut>
interface Ethernet2/4
description "Linux-C3260-n2.enp9s0"
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 1,10,200,210,220,300-399
channel-group 31 mode active
no shutdown
<cut>
interface Ethernet2/6
description "Linux-C3260-n2.enp10s0"
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 1,10,100,110,120,130,200-299
channel-group 31 mode active
no shutdown
<cut>

[Steven Tardy]

I think you need to configure the Cisco VIC (through the UCS CIMC) to tag VLAN 10.

See doc: https://www.cisco.com/c/en/us/support/docs/servers-unified-computing/ucs-c-series-rack-servers/117637-technote-UCS-00.html

Section:

Upstream Switch-Dependent Teaming

In that doc:

The VIC always tags packets with an 802.1p header.

To expand upon that. . . . All non-Cisco VIC NICs send OS untagged packets without a tag (untagged or native is what upstream switches call this). All Cisco VIC NICs send OS untagged packets with a tag. . . . a tag of VLAN 0. This is the source of a lot of headache and head scratching for those new to configuring UCS VIC with upstream devices.

What is happening now is:

• Linux sends LACP BPDUs without a VLAN tag.
• The VIC changes the untagged packet to tag VLAN 0.
• Upstream switch see a LACP BPDU in VLAN 0 and ignores it.

In the other direction:

• Upstream switch sends LACP BDPU in its native VLAN of VLAN 10.
• The VIC sends along the BPDU in VLAN 10.
• The OS is expecting an untagged (NOT VLAN 10) LACP BPDU.
• The teaming never sees the upstream LACP BPDU.

Can either:

1. Configure the UCS VIC vNIC to tag/untag VLAN 10 detailed in the doc above.
2. Or configure the Linux OS to tag VLAN 10.

Hope that helps.