06-14-2016 05:22 AM - edited 03-08-2019 06:11 AM
I have two lacp bonds configured on my linux box, identical to each other. One is on an iscsi vlan, the other on my access vlan.
The iscsi bond looks fine:
# cat /proc/net/bonding/bond1
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer3+4 (1)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 2
Actor Key: 9
Partner Key: 12
Partner Mac Address: 00:13:80:73:2f:80
Slave Interface: eth2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:f2:e9:0d:e1:de
Aggregator ID: 1
Slave queue ID: 0
Slave Interface: eth3
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:f2:e9:0d:e1:df
Aggregator ID: 1
Slave queue ID: 0
but the access bond is funny in that it shows Number of ports: 1
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)
Bonding Mode: IEEE 802.3ad Dynamic link aggregation
Transmit Hash Policy: layer2 (0)
MII Status: up
MII Polling Interval (ms): 0
Up Delay (ms): 0
Down Delay (ms): 0
802.3ad info
LACP rate: slow
Min links: 0
Aggregator selection policy (ad_select): stable
Active Aggregator Info:
Aggregator ID: 1
Number of ports: 1
Actor Key: 9
Partner Key: 11
Partner Mac Address: 00:13:80:73:2f:80
Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:f2:e9:0d:e1:dc
Aggregator ID: 1
Slave queue ID: 0
Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 40:f2:e9:0d:e1:dd
Aggregator ID: 2
Slave queue ID: 0
From both the host and the switch side, both bonds are configured identically:
# cat /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
DEFROUTE=yes
IPV6INIT=no
NAME="System bond0"
PEERDNS=yes
PEERROUTES=yes
IPADDR=10.1.11.60
NETMASK=255.255.255.0
GATEWAY=10.1.11.1
BONDING_OPTS=mode=4 miimon=100 xmit_hash_policy=layer3+4
# cat /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
TYPE=Ethernet
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=static
IPV6INIT=no
IPADDR=192.43.216.60
NETMASK=255.255.255.0
MTU=9000
BONDING_OPTS="mode=4 miimon=100 xmit_hash_policy=layer3+4"
and from the switch side, it looks fine
interface Port-channel11
description 'access'
switchport mode access
switchport nonegotiate
load-interval 30
spanning-tree portfast
!
interface Port-channel12
description 'iscsi'
switchport access vlan 216
switchport mode access
switchport nonegotiate
load-interval 30
spanning-tree portfast
!
interface GigabitEthernet1/0/17
switchport access vlan 216
switchport mode access
switchport nonegotiate
load-interval 30
channel-group 12 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/18
switchport access vlan 216
switchport mode access
switchport nonegotiate
load-interval 30
channel-group 12 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/19
switchport mode access
switchport nonegotiate
load-interval 30
channel-group 11 mode active
spanning-tree portfast
!
interface GigabitEthernet1/0/20
switchport mode access
switchport nonegotiate
load-interval 30
channel-group 11 mode active
spanning-tree portfast
switch#sho lacp counters
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
Channel group: 11
Gi1/0/19 90 20 0 0 0 0 0
Gi1/0/20 75 45 0 0 0 0 0
LACPDUs Marker Marker Response LACPDUs
Port Sent Recv Sent Recv Sent Recv Pkts Err
---------------------------------------------------------------------
Channel group: 12
Gi1/0/17 93 57 0 0 0 0 0
Gi1/0/18 89 57 0 0 0 0 0
So - why is the host showing Number of ports: 1 for the access bond?
06-14-2016 06:17 AM
The problem appears to be on the linux side. eth1 is not being used to send any data. Both ports are receiving. I swapped cables at the host side and the problem stayed with eth1, then I swapped them on the switch side and the problem stayed with eith1, so the switch and the cables are no longer suspect. Rebooting failed to change the behavior.
# dmesg | grep eth1
igb 0000:06:00.1: added PHC on eth1
igb 0000:06:00.1: eth1: (PCIe:5.0Gb/s:Width x4) 40:f2:e9:0d:e1:dd
igb 0000:06:00.1: eth1: PBA No: 106100-000
bond0: Adding slave eth1
ADDRCONF(NETDEV_UP): eth1: link is not ready
bond0: Enslaving eth1 as a backup interface with an up link
igb 0000:06:00.1: eth1: igb: eth1 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide