All LACP links are suspended

[Graham Murison]

I have configured an 4 interface etherchannel with a NetApp storage device.
This is the second bundle to the second controller of the NetApp device, the other one works fine with the same config.

I am getting all the interfaces with a status of suspended when trying force LACP (the only one supported by NetApp)
The stack in question are Catalyst 3650 running 03.06.05E

Here is the status of the enterchannel:
HQC4#sh etherchannel 2 summ
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator

M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

Number of channel-groups in use: 3
Number of aggregators: 3

Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
2 Po2(SD) LACP Gi2/0/1(s) Gi2/0/2(s) Gi2/0/3(s)
Gi2/0/4(s)

I am seeing this in the logs:
Feb 21 05:47:31.378: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to down
Feb 21 05:47:31.419: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to down
Feb 21 05:47:31.457: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to down
Feb 21 05:47:31.497: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to down
Feb 21 05:47:35.367: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to up
Feb 21 05:47:35.517: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to up
Feb 21 05:47:35.668: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to up
Feb 21 05:47:35.818: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Feb 21 05:47:41.610: %EC-5-L3DONTBNDL2: Gi2/0/1 suspended: LACP currently not enabled on the remote port.
Feb 21 05:47:41.908: %EC-5-L3DONTBNDL2: Gi2/0/2 suspended: LACP currently not enabled on the remote port.
Feb 21 05:47:41.937: %EC-5-L3DONTBNDL2: Gi2/0/4 suspended: LACP currently not enabled on the remote port.
Feb 21 05:47:42.537: %EC-5-L3DONTBNDL2: Gi2/0/3 suspended: LACP currently not enabled on the remote port.

Below is some of my config.
interface GigabitEthernet2/0/1
description Storage01 B E0A
switchport trunk allowed vlan 101,102
switchport mode trunk
switchport nonegotiate
channel-group 2 mode on
spanning-tree bpduguard enable
spanning-tree guard root
end

Switch4#sh run int g2/0/2
Building configuration...

Current configuration : 242 bytes
!
interface GigabitEthernet2/0/2
description Storage01 B E0B
switchport trunk allowed vlan 101,102
switchport mode trunk
switchport nonegotiate
channel-group 2 mode on
spanning-tree bpduguard enable
spanning-tree guard root
end

Switch4#sh run int g2/0/3
Building configuration...

Current configuration : 242 bytes
!
interface GigabitEthernet2/0/3
description Storage01 B E0C
switchport trunk allowed vlan 101,102
switchport mode trunk
switchport nonegotiate
channel-group 2 mode on
spanning-tree bpduguard enable
spanning-tree guard root
end

Switch4#sh run int g2/0/4
Building configuration...

Current configuration : 242 bytes
!
interface GigabitEthernet2/0/4
description Storage01 B E0D
switchport trunk allowed vlan 101,102
switchport mode trunk
switchport nonegotiate
channel-group 2 mode on
spanning-tree bpduguard enable
spanning-tree guard root
end

Switch4#sh run int p2
Building configuration...

Current configuration : 138 bytes
!
interface Port-channel2
description Storage01 B
switchport trunk allowed vlan 101,102
switchport mode trunk
switchport nonegotiate
end

Switch4#sh int g2/0/1
GigabitEthernet2/0/1 is up, line protocol is down (suspended)
Hardware is Gigabit Ethernet, address is 00a6.ca7e.5d01 (bia 00a6.ca7e.5d01)
Description: Storage01 B E0A (HQC04-B)
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:49:16, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
105 packets input, 13440 bytes, 0 no buffer
Received 105 broadcasts (105 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 105 multicast, 0 pause input
0 input packets with dribble condition detected
45089 packets output, 5883107 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Switch4#sh int g2/0/2
GigabitEthernet2/0/2 is up, line protocol is down (suspended)
Hardware is Gigabit Ethernet, address is 00a6.ca7e.5d02 (bia 00a6.ca7e.5d02)
Description: Storage01 B E0B (HQC04-B)
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:49:18, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 1000 bits/sec, 3 packets/sec
105 packets input, 13440 bytes, 0 no buffer
Received 105 broadcasts (105 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 105 multicast, 0 pause input
0 input packets with dribble condition detected
60747 packets output, 30427580 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Switch4#sh int g2/0/3
GigabitEthernet2/0/3 is up, line protocol is down (suspended)
Hardware is Gigabit Ethernet, address is 00a6.ca7e.5d03 (bia 00a6.ca7e.5d03)
Description: Storage01 B E0C (HQC04-B)
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:49:20, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
105 packets input, 13440 bytes, 0 no buffer
Received 105 broadcasts (105 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 105 multicast, 0 pause input
0 input packets with dribble condition detected
135031 packets output, 14919505 bytes, 0 underruns
0 output errors, 0 collisions, 5 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
Switch4#sh int g2/0/4
GigabitEthernet2/0/4 is up, line protocol is down (suspended)
Hardware is Gigabit Ethernet, address is 00a6.ca7e.5d04 (bia 00a6.ca7e.5d04)
Description: Storage01 B E0D (HQC04-B)
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:49:22, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 13000 bits/sec, 27 packets/sec
106 packets input, 13568 bytes, 0 no buffer
Received 106 broadcasts (106 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 106 multicast, 0 pause input
0 input packets with dribble condition detected
121903 packets output, 11575760 bytes, 0 underruns
0 output errors, 0 collisions, 6 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

I have tried changing the "channel-group 2 mode" to on, however it is not able to detect the protocol:
HQC4#show etherchannel 2 detail
Group state = L2
Ports: 4 Maxports = 8
Port-channels: 1 Max Port-channels = 1
Protocol: -
Minimum Links: 0

Ports in the group:
-------------------
Port: Gi2/0/1
------------

Port state = Up Mstr In-Bndl
Channel group = 2 Mode = On Gcchange = -
Port-channel = Po2 GC = - Pseudo port-channel = Po2
Port index = 0 Load = 0x00 Protocol = -

Age of the port in the current state: 0d:00h:29m:39s

Port: Gi2/0/2
------------

Port state = Up Mstr In-Bndl

I am a complete loss and right now I am in NetApp failover mode. If this is a NetApp issue please let me know I and I will throw it their way!

Thanks.

[marce1000]

>...if this is a netapp issue....

  Well, indeed one thing to check is whether the interfaces on the 'second controller' are correctly configured and or are aware that they are to be used in a bundled mode. Compare with the interface configuration on the first controller (e.g.).

M.

[Jaderson Pessoa]

Well, maybe the problem is on the NetApp, check if configurations are correctly.

I hope that this documentation can help you: http://docs.netapp.com/ontap-9/index.jsp?topic=%2Fcom.netapp.doc.dot-cm-cmpr-910%2Fnetwork__port__ifgrp__create.html

[balaji.bandi]

If you using NetApp Ontab 7.X  on wards here is the sample working config.

Netapp Side :
vif create lacp <vif name> -b ip <interface list>

Cisco switch Side :

#channel-protocol lacp  
#channel-group X mode active

 Test and advise.

some FAQ :

https://kb.netapp.com/app/answers/answer_view/a_id/1001343/~/faq%3A-troubleshooting-lacp-port-channel%2F-interface-groups-

[Graham Murison]

Thanks to all for the suggestions.  

Background:

We had two identical NetApp FAS 2000 series controllers connected to separate C3560X switches.  In the process of switching out these switch we installed a stack of two C3650X.  I managed to get the first controller head swung over to the new switch stack fine (all ports connected to switch #1 in the stack), the second is proving to be the problem child.

We have NetApp support, but didn't get far since nothing has changed on the controller end.  When I add the line channel-protocol lacp there is no change, still suspended with amber LEDs.  When I try channel-group 2 mode on without channel-protocol lacp the links come active (green), but LACP is not negotiated or sensed on the link.

When I try both channel-protocol lacp & channel-group 2 mode on, the command is rejected: Command rejected (Channel protocol mismatch for interface Gi1/0/5 in group 2): the interface can not be added to the channel group.

I have also tried new network cables with ones I know are good, but the situation is the same.

If anyone has any suggestions that would be appreciated since I am now going to take the plunge with Cisco TAC.

Thanks,

Graham

[marce1000]

 - Could you also have a test with passive-mode LACP on the switch side (as stated as-a-test).

M.

[Graham Murison]

Thanks.

Updated the config as follows:

interface GigabitEthernet2/0/1
 switchport trunk allowed vlan 101,102
 switchport mode trunk
 switchport nonegotiate
 channel-group 2 mode passive
 spanning-tree bpduguard enable
 spanning-tree guard root

This resulted in suspended links.  

#sh int g2/0/1
GigabitEthernet2/0/1 is up, line protocol is down (suspended)
Hardware is Gigabit Ethernet, address is 00a6.ca7e.5d01 (bia 00a6.ca7e.5d01)
Description: 
MTU 9000 bytes, BW 1000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 1000Mb/s, media type is 10/100/1000BaseTX
input flow-control is off, output flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 4d15h, output never, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
105 packets input, 13440 bytes, 0 no buffer
Received 105 broadcasts (105 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 105 multicast, 0 pause input
0 input packets with dribble condition detected
59165 packets output, 7675454 bytes, 0 underruns
0 output errors, 0 collisions, 8 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

sh logg
.......
Feb 25 20:18:12.217: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to down
Feb 25 20:18:12.248: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to down
Feb 25 20:18:12.287: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to down
Feb 25 20:18:12.328: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to down
Feb 25 20:18:16.150: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to up
Feb 25 20:18:16.378: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Feb 25 20:18:16.530: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to up
Feb 25 20:18:16.679: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to up
Feb 25 20:18:22.457: %EC-5-L3DONTBNDL2: Gi2/0/1 suspended: LACP currently not enabled on the remote port.
Feb 25 20:18:22.470: %EC-5-L3DONTBNDL2: Gi2/0/4 suspended: LACP currently not enabled on the remote port.
Feb 25 20:18:22.471: %EC-5-L3DONTBNDL2: Gi2/0/3 suspended: LACP currently not enabled on the remote port.
Feb 25 20:18:23.279: %EC-5-L3DONTBNDL2: Gi2/0/2 suspended: LACP currently not enabled on the remote port.
#sh etherchannel 2 sum
Flags: D - down P - bundled in port-channel
I - stand-alone s - suspended
H - Hot-standby (LACP only)
R - Layer3 S - Layer2
U - in use f - failed to allocate aggregator
M - not in use, minimum links not met
u - unsuitable for bundling
w - waiting to be aggregated
d - default port

Number of channel-groups in use: 3
Number of aggregators: 3
Group Port-channel Protocol Ports
------+-------------+-----------+-----------------------------------------------
2 Po2(SD) LACP Gi1/0/5(D) Gi2/0/1(s) Gi2/0/2(s)
Gi2/0/3(s) Gi2/0/4(s)

[Graham Murison]

Okay I I am approaching this from a different direction by: shutting the interfaces, "Debug etherchannel all" and then "no shutdown" the interfaces.

It produces some surprising oputput, some of which includes pagp????

Feb 25 23:16:56.480: %LINK-5-CHANGED: Interface GigabitEthernet2/0/1, changed state to administratively down
Feb 25 23:16:56.518: %LINK-5-CHANGED: Interface GigabitEthernet2/0/2, changed state to administratively down
Feb 25 23:16:56.548: %LINK-5-CHANGED: Interface GigabitEthernet2/0/3, changed state to administratively down
Feb 25 23:16:56.578: %LINK-5-CHANGED: Interface GigabitEthernet2/0/4, changed state to administratively down
Feb 25 23:16:57.451: %SYS-5-CONFIG_I: Configured from console by ssh on vty0 (10.96.1.246)
Feb 25 23:24:37.896: %SYS-5-CONFIG_I: Configured from console by ssh on vty0 (10.96.1.246)
Feb 25 23:24:39.009: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to down
Feb 25 23:24:39.037: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to down
Feb 25 23:24:39.077: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to down
Feb 25 23:24:39.118: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to down
Feb 25 23:24:40.866: FEC: pagp_switch_port_up: Gi2/0/1
Feb 25 23:24:40.866: FEC: pagp_switch_invoke_port_up: Gi2/0/1
Feb 25 23:24:40.867: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/1 Po2 flag = 1 1
Feb 25 23:24:40.867: FEC: pagp_switch_port_attrib_diff: Gi2/0/1 Po2 same
Feb 25 23:24:40.867: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:40.867: FEC: pagp_switch_find_master: group 2
Feb 25 23:24:40.867: FEC: lacp_switch_calculate_oper_key: oper_key for port Gi2/0/1 is 0x2
Feb 25 23:24:41.029: FEC: pagp_switch_port_up: Gi2/0/2
Feb 25 23:24:41.029: FEC: pagp_switch_invoke_port_up: Gi2/0/2
Feb 25 23:24:41.029: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/2 Po2 flag = 1 1
Feb 25 23:24:41.029: FEC: pagp_switch_port_attrib_diff: Gi2/0/2 Po2 same
Feb 25 23:24:41.030: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:41.030: FEC: pagp_switch_find_master: group 2
Feb 25 23:24:41.030: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/2 Gi2/0/1 flag = 1 1
Feb 25 23:24:41.030: FEC: pagp_switch_port_attrib_diff: compare LACP modes for Gi2/0/2
Feb 25 23:24:41.030: FEC: pagp_switch_port_attrib_diff: Gi2/0/2 Gi2/0/1 same
Feb 25 23:24:41.030: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:41.030: FEC: lacp_switch_calculate_oper_key: oper_key for port Gi2/0/2 is 0x2
Feb 25 23:24:41.175: FEC: pagp_switch_port_up: Gi2/0/3
Feb 25 23:24:41.175: FEC: pagp_switch_invoke_port_up: Gi2/0/3
Feb 25 23:24:41.175: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/3 Po2 flag = 1 1
Feb 25 23:24:41.175: FEC: pagp_switch_port_attrib_diff: Gi2/0/3 Po2 same
Feb 25 23:24:41.176: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:41.176: FEC: pagp_switch_find_master: group 2
Feb 25 23:24:41.176: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/3 Gi2/0/1 flag = 1 1
Feb 25 23:24:41.176: FEC: pagp_switch_port_attrib_diff: compare LACP modes for Gi2/0/3
Feb 25 23:24:41.176: FEC: pagp_switch_port_attrib_diff: Gi2/0/3 Gi2/0/1 same
Feb 25 23:24:41.176: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:41.176: FEC: lacp_switch_calculate_oper_key: oper_key for port Gi2/0/3 is 0x2
Feb 25 23:24:41.329: FEC: pagp_switch_port_up: Gi2/0/4
Feb 25 23:24:41.329: FEC: pagp_switch_invoke_port_up: Gi2/0/4
Feb 25 23:24:41.330: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/4 Po2 flag = 1 1
Feb 25 23:24:41.330: FEC: pagp_switch_port_attrib_diff: Gi2/0/4 Po2 same
Feb 25 23:24:41.330: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:41.330: FEC: pagp_switch_find_master: group 2
Feb 25 23:24:41.330: FEC: pagp_switch_agc_compatable: comparing GC values of Gi2/0/4 Gi2/0/1 flag = 1 1
Feb 25 23:24:41.330: FEC: pagp_switch_port_attrib_diff: compare LACP modes for Gi2/0/4
Feb 25 23:24:41.330: FEC: pagp_switch_port_attrib_diff: Gi2/0/4 Gi2/0/1 same
Feb 25 23:24:41.330: FEC: pagp_switch_agc_compatable: GC values are compatable
Feb 25 23:24:41.330: FEC: lacp_switch_calculate_oper_key: oper_key for port Gi2/0/4 is 0x2
Feb 25 23:24:42.859: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/1, changed state to up
Feb 25 23:24:43.030: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/2, changed state to up
Feb 25 23:24:43.169: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/3, changed state to up
Feb 25 23:24:43.329: %LINK-3-UPDOWN: Interface GigabitEthernet2/0/4, changed state to up
Feb 25 23:24:49.438: FEC: lacp_fec_dontbundle_internal: Gi2/0/2
Feb 25 23:24:49.438: FEC: fec_dontbundle: Gi2/0/2
Feb 25 23:24:49.438: %EC-5-L3DONTBNDL2: Gi2/0/2 suspended: LACP currently not enabled on the remote port.
Feb 25 23:24:49.597: FEC: lacp_fec_dontbundle_internal: Gi2/0/3
Feb 25 23:24:49.597: FEC: fec_dontbundle: Gi2/0/3
Feb 25 23:24:49.597: %EC-5-L3DONTBNDL2: Gi2/0/3 suspended: LACP currently not enabled on the remote port.
Feb 25 23:24:49.777: FEC: lacp_fec_dontbundle_internal: Gi2/0/1
Feb 25 23:24:49.777: FEC: fec_dontbundle: Gi2/0/1
Feb 25 23:24:49.777: %EC-5-L3DONTBNDL2: Gi2/0/1 suspended: LACP currently not enabled on the remote port.
Feb 25 23:24:50.137: FEC: lacp_fec_dontbundle_internal: Gi2/0/4
Feb 25 23:24:50.137: FEC: fec_dontbundle: Gi2/0/4
Feb 25 23:24:50.137: %EC-5-L3DONTBNDL2: Gi2/0/4 suspended: LACP currently not enabled on the remote port.

So confused.

[Graham Murison · Accepted solution]

So I thought I would update in case someone comes up with this issue or for those that really need to know.

I managed to find the resolution and document here.

The issue was that when a couple of storage nodes are setup in a HA cluster and you manually failover (say node B), then B will not send any LACP traffic over the bundled links.  In the older switch (C3560X) the physical links would remain active in standalone (individual) state.  With the new switches (C3650X), when this happens the links become suspended until manually unshut.

The resolution for my case is to set the command "no port-channel standalone-disable" on the port channel and now very is working as intended.  

Here is the resulting configuration:

interface GigabitEthernet2/0/1
switchport trunk allowed vlan 101,102
switchport mode trunk
channel-group 2 mode active
spanning-tree bpduguard enable
spanning-tree guard root
!
interface GigabitEthernet2/0/2
switchport trunk allowed vlan 101,102
switchport mode trunk
channel-group 2 mode active
spanning-tree bpduguard enable
spanning-tree guard root
!
interface Port-channel2
switchport trunk allowed vlan 101,102
switchport mode trunk
no port-channel standalone-disable
end

 I hope this may help someone else.