LACP coming up in suspended mode but can't find mismatch

[MattB_765]

Hey there - been staring at this for a while and just can't see why one of my interfaces won't come up in an LACP aggregate.  I've checked the individual port settings one by one and they all look the same to me.

I'm trying to LACP trunk a pair of Nexus3000 C3064PQ Chassis running 7.0(3)I7(9) with a Fortigate 300D running it's ports in an 802.3ad aggregate.  On switch 2 both ports come up fine (P/P) but on switch 1 I get (P/s)

--------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
3     Po3(SU)     Eth      LACP      Eth1/33(P)   Eth1/35(s)

I have the following debug output from `debug lacp trace` but can't spot the speficif reason for the mismatch

2022 May  4 11:25:36.586833 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(295): pcm_lacp_proto_up_resp_rcvd: port suspended due to misconfiguration

2022 May  4 11:25:34.362482 eth_port_channel: pcm_eth_seq_lacp_bringup(127): lacp protocol bringup request for port Ethernet1/35[0x1a022000]: bdl Ethernet1/35[0x16000002] mode=5
2022 May  4 11:25:36.586648 eth_port_channel: pcm_get_data_from_queue(1411): (1411): dequeued mts msg (5516875), MTS_OPC_PCM_PROTOCOL_UP
2022 May  4 11:25:36.586741 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(221): lacp proto up resp rcvd for port Ethernet1/35[0x1a022000]
2022 May  4 11:25:36.586792 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(259): Msg Opc MTS_OPC_PCM_PROTOCOL_UP: rr_token 5516818: response handler received     LACP rsp for UP msg: err lacp: LACP port suspended due to misconfigurarion (err_id 0x4132001A)     bdl 0x0: MemPort Ethernet1/35: MemPort
2022 May  4 11:25:36.586813 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(290): pcm_lacp_proto_up_resp_rcvd: going to hit assert
2022 May  4 11:25:36.586833 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(295): pcm_lacp_proto_up_resp_rcvd: port suspended due to misconfiguration
2022 May  4 11:25:36.586867 eth_port_channel: pcm_es_seq_generic_cbk(224): Sequence callbback for key 0, data:11a7933c
2022 May  4 11:25:36.586894 eth_port_channel: pcm_create_new_fsm_event(99): Create event 73 for interface 0x1a022000 (rid 0x20000001a022000)
2022 May  4 11:25:36.586944 eth_port_channel: pcm_get_data_from_queue(1438): dequeued pending queue 8 msg: rid (0x20000001a022000), event_id (73), event_cat (11)
2022 May  4 11:25:36 eudc01tsw00101 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/35: Ethernet1/35 is suspended
2022 May  4 11:25:36.587036 eth_port_channel: pcm_eth_port_ac_drop_all_txns(1092): To drop the event 73
2022 May  4 11:25:36.587056 eth_port_channel: pcm_pss_save_data(895): port-channel port-channel3
2022 May  4 11:25:36.587075 eth_port_channel: pcm_search_pc(798): found id=2 and value=0x11a68ab4
2022 May  4 11:25:36.587205 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 13
2022 May  4 11:25:36.587244 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 14
2022 May  4 11:25:36.587258 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 15
2022 May  4 11:25:36.587270 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 16
2022 May  4 11:25:36.587288 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 28
2022 May  4 11:25:36.587301 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 29
2022 May  4 11:25:36.587314 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 30
2022 May  4 11:25:36.587326 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 31
2022 May  4 11:25:36.587339 eth_port_channel: pss_create_pc_tlv(624): unknown runtime type for pc_pss 32
2022 May  4 11:25:36.587354 eth_port_channel: pss_create_pc_tlv(604): pc delayed lacp: type 36 value = 0x0 for global_pss
2022 May  4 11:25:36.587394 eth_port_channel: pcm_pss_save_data(901): port Ethernet1/35

Any input gratefully received, thanks.

[marce1000]

 - Some hints, what is the particular device model and software version (use advisory release if applicable). 'Who' is the LACP partner (same series of questions). Check it's logs too when the lacp bundle is supposed to get going or for that particular interface at the partner-side.

 M.

[MattB_765]

Quite right Marca - conspicious lack of detail - I've edited the question to add some detail.  Logilfe output below:

2022 May  4 11:25:30 eudc01tsw00101 %ETHPORT-5-IF_ADMIN_UP: Interface Ethernet1/35 is admin up .
2022 May  4 11:25:34 eudc01tsw00101 %ETHPORT-5-SPEED: Interface Ethernet1/35, operational speed changed to 1 Gbps
2022 May  4 11:25:34 eudc01tsw00101 %ETHPORT-5-IF_DUPLEX: Interface Ethernet1/35, operational duplex mode changed to Full
2022 May  4 11:25:34 eudc01tsw00101 %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface Ethernet1/35, operational Receive Flow Control state changed to off
2022 May  4 11:25:34 eudc01tsw00101 %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface Ethernet1/35, operational Transmit Flow Control state changed to off
2022 May  4 11:25:36 eudc01tsw00101 %ETH_PORT_CHANNEL-5-PORT_SUSPENDED: Ethernet1/35: Ethernet1/35 is suspended

 Here is the port configuration, same as the config on the working switch 2.

interface Ethernet1/33
  switchport mode trunk
  channel-group 3 mode active

interface Ethernet1/35
  switchport mode trunk
  channel-group 3 mode active

interface port-channel3
  switchport mode trunk
  vpc 3

[MHM Cisco World]

show vpc brief

show vpc 

share it here

[MattB_765]

Legend:
                (*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 10
Peer status                       : peer adjacency formed ok
vPC keep-alive status             : peer is alive
Configuration consistency status  : success
Per-vlan consistency status       : success
Type-2 consistency status         : success
vPC role                          : secondary, operational primary
Number of vPCs configured         : 4
Peer Gateway                      : Disabled
Dual-active excluded VLANs        : -
Graceful Consistency Check        : Enabled
Auto-recovery status              : Disabled
Delay-restore status              : Timer is off.(timeout = 30s)
Delay-restore SVI status          : Timer is off.(timeout = 10s)
Operational Layer3 Peer-router    : Disabled

vPC Peer-link status
---------------------------------------------------------------------
id    Port   Status Active vlans
--    ----   ------ -------------------------------------------------
1     Po1    up     1,75,99,110,115,120-121,127,300

vPC status
----------------------------------------------------------------------------
Id    Port          Status Consistency Reason                Active vlans
--    ------------  ------ ----------- ------                ---------------
2     Po2           up     success     success               1,75,99,110,11
                                                             5,120-121,127,
                                                             300
3     Po3           up     success     success               1,75,99,110,11
                                                             5,120-121,127,
                                                             300
4     Po4           up     success     success               1,75,99,110,11
                                                             5,120-121,127,
                                                             300
150   Po150         up     success     success               1,75,99,110,11
                                                             5,120-121,127,
                                                             300

[MHM Cisco World]

are FW support LACP ?? Nexus only support LACP.

[MattB_765]

The Fortigate 300D does support 802.3ad - as per the OP, it's working fine on switch 2, and one of the ports on this switch 1, it's this last port that won't come up for some reason.

[marce1000]

             -  Check  Fortigate logs and or network events too

 M.

[MHM Cisco World]

show interfaces port-channel compatibility-parameters

 can you share output of this command 

[David Ruess]

Hello,

Usually if a link is suspended it has not received a LACP message from its peer on that link.  (Make sure both links are in LACP active mode to force the messages)

A couple things:

1.) Suspended just means it couldn't be bundled with the LACP port channel for some reason. It may still be forwarding traffic. Can you clear counters on the interface and then a "show interface <int/type>" to see if the counters go up on the suspended link. 

2.) You could try doing port-channel mode on and see if its just an issue with LACP messages or link status. (Not really recommended if you don't know the implications of setting a port bundle unconditionally)

3.) You can try removing the ports and adding back one by one.

-David

[paul driver]

Hello

Have you tried using a static PC (ON) instead of LACP?

[MattB_765]

Thanks for all the replies.  I think what I was really looking for was, given that the switch was giving me a clear "port suspended due to misconfiguration" message, whether it was possible to get further information on specifically, what the switch thought the misconfiguration was.  It seems that there isn't.

As it turned out, a cable monkey had swapped cables across port channels while swapping out the switch.  As the switch was replaced and the configuration blasted back on, I'd assumed wrongly that was the problem.   Ah well.

[MHM Cisco World]

Thanks