FI6536 Port Channel member suspended

[a12288]

I am setting up a port channel from FI6536 to upstream ACI leaf switches, but somehow always have 1 FI member port being suspended. Is there any way to figure out why it's been suspended? Thanks.

Leo

[Riaan van Niekerk]

We configure a Port Channel Policy with Suspend Individual set to True and configure this policy in the FI Port Policy -> Port Roles -> Port Channels.

We were told by our network admins to configure this setting as part of the ACI requisites. On our FabricPath port channels the uplinks worked with this set to the default of False.

Please let me know if this solves your issue.

[Steven Tardy]

https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Network-Mgmt/4-2/b_UCSM_Network_Mgmt_Guide_4_2/b_UCSM_Network_Mgmt_Guide_chapter_011.html

In end-host mode, the fabric interconnect does not run the Spanning Tree Protocol (STP)

 https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/ucs-manager/GUI-User-Guides/Network-Mgmt/4-2/b_UCSM_Network_Mgmt_Guide_4_2/b_UCSM_Network_Mgmt_Guide_chapter_01010.html#concept_D723D67847C54018B50688E32AC0FA2C

Suspended-individual: If you do not configure the ports on an upstream switch for lacp, 
the fabric interconnects treat all ports as uplink Ethernet ports to forward packets. 
You can place the lacp port in suspended state to avoid loops. 
When you set suspend-individual on a port-channel with LACP, 
if a port that is part of the port-channel does not receive PDUs from the peer port, 
it will go into suspended state.

The UCSM default seems to be (UCSM / LAN / Policies / LACP Policies / default):

• Suspend Individual :: False

The UCSM online help shows:

Suspend Individual	
This can be one of the following:
    False— Port is active even when not receiving protocol data units (PDUs) from the peer port. 
           The port goes into individual state and continues to forward traffic.
    True— Port goes into suspended mode when the PDUs are not received from the peer port. 
          Port does not forward any traffic.

Was UCSM Suspend Individual changed from the default?
Which side is doing the suspending?
`show port-channel summary` may provide some insight.
In my experience, upstream from UCSM needs to be configured appropriately and rarely does this nerd knob on the UCS side need to change.
Sounds like ACI doesn't send PDUs. Should it?

https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-application-centric-infrastructure-design-guide.html

If the LACP "Suspend Individual Port" Control option is not selected: the port is kept in the Individual state. 
This means that it operates the same as any other switch port. 
This option can be useful when the port channel is connected to a server, because if the server performs a PXE boot, 
the server is not able to negotiate the port channel at the very beginning of the boot up phase. 
In addition, a server typically won’t switch traffic across the NIC teaming interfaces of the port channel, 
hence keeping the port in the Individual state while waiting for the server bootup, which should not introduce any loops.

This may need to change on the ACI side.

Consider the UCSM Fabric Interconnect a "server" and not a "switch" regarding STP/LACP/etc.
That usually makes upstream connectivity work reliably.
(I'm looking at you UCS upstream switch missing `spanning-tree port type edge trunk` causing numerous outages.)

[a12288]

I explicitly setup "Suspend Individual" to "True" at FI through InterSight. Also, at ACI LEAF I created a separate policy and also explicitly enable "Suspend Individual Port" as well. But It always ending up with one member port suspended and another member port up.

As for "spanning-tree port type edge trunk" at the ACI, I can only explicitly specify "BPDU filter enabled" and "BPDU Guard enabled", and by default, both of them are unchecked.

By the way, the default ACI policy works well with FI6300/UCSM, but somehow cannot get the vPC port channel works with FI6536. From the FI6536 debug, I got the following:

2024 Oct 24 16:39:51.950389 eth_port_channel: pcm_eth_seq_lacp_bringup(128):  lacp protocol bringup request for port Ethernet1/36[0x1a004600]: bdl port-channel1[0x16000000] mode=5

2024 Oct 24 16:39:51.950485 eth_port_channel: fu_fsm_add_outstanding_response: added entry for mts msg rr token(171543451)

2024 Oct 24 16:39:51.950509 eth_port_channel: fu_mts_drop ref 0xffa10aeb opc 63496

2024 Oct 24 16:39:51.950547 eth_port_channel: 0) Event:ESQ_REQ length:38, at 950523 usecs after Thu Oct 24 16:39:51 2024

2024 Oct 24 16:39:51.950566 eth_port_channel:     Instance:0x1A004600, Seq Id:0x1, Ret:SUCCESS

2024 Oct 24 16:39:51.950591 eth_port_channel:     [E_MTS_TX] Dst:MTS_SAP_LACP(347), Opc:MTS_OPC_PCM_PROTOCOL_UP(63496),     RRtoken:0x0A398B9B

2024 Oct 24 16:39:51.950605 eth_port_channel:

2024 Oct 24 16:39:51.950626 eth_port_channel: Free ptr in fu_fsm_execute@../utils/fsmutils/fsm.c[1166] for addr 0x12598f2c

2024 Oct 24 16:39:51.950645 eth_port_channel: fu_fsm_execute_all: done processing event PCM_PORT_EV_COMPAT_OK_ACTIVE

2024 Oct 24 16:39:51.950661 eth_port_channel: fu_fsm_engine_post_event_processing

2024 Oct 24 16:39:51.950677 eth_port_channel:  end of while in fu_fsm_engine

2024 Oct 24 16:39:51.950695 eth_port_channel: begin fu_fsm_engine: line[2416]

2024 Oct 24 16:39:51.950715 eth_port_channel:   - setting fd[6] for select call

2024 Oct 24 16:39:54.954373 eth_port_channel:   - found data in FU_PSEL_Q_CAT_MTS queue, fd(6), usr_q_info(1)

2024 Oct 24 16:39:54.954400 eth_port_channel: fu_priority_select_select_queue: round credit(9)

2024 Oct 24 16:39:54.954411 eth_port_channel:     curr_q - FU_PSEL_Q_CAT_CQ, usr_q_info(4), priority(7), credit(3), empty

2024 Oct 24 16:39:54.954420 eth_port_channel: fu_priority_select: returning FU_PSEL_Q_CAT_MTS queue, fd(6), usr_q_info(1)

2024 Oct 24 16:39:54.954455 eth_port_channel: pcm_get_data_from_queue(1411):  (1411): dequeued mts msg (171543485), MTS_OPC_PCM_PROTOCOL_UP

2024 Oct 24 16:39:54.954527 eth_port_channel: fu_sdb_publisher_invoke_app_callback:OPC(63496/MTS_OPC_PCM_PROTOCOL_UP) is NOT all-drop;Bail-out.

2024 Oct 24 16:39:54.954547 eth_port_channel: fu_fsm_engine: fsrv_sdb_process_msg ret 0x0

2024 Oct 24 16:39:54.954558 eth_port_channel: fu_fsm_engine: fsrv_sdb_process_msg continue ret 0x0

2024 Oct 24 16:39:54.954567 eth_port_channel: fu_sync_pss_to_standby_apply:Set of checks failed

2024 Oct 24 16:39:54.954576 eth_port_channel: fu_sdb_handle_update: validation fail

2024 Oct 24 16:39:54.954591 eth_port_channel: fu_sdb_handle_update: validation fail,                        fu_is_state_active = 1,                        fu_is_sync_pss_to_standby_enabled = 1,                        mts_sync_event_get(mts_msg) = 0,                             create_del) = 0

2024 Oct 24 16:39:54.954612 eth_port_channel: fu_fsm_match_outstanding_response_in_context: id->msg_type[1]

2024 Oct 24 16:39:54.954624 eth_port_channel: fu_vdc_delete_outstanding_response: purging entry for mts msg rr token(171543451)

2024 Oct 24 16:39:54.954634 eth_port_channel: Free ptr in fu_hashtable_64_delete@../utils/fsmutils/hashtable.c[894] for addr 0x125846fc

2024 Oct 24 16:39:54.954648 eth_port_channel: Free ptr in fu_out_res_node_destroy@../utils/fsmutils/fsm.c[4795] for addr 0x12579894

2024 Oct 24 16:39:54.954661 eth_port_channel: Free ptr in fu_msg_id_node_wrap_destroy@../utils/fsmutils/fsm.c[4788] for addr 0x1258472c

2024 Oct 24 16:39:54.954669 eth_port_channel: fu_fsm_match_outstanding_response_in_context: match found for mts msg rr token(171543451)

2024 Oct 24 16:39:54.954689 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(222):  lacp proto up resp rcvd for port Ethernet1/36[0x1a004600]

2024 Oct 24 16:39:54.954772 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(260):  Msg Opc MTS_OPC_PCM_PROTOCOL_UP: rr_token 171543451: response handler received     LACP rsp for UP msg: err lacp: LACP port suspended due to misconfigurarion (err_id 0x4132001A)     bdl 0x0: MemPort Ethernet1/36: MemPo

2024 Oct 24 16:39:54.954813 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(291):  pcm_lacp_proto_up_resp_rcvd: going to hit assert

2024 Oct 24 16:39:54.954844 eth_port_channel: pcm_lacp_proto_up_resp_rcvd(296):  pcm_lacp_proto_up_resp_rcvd: port suspended due to misconfiguration

2024 Oct 24 16:39:54.954888 eth_port_channel: 0) Event:ESQ_RSP length:38, at 954873 usecs after Thu Oct 24 16:39:54 2024

2024 Oct 24 16:39:54.954901 eth_port_channel:     Instance:0x1A004600, Seq Id:0x1, Ret:SUCCESS

2024 Oct 24 16:39:54.954915 eth_port_channel:     [E_MTS_RX] Src:MTS_SAP_LACP(347), Opc:MTS_OPC_PCM_PROTOCOL_UP(63496),     RRtoken:0x0A398B9B

2024 Oct 24 16:39:54.954923 eth_port_channel:

2024 Oct 24 16:39:54.954936 eth_port_channel: pcm_es_seq_generic_cbk(222):  Sequence callbback for key 0, data:12595724

2024 Oct 24 16:39:54.954975 eth_port_channel: pcm_create_new_fsm_event(99):  Create event 73 for interface 0x1a004600 (rid 0x20000001a004600)

2024 Oct 24 16:39:54.955014 eth_port_channel: fu_fsm_execute_all: match_msg_id(0), log_already_open(0)

2024 Oct 24 16:39:54.955025 eth_port_channel: fu_fsm_execute_all: null fsm_event_list

2024 Oct 24 16:39:54.955033 eth_port_channel: fu_fsm_engine_post_event_processing

2024 Oct 24 16:39:54.955045 eth_port_channel: fu_mts_drop ref 0x1242ecf0 opc 63496

2024 Oct 24 16:39:54.955064 eth_port_channel: fu_fsm_engine_post_event_processing: mts msg MTS_OPC_PCM_PROTOCOL_UP(msg_id 171543485) dropped

2024 Oct 24 16:39:54.955075 eth_port_channel:  end of while in fu_fsm_engine

2024 Oct 24 16:39:54.955084 eth_port_channel: begin fu_fsm_engine: line[2416]

2024 Oct 24 16:39:54.955102 eth_port_channel: fu_priority_select  - returning absolute priority FU_PSEL_Q_CAT_CQ queue, usr_q_info(8)

2024 Oct 24 16:39:54.955115 eth_port_channel: Free ptr in fu_cq_node_free@../utils/fsmutils/cqueue.c[958] for addr 0x125798d4

2024 Oct 24 16:39:54.955132 eth_port_channel: pcm_get_data_from_queue(1438):  dequeued pending queue 8 msg: rid (0x20000001a004600), event_id (73), event_cat (11)

2024 Oct 24 16:39:54.955167 eth_port_channel: fu_fsm_handle_sysmgr_msg: Not mts event

2024 Oct 24 16:39:54.955179 eth_port_channel: fu_sync_pss_to_standby_apply:Input event not MTS msg

2024 Oct 24 16:39:54.955191 eth_port_channel: fu_fsm_execute_all: match_msg_id(0), log_already_open(0)

2024 Oct 24 16:39:54.955205 eth_port_channel: fu_fsm_execute: (Ethernet1/36)

2024 Oct 24 16:39:54.955216 eth_port_channel:     current state [PCM_ETH_PORT_ST_WAIT_PROTO]

2024 Oct 24 16:39:54.955224 eth_port_channel:     current event [PCM_PORT_EV_PCP_DONE_SUSPENDED]

2024 Oct 24 16:39:54.955231 eth_port_channel:     next state    [PCM_ETH_PORT_ST_INIT_DOWN]

2024 Oct 24 16:39:54.955269 eth_port_channel: pcm_eth_port_ac_drop_all_txns(1065):  Interface Ethernet1/36 suspended by protocol

2024 Oct 24 16:39:54.955326 eth_port_channel: pcm_proc_response(427):  Ethernet1/36 (0x1a004600): Setting response status to 0x402b000c (port not compatible) for MTS_OPC_ETHPM_PORT_BRINGUP (61442) response