Firepower 1000 series causes LACP link flaps with Nexus 9k Switches

Network Diver · ‎05-18-2025

Hi,

We're having encountered this bug here on all Firepower devices: https://bst.cisco.com/bugsearch/bug/CSCwn92248

Firepower 1120 (HA active/standby), running FTD 7.4.2.2, connected with 4x1 Gb/s vPC to a pair of N9K-C9348GC-FXP, running NXOS 10.2(4)
Firepower 1150 (HA active/standby), running FTD 7.4.2.2, connected with 2x10 Gb/s vPC to a pair of N9K-C93180YC-FXrunning NXOS 10.2(4)

Logs on Nexus switches:

May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel25: first operational port changed from Ethernet1/25 to none
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETH_PORT_CHANNEL-5-PORT_DOWN: port-channel25: Ethernet1/25 is down
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel25 is down (No operational members)
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent (message repeated 2 times)
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel25,bandwidth changed to 100000 Kbit
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_DOWN_INITIALIZING: Interface Ethernet1/25 is down (Initializing)
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_DOWN_PORT_CHANNEL_MEMBERS_DOWN: Interface port-channel25 is down (No operational members)
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-SPEED: Interface port-channel25, operational speed changed to 10 Gbps
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_DUPLEX: Interface port-channel25, operational duplex mode changed to Full
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_RX_FLOW_CONTROL: Interface port-channel25, operational Receive Flow Control state changed to off
May 16 00:05:24 nexus-1 2025 May 16 02:05:24 CEST: %ETHPORT-5-IF_TX_FLOW_CONTROL: Interface port-channel25, operational Transmit Flow Control state changed to off
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent (message repeated 1 time)
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-VPC_UP: vPC 25 is up
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent (message repeated 3 times)
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %ETH_PORT_CHANNEL-5-PORT_UP: port-channel25: Ethernet1/25 is up
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %ETH_PORT_CHANNEL-5-FOP_CHANGED: port-channel25: first operational port changed from none to Ethernet1/25
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %ETHPORT-5-IF_BANDWIDTH_CHANGE: Interface port-channel25,bandwidth changed to 10000000 Kbit
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %ETHPORT-5-IF_UP: Interface Ethernet1/25 is up in mode trunk
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent (message repeated 1 time)
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %ETHPORT-5-IF_UP: Interface port-channel25 is up in mode trunk
May 16 00:05:27 nexus-1 2025 May 16 02:05:27 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent
May 16 00:05:34 nexus-1 2025 May 16 02:05:34 CEST: %VPC-5-INTF_CONSISTENCY_SUCCESS: In domain 1, vPC 25 configuration is consistent (message repeated 1 time)

Logs on Firepower device:

May 16 00:05:24 firepower %FTD-6-199018: port-manager: Portchannel set po member bundle state for Eth1/9 to 2
May 16 00:05:24 firepower %FTD-6-199018: port-manager: ethch_pd_trunk_member_disable:Po-9 mem_port:0x18
May 16 00:05:24 firepower %FTD-6-199018: port-manager: cmd IOCTL_SW_TRUNK_XYZ (8207)
May 16 00:05:24 firepower %FTD-6-199018: port-manager: (caller by mrvl_esw_ioctl), dev 0, port 24
May 16 00:05:24 firepower %FTD-6-199018: port-manager: cmd IOCTL_SW_TRUNK_XYZ (8207)
May 16 00:05:24 firepower %FTD-6-199018: port-manager: (caller mrvl_esw_ioctl_portchannel_cfg), cmd 8207, trunk_id 9
May 16 00:05:24 firepower %FTD-6-199018: port-manager: ecport: member:Eth1/9 unbundled, bundle cnt 1
May 16 00:05:24 firepower %FTD-6-199018: port-manager: unbundle member Eth1/9 success
May 16 00:05:27 firepower %FTD-6-199018: port-manager: Portchannel set po member bundle state for Eth1/9 to 1
May 16 00:05:27 firepower %FTD-6-199018: port-manager: ethch_pd_trunk_member_enable:Po-9 mem_port:0x18
May 16 00:05:27 firepower %FTD-6-199018: port-manager: cmd IOCTL_SW_TRUNK_XYZ (8207)
May 16 00:05:27 firepower %FTD-6-199018: port-manager: (caller by mrvl_esw_ioctl), dev 0, port 24
May 16 00:05:27 firepower %FTD-6-199018: port-manager: cmd IOCTL_SW_TRUNK_XYZ (8207)
May 16 00:05:27 firepower %FTD-6-199018: port-manager: (caller mrvl_esw_ioctl_portchannel_cfg), cmd 8207, trunk_id 9
May 16 00:05:27 firepower %FTD-6-199018: port-manager: ecport: member:Eth1/9 bundled, bundle cnt 2
May 16 00:05:27 firepower %FTD-6-199018: port-manager: bundle member Eth1/9 success
May 16 00:05:27 firepower %FTD-6-199018: port-manager: ethch_send_pc_state_change: PortMgr- l2 state up change event sent for pc9

VPC configuration on Nexus switch:

interface Ethernet1/25
  description firepower gi1
  switchport mode trunk
  switchport trunk allowed vlan 10-11,151,1140
  spanning-tree port type edge trunk
  spanning-tree guard root
  mtu 9216
  channel-group 25 mode active
  no shutdown

interface Ethernet1/26
  description firepower gi2
  switchport mode trunk
  switchport trunk allowed vlan 10-11,151,1140
  spanning-tree port type edge trunk
  spanning-tree guard root
  mtu 9216
  channel-group 25 mode active
  no shutdown

interface port-channel25
  description firepower
  switchport mode trunk
  switchport trunk allowed vlan 10-11,151,1140
  spanning-tree port type edge trunk
  spanning-tree guard root
  mtu 9216
  vpc 25

Port-channel configuration on Firepower:

firepower# connect local-mgmt
firepower(local-mgmt)# show portchannel summary
Flags:  D - Down        P - Up in port-channel (members)
I - Individual  H - Hot-standby (LACP only)
s - Suspended   r - Module-removed
S - Switched    R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port-       Type     Protocol  Member Ports
      Channel
--------------------------------------------------------------------------------
7     Po7(U)      Eth      LACP      Eth1/7(P)    Eth1/8(P)
1     Po1(U)      Eth      LACP      Eth1/1(P)    Eth1/2(P)    Eth1/3(P)
                                     Eth1/4(P)

LACP KeepAlive Timer:
--------------------------------------------------------------------------------
      Channel  PeerKeepAliveTimerFast
--------------------------------------------------------------------------------
7     Po7(U)      False
1     Po1(U)      False

Cluster LACP Status:
--------------------------------------------------------------------------------
      Channel  ClusterSpanned  ClusterDetach  ClusterUnitID  ClusterSysID
--------------------------------------------------------------------------------
7     Po7(U)      False          False          0
1     Po1(U)      False          False          0

The bug workaround suggests to configure port-channel to mode "on". When doing that only on the Firepower but not on the Nexus side, all the ports are suspended and the Firepower does a failover and is unreachable. I didn't try yet to change LACP mode on Nexus side, but I guess when doing that there also then only 1 uplinks becomes active and all the others are suspended. We need 4 x 1 Gb/s as we have two ISP routers with each 1 Gb/s.

nexus-1(config-if)#   channel-group 25 mode ?
  active   Set channeling mode to ACTIVE
  on       Set channeling mode to ON
  passive  Set channeling mode to PASSIVE

So far no network outage has registered by our monitoring and no user complaints. But I guess if more than one link goes down at the same time, there will be an outage or trigger a FTD failover.

TAC case has been opened.

If you have the same setup, check your logs.

Network Diver · ‎05-21-2025

So far it looks that only one member of a port-channel goes down at the same time and comes back 3 seconds later. There's also no distinct interval when one interface goes down. Happens roughly once a day per port and device.

Aref Alsouqi · ‎05-21-2025

You would need two port channels to be configured on the switches stack, one connected to each firewall. From the firewalls perspective you will have one port channel.

Active FTD port 1 > Switch 1 port 1 - Port Channel 1
Active FTD port 2 > Switch 2 port 1 - Port Channel 1

Passive FTD port 1 > Switch 1 port 2 - Port Channel 2
Passive FTD port 2 > Switch 2 port 2 - Port Channel 2

Network Diver · ‎05-21-2025

We've connected the Firepower 1120 with active LACP and vPC to Nexus switches as follows. The Firepower 1150 use the 2 x 10 Gb/s uplink ports. We also have older Firepower 2120 running ASA and same uplink configuration as well as plenty of other devices that work perfectly that way.

Network Diver · ‎05-21-2025

Captured packets on Nexus switch.

ethanalyzer local interface inband capture-filter "ether proto 0x8809" limit-captured-frames 0 autostop duration 86400 write bootflash:///capture-lacp.pcap display

It shows that the FTD is sending a LACPDU packed with flax expired set before the link is torn down an built again.

Network Diver · ‎05-21-2025

Port-channels are managed in FXOS. Affected FXOS version is 2.14, running on our new Firepower 1120 and 1150 firewalls.

firepower11xx# show version
  Version: 2.14(1.187)
  Startup-Vers: 2.14(1.187)

Our older Firepower 2120 running ASA is running older FXOS releases and have no problems with LACP.

firepower2120-1# show version
  Version: 2.12(1.73)
  Startup-Vers: 2.12(1.73)

firepower2120-2# show version
  Version: 2.10(1.214)
  Startup-Vers: 2.10(1.214)

Aref Alsouqi · ‎05-22-2025

The thing is that if you set the mode to "on" on both ends the negotiation won't happen at all. Probably I would try to set the FTD site to passive leaving the switches set to active and see if that helps.

Network Diver · ‎05-22-2025

FTD 7.4.x supports only LACP modes "active" or "on".

From the Firewall Management Center 7.4 config guide:

Active—Sends and receives LACP updates. An active EtherChannel can establish connectivity with either an active or a passive EtherChannel. You should use the active mode unless you need to minimize the amount of LACP traffic.
Passive—Receives LACP updates. A passive EtherChannel can only establish connectivity with an active EtherChannel. Not supported on hardware models.
On—The EtherChannel is always on, and LACP is not used. An “on” EtherChannel can only establish a connection with another “on” EtherChannel.

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/management-center/device-config/740/management-center-device-config-74/interfaces-settings-ifcs-overview.html#ID-2077-0000003c

Mode ON: All static EtherChannels, that is, the port is not running LACP messaging but static bundling. With this mode, the switch or the other end switch will not recognize any problem with the EtherChannel and will not tell about the problem.

https://www.grandmetric.com/knowledge-base/design_and_configure/how-to-configure-lacp-on-cisco/

Network Diver · ‎05-22-2025

Response from Cisco TAC:

Can more than one port-channel member expire at once?

Yes. Due to the timer-management issue in the current FXOS/FTD code, when one member’s LACP state machine expires it can inadvertently drive the expired flag on all members of that same channel. When that happens, the switch will take down each affected link and, if enough members drop, the port-channel itself will clear and the FTD will trigger failover (since it’s monitoring link state).

Timing for a software correction

Engineering has prioritized CSCwn92248 for the upcoming 7.4.3 maintenance release (FXOS 2.15-series). At this time, QA is validating the fix and we expect it to become generally available in the July-timeframe (However, we do not have a definitive date yet). Once the maintenance train is officially posted, that build will include the corrected LACP-timer behavior so that members don’t all receive the expired indication simultaneously.

Network Diver · ‎05-22-2025

Anyone experience with LACP mode "on" instead of "active"? Must be the same on FTD and on Nexus side and if I understood it correctly, a down link of a port-channel member will be detected, but not when one side removes a member of the port-channel and the link stays up? Then it will still send traffic and cause a black hole? Will all links be active and forwarding traffic or just one and the others are standby?

http://www.ccnaccnplinux.com/2016/10/etherchannel-port-option.html

Network Diver · ‎05-22-2025

Will switching between FTD LACP modes from "active" to "on" work without or minimum downtime with these steps?

On FTD disable monitoring for all port-channel VLAN subinterfaces and deploy change.
On Nexus switch 1 set all ethernet ports of FTD HA pair to "channel-group xx mode on"
The ethernet ports on switch 1 should go to suspended, the ones on switch 2 should still be up. The vPC remains up
On FTD set port-channel LACP mode to on and deploy change
The ethernet ports on switch 1 should now go up and the ones on switch 2 should go to suspended. The vPC should remain up or go down for a very short time.
⁠On Nexus switch 2 set all ethernet ports of FTD pair to "channel-group xx mode on" and deploy change.
⁠On FTD enable monitoring for all port-channel VLAN subinterfaces again.

I haven't tried that yet.