LACP Port-channel Failure on FPR-2140

[ida71]

I have a pair of 2140's running v7.2.5+Hotfix code, setup as HA Pair. 10Gbps Cisco SFP's to Cisco switch (same setup in 3 other locations on v7.0.5 with no issues). 

My switch reports no LACP on remote end when shut/no shut issued on the etherchannel connected ports, but FMC says FTD setup is correct. The Primary 1st interface in port-channel is up, but second interface down/down on switch, same for both on Standby unit !  Been waiting days for TAC to come up with something useful

Any info from previous experience, appreciated.

Chris

[balaji.bandi]

have you checked any BUG associated :

https://www.cisco.com/c/en/us/td/docs/security/secure-firewall/release-notes/threat-defense/720/threat-defense-release-notes-72/bugs.html

since you have TAC case, i would suggest to follow up with TAC, since the issue was with new Code.

we are running 7.2.5 using FPR 3K not seen this issue.

[MHM Cisco World]

Sure it not work Port channel for FW HA to one SW' one leg connect to each FW.

The FW cluster support this FW HA active/standby or active/active dont support connect two FW to one SW (or stack or VSS or vPC)

The reason the SW see two FW and only active send lacp.

MHM

[ida71]

Switches are 4500 2 unit stack. We have same setup in 4 locations.

FTD-1 ETH13 attached to SW1/1/1 ETH14 to SW2/1/1, Port-channel1 both ports setup for channel group1

FTD-2 ETH13 attached to SW1/1/2 ETH14 to SW2/1/2, Port-channel2 both ports setup for channel group2

All other config is correct & works with exact same setup in the other 3 locations. 

Only difference here, is the Ports (eth13/14) were tested as access ports before having config stripped in FMC & reallocated to Port-Channel. Ports on switch show no LACP on remote when shut/no shut issued. Port-Channel1 is up & Eth13 is UP/UP & shows as member of PO1.

I have 3x 2120's with v7.2.5+HF in HA pairs in service but they are using multiple onboard 1Gbps ports as only in low traffic locations.

I have found reference under v6.x that Port-Channel interfaces can NOT have been configured before being allocated to Port-channel. But that would be very bad of Cisco, or normal now as this sh!t is so bad

One possible course of action is to swap the SFP's to ports 15/16 & then allocate them to the port channel after dropping 13/14 but no guarantee this will work & unfortunately the unit is now in live service, so need a solution ASAP.

[MHM Cisco World]

AHH 

So you PO one FW to two SW (stack) that work with FW HA.

But make port number different for second FW' use different port number for example port E15/16

If not working share show etherchannel summary in SW 

MHM

[ida71]

NO both FTD's are in HA, you can only assign the same FTD ports to a Port-channel (PC). 

What I meant is that current PC is using port 13/14 on both FTD's & not working. Based on early version (v6x) info that says FTD PC's interfaces MUST be virgin, as in never before used. Assuming same may be true for V7, then removing 13/14 for PC then adding 15/16 & physically moving the SFP's in the FTD from 13/14 to 15/16  would give the PC virgin interfaces, as 15/16 have Never been configured !

[MHM Cisco World]

I Know this info friend' but the port-channel is different 

So either use different port member or use same port-channel number 

MHM

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw/215351-configure-verify-and-troubleshoot-port.html

check this Guide, 
FW HA 
the PO must same in both FW and different PO in SW 

MHM

[ida71]

Yeah seen that did all that before this post. All is correct according this that, but here's some sanitized output if it helps. All the original stuff has been supplied to TAC.

>>>

Error on my Cisco Switch Port when doing a shut/no shut on FTD Interfaces.
*Dec 10 00:35:25.258: %EC-5-L3DONTBNDL2: Te2/1/1 suspended: LACP currently not enabled on the remote port.
*Dec 10 00:35:26.266: %EC-5-L3DONTBNDL2: Te2/1/2 suspended: LACP currently not enabled on the remote port.

CLi Output
FTD-1# connect local-mgmt
FTD-1(local-mgmt)# show portchannel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
3 Po3(U) Eth LACP Eth1/13(P) Eth1/14(D)

LACP KeepAlive Timer:
--------------------------------------------------------------------------------
Channel PeerKeepAliveTimerFast
--------------------------------------------------------------------------------
3 Po3(U) False

Cluster LACP Status:
--------------------------------------------------------------------------------
Channel ClusterSpanned ClusterDetach ClusterUnitID ClusterSysID
--------------------------------------------------------------------------------
3 Po3(U) False False 0
FTD-1(local-mgmt)#
FTD-1(local-mgmt)# exit
FTD-1# scope eth-uplink
FTD-1 /eth-uplink # scope fabric a
FTD-1 /eth-uplink/fabric # show port-channel

Port Channel:
Port Channel Id Name Port Type Admin State Oper State State Reason
--------------- ---------------- ------------------ ----------- ---------------- ------------
3 Port-channel3 Data Enabled Up Up
FTD-1 /eth-uplink/fabric #
FTD-1 /eth-uplink/fabric/port-channel # show member-port detail

Member Port:
Port Name: Ethernet1/13
Membership: Up
Admin State: Enabled
Oper State: Up
State Reason: Up
Ethernet Link Profile name: default
Oper Ethernet Link Profile name: fabric/lan/eth-link-prof-default
Udld Oper State: Unknown
Current Task:

Port Name: Ethernet1/14
Membership: Down
Admin State: Enabled
Oper State: Up
State Reason: Up
Ethernet Link Profile name: default
Oper Ethernet Link Profile name: fabric/lan/eth-link-prof-default
Udld Oper State: Unknown
Current Task:
FTD-1 /eth-uplink/fabric/port-channel #

=========================================================================================================
FTD-2# connect local-mgmt
FTD-2(local-mgmt)# show portchannel summary
Flags: D - Down P - Up in port-channel (members)
I - Individual H - Hot-standby (LACP only)
s - Suspended r - Module-removed
S - Switched R - Routed
U - Up (port-channel)
M - Not in use. Min-links not met
-------------------------------------------------------------------------------
Group Port- Type Protocol Member Ports
Channel
--------------------------------------------------------------------------------
3 Po3(D) Eth LACP Eth1/13(D) Eth1/14(D)

LACP KeepAlive Timer:
--------------------------------------------------------------------------------
Channel PeerKeepAliveTimerFast
--------------------------------------------------------------------------------
3 Po3(D) False

Cluster LACP Status:
--------------------------------------------------------------------------------
Channel ClusterSpanned ClusterDetach ClusterUnitID ClusterSysID
--------------------------------------------------------------------------------
3 Po3(D) False False 0
FTD-2(local-mgmt)#

FTD-2# scope eth-uplink
FTD-2 /eth-uplink # scope fabric a
FTD-2 /eth-uplink/fabric # show portchannel
^
% Invalid Command at '^' marker
FTD-2 /eth-uplink/fabric # show port-channel

Port Channel:
Port Channel Id Name Port Type Admin State Oper State State Reason
--------------- ---------------- ------------------ ----------- ---------------- ------------
3 Port-channel3 Data Enabled Link Down Down
FTD-2 /eth-uplink/fabric #
FTD-2 /eth-uplink/fabric/port-channel # show member-port detail

Member Port:
Port Name: Ethernet1/13
Membership: Down
Admin State: Enabled
Oper State: Up
State Reason: Up
Ethernet Link Profile name: default
Oper Ethernet Link Profile name: fabric/lan/eth-link-prof-default
Udld Oper State: Unknown
Current Task:

Port Name: Ethernet1/14
Membership: Down
Admin State: Enabled
Oper State: Up
State Reason: Up
Ethernet Link Profile name: default
Oper Ethernet Link Profile name: fabric/lan/eth-link-prof-default
Udld Oper State: Unknown
Current Task:
FTD-2 /eth-uplink/fabric/port-channel #

<<<

[ida71]

Note Membership down on the affected interfaces !?

[MHM Cisco World]

this from cisco guide 
the PO in both active ans standby using same port-channel number PO10 
and the PO channel in SW (your case is stack here it VSS but it same) use different port channel 

you config as you mention use two different PO one is PO1 and other is PO2 
that what I meaning by config it same 
MHM

[tvotna]

@MHM Cisco World, @ida71 wrote the following:

Switches are 4500 2 unit stack. We have same setup in 4 locations.

FTD-1 ETH13 attached to SW1/1/1 ETH14 to SW2/1/1, Port-channel1 both ports setup for channel group1

FTD-2 ETH13 attached to SW1/1/2 ETH14 to SW2/1/2, Port-channel2 both ports setup for channel group2

This means that there are TWO port-channels created in the 4500 stack, no? @MHM Cisco World , what's wrong with this or maybe we are all missing something here?

@ida71, can you provide configuration from 4500 including "switch virtual domain", VSL link, physical and port-channel interfaces along with:

show switch virtual
show switch virtual role
show switch virtual link
show port-channel summary

(from the top of my head). Perhaps there is indeed a misconfiguration on 4500?

[ida71]

I'll state again NO mis-configuration, its an FTD fault. Exact same config works in 13 separate Data centres globally on Cisco or Dell switches. The image posted above by "MHM Cisco World" exactly matches my config. I get the feeling I'm TAC's best friend, I think they have a dedicated support channel for my Bleeding Edge FTD/FMC failure finds   I believe I have my own Bug List now, maybe I should charge them a Bounty

[tvotna]

Ok. First, just in case, check that *all* physical ports are in full-duplex mode on FTD (connect local-mgmt):

show portmanager switch status

If I'm not mistaken, physical ports range is from 0/0 till 0/59 (except 0/52). (This is not a typo).

Then use

show lacp counters

many times on both sides to understand which side sends LACP PDUs and which one receives. On FTD you can also use:

show pktmgr counters

for the same, but I don't have sample output in hands.

Finally check if /opt/cisco/lacp/lacp and /opt/cisco/pm/portmgr are running:

show processes | egrep 'port|lacp'

And:

show lacp neighbor

can be used on both sides two. TAC will ask you to collect all of those outputs anyway. Not much we can do on FP2k, because FXOS CLI is in read-only mode there and settings like slow/fast LACP are unavailable in the GUI, so far as I remember.

HTH

[ida71]

That image exactly matches my setup