FMC Deployment is causing loss of failover link on FTD-HA

[sabienzia5500]

Hi there,
We have 2 FTD 2120 in HA, everything works fine and everything is green but since we have updated our FMCs last week, whenever we try to deploy something by FMC to FTD-HA, the HA on FTDs breaks down, in the logs you can see: (Secondary) Failover interface failed" and the whole deployment failed.
After a few minutes the failover link comes backagain, both FTDs are then in sync and everything is green again, but the Deployment Failed.
We have performed successfully a failover, but again at deployment, we lost a short failover link and deployment failed again.

[balaji.bandi]

what version of FTD FMC before, what version you upgraded to FMC - have you check the compatibility matrix, and read the release notes of FMC before upgrading ?

[sabienzia5500]

HI
we have updated our FMC(HA) from v7.3.1.1 (build 19) to v7.3.1.1 (build 83).
Our FTDs are both on Threat Defense v7.3.1 (build 19).
So it was not an upgrade , we only "update" the FMCs.

[balaji.bandi]

what is the reason of update due any bug or security suggestions ?

[MHM Cisco World]

show failover state

Can you share this if you can access to ftd.

MHM

[sabienzia5500]

The reason for update was really security suggestions.
Here is the out put:

State Last Failure Reason Date/Time
This host - Secondary
Active Comm Failure 01:14:12 UTC Dec 1 2023
Other host - Primary
Standby Ready Comm Failure 02:28:54 UTC Dec 1 2023

====Configuration State===
Sync Done
Sync Done - STANDBY
====Communication State===
Mac set

[MHM Cisco World]

show failover history

I need to know the reason of failure 

Share output of above command

MHM

[MHM Cisco World]

Do you get the exact reason?

MHM

[sabienzia5500]

our team tried a lot of things on 01th dec, including suspending, rebooting and resuming the secondary, we also had performed a lot of failovers , you can see all of them in failover history:
==========================================================================
From State To State Reason
==========================================================================
01:14:11 UTC Dec 1 2023
Cold Standby Failed Other unit reports that I am failed

01:14:12 UTC Dec 1 2023
Failed Cold Standby Recovered from communication failure

01:14:16 UTC Dec 1 2023
Cold Standby App Sync Recovered from communication failure

01:17:25 UTC Dec 1 2023
App Sync Sync Config Recovered from communication failure

01:17:30 UTC Dec 1 2023
Sync Config Sync File System Recovered from communication failure

01:17:30 UTC Dec 1 2023
Sync File System Bulk Sync Recovered from communication failure

01:17:47 UTC Dec 1 2023
Bulk Sync Standby Ready Recovered from communication failure

01:32:35 UTC Dec 1 2023
Standby Ready Disabled Set by the config command
(no failover)

01:36:55 UTC Dec 1 2023
Disabled Negotiation Set by the config command
(failover)

01:36:58 UTC Dec 1 2023
Negotiation Cold Standby Detected an Active peer

01:36:59 UTC Dec 1 2023
Cold Standby App Sync Detected an Active peer

01:40:05 UTC Dec 1 2023
App Sync Sync Config Detected an Active peer

01:48:59 UTC Dec 1 2023
Sync Config Sync File System Detected an Active peer

01:48:59 UTC Dec 1 2023
Sync File System Bulk Sync Detected an Active peer

01:49:11 UTC Dec 1 2023
Bulk Sync Standby Ready Detected an Active peer

02:17:24 UTC Dec 1 2023
Standby Ready Just Active Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Just Active Active Drain Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Active Drain Active Applying Config Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Active Applying Config Active Config Applied Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Active Config Applied Active Other unit wants me Active
(Set by the config command)

[MHM Cisco World]

====Configuration State===
Sync Done
Sync Done - STANDBY

Two Sync one completed and other empty!!

Open TAC with cisco. 

MHM

[MHM Cisco World]

I think the history is from active unit?

If yes can you share history from standby unit (failed unit)

Thanks 

MHM

[MHM Cisco World]

/ngfw/var/log/action_queue.log

Another location for log message' check it to see the excat reason of ha failed.

MHM

[sabienzia5500]

We have checked everything again and we don't see any errors, but we found out that the Deployment Failed in phase 6 and also the allocated memory for snort on the active FTD has increased to 13.3G now , on the passive is still on 7.3G.
we will try to reboot the standby next time , and then try to deploy after a hour .
We also considering to rollback the FMC to version 7.1.1 19 again , what do you think ?

[MHM Cisco World]

The FMC ver. That work before

Yes you can if it urgently case.

MHM

[sabienzia5500]

Hi,
Yesterday we have rolled back the FMC version to 7.3.1 (19) again and since then everything back working fine again; We can now complete a deployment without losing the HA-Link.

At the end of the day the whole problem was caused by buggy FMC version 7.3.1.1(83) which is indeed very disappoint.