topic Re: FMC Deployment is causing loss of failover link on FTD-HA in Network Security

FMC Deployment is causing loss of failover link on FTD-HA

sabienzia5500 — Sun, 03 Dec 2023 19:20:31 GMT

Hi there,
We have 2 FTD 2120 in HA, everything works fine and everything is green but since we have updated our FMCs last week, whenever we try to deploy something by FMC to FTD-HA, the HA on FTDs breaks down, in the logs you can see: (Secondary) Failover interface failed" and the whole deployment failed.
After a few minutes the failover link comes backagain, both FTDs are then in sync and everything is green again, but the Deployment Failed.
We have performed successfully a failover, but again at deployment, we lost a short failover link and deployment failed again.

Re: FMC Deployment is causing loss of failover link on FTD-HA

balaji.bandi — Sun, 03 Dec 2023 19:22:20 GMT

what version of FTD FMC before, what version you upgraded to FMC - have you check the compatibility matrix, and read the release notes of FMC before upgrading ?

Re: FMC Deployment is causing loss of failover link on FTD-HA

sabienzia5500 — Sun, 03 Dec 2023 19:31:32 GMT

HI
we have updated our FMC(HA) from v7.3.1.1 (build 19) to v7.3.1.1 (build 83).
Our FTDs are both on Threat Defense v7.3.1 (build 19).
So it was not an upgrade , we only "update" the FMCs.

Re: FMC Deployment is causing loss of failover link on FTD-HA

balaji.bandi — Sun, 03 Dec 2023 19:37:35 GMT

what is the reason of update due any bug or security suggestions ?

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Sun, 03 Dec 2023 19:52:30 GMT

show failover state

Can you share this if you can access to ftd.

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

sabienzia5500 — Sun, 03 Dec 2023 20:07:00 GMT

The reason for update was really security suggestions.
Here is the out put:

State Last Failure Reason Date/Time
This host - Secondary
Active Comm Failure 01:14:12 UTC Dec 1 2023
Other host - Primary
Standby Ready Comm Failure 02:28:54 UTC Dec 1 2023

====Configuration State===
Sync Done
Sync Done - STANDBY
====Communication State===
Mac set

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Sun, 03 Dec 2023 20:28:12 GMT

show failover history

I need to know the reason of failure

Share output of above command

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Sun, 03 Dec 2023 20:55:27 GMT

Do you get the exact reason?

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

sabienzia5500 — Sun, 03 Dec 2023 21:06:24 GMT

our team tried a lot of things on 01th dec, including suspending, rebooting and resuming the secondary, we also had performed a lot of failovers , you can see all of them in failover history:
==========================================================================
From State To State Reason
==========================================================================
01:14:11 UTC Dec 1 2023
Cold Standby Failed Other unit reports that I am failed

01:14:12 UTC Dec 1 2023
Failed Cold Standby Recovered from communication failure

01:14:16 UTC Dec 1 2023
Cold Standby App Sync Recovered from communication failure

01:17:25 UTC Dec 1 2023
App Sync Sync Config Recovered from communication failure

01:17:30 UTC Dec 1 2023
Sync Config Sync File System Recovered from communication failure

01:17:30 UTC Dec 1 2023
Sync File System Bulk Sync Recovered from communication failure

01:17:47 UTC Dec 1 2023
Bulk Sync Standby Ready Recovered from communication failure

01:32:35 UTC Dec 1 2023
Standby Ready Disabled Set by the config command
(no failover)

01:36:55 UTC Dec 1 2023
Disabled Negotiation Set by the config command
(failover)

01:36:58 UTC Dec 1 2023
Negotiation Cold Standby Detected an Active peer

01:36:59 UTC Dec 1 2023
Cold Standby App Sync Detected an Active peer

01:40:05 UTC Dec 1 2023
App Sync Sync Config Detected an Active peer

01:48:59 UTC Dec 1 2023
Sync Config Sync File System Detected an Active peer

01:48:59 UTC Dec 1 2023
Sync File System Bulk Sync Detected an Active peer

01:49:11 UTC Dec 1 2023
Bulk Sync Standby Ready Detected an Active peer

02:17:24 UTC Dec 1 2023
Standby Ready Just Active Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Just Active Active Drain Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Active Drain Active Applying Config Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Active Applying Config Active Config Applied Other unit wants me Active
(Set by the config command)

02:17:26 UTC Dec 1 2023
Active Config Applied Active Other unit wants me Active
(Set by the config command)

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Sun, 03 Dec 2023 21:19:46 GMT

====Configuration State===
Sync Done
Sync Done - STANDBY

Two Sync one completed and other empty!!

Open TAC with cisco.

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Sun, 03 Dec 2023 22:05:34 GMT

I think the history is from active unit?

If yes can you share history from standby unit (failed unit)

Thanks

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Sun, 03 Dec 2023 22:43:54 GMT

/ngfw/var/log/action_queue.log

Another location for log message' check it to see the excat reason of ha failed.

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

sabienzia5500 — Tue, 05 Dec 2023 15:47:14 GMT

We have checked everything again and we don't see any errors, but we found out that the Deployment Failed in phase 6 and also the allocated memory for snort on the active FTD has increased to 13.3G now , on the passive is still on 7.3G.
we will try to reboot the standby next time , and then try to deploy after a hour .
We also considering to rollback the FMC to version 7.1.1 19 again , what do you think ?

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Tue, 05 Dec 2023 15:54:23 GMT

The FMC ver. That work before

Yes you can if it urgently case.

MHM

Re: FMC Deployment is causing loss of failover link on FTD-HA

sabienzia5500 — Thu, 07 Dec 2023 13:22:14 GMT

Hi,
Yesterday we have rolled back the FMC version to 7.3.1 (19) again and since then everything back working fine again; We can now complete a deployment without losing the HA-Link.

At the end of the day the whole problem was caused by buggy FMC version 7.3.1.1(83) which is indeed very disappoint.

Re: FMC Deployment is causing loss of failover link on FTD-HA

MHM Cisco World — Thu, 07 Dec 2023 13:37:40 GMT

Thank to you so much for update me

Glad issue is solve even with downgrade.

Have a nice day

MHM