FMC/FTD HA failover

[Groto]

Hi,

I have a few questions about FTD HA failover and FMC and FTD communication in general.

I have 2 FTDs in HA failover (Active/Standby) pair and they are being managed by FMC.

Access and platform settings policy are assigned to HA.

My question about making changes to those policies and deploying them: 

1. How is FMC sending changes to FTD? Does it send only the changes or the entire new running-config? If it sends only the changes, how are they implemented on FTD- does FMC have some internal system user who then executes those commands on FTD? If yes, how is that user called?
2. Is FMC sending changes to both devices in HA pair or just to the active device that synchronizes changes to the standby device?

My question about HA failover:

1. Given that devices in HA are synchronized when configuration changes, when a failover event occurs, shouldn't devices already have the same configuration?
2. Which device starts the synchronization? Active or standby?
3. Does FTD check if the configurations are the same and if they turn out to be, does it skip the synchronization process? 
4. How is configuration synchronization implemented? Only the differences are synchronized or does the standby device just copy the entire running-config from the active device? If only the changes, how are they implemented? Does some internal system user execute commands? If so, how is that user called? If it copies the entire running-config, how does it do it? Executes all commands or just copies the file?

I appreciate any help you can provide.

[tvotna]

@Groto, this is complicated. I believe nobody will give you definitive answers. Also, answers depend on manager in use: FMC or FDM, and software versions. E.g. new versions support incremental NGFW/Snort deployments. Also, each deployment consists of two parts: NGFW (aka Snort) part and Lina (aka ASA) part and they use different mechanisms. These links will help you:

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2020/pdf/BRKSEC-3455.pdf

https://www.cisco.com/c/en/us/support/docs/security/firepower-ngfw-virtual/215258-troubleshooting-firepower-threat-defense.html

Classic ASA HA works as follows:
- sync from active to standby only, push model
- incrementally after CLI/ASDM changes or full config after bootup or "write standby"
- received commands are interpreted and applied by internal 'failover' user in both cases
- capability to skip config sync after bootup if configs are identical is being implemented by Cisco
- CLI is not blocked on standby unit, so you can configure it and it will start processing traffic with its configuration (which differs from active) should switchover happen. This capability is used in certain specific cases. Or you can block conf t access on standby if you need to.

On FTD above is not 100% true.

[Marius Gunnerud]

How is FMC sending changes to FTD? Does it send only the changes or the entire new running-config? If it sends only the changes, how are they implemented on FTD- does FMC have some internal system user who then executes those commands on FTD? If yes, how is that user called?

This document has a good explanation on how the FMC deploys configurations to the FTD.   https://www.cisco.com/c/en/us/td/docs/security/firepower/70/configuration/guide/fpmc-config-guide-v70/policy_management.html

In newer versions we are able to do partial deployments, so I am assuming that the configuration that is pushed from the FMC is merged with the existing configuration on the FTD.

Is FMC sending changes to both devices in HA pair or just to the active device that synchronizes changes to the standby device?

I have yet to find any documentation that explains how this works.  But my assumption is that configuration is only pushed to the Active firewall and then synched to the standby.  A full deployment to both firewalls at the same time would end up being very time consuming.  It is possible that this is how it worked with FMC and FTD prior to 6.6.

My question about HA failover:

Given that devices in HA are synchronized when configuration changes, when a failover event occurs, shouldn't devices already have the same configuration?

Yes, the configuration should be the same.

Which device starts the synchronization? Active or standby?

The standby device anounces it is online and starts sending heartbeats / hello packets to the active FTD.  Once the Active FTD realizes there is a standby device online and connected, it initiates a configuration synchronization to the standby FTD.

Does FTD check if the configurations are the same and if they turn out to be, does it skip the synchronization process? 

As it is not possible to configure the standby device alone in an FTD HA pair managed by FMC this is irrelevant.  With ASA however, if you configure the standby ASA on the CLI you will have devices that are no longer in synch.  So no, there is no check to make sure the configuration is the same on both devices.

How is configuration synchronization implemented? Only the differences are synchronized or does the standby device just copy the entire running-config from the active device? If only the changes, how are they implemented? Does some internal system user execute commands? If so, how is that user called? If it copies the entire running-config, how does it do it? Executes all commands or just copies the file?

On the ASA changes are merged with the standby device configuration.  If you issue the write standby command a complete copy of the configuration is pushed to the standby device.

[Marvin Rhoads]

"pushed to the Active firewall and then synched to the standby." This holds true for FTD - you can watch it happening with a packet capture seeing the FMC communicate with the Active device in an HA pair or pigtail on the managed devices (very verbose but the transactions are there).

[MHM Cisco World]

check below
MHM

[Groto]

Are you sure about that? Answer from Marvin says differently - also I checked FMC deploy transcript and syslog and I saw the following:

FMC Deploy Transcript-> CLI APPLY section = FMC >> policy-map global_policy
Syslog -> Active FW-> User 'Config' executed the 'policy-map global_policy' command
Syslog -> Standby FW -> User 'failover' executed the 'policy-map global_policy' command.

Doesn't that mean FMC send CLI command to active FW, internal system user Config executed it on it and then internal system user failover sync configuration changes on standby FW?

[MHM Cisco World]

check below
MHM

[tvotna · Accepted solution]

@Marvin Rhoadsis of course right and @MHM Cisco World is wrong. CiscoLive presentation (the .pdf I mentioned) has lots of interesting details about deployment which have never been documented officially. Interestingly, it says that FTD part of config is pushed from active to standby via file copy Lina mechanism (wow, what a kludge!) and as of 6.5 it is pulled by standby from active via another file-copy-like mechanism. Trust Veronika.

[MHM Cisco World]

check below
MHM

[Groto]

It says: "... changes are also transmitted to the standby unit"- it doesn't explicitly say that those changes are transmitted from FMC; couldn't it also mean that those changes are transmitted from an active unit instead?

[MHM Cisco World · Accepted solution]

@Marvin Rhoads @Groto 
check below it confirm the config is send from FMC to active and active sync it to standby 
thanks to all 
MHM