Firepower deployments really slow

[ncowger]

I have new pair of NGFW 2110's.  I have a virtual FPMC.  This is a new build with relatively few rules (10) and NAT statements (14).  If I make a simple change to the policy and deploy it, it seems to take a really long time.  I'm regularly seeing 7+ minutes.  Is this normal?  Why?     

[Marvin Rhoads]

I'd expect under a minute unless:

a. A congested WAN is between your FMC and the sensors or

b. The FMC is on underpowered compute resources (check the FMC status page for details).

I'd recommend opening a TAC case to have them drill into the root cause if neither of the above is the case.

[ncowger]

FMC and Management port of both firewalls is on the same LAN.  FMC is virtual on a UCS that is currently way under utilized.  I'm seeing that the only statistic that is high on the FMC statistics page is that Memory is at 80%.  Can I simply add more memory since it was an OVF deployment?

[Marvin Rhoads]

You can shutdown the server, add memory to the VM and restart but I was thinking more about CPU and storage IOPS. If it has the recommended 8 GB you may get some incremental improvement by going up to 12 or 16 GB but a deployment would not normally be a memory-intensive process.

[ncowger]

I agree.  But CPU is fine and storage has a long way to go before I am pushing IOPS.  It's a Nimble / Cisco Smartstack.

[Marvin Rhoads]

Are you running 6.2.1 with the 2110s?

I haven't done any production deployments of those and there may be a not yet publicly-documented bug. I know 6.2.2. is about to be released - I'd reach out to the TAC to see if they can shed some light.

[ncowger]

Yes, 6.2.1.  I will open a case.

[danhed7400]

what did you find out ?

i am seeing the same thing on a pair of 2120 with a vFMC running 6.2.1.

when navigating in the FMC it is very slow especially when you go want to use Connection/events. deployents takes 5-10min

[Marvin Rhoads]

Just did my first production 2110s last week. In this case we ran 6.2.2.

I found deployments to take about 1 minute. I recommend upgrading to 6.2.2. to see if that helps. Even if it doesn't, there are many bug fixes there for other things.

[Rodrigo Rosa da Silva]

Hi Marvin,

I have installed a pair of 2110 (in HA) and running FMC 6.2.2 code.

The FMC is taking about 8 to 11 minutes each deploy.

I checked the FMC health and everything is ok.

CPU Usage - User	0.10%
CPU Usage - System	0.07%

*** This environment isn't in production, no data passing through interfaces.

[WC615]

Hi all,

May i know if you are using the hard appliance or virtual FMC?

Because i tried upgrading my FMCv to 6.2.2 but still experience slow deployment timing on FTD 5506X

Standalone deployment takes around 4mins and HA deployments takes around 8 mins.

[dspender]

Firepower 2110 HA, 6.2.2.1 code

Also taking 7+ minutes for each deployment. Somewhat frustrating.

Any progress on this?

[dspender]

For anyone searching on this. Here is the result of my TAC Case - I have TWO Firepower 2110 devices in HA running on most recent code:

Hello,

I reviewed the troubleshoot file and I was not able to find any issue.

As I explained in my previous email this time depends on the bandwidth and the Policy (rules, sensors and so on). I do not consider this time - 7 minutes for deploy as a problem.

Please let me know if you have any other concerns or questions.

Business day hours:  Mon - Fri - 8AM - 5PM (EST)

Kind Regards,

XXXXXXXXXXX

Cisco Firewall TAC engineer

[Marvin Rhoads]

I haven't deployed to 2110's but I agree that 7 minutes is excessive. I'd push back on the TAC and request escalation to get another pair of eyes on it.

Right now I am working with a couple of vFTD instances and an FMC VM (all on the same ESXi host which is running exclusively SSD storage) and deployments complete in about 1-1/2 minutes.

You had indicated this is a new deployment with minimal policies. Are they in production at this point? I ask because I'm wondering if them being in an HA pair is affecting the time.

Is there any possibility of network issues between your FMC and the appliances? You might grab a tcpdump or spanned capture during deployment and see if Wireshark shows any tcp retransmissions or such.

[workforcesoftware]

We run a few FTD devices, along with several ASA w/FirePower services and a vFMC.  I've found that the deployment times are very sporadic for FTD devices.  The two devices that have the longest deployment times are our 2110's running in Active/Failover.  Depending on the changes being made, they can take about up to 10 minutes.  I've found that 5 minutes is the average, especially for changes to NAT and Access Policy whereas VPN changes seem to push in just a few minutes.  

I've had several long talks and multiple tickets open for issues/questions with FTD, but I'm at the point where I'm just attributing this to platform maturity.  I'm at peace with the length of deployment due to the security the system provides us.  We used CSM to manage our ASA firewalls for a long time, so longer deployments I'm used to.