topic Re: Firepower deployments really slow in Network Security

Firepower deployments really slow

ncowger — Tue, 12 Mar 2019 13:29:15 GMT

I have new pair of NGFW 2110's. I have a virtual FPMC. This is a new build with relatively few rules (10) and NAT statements (14). If I make a simple change to the policy and deploy it, it seems to take a really long time. I'm regularly seeing 7+ minutes. Is this normal? Why?

I'd expect under a minute

Marvin Rhoads — Tue, 15 Aug 2017 02:31:22 GMT

I'd expect under a minute unless:

a. A congested WAN is between your FMC and the sensors or

b. The FMC is on underpowered compute resources (check the FMC status page for details).

I'd recommend opening a TAC case to have them drill into the root cause if neither of the above is the case.

FMC and Management port of

ncowger — Tue, 15 Aug 2017 02:46:17 GMT

FMC and Management port of both firewalls is on the same LAN. FMC is virtual on a UCS that is currently way under utilized. I'm seeing that the only statistic that is high on the FMC statistics page is that Memory is at 80%. Can I simply add more memory since it was an OVF deployment?

You can shutdown the server,

Marvin Rhoads — Tue, 15 Aug 2017 02:50:39 GMT

You can shutdown the server, add memory to the VM and restart but I was thinking more about CPU and storage IOPS. If it has the recommended 8 GB you may get some incremental improvement by going up to 12 or 16 GB but a deployment would not normally be a memory-intensive process.

I agree. But CPU is fine and

ncowger — Tue, 15 Aug 2017 02:52:38 GMT

I agree. But CPU is fine and storage has a long way to go before I am pushing IOPS. It's a Nimble / Cisco Smartstack.

Are you running 6.2.1 with

Marvin Rhoads — Tue, 15 Aug 2017 02:56:53 GMT

Are you running 6.2.1 with the 2110s?

I haven't done any production deployments of those and there may be a not yet publicly-documented bug. I know 6.2.2. is about to be released - I'd reach out to the TAC to see if they can shed some light.

Yes, 6.2.1. I will open a

ncowger — Tue, 15 Aug 2017 03:25:00 GMT

Yes, 6.2.1. I will open a case.

Re: Yes, 6.2.1. I will open a

danhed7400 — Sat, 14 Oct 2017 17:32:11 GMT

what did you find out ?

i am seeing the same thing on a pair of 2120 with a vFMC running 6.2.1.

when navigating in the FMC it is very slow especially when you go want to use Connection/events. deployents takes 5-10min

Re: Yes, 6.2.1. I will open a

Marvin Rhoads — Sun, 15 Oct 2017 02:47:57 GMT

Just did my first production 2110s last week. In this case we ran 6.2.2.

I found deployments to take about 1 minute. I recommend upgrading to 6.2.2. to see if that helps. Even if it doesn't, there are many bug fixes there for other things.

Re: Yes, 6.2.1. I will open a

Rodrigo Rosa da Silva — Wed, 18 Oct 2017 18:11:03 GMT

Hi Marvin,

I have installed a pair of 2110 (in HA) and running FMC 6.2.2 code.

The FMC is taking about 8 to 11 minutes each deploy.

I checked the FMC health and everything is ok.

CPU Usage - User	0.10%
CPU Usage - System	0.07%

*** This environment isn't in production, no data passing through interfaces.

Re: Yes, 6.2.1. I will open a

WC615 — Fri, 20 Oct 2017 00:22:41 GMT

Hi all,

May i know if you are using the hard appliance or virtual FMC?

Because i tried upgrading my FMCv to 6.2.2 but still experience slow deployment timing on FTD 5506X

Standalone deployment takes around 4mins and HA deployments takes around 8 mins.

Re: Firepower deployments really slow

dspender — Tue, 09 Jan 2018 15:03:24 GMT

Firepower 2110 HA, 6.2.2.1 code

Also taking 7+ minutes for each deployment. Somewhat frustrating.

Any progress on this?

Re: Firepower deployments really slow

dspender — Wed, 10 Jan 2018 18:07:57 GMT

For anyone searching on this. Here is the result of my TAC Case - I have TWO Firepower 2110 devices in HA running on most recent code:

Hello,

I reviewed the troubleshoot file and I was not able to find any issue.

As I explained in my previous email this time depends on the bandwidth and the Policy (rules, sensors and so on). I do not consider this time - 7 minutes for deploy as a problem.

Please let me know if you have any other concerns or questions.

Business day hours: Mon - Fri - 8AM - 5PM (EST)

Kind Regards,

XXXXXXXXXXX

Cisco Firewall TAC engineer

Re: Firepower deployments really slow

Marvin Rhoads — Wed, 10 Jan 2018 18:35:07 GMT

I haven't deployed to 2110's but I agree that 7 minutes is excessive. I'd push back on the TAC and request escalation to get another pair of eyes on it.

Right now I am working with a couple of vFTD instances and an FMC VM (all on the same ESXi host which is running exclusively SSD storage) and deployments complete in about 1-1/2 minutes.

You had indicated this is a new deployment with minimal policies. Are they in production at this point? I ask because I'm wondering if them being in an HA pair is affecting the time.

Is there any possibility of network issues between your FMC and the appliances? You might grab a tcpdump or spanned capture during deployment and see if Wireshark shows any tcp retransmissions or such.

Re: Firepower deployments really slow

workforcesoftware — Fri, 19 Jan 2018 16:22:48 GMT

We run a few FTD devices, along with several ASA w/FirePower services and a vFMC. I've found that the deployment times are very sporadic for FTD devices. The two devices that have the longest deployment times are our 2110's running in Active/Failover. Depending on the changes being made, they can take about up to 10 minutes. I've found that 5 minutes is the average, especially for changes to NAT and Access Policy whereas VPN changes seem to push in just a few minutes.

I've had several long talks and multiple tickets open for issues/questions with FTD, but I'm at the point where I'm just attributing this to platform maturity. I'm at peace with the length of deployment due to the security the system provides us. We used CSM to manage our ASA firewalls for a long time, so longer deployments I'm used to.

Re: Firepower deployments really slow

mikael.lahtela — Fri, 19 Jan 2018 17:15:40 GMT

Hi everyone,

I'm working with many different deployments and I would say 8 minutes with FMCv and HA pair 2110 is normal.

There is a big difference on a empty box, stand alone or ha pair. ranging from 2 minutes to 10 minutes.

I believe Cisco will be doing something about this in coming releases.

br, Micke

Re: Firepower deployments really slow

adammckay1 — Mon, 22 Jan 2018 08:15:19 GMT

It's the same for me on a physical FPMC 1000 with around 15 rules and some very basic NAT & HA configuration, for a single FPR2110 pair - somewhere between 5-7 minutes per deploy even with a single change. I wouldn't say this is a FMCv-specific issue at all and from the horses mouth I was told this was "normal".

It's frustrating because under some circumstances traffic may be dropped during a deploy (~~the circumstances where this can happen are vague and the documentation has conflicting information with the on-box help, which has information that conflicts with other on-box help~~ I just double-checked and it looks like the documentation has been updated to be clearer). We're scheduling any policy change for after-hours as a result, even if it's a single access policy item addition or removal.

Re: Firepower deployments really slow

workforcesoftware — Tue, 23 Jan 2018 14:59:24 GMT

Yeah, I've also heard this is normal from several resources within Cisco. The issue of traffic dropping on deployment is the biggest issue I have with the new system. Gone are the days of making changes during production hours, with little to no impact on the end-user. That was the one thing I loved the most about the ASAs, especially at our headquarters.

Re: Firepower deployments really slow

elcommunication — Mon, 11 Jun 2018 19:22:30 GMT

I'm new into the ASA firepower stuff and I think the deployment times are really slow up to 5 minutes. I'm getting gray hair before they're done. And if I deploy a change on a live environment and figure out the rule breaks connectivity for my users it takes at least 5 minutes to revert the changes

Re: Firepower deployments really slow

Nikolaj Pabst — Mon, 18 Jun 2018 08:33:35 GMT

Hi,

Are you running 6.2.3.X and is it a cluster?

In general 6.2.3 are MUCH faster than previous releases, and will give you a much better experience.