firepower decrypt known key stability

[tato386]

Anybody out there using this with good results?  I have been messing with it for almost a year now and it seems like it's just not ready for prime time.  I have tried with ASA firepower services (6.x, 7.0.x) and now with FTD (7.0, 7.1) and it is constantly giving me headaches.  In the logs everything always looks fine in that packets show as being allowed, decrypted, etc but encrypted connections are constantly dropping (https based VPNs), pages not loading correctly (https to internal web servers)  and even issues with it flat out blocking SMTP/TLS traffic.  Again, the logs do not show any of this.  Only way to get stability is to remove the SSL policy and/or use prefilter.  TAC is working on it and saying something about "memory blocks" and resource issues but FMC health statistics do not support this theory and like I mentioned earlier, the symptoms show on different hardware and software versions which lead me to believe the problem lies in buggy SSL decryption code?  I think it's super important to use this feature but I am close to giving up on it.

Is anybody else having similar issues?

[Marvin Rhoads]

I've used it with several customers for https to internal web servers. It worked as advertised in those cases. They were all using FTD image and mostly on 2k/4k series appliances. ASA with Firepower services is pretty underpowered for anything but the lightest use of decryption since it all occurs strictly in software.

[tato386]

Hello Marvin,

What would be the definition of "anything but the lightest use"?  In my case normal use shows as CPU never goes above 50% and usually is hovering in the teens to 20% range.  BW is 100MB circuit and we do spike to that at times but usual load is maybe 50-60MB.  Connections are maybe several hundred.  Memory use less than 50%.  Besides, the issues even happen during non-peak times where resources are even less loaded than what I mention above.

[Marvin Rhoads]

I don't have hard metrics to support the use but just remember when we learned of this feature and were highly discouraged from using it unless we wanted to see an 80%+ performance hit.

[tato386]

Does that 80% hold across the entire family of devices? If it does, I would think it makes it cost prohibitive to use this feature in most scenarios.  It would mean clients would have to purchase 4-5 times as much performance as would be needed without using it??.  Sounds kinda crazy

[Marvin Rhoads]

That's overall - if you tried to decrypt everything. For a given flow/server, it would only have that effect for that particular subset of traffic. So the overall effect(s) would depend on how much of the traffic through the appliance your are potentially decrypting.

[tato386]

That's a good point and I guess that might help a tad but what Internet traffic isn't encrypted nowadays? 

I have been monitoring my 5516 with Firepower all day today and it's actually more lightly used that I thought.  CPU has been in single digits and highest BW burst was 70MB.  I am fairly confident this is not a resource issue.  I also have an FTD-1120 which has a bit heavier load but still would classify as lightly used in my book.

What stands out is that they are different platforms and have been running different Firepower versions but symptoms and glitches are strikingly similar.  That is why I suspect defects in the underlying code.

TAC is working on the 1120.  Hopefully they will find a fix.   Depending on what we find I suspect a similar fix or mitigation would work on the ASA as well.

Thank you