Firepower 6.1 - SSL decryption random drops - ASA 5516 + FMC

[Raphael Chevalier]

Hey,

I'm trying to make outbound SSL decryption work but have been facing many issues.

When I apply an SSL inspection policy I'm having really weird results (see issue below). Right now I only have one rule in my SSL policy : decrypt everything from a single ip address (test user) and then default action : do not decrypt.

I assumed this policy would have absolutely no impact beside my test user but clearly I was wrong :-)

Issue:

The ASA is dropping SSL packets randomly, this happens not only on my test user but on several web servers hosted by my customer (inbound connections).

In the FMC logs I can see all packets as "Allow". But in the ASA logs I get "SFR requested to drop TCP packet from outside:xxx.xxx.xxx.xxx/50000 to dmz:xxx.xxx.xxx.xxx/443" on about 2 out of 3 packets. And sometimes "SFR requested ASA to bypass further packet redirection and process TCP flow from [...]"

The end user result is very slow https websites since most packets are dropped. I really don't understand how this is possible as I only have a single rule matching a single private ip and everything else is set to not be decrypted.

I thought it could be a performance issue as we have a lot running on these ASA (malware&file, ips, identity, vpn...) but they are just fine CPU & memory wise.

Any idea ?

Raphael

[Dennis Perto]

6.1 has a lot of SSL bugs. Please update to 6.1.0.2 if you have not already. :)

[Raphael Chevalier]

Thanks for your answer Dennis. I've already done so on the FMC (6.1.0.3 even). Should I also upgrade the firewalls ? Could that be the reason ?

[Dennis Perto]

Great. I don't know if you will need to upgrade the ASA version, but anything could be the cause of this. :)

Your ASA just need to be at either version;

9.5(2) or later

9.6(1) or later

9.7(1) or later

As per this page:

http://www.cisco.com/c/en/us/td/docs/security/firepower/compatibility/firepower-compatibility.html

[Raphael Chevalier]

Alright but what about the firepower modules on the ASA ? Does that need a separate upgrade too ?

[Dennis Perto]

In the best scenario they are the same version as the FMC :) 

[HQuest]

I'm on the same boat too, however with a slightly different setup for my lab. A 5506-X ASA 9.7(1) and FirePower 6.2.0.1. The HTTPS performance is mediocre past the SFR requests to ASA for packets to be dropped. Doesn't matter if the ASA+FP is managed locally via ASDM+IDM, or via FMC.

Now, I noticed some of my Access Control Policies were a little too aggressive. For "Very High Risk" sites I made a "Maximum Security" IPS policy, for "High and Medium Risk" sites, a "Security over Connection" IPS policy, and so on. I had to make exception rules prior to those in order to lower specific higher risk sites to a lower IPS policy. That helped with a few packets being blocked for public services, ie Facebook, Twitter, Apple AppStore and Amazon Videos, however other sites classified as "Low Risk" or even Very Low Risk were still being dropped.

Then there were a few sites I had to turn SSL inspection off. Sourcefire was such site I added a new DN object, otherwise the appliance would fail to update their own GeoLocation and Rules. My VMWare Horizon portal was too such a location I had to make a network object exception as the tunnel would not be validated because of the man-in-the-middle PKI. Even while I was only actively re-signing Very High and High Risk sites, and leaving everything else go their merry way.

I have a TAC open, will see what Cisco replies back from this. Had to turn SSL Inspection off in order to open the TAC page to update my cases... However I'm still up to find out more about it from any fronts.

[Dennis Perto]

Just as a side note to this. How did you configure your Network Analysis Policy? 

If it is not inherited from "Maximum Security" you are wasting resources on your current policy. 

Be advised that the Maximum Security policies are very resource heavy.

[HQuest]

Monitor All;

Inspection Exception - High (to downgrade applications listed as Very High risk)

Inspection Exception - Med (to downgrade applications listed as Very High/High risk)

Inspection Exception - Low (to downgrade applications listed as Very High/High/Medium risk)

Inspection Very High risk (with applications listed as Very High risk)

Inspection High/Medium risk (with applications listed as High/Medium risk)

Inspection Low risk (with applications listed as Low risk)

Inspection Very Low risk (with applications listed as Very Low risk)

Default Action: Maximum Security

Similar on the SSL policies:

Monitor All;

Do not decrypt (Manual sites)

Do not decrypt (Subject DN)

Re-Sign Very High/High risk sites

Default action: Do not decrypt

And I can see the classification happening, with many sites on the Medium and lower risks still being inaccessible with a test network of one ASA and two clients. 

[Michael Braun]

Well at least i am not alone.

Just went through a day playing with different setting within the SSL Policy.

I noticed the same SFR drops, especially the returning traffic seems to be affected. While Opera and IE seem to handle it, Firefox was not going to play along. (Certificate store etc was done in FX, so no that is not it)

Even if i make an exception for the source IP of the management center, it still shows the same drops from it to the WAN. ??? Even limiting to just ONE source ip for testing, the ASA showed massive SFR drops all with port 443, mainly return traffic. It seemed to work thou, the certificates did get rewritten, it makes no sense having all the sfr drops show up on the ASA even thou they clearly should pass without matching anything.

I am on 6.1.x and for right now i have turned off ssl inspection completely.

It looks like the FP module is not correctly matching the ssl rule.

Traffic should only be affected with a match of the rule and nothing else - not the case.

Strange thou, it is still passing traffic even thou the returning traffic gets dropped - according to the ASA log. So maybe it is partial cosmetic?

I am curious what support has to say about that. Please post it when you get an answer. I will postpone hitting up Cisco support for right now, at least wait til the next update comes out.

[Raphael Chevalier]

Hi Michael,

I plan to upgrade the firewalls to 6.1.0.2 and see if that fixes the issue.

If it doesn't then I'll open a TAC ticket because I see no other solution.

[Raphael Chevalier]

SSL packets are still being randomly dropped after upgrading FMC and firewalls to 6.1.0.2.

I'm opening a ticket and will post back.

[HQuest]

I'm still working with TAC on this. So far, nothing moving anywhere.

However, I was pointed to this topic today - https://supportforums.cisco.com/discussion/12959476/5506-x-firepower-blocks-traffic-prior-policy-processing-and-does-not-record

Seems this is an old issue crippling the product. Also seems bigger ASAs are not seeing much impact, which tends to tell me the 5506 has the SSL Inspection feature but it cannot handle traffic of a single user.