cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
15516
Views
3
Helpful
12
Replies

5506-X with FirePOWER blocks traffic prior to policy processing and does not record an event?

I have some pretty severely broken behavior from the 5506-X with FirePOWER that has made inline mode completely unusable for me. In real world terms, its behavior makes it impossible to reliably use Apple iOS or OS X systems if their traffic is being processed by the FirePOWER module.  From what I can tell, the FirePOWER can respond to traffic it is processing prior to processing it through its defined policies.  Here is what happens as soon as I put it into inline mode.  (Note I am running 9.5.2(6) and 6.0.0.1 but this has happened under every version released since I acquired this 5506X about a year ago).


I take the 5506-X FirePOWER out of monitor mode.  Note that there are probably a dozen iOS and OS X systems on the network trying to maintain connections to receive push etc.  With this number of devices, I see entries like this generated 2-3 times per second as they endlessly retry.  A side effect of this blocking is that it generates TONS of additional DNS queries, which return ever more 17.x.x.x IP’s to attempt to connect to, so this never idles down and never stops.  Apple seems to provides pseudo-random responses to DNS queries that cover an enormous part of their class A.

So:
As soon as I take it out of monitor mode, it immediately starts generating entries in the ASA log like this 2-3 times per second:
SFR requested to drop TCP packet from inside: 192.168.1.81/49632 to 17.143.161.157/5223
SFR requested to drop TCP packet from outside: 17.143.161.157/5223 to inside 108.28.98.61/49632
SFR requested to drop TCP packet from outside: 17.143.161.157/5223 to inside 108.28.98.61/49632
SFR requested to drop TCP packet from outside: 17.143.161.157/5223 to inside 108.28.98.61/49632

It appears Apple retries three times on the response.

I have the logging level set to debugging, and the Syslog details say “No information is available on this syslog ID”

So that’s useless.

Then I choose ASA FirePOWER Monitoring in the ASDM and select Real Time Eventing. Despite the ASA being told by SFR to drop packets several times a second there are ZERO events listed that are not “Allow”.  Nothing but allows.  I refresh, I add new columns to try to filter by, etc, no events listed other than "allow".

So I run the Packet Tracer for that exact outbound path. Packet tracer shows it passing through FirePOWER inspection, and then “RESULT - The packet is allowed”.

Um, what?  I look closer at the logs and every once in a while I see that it DID allow a connection.  This explains the erratic pseudo-random delays in users receiving iMessage traffic and whatnot.  So the FirePOWER's behavior is non-deterministic, and UNLOGGED. So it’s @#$% BROKEN.

Here, I think, is the kicker.

In an attempt to fix this, I created a policy in the FirePOWER to trust all traffic to and from the 17.x.x.x network.  It is called “Trust Apple Class A”.  It is pretty simple really.  It trusts all traffic to and from that class A.  If I have the FirePOWER in monitor only, this policy events a few times an hour as things like iPhones and iPads and OS X Server etc try to keep a port hung open to Apple for purposes of getting push notifications etc.

When I take the FirePOWER out of monitor mode, this policy still generates hits ONLY when the FirePOWER randomly decides not to tell the ASA to drop the traffic.

Since traffic in the trust category is supposed to bypass inline blocking, and it is clear that the FirePOWER is processing traffic and making decision before it hits my trust policy, this implies that the published flow path through the FirePOWER is wrong.  I suspect that this explains why there are no events for blocking this traffic listed when you use monitor to watch FirePOWER events.

12 Replies 12

nspasov
Cisco Employee
Cisco Employee

Hi Justin, I had a lot of issues with 6.0.0.1 and my 5506 in my lab. Upgrading to 6.0.1 seems to have made things a lot more stable and my ASA does not randomly drops packets anymore :) I would suggest you upgrade yours as well.

Thank you for rating helpful posts!

There are several issues that got fixed in 6.0.0.1 version and thus its highly recommended to upgrade the device to the most stable version.

I updated to 6.0.1 and the behavior is the same.  As soon as I take the FirePOWER out of monitor mode, I start seeing entries in the ASA log like the one I posted above "SFR requested to drop TCP packet from inside: 192.168.1.81/49632 to 17.143.161.157/5223", and if I look in the FirePOWER event logs, there are no related events recorded.

Are you using SSL policy by any chance ?

If SSL decryption is used , original traffic is dropped while new traffic is injected causing ASA to see messages like these. But there shouldn't be any disruption in traffic.

Or you can check global security intelligence block list to see if any of the IP's is in the block list which is being dropped.

Is the traffic being dropped there ? I mean any outage or its just the message keeps coming but there are no actual drops.

Obviously I've been hoping Cisco would do something about this behavior for some time.  What I can say is that with 6.0.1.1-24, and having switched to a 5516 from a 5506, I still get the "SFR requested..." entries in the firewall logs, but it no longer actually drops the traffic most of the time.  This has allowed me to switch the FirePOWER into inline mode.  While TAC doesn't give a satisfactory explanation, I think what the message is intended to mean is that SFR has requested that the firewall stop handling the traffic so that it can.  IE "Drop that packet so I can play with it".  If so, this is pants on head retarded, but the ASDM is packed with bad grammar and spelling mistakes and inexplicably is still a java app, thus requiring us to maximally increase the attack surface of the console we use to administer our security appliances, so no surprises there.

Anyway, this 6.0.1.1 behavior is still not perfect. Every once in a while it seems like multiple simultaneous connection attempts (such as a client that is logging into several exchange accounts at once on cloud hosting) will trigger it to drop traffic for one of the connections, I assume due to some kind of rate limiting for the number of SYN's or NAT actions with the same source and destination IP.  This is rare enough that it's sort of tolerable (maybe once per day for a client that tries to stand up 7 POP connections at the same time), but it is undeniably broken that the FirePOWER will modify traffic flows for reasons that the operator cannot see or control.

WOW. I'm floored. I'm in the process of replacing our Checkpoint cluster in our Datacenter and went with Cisco because I use there routing and switching throughout our environment and its rock solid. Even use ASA's to terminate VPN's. If I wasn't working on my second attempt to cut-over at his point, Id send them back and get Fortigates.

Thanks for the responses. At least I don't feel like I'm the only one seeing these drops and abnormal behavior for a Firewall.

Anthony

I think the ASA and even the FirePOWER are good solutions for the edge of a datacenter or business network that does not have BYOD activity on it.  Reliability has been very good (I have 5506, 5516, and 5555's in the mix) and I do like the automated signature updates etc, though the pace of them is entirely too slow.  Being able to define IP geo block lists and let the FirePOWER deal with the changes automatically is also nice for a DC or tightly controlled corporate network.

As soon as you bring BYOD into the mix though, these problems with behavior and logging mean that the savvier a BYOD user is, the more likely they are to run into uncontrollable problems with the FirePOWER and then complain to their admin.  When I made my original post it was right after this exact experience.  Having it constantly block push notifications to iOS and thus iMessage traffic in an office environment that is 100% iPhone users generated complaints so fast after I took the FirePOWER out of monitor mode that you'd think they were all looking at the logs at the same time I was.  Since the traffic was being blocked before the FirePOWER's flow processing hit a user-controllable feature (namely setting traffic to "trust"), that meant that the choices were to completely disable the FirePOWER and thus gain no benefit from it whatsoever, or to cripple communications for users, all because it does not actually process traffic the way Cisco says it does and can still block traffic that should be trusted for reasons that a user cannot change or control. Cisco is also not doing an acceptable job of keeping their traffic signatures up to date for operating systems.  Go ahead and poke through the operating system traffic signatures it has and try to find mention of iOS or Android, or a version of OS X newer than a decade or so ago.

So I run it in my test environment, which is quite noisy in terms of traffic, and with each update I would try taking it out of monitor mode to see if it crippled mobile device communications or caused email clients that generated a lot of simultaneous outbound NAT translates to throw password error messages etc when packets were arbitrarily dropped.  With 6.0.1.1-24, this behavior became good enough to be acceptable and I have switched it out of monitor at client sites, but I know that I have to check that with each release now to ensure that it does not regress.

Jetsy,

I am seeing the same issue with our 5555 running 6.0.1.1. I see several "SFR requested to drop TCP packet" messages in the syslogs for traffic that should not be dropped and as far as I can tell is not actually being dropped. No drops recorded in Firesight management either. I read somethings about inline mode and or pre processor but inline mode is off. My experience has always been when a firewall (any vendor) says its dropping a packet there is something incorrect. Tac cant give me a good explanation. Very frustrating...

Hello Anthony,

We have several scenarios for this message and infact we have few bugs reported for this kind of messages in the ASA logs. We have 3 bugs reference reported on this behavior. Unfortunately apart from the below one , the other two you cannot access from outside or visible. Are you actually facing any outage or just you see those messages in logs ? In that case it can be due to the following bug.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz32624/?reffering_site=dumpcr

Have you provided the troubleshoot file to the TAC ? In that case they should be able to give you a explanation.Please check above bug and let me know.

IS your ASA in multi context mode ?

Do you have any default action as trust in AC policy ?

Rate and mark correct if the post helps you

Regards

Jetsy

Jetsy,

Wow what a quick reply. As far as I can tell with my testing there is no outage or unable to connect to hosts. I am using only a few test servers now as I'm planning a second attempt at a cut over from our existing Firewalls.  I have provided one Troubleshoot to TAC and just sent another. I am waiting to hear back but so far I haven't gotten any good explanation. Ive even discovered issues when using Applications in rules causing traffic not to pass. Ive since removed all Applications from my rules and replaced them with Ports and port groups.

IS your ASA in multi context mode ?  - No. Failover HA Active/Passive

Do you have any default action as trust in AC policy ? -  No. My Default Action is Block All Traffic

Anthony

Hello Enrico,

Since you are not facing any outage , it can be the above mentioned bug .

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz32616

(its a duplicate bug of what mentioned above ).

Rate and mark correct if the post helps you.

Regards

Jetsy 

Abraão Marques
Level 1
Level 1

I have a problem too, 

You fix this ? 

 

Asa firepower 

SFR requested to drop TCP packet from outside: 10.90.7

69/80 to inside 189.8.76.86/80

Review Cisco Networking for a $25 gift card