FTD 1010 / 6.7: Seems to drop packets/connections on some clients in the network

[mcgiga]

Hi all,

we are testing a FTD with version 6.7 since this week. It's the first release we are testing because the AnyConnect port is changeable now, that means we don't have experiences with previous versions regarding bugs etc.

Today we noticed that on some clients on the network several websites are not loading. Most of them are loading very slow until we actually can see the content. Then there are some websites like www.techdata.de / de.techdata.com which can never be reached.

Some clients on the network don't have this issue. First we thought it has something to do with DNS on the domain controller, the switches or anything like that but that's not the case.

We removed the FTD 1010 and replaced it with a FritzBox router. The issue is gone and not reproduceable anymore.

Then we had a look into it with Wireshark on the affected clients. When we use FTD 1010 and try to browse www.techdata.de is seems that the browser doesn't get a response and always tries to connect to the destination with TLSv1. When using the FritzBox the connection to the website looks completely different in Wireshark, no issues with loading the website. The connection happens with TLSv1.2 according to Wireshark.

FTD 1010 with 6.7:

FritzBox:

We don't have anything special configured in FTD, one NAT rule, some port forwardings from the outside and of course some ACLs, nothing else.

For now it looks like FTD 6.7 is somehow dropping packets/connections on some clients, but not all clients.

[mcgiga]

Additionally I can say the problem occurs on mobile devices like iPhones and Androids too.

[HQuest]

I've just experienced a very similar issue, in between two internal network segments. Although on v6.6.1, the sensor (FPR1010) just stopped forwarding packets in between its interfaces. From the FTD CLI I could reach destinations on both interfaces/zones, but from zone A to B, traffic wasn't moving. Seems the way the FTD and the FXOS interact in newer code is different from v6.2.3 found on ASA 5500-X series, so I'll need to dig a bit more into its internals to do better troubleshooting. I just played lazy and reload the unit. As expected, everything came back up. Not a solid start for a code marked as "recommended for quality, stability and longevity": it is the 3rd random issue we've experiencing in less than 72-hours. Our next call with our Cisco account manager and product team will be fun...

[Edit] about two hours later, it happened again. No direct clues from the regular, common CLI options. I found CPU03 is eating 100% CPU during the time the device gets unresponsive; now it is time to find out why.

[HQuest]

For my own issue, I reviewed some old TAC cases and found a possible link. Way early in the days of the new ASA5500-X series, Cisco once told me some of my devices were too small for SSL decryption (even while decryption was an advertised feature; it is now consented to expect 80% drop of performance - or random crashes with symptoms including high CPU utilization - when this feature is on). I found decryption to one pretty quiet web server was enabled on the inspection policy for this particular device, turned it off on and guess what, the sensor has been acting pretty normal since then. Might be coincidence? Maybe.

The data sheet of the smallest FPR 1010 claims it can do at least 150Mbps of TLS security, but seems this is, once again, marketing material disconnected from the engineering reality, and I’ll stick with other vendors products for a long time before even thinking on enabling this on any of my beefier 4100/9300 series.

[Aref Alsouqi]

This is very interesting, I've never seen something similar, although I'd deployed a few Firepower 1000 appliances. It smells as a bug. What is the IPS base policy you have? is the IPS policy applied to the affected traffic?

[mcgiga]

A reboot doesn't fix the issue. Not matter which devices I restart (Firepower, switches, PCs, mobile phones) the issue occurs on the identical devices every time.

I have configured Identity policy, NAT and ACL only. There are no IPS rules.

I would like to try an older firmware but we need the possibility to change the AnyConnect port to a different port, so in production it would be no real solution.

[Aref Alsouqi]

Very interesting! Not sure what traffic those devices would be generating to cause that issue. Out of interest, could you please try to change the IP address of one or more of those devices and see if that makes any difference?

[mcgiga]

I tried, unfortunately the issue is still there. I could not find the cause at the PCs and mobile devices for the behaviour of the Firepower.

[Aref Alsouqi]

Do you see any interesting events in the Monitoring for those devices?

[HQuest]

Even if you have not configured SSL Decryption, the simple fact a default SSL policy is listed on your Access Policies will cause the SSL inspection to be turned on, and possibly random behaviors such as connection errors and soft freezes of the sensor DPI. Also, I've seen previous FMC versions without support to TLS v1.3 to break communications with this protocol - and so did Cisco, more at https://www.cisco.com/c/en/us/td/docs/security/firepower/SA/SW_Advisory_CSCvh22181.html

[Aref Alsouqi]

FDM sows all the supported policies in the Policies section, however, any policy showing in italic grey is disabled, so I don't believe in this case the SSL decryption feature can affect in anyway.

[mcgiga]

Since Cisco three different engineers were not able to resolve the issue yet, I did some more testing. As of yet I know that the issue occurs only with specific LAN controllers on every kind of device (computer, notebook, smartphone).

That means a client computer which faces this issue can be fixed with a different LAN controller (i. e. USB to LAN adapter).

So the issue on the Firepower device is triggered by specific LAN controllers.

Like I wrote the issue is not fixed, I did more testing today. I have set up a SSL Decryption policy for the inside LAN network. When this policy is enabled the affected clients can browser all websites, can connect to services like virus definition update server etc. by SSL/TLS.

Then I removed the policy but left SSL Decryption turned on. The results are the same, all SSL/TLS connections are still working. When I disable the SSL Decryption and deploy it the issue reappears.

I am amazed that no one in this forum has this issue nor Cisco engineers seem to know this issue.