FTD's - Firepower dropping HTTPS traffic using TLS 1.3 Hybridized Kybe

[Jon Are Endrerud]

Hello

We have a lot of clients getting the following error when contacting diffrent sites:  ERR_SSL_PROTOCOL_ERROR, we have read that SonicWall and Palo Alto also have these problemes. Solution is to turn off "TLS 1.3 Hybridized Kyber Support" in chromium web browser, and/or I have tried to disable all SSL and "Early application detection and URL categorization" for 1.3 in FirePower.

We are using fw: 7.2.5, have created a TAC case and are waiting for answer.

Anybody else getting this ?

Regards 

J.

[Jon Are Endrerud]

Thank you for this verification. I still havent got an initial answer from TAC on this.

[Jon Are Endrerud]

We have the problem when traffic is fastpath also, is this a browser problem alone or what do you think ?

[MHM Cisco World]

Fastpath of prefilter is done and you face same issue?

MHM

[Jon Are Endrerud]

Yes, some helpdesk cases just came in from IP's in the prefilter-fastpath rules.

[Jon Are Endrerud]

Forget this post, after som tshoot, there was an error in the interface list.

[georgegriggs2]

We worked with TAC on this problem.  We are experiencing the same issue with both Edge, and Chrome.  We ended up working with a Cisco Developer.  He stated there were two Cisco Bugs listed for this: CSCwj82366 and CSCwj82736.  One bug is for if you're running Snort2, and the other is for Snort3.  Both Bugs have the workarounds for this.  See below.  

The Developer also said that Cisco is currently working on a fix for these bugs.  He doesn't know if it'll be a hotfix, or a code change.  He did say that the fix will be for Snort3 for the time being.  As Snort2 will take much longer.  

Ultimately, we ended up using Workaround #5 to resolve our problem.  Hope this helps!

1. Resolve out of order packets
-or-
2. If unable to resolve, disable TSID
-or-
3. Create a prefilter fastpath rule for TCP 443
-or-
4. Use a browser not based on Chromium 124+ (to avoid Client Hello fragmentation)
-or-
5. In the Chromium 124+ browser, disable TLS1.3 hybridized Kyber support chrome://flags/

[tvotna]

@georgegriggs2, Out of curiosity, can you share technical explanation of the issue, either details from TAC, if any, or some link? Bugs are not opening yet.

For others: TSID is TLS Server Identity Discovery: Access Control Policy > Advanced > TLS Server Identity Discovery, which is a 6.7 feature which sends TLS1.2 probes to destination server when firewall sees TLS1.3 connection. This is to get server certificate and cache it.

Also, I guess that all decryption policies became broken too now, even if TSID is disabled, right? Or it is possible to tweak Decryption Policy > Advanced > Enable adaptive TLS server identity probe ?

[georgegriggs2]

Here is the summary from TAC.

This is a brief summary of the Webex meeting:
-By collaborating with the engineering team, after further session analysis we determined that the cause of the problem is related to two different bugs which are currently internal, their IDs are as follows: CSCwj82366 and CSCwj82736
-This issue is seen on web browsers derived from Chromium 124, where hello packets are too large, causing web pages fail to load when accessing https websites.
-As per engineering team, we can apply one of the following 5 workarounds to mitigate this issue :
1. Resolve out of order packets
-or-
2. If unable to resolve, disable TSID
-or-
3. Create a prefilter fastpath rule for TCP 443
-or-
4. Use a browser not based on Chromium 124+ (to avoid Client Hello fragmentation)
-or-
5. In the Chromium 124+ browser, disable TLS1.3 hybridized Kyber support chrome://flags/

-As per engineering team, it is also suggested try running snort 3.

-It was mentioned that there will be a fix included in a future release; however this process will take months.

[MHM Cisco World]

but some engineer try two or more of solution and even fastpath the traffic and still same issue?

did you still open TAC with cisco can you get more info

MHM

[georgegriggs2]

We created a FastPath rule to regain initial connectivity.  This also told us that it was our firewall blocking.  Snort was blocking it.  Once we opened a TAC case, they told us the work arounds.  We've since removed the FastPath rule.

[ERIK ELMER]

It is a pity these bugs IDs are not published yet. (not accessible in Bug Search Tool). I would really like to see the "Conditions" here.

"1. Resolve out of order packets"

So, out of order packets is a strict condition?

[storm-syscontacts]

We've been having this issue on both 7.2.5 & 7.2.6 and found that an upgrade to 7.4.1 has resolved it.

[swilke318]

Nice. Looks like they released a 7.2.7 yesterday as well. Not sure I'm ready to make that jump to 7.4.1 but we could always try it at one of our branches first.

[swilke318]

No change on 7.2.7 as far as this goes.

[davidb84]

We were seeing ERR_CONNECTION_RESET from the client end.  Technically this issue is twofold.  The sender is sending packets out of order and there is a bug with the Snort. I will also add that we didn't see any blocking from the firewall through FMC.  It wasn't until TAC a asp-drop cap that we saw them.