FTD/FMC - DNS replies being blocked by ACL - Need help understanding

[NA-School]

- Environment overview
-- FTD 2110 on 7.2.4
-- FMC on 1600 7.2.4 running Snort 3.0
-- 2 - DCs on Windows Server 2016 both running DNS
---- DNS Forwarders pointing towards ContentFilterDNSServers
-- The content filter (4 public ipv4) offer DNS response when they have it cached (assumption), if the content filter is triggered, DNS responds and the user is redirected to a "Restricted Page", this works as expected.

-- Access Control Policy rule set as follows:
Source Zone: Inside
Destination Zone: Outside
Source Networks: DC1,DC2
Destination Networks: ContentFilterDNSServers
Applications: DNS
Destination Port: UDP (17):53
Logging: Log at end
Intrusion policy: Balanced

-- When the DNS request isn't cached on the content filter's DNS servers, it forwards that request to cloudflare/google/others - the packet is then returning to the firewall from a random DNS server
-- The final rule in my ACP is called "Explicit Block" Outside -> Inside BLOCK any
-- The content filters DNS servers are under heavy load in the mornings (assumption), and a lot of requests get passed along, so during the first 2 hours of the day here, I will see my firewall with 100s of dns responses (that were requested) being blocked in the connection events, with "Explicit Block" being the reason.

My understanding is that the firewall's ACP is stateful, if a connection is on the allow from inside to outside, that it will allow the response in the opposite direction as long as the connection is initiated from inside. Does this apply for DNS if the response packet is from a different IP?

Last week, I was seeing a lot of invalid response errors in my DC's event viewer, and I disabled EDNS, this seemed to make everything better, but the problem returned today and EDNS is still disabled.

I went ahead and created a new ACL in my ACP, but I want to avoid doing this unless it's necessary..

Source Zone: Outside
Destination Zone: Inside
Source Networks: ContentFilterDNSServers, 9.9.9.9, 1.1.1.1, 8.8.8.8, 8.8.4.4
Destination Networks: DC1,DC2
Applications: DNS
Source Port: UDP (17):53
Logging: Log at end
Intrusion policy: Balanced

This feels wrong to me. Is there a best practices guide or any recommendations in getting DNS to not break in my environment?

I'm feeling like the "aha!" moment is just out of reach here, and I apologize for being ignorant to how DNS works in conjunction with the ACP when packets are coming back from a different IP. 

[MHM Cisco World]

What is the dns server use by client?

Is it 8.8.8.8/8.8.4.4 or Inside FPR ip?

Did you check by packet tracer? if yes share output here.

MHM

[NA-School]

DNS by client is DC1, DC2.

[tvotna]

@NA-School, in my opinion, your content filter's DNS servers do something wrong. They should not forward original requests to 8.8.8.8 / 8.8.4.4 / etc. Instead, they should process requests and act as regular DNS forwarders, whether the load is high or low. They should initiate new DNS request from their own IP address, so that response comes in to them, then forward the response back to DC1/DC2. Otherwise FTD has to drop the response if it doesn't have "allow" rule to let it through.

And even though you created new inbound rule to allow responses through, this should not help in my opinion, until you also disable DNS inspection on FTD. This is because FTD DNS inspection by default matches DNS responses with DNS requests previously seen. It uses DNS ID to do this. Inspection can be disabled via FTD CLI: configure inspection dns disable. Incomprehensible, but the product still lacks this feature in GUI.

[NA-School]

I agree wholeheartedly, I was scratching my head on this one. I'm guessing it is related to load constraints on their part. I will say the ACP rule I added did seem to fix it, but in my haste and frustration to resolve the issue yesterday, I just used cloudflare as forwarders. 

I will go back to the ContentFilterDNSServers as forwarders for the DCs, and try again to confirm my assumptions.

If it was being blocked by the default DNS inspection, would connection events still be calling out the rule name for my explicit block or would it say something else? 

[Marvin Rhoads]

I agree with @tvotna on this one. No firewall should be expected to allow random incoming DNS replies that are not explicitly returning an answer to an outbound lookup. I would take it up with the content filter vendor support team.

[MHM Cisco World]

Can I see 

Policy -> dns ->edit defualt dns policy 

MHM

[NA-School]

[MHM Cisco World]

No need that.

MHM

[MHM Cisco World]

Three days thinking about issue 

Retrun back traffic drop!!

Let start 

DNS DC1/2 send DNS query to google DNS the traffic pass as usual.

So

Prefilter is OK

ACP is OK 

DNS policy (as you share I see default you dont modify DNS policy' and even if the DNS policy drop query then we must see traffic going not return traffic drop)

So there are still two points still 

1- QoS 

2- the DNS use https 

So can you check these two points 

Thanks

MHM