ASA 5525X dropping all DNS queries

[Hermanus Janse van Vuuren]

Hi All,

I have a problem since Friday and no shared support has not been renewed so I cannot open a TAC case.

We have found that the ASA 5525X on ver 9.4.3(8) just started to deny all DNS requests.

We have an ACL that allows the Internal DNS servers to communicated with the internet on port 53 on both udp/tcp. 

After I have rebooted the Firewall everything was working 100% for between 20 - 40 min then just suddenly all UDP DNS queries gets dropped from the internal DNS servers to the external DNS servers. You reboot the FW and everything works again for 20 - 40 min.

I have tried ver 9.4.4 and ver 9.7.1 and even downgraded to ver 8.6.1 all giving exactly the same problem, after the reboot all DNS queries is allowed and then suddenly everything is denied. 

It is as if the FW skips all the ACL's created and denies the request on the global access ACL.

What is even strangers is I have an Active/Standby setup, switch the 2 FW's around and the same happens, only after a reboot the DNS queries hits the correct ACL but only for 20 - 40 minutes. I have switched off the 1 FW now and is running on only 1 firewall.

Any suggestions would be welcome on where or what to look at.

No environmental changes was made only a few Office365 FQDN's that was added before all of this started.

Thank you in advance.

[ajay chauhan]

You should collect logs from ASA to debug what exactly is going on . Is that just DNS does not work or other services or also getting stopped ?

Number of connection ,syslog ,cpu ...all these things will help to debug ?where is you DNS server located even packettracer will help .

Ajay

[Philip D'Ath]

+1.  What does the log say as the reason for the traffic being blocked?

[Hermanus Janse van Vuuren]

Thank you Sir for actually pointing me in the right direction with your question here, I never looked at the end of the deny output.

[Hermanus Janse van Vuuren]

Hi,

 I have done a show tech support report on the asa but cannot see myself any thing odd on the firewall.

Yes all DNS queries gets dropped so we do not have any internet access out making use of the proxy server as it comes back saying DNS cannot be resolved, how ever if you bypass the proxy and open your IP on the FW for direct access and make use of an external DNS like 8.8.4.4 you are able to browse the internet.

My problem here is the internal DNS that sits on the internal network is being blocked accessing the external DNS for queries like for example Internal DNS is querying 8.8.8.8 but gets denied.

Deny udp src Inside_interface:x.x.x.x/53432 dst ISP_Interface 8.8.8.8/53 by access-group "global_access"

This is one example I get similar denies for the same interface with different acces-groups.

My CPU is running on 76% and my Memory usage is 3.9GB on the device.

[ajay chauhan]

This seems to be issue with ACL's . Do you have any specific ACL applied on interface where your DNS reside and UDP 53 is allowed ?

Global access list applies logically to the entire firewall in inbound direction to all interface.

You may paste device configuration here.

Ajay

[Hermanus Janse van Vuuren]

Hi Ajay,

Yes I have an ACL explicit for my DNS servers to access the internet on UDP/TCP port53. As I said this works well after a reboot for about 20 - 40 minutes then it skips all ACL's and go straight down to the global_access ACL.

I have now allowed my DNS servers on the global_Access ACL to query DNS on port 53 TCP/UDP and problem has been resolved.

Not the ideal solution but at-least we can browse now again and send email.

I suspect some changes was made the day before this started adding a new VPN group, DMZ for them and some ACL's but I cannot see how this would stop the Firewall from reading the spesific interface ACL's and just go straight down to the Global one.

[Philip D'Ath · Accepted solution]

I just found something interesting.  There is an ASA DNS memory leak bug in the version of code you are running.

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvd71473

It is not showing any fixed releases, but I would upgrade to asa963-1-smp-k8.bin to try and help.

[Hermanus Janse van Vuuren]

I have upgraded to the version you have mentioned below 9.6.3 and it seems to have resolved my issue. Just a point of note it would seem as if ver 9.7.1 also has the memory leak then as this did not resolve the issue when I ran it on that version.

Thank you very much for the kind advice.