Forced to reboot ASAs to see new DNS server and be able to resolve local resources, why?

[Mike Bowers]

                   Hello all. I ran into a very interesting problem that occurred today and I'm trying to figure out why it happened. If it was one ASA 5505 that just required the reboot, then I'd have just chalked it up to a glitch, but when we built a new AD/DNS server on the main network at the main site and changed the 3 Remote site ASAs to point to the new DNS server in the DHCPD options, none of them could ping any local hostnames to the DNS server at the main site they were now pointing too, but external hostnames (www.yahoo.com, www.google.com, etc) all translated and pinged fine.

From a laptop on one of the remote sites, we could ping the new AD/DNS server(192.168.0.3) and the old AD/DNS server(192.168.0.2) and everything else at the main site, and telnet to port 53 showed successful across the EasyVPN from the Remote site to the new server at the main site. When wireshark was added to the new DNS server at the main site, the DNS request and replies for www.google.com, for example, came and worked fine, but any requests for local resources never made it to the server from the remote sites.

A reboot of one of the Remote Site ASA's corrected the issue. Then I rebooted the other two remote site ASAs, and now DNS was working fine for everybody.

I had also tried clearing the ARP cache on the ASAs before resorting to rebooting them. I also tried rebooting the laptop thinking the local DNS cache needed cleared before resorting to rebooting the ASAs.

I'm struggling to understand why external, public hostnames made it through and resolved from the remote sites to the new server at the main site, but anything local failed before even reaching the new server(The new DNS server could resolve requests made by computers at the main site, but the remote sites that traverse the EasyVPN from the ASAs failed).  The new AD/DNS server is the only server configured for DNS for all remote site computers.

Is any of this making sense? I'm wondering if clearing the xlate or local host tables would have corrected it without having to reboot. I'm just trying to grasp the understanding here and figure out what happened.

Please let me know if I can explain the issue better or if anybody is seeing the issue, because I'd love to know!

[Julio Carvajal]

Hello ,

As you already explained it does sounds weird,

But for this kinds of scenarios where nothing makes sense for us as networking people what we could use are : logs and captures from the ASA itself,

Is there a possiblity that you have that I would love to see that because indeed it's really interesting,

Hope you have them

Regards,

[Mike Bowers]

Hello JCarvaja,

I apologize but I unfortunately do not. Up until rebooting the first firewall we were assuming it was the server since it was a new install. Once we saw the packet trace we tried rebooting a firewall, and since it worked, we rebooted the other two remote firewalls and everybody was fine. We were kind of in a hurry to get them up and running and the buffers were cleared.

I'd have liked to take a few extra minutes before to go over them if I could go back just to see if there was anything in them regarding this issue and why only local DNS request/replies weren't getting through.

MDF is just the default policy for DNS inspection. Everything is pretty much default on the remote ASAs except for the easyvpn set up.

[Julio Carvajal]

Hello,

Yeah, that is a shame. I mean I would love to determine the root cause of that problem but without any inputs from the ASA at the time of the problem I would not be able to get any further,

I hope you understand this and if by any chance this happens in the future you get the right information

Regards,

Remember to rate all of the helpful posts

[Mike Bowers]

Oh I understand.

However, I've been thinking, and considering it is a pretty basic configuration. I'm trying to look into DNS inspect and how it inspects DNS queries and responses.

Is there any way in DNS inspect an established connection can keep a host name (for example, server.domain.com) pointed to a specific IP address?

So DNS queries to the new server work when going externally, but internally, all the clients going to server.domain.local..the firewall is possibly blocking because it sees server.domain.local still linked to the old IP address.

The ARP table links the MAC with an IP, but is there any connections that link an FQDN with an IP that may need to time out or be cleared? 

I don't know, taking a last stab at it before I go on my merry way   Thanks for the consideration!

[Julio Carvajal]

Man you said is to easy but not sure that I follow you on this requirement,

Can you explain it once again