06-09-2020 10:38 AM - edited 06-23-2020 01:39 PM
I have a problem where DNS responses over IPv4 are getting dropped between my firewall and switch and are not getting back to the host. This is resulting in failed DNS lookups. Below is my network layout:
The DNS standard query responses are getting dropped somewhere between the ASA's inside interface and the Catalyst's main uplink interface (Port-channel 1). This is only happening on IPv4. I have the 10.2.28.0/22 network NAT'ed on the ASA. I'm not sure if this is a routing, NAT, packet inspection, or ACL issue for the ASA or an ACL issue for the Catalyst. Details are below:
root@server1:~# ip --brief -4 addr show veth-mgmt veth-mgmt@if273 UP 10.2.28.2/22 root@server1:~# dig A archive.ubuntu.com @91.189.94.173 ; <<>> DiG 9.11.3-1ubuntu1.12-Ubuntu <<>> A archive.ubuntu.com @91.189.94.173 ;; global options: +cmd ;; connection timed out; no servers could be reached
root@server1:~# ping -c 4 91.189.94.173 PING 91.189.94.173 (91.189.94.173) 56(84) bytes of data. 64 bytes from 91.189.94.173: icmp_seq=1 ttl=53 time=120 ms 64 bytes from 91.189.94.173: icmp_seq=2 ttl=53 time=120 ms 64 bytes from 91.189.94.173: icmp_seq=3 ttl=53 time=120 ms 64 bytes from 91.189.94.173: icmp_seq=4 ttl=53 time=120 ms --- 91.189.94.173 ping statistics --- 4 packets transmitted, 4 received, 0% packet loss, time 3002ms rtt min/avg/max/mdev = 120.767/120.826/120.957/0.355 ms root@server1:~# telnet 91.189.94.173 53 Trying 91.189.94.173... Connected to 91.189.94.173. Escape character is '^]'. ^] telnet> quit Connection closed.
ASA# capture asadnscap interface inside match udp host 10.2.28.2 host 91.189.94.173 ASA# show capture asadnscap 6 packets captured 1: 16:58:50.489552 10.2.28.2.56438 > 91.189.94.173.53: udp 59 2: 16:58:50.610701 91.189.94.173.53 > 10.2.28.2.56438: udp 191 3: 16:58:55.489736 10.2.28.2.56438 > 91.189.94.173.53: udp 59 4: 16:58:55.610808 91.189.94.173.53 > 10.2.28.2.56438: udp 191 5: 16:59:00.490132 10.2.28.2.56438 > 91.189.94.173.53: udp 59 6: 16:59:00.611098 91.189.94.173.53 > 10.2.28.2.56438: udp 191 6 packets shown
SWITCH#monitor capture swdnscap buffer size 10 interface GigabitEthernet 1/1/1 both match ipv4 host 10.2.28.2 host 91.189.94.173 SWITCH#monitor capture swdnscap start SWITCH#monitor capture swdnscap stop SWITCH#show monitor capture swdnscap buffer brief 0.000000 10.2.28.2 -> 91.189.94.173 DNS Standard query A archive.ubuntu.com 5.005996 10.2.28.2 -> 91.189.94.173 DNS Standard query A archive.ubuntu.com 10.005996 10.2.28.2 -> 91.189.94.173 DNS Standard query A archive.ubuntu.com
If I had to guess it has something to do with packet inspection at the ASA level, but I'm not sure how to confirm this or what to do about it. Also, my switch actually uses a port channel and I don't have a recent enough version of IOS-XE to monitor PC's so I just picked one of the ethernet devices in the PC. Not sure if that would effect my packet capture. I ran the test multiple times to make sure I always saw the same results (which I did).
Finally I also tried to see if I could capture dropped packets on the ASA (i.e. capture asadropdns type asp-drop all). I exported the pcap and tried to find any packets from 91.189.94.173 that were dropped, but couldn't find any that specifically contained DNS responses (there were some administrative ICMP drops but their source was not from the DNS server).
Here is my ASA configuration:
interface Port-channel2 lacp max-bundle 8 nameif inside security-level 100 ip address 10.2.1.1 255.255.255.0 ipv6 address 2001:db8::1/64 ipv6 enable ! object network virt-mgmt subnet 10.2.28.0 255.255.252.0 access-list outside_ingress extended permit ip any 203.0.113.0 255.255.255.0 access-list outside_ingress extended permit icmp any 198.51.100.0 255.255.255.248 access-list outside_ingress extended permit tcp any host 203.0.113.84 eq imap4 access-list outside_ingress extended permit tcp any host 203.0.113.85 eq 587 access-list outside_ingress extended permit tcp any host 203.0.113.85 eq smtp access-list outside_ingress extended permit ip any 2001:db8:1::/48 access-list outside_ingress extended permit icmp6 any 2001:db8:2::1/64 access-list outside_ingress extended permit tcp any host 2001:db8:1:1::1 eq imap4 access-list outside_ingress extended permit tcp any host 2001:db8:1:1::2 eq 587 access-list outside_ingress extended permit tcp any host 2001:db8:1:1::3 eq smtp object network virt-mgmt nat (inside,outside) dynamic interface access-group outside_ingress in interface outside policy-map type inspect dns preset_dns_map parameters message-length maximum client auto message-length maximum 512 no tcp-inspection service-policy global_policy global policy-map global_policy class inspection_default inspect dns preset_dns_map inspect ftp inspect h323 h225 inspect h323 ras inspect rsh inspect rtsp inspect esmtp inspect sqlnet inspect skinny inspect sunrpc inspect xdmcp inspect sip inspect netbios inspect tftp inspect ip-options inspect icmp !
And my switch access list:
Extended IP access list harm_acl_ipv4 9 permit ip any 203.0.113.0 0.0.0.255 (205276184 matches) 10 permit tcp any any established 11 permit udp any 10.2.0.0 0.0.255.255 eq domain 15 permit icmp any 10.2.0.0 0.0.255.255 (102 matches) 60 permit tcp any host 203.0.113.84 eq 143 70 permit tcp any host 203.0.113.85 eq 587 80 permit tcp any host 203.0.113.85 eq smtp 90 permit ip 10.2.0.0 0.0.255.255 any (3 matches)
Solved! Go to Solution.
06-11-2020 02:11 PM
I figured out the problem. It was an ACL issue on the switch. I had the port in the wrong location on the ACL. It was:
permit udp any 10.2.0.0 0.0.255.255 eq domain
But needed to be:
permit udp any eq domain 10.2.0.0 0.0.255.255
Once I updated the ACL the hosts were resolving DNS queries.
06-09-2020 07:53 PM
06-09-2020 11:14 PM
I'm running 9.8(2). I ran the debug inspect dns errors and debug inspect dns events and nothing shows up, even after running my dig request again. When I run the debug inspect dns packets a lot of information is constantly spewed to the screen with information like the below. I can provide the full output if necessary. I will remove the dns inspect from the policy and see if that changes anything.
Flow is regular new udp len is 40 DNS request: Flags=0 (Qs=1 An=0 Au=0 Ad=0) Flow is regular new udp len is 40 DNS reply: Flags=8400 (Qs=1 An=2 Au=3 Ad=3) Flow is regular cptr=1c offset=16 cnt=0, dnsdp=0x00002aaab22251e8 cptr=1c offset=16 cnt=1, dnsdp=0x00002aaab2225204 cptr=2 offset=16 cnt=2, dnsdp=0x00002aaab2225220 cptr=2 offset=6 cnt=3, dnsdp=0x00002aaab2225232 cptr=2 offset=6 cnt=4, dnsdp=0x00002aaab2225244 cptr=1 offset=4 cnt=5, dnsdp=0x00002aaab2225254 cptr=1 offset=4 cnt=6, dnsdp=0x00002aaab2225264 cptr=1 offset=4 cnt=7, dnsdp=0x00002aaab2225274 new udp len is 208 DNS reply: Flags=8400 (Qs=1 An=4 Au=3 Ad=3) Flow is regular cptr=1 offset=4 cnt=0, dnsdp=0x00002aaab1ad75dc cptr=1 offset=4 cnt=1, dnsdp=0x00002aaab1ad75ec cptr=1 offset=4 cnt=2, dnsdp=0x00002aaab1ad75fc cptr=1 offset=4 cnt=3, dnsdp=0x00002aaab1ad760c cptr=2 offset=16 cnt=4, dnsdp=0x00002aaab1ad7628 cptr=2 offset=6 cnt=5, dnsdp=0x00002aaab1ad763a cptr=2 offset=6 cnt=6, dnsdp=0x00002aaab1ad764c cptr=1 offset=4 cnt=7, dnsdp=0x00002aaab1ad765c cptr=1 offset=4 cnt=8, dnsdp=0x00002aaab1ad766c cptr=1 offset=4 cnt=9, dnsdp=0x00002aaab1ad767c new udp len is 216 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS reply: Flags=8410 (Qs=1 An=0 Au=1 Ad=1) Flow is regular cptr=6 offset=39 cnt=0, dnsdp=0x00002aaab1ef9704 cptr=29 offset=28 cnt=1, dnsdp=0x00002aaab1ef972b new udp len is 135 DNS reply: Flags=8410 (Qs=1 An=0 Au=1 Ad=1) Flow is regular cptr=6 offset=39 cnt=0, dnsdp=0x00002aaab1e73484 cptr=29 offset=0 cnt=1, dnsdp=0x00002aaab1e7348f new udp len is 107 DNS reply: Flags=8410 (Qs=1 An=0 Au=1 Ad=1) Flow is regular cptr=6 offset=39 cnt=0, dnsdp=0x00002aaab1ad9e84 cptr=29 offset=28 cnt=1, dnsdp=0x00002aaab1ad9eab new udp len is 135 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 84 DNS reply: Flags=8010 (Qs=1 An=0 Au=15 Ad=27) Flow is regular cptr=2 offset=20 cnt=0, dnsdp=0x00002aaab27df905 cptr=2 offset=4 cnt=1, dnsdp=0x00002aaab27df915 cptr=2 offset=4 cnt=2, dnsdp=0x00002aaab27df925 cptr=2 offset=4 cnt=3, dnsdp=0x00002aaab27df935 cptr=2 offset=4 cnt=4, dnsdp=0x00002aaab27df945 cptr=2 offset=4 cnt=5, dnsdp=0x00002aaab27df955 cptr=2 offset=4 cnt=6, dnsdp=0x00002aaab27df965 cptr=2 offset=4 cnt=7, dnsdp=0x00002aaab27df975 cptr=2 offset=4 cnt=8, dnsdp=0x00002aaab27df985 cptr=2 offset=4 cnt=9, dnsdp=0x00002aaab27df995 cptr=2 offset=4 cnt=10, dnsdp=0x00002aaab27df9a5 cptr=2 offset=4 cnt=11, dnsdp=0x00002aaab27df9b5 cptr=2 offset=4 cnt=12, dnsdp=0x00002aaab27df9c5 cptr=2b offset=36 cnt=13, dnsdp=0x00002aaab27df9f5 cptr=2e offset=275 cnt=14, dnsdp=0x00002aaab27dfb14 cptr=1 offset=4 cnt=15, dnsdp=0x00002aaab27dfb24 cptr=1 offset=4 cnt=16, dnsdp=0x00002aaab27dfb34 cptr=1 offset=4 cnt=17, dnsdp=0x00002aaab27dfb44 cptr=1 offset=4 cnt=18, dnsdp=0x00002aaab27dfb54 cptr=1 offset=4 cnt=19, dnsdp=0x00002aaab27dfb64 cptr=1 offset=4 cnt=20, dnsdp=0x00002aaab27dfb74 cptr=1 offset=4 cnt=21, dnsdp=0x00002aaab27dfb84 cptr=1 offset=4 cnt=22, dnsdp=0x00002aaab27dfb94 cptr=1 offset=4 cnt=23, dnsdp=0x00002aaab27dfba4 cptr=1 offset=4 cnt=24, dnsdp=0x00002aaab27dfbb4 cptr=1 offset=4 cnt=25, dnsdp=0x00002aaab27dfbc4 cptr=1 offset=4 cnt=26, dnsdp=0x00002aaab27dfbd4 cptr=1 offset=4 cnt=27, dnsdp=0x00002aaab27dfbe4 cptr=1c offset=16 cnt=28, dnsdp=0x00002aaab27dfc00 cptr=1c offset=16 cnt=29, dnsdp=0x00002aaab27dfc1c cptr=1c offset=16 cnt=30, dnsdp=0x00002aaab27dfc38 cptr=1c offset=16 cnt=31, dnsdp=0x00002aaab27dfc54 cptr=1c offset=16 cnt=32, dnsdp=0x00002aaab27dfc70 cptr=1c offset=16 cnt=33, dnsdp=0x00002aaab27dfc8c cptr=1c offset=16 cnt=34, dnsdp=0x00002aaab27dfca8 cptr=1c offset=16 cnt=35, dnsdp=0x00002aaab27dfcc4 cptr=1c offset=16 cnt=36, dnsdp=0x00002aaab27dfce0 cptr=1c offset=16 cnt=37, dnsdp=0x00002aaab27dfcfc cptr=1c offset=16 cnt=38, dnsdp=0x00002aaab27dfd18 cptr=1c offset=16 cnt=39, dnsdp=0x00002aaab27dfd34 cptr=1c offset=16 cnt=40, dnsdp=0x00002aaab27dfd50 cptr=29 offset=28 cnt=41, dnsdp=0x00002aaab27dfd77 new udp len is 1215 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS reply: Flags=8010 (Qs=1 An=0 Au=8 Ad=5) Flow is regular cptr=2 offset=2 cnt=0, dnsdp=0x00002aaab26e6a73 cptr=2 offset=6 cnt=1, dnsdp=0x00002aaab26e6a85 cptr=2 offset=6 cnt=2, dnsdp=0x00002aaab26e6a97 cptr=2 offset=6 cnt=3, dnsdp=0x00002aaab26e6aa9 cptr=32 offset=35 cnt=4, dnsdp=0x00002aaab26e6af9 cptr=2e offset=183 cnt=5, dnsdp=0x00002aaab26e6bbc cptr=32 offset=34 cnt=6, dnsdp=0x00002aaab26e6c0b cptr=2e offset=183 cnt=7, dnsdp=0x00002aaab26e6cce cptr=1 offset=4 cnt=8, dnsdp=0x00002aaab26e6cde cptr=1 offset=4 cnt=9, dnsdp=0x00002aaab26e6cee cptr=1 offset=4 cnt=10, dnsdp=0x00002aaab26e6cfe cptr=1 offset=4 cnt=11, dnsdp=0x00002aaab26e6d0e cptr=29 offset=0 cnt=12, dnsdp=0x00002aaab26e6d19 new udp len is 737 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS request: Flags=10 (Qs=1 An=0 Au=0 Ad=1) Flow is regular new udp len is 68 DNS request: Flags=0 (Qs=1 An=0 Au=0 Ad=0) Flow is regular new udp len is 40 DNS request: Flags=0 (Qs=1 An=0 Au=0 Ad=0) Flow is regular new udp len is 40 DNS reply: Flags=8410 (Qs=1 An=0 Au=1 Ad=1) Flow is regular cptr=6 offset=35 cnt=0, dnsdp=0x00002aaaabe1e280 cptr=29 offset=28 cnt=1, dnsdp=0x00002aaaabe1e2a7 new udp len is 131 DNS reply: Flags=8410 (Qs=1 An=0 Au=1 Ad=1) Flow is regular cptr=6 offset=39 cnt=0, dnsdp=0x00002aaab223e704 cptr=29 offset=0 cnt=1, dnsdp=0x00002aaab223e70f new udp len is 107 DNS reply: Flags=8410 (Qs=1 An=0 Au=1 Ad=1) Flow is regular cptr=6 offset=39 cnt=0, dnsdp=0x00002aaab24c1604 cptr=29 offset=0 cnt=1, dnsdp=0x00002aaab24c160f new udp len is 107 DNS reply: Flags=8400 (Qs=1 An=2 Au=3 Ad=3) Flow is regular cptr=1c offset=16 cnt=0, dnsdp=0x00002aaaac50e3e8 cptr=1c offset=16 cnt=1, dnsdp=0x00002aaaac50e404 cptr=2 offset=16 cnt=2, dnsdp=0x00002aaaac50e420 cptr=2 offset=6 cnt=3, dnsdp=0x00002aaaac50e432 cptr=2 offset=6 cnt=4, dnsdp=0x00002aaaac50e444 cptr=1 offset=4 cnt=5, dnsdp=0x00002aaaac50e454 cptr=1 offset=4 cnt=6, dnsdp=0x00002aaaac50e464 cptr=1 offset=4 cnt=7, dnsdp=0x00002aaaac50e474 new udp len is 208 DNS reply: Flags=8400 (Qs=1 An=4 Au=3 Ad=3) Flow is regular cptr=1 offset=4 cnt=0, dnsdp=0x00002aaab1baa4dc cptr=1 offset=4 cnt=1, dnsdp=0x00002aaab1baa4ec cptr=1 offset=4 cnt=2, dnsdp=0x00002aaab1baa4fc cptr=1 offset=4 cnt=3, dnsdp=0x00002aaab1baa50c cptr=2 offset=16 cnt=4, dnsdp=0x00002aaab1baa528 cptr=2 offset=6 cnt=5, dnsdp=0x00002aaab1baa53a cptr=2 offset=6 cnt=6, dnsdp=0x00002aaab1baa54c cptr=1 offset=4 cnt=7, dnsdp=0x00002aaab1baa55c cptr=1 offset=4 cnt=8, dnsdp=0x00002aaab1baa56c cptr=1 offset=4 cnt=9, dnsdp=0x00002aaab1baa57c new udp len is 216
06-11-2020 01:06 PM
I removed the dns inspect from the global_policy policy-map but I'm still unable to resolve DNS queries:
class-map inspection_default match default-inspection-traffic ! ! policy-map type inspect dns preset_dns_map parameters message-length maximum client auto message-length maximum 512 no tcp-inspection policy-map global_policy class inspection_default inspect ftp inspect h323 h225 inspect h323 ras inspect rsh inspect rtsp inspect esmtp inspect sqlnet inspect skinny inspect sunrpc inspect xdmcp inspect sip inspect netbios inspect tftp inspect ip-options inspect icmp ! service-policy global_policy global
06-11-2020 02:11 PM
I figured out the problem. It was an ACL issue on the switch. I had the port in the wrong location on the ACL. It was:
permit udp any 10.2.0.0 0.0.255.255 eq domain
But needed to be:
permit udp any eq domain 10.2.0.0 0.0.255.255
Once I updated the ACL the hosts were resolving DNS queries.
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide