Unable to resolve DNS between VLANs across an ASA5510

[RednaxelaJR]

Hello experts, first time poster here. I have quite a perplexing problem that I have been banging my head against the wall about for awhile now. Short explanation: I have a corporate network running on data VLAN 100 in the IP range 10.0.1.0/24, with the gateway 10.0.1.1 being the ASA5510 and a layer 3 switch behind it doing the routing with an IP of 10.0.1.218 (don't ask about the random IP, that's not important right now).

Recently, I have configured another layer 3 switch with the VLAN 1000 and IP range 10.0.100.128/25. The second switch (we'll refer to it as noncorp) is assigned the IP 10.0.100.130/25 with the gateway 10.0.100.129 configured on the corporate core switch. Connectivity between the 10.0.1.0 and 10.0.100.128 networks is working just fine, hosts on one network can ping, ssh into, and connect with hosts on the other and vice versa. There's quite a bit of NAT exempting and static routing going on at the ASA to accomplish that, but it all works.

My problem is the corporate DNS server is at IP 10.0.1.252. Hosts on the 10.0.100.128 network are unable to resolve DNS using this DNS server and I cannot for the life of me figure out why. Hosts on 10.0.100.128/25 can reach any external or internal address on either network via IP, but name resolution is not working at ALL. In the DNS server itself, I see none of these hosts being picked up, though I can ping from a host to the DNS server and vice versa. I've been messing with same-securirty-traffic, static NAT, ACLs...nothing seems to work. At first I thought it was a routing issue, but again ICMP and traces show that getting to the server is just fine, it's getting DNS from the server that is not working, and only for hosts in the 10.0.100.128/25 network. I think it may be a DNS server configuration issue, but it could still be a routing issue either in the ASA or the corp & noncorp switches as well.

Oh yes, the noncorp switch is connected to the corp switch via a gigabit ethernet port configured as an access port on both switches, in the VLAN 1000 which exists on both switches. The corp switch is still doing all inter-VLAN routing for traffic coming from or to the noncorp network. Any advice or suggestions at all would be appreciated! Please find a truncated view of my ASA below, and let me know if switch configs are needed for this as well:

# sho run
: Saved
:
ASA Version 8.2(5)
!
hostname
domain-name .local
enable
passwd
no names
!
interface Ethernet0/0
 nameif outside
 security-level 0
 ip address 192.168.1.2 255.255.255.0
 ospf cost 10
!
interface Ethernet0/1
 shutdown
 no nameif
 no security-level
 no ip address
!
interface Ethernet0/2
 nameif inside
 security-level 100
 ip address 10.0.1.1 255.255.255.0
 ospf cost 10
!
interface Ethernet0/3
 nameif MPLSVPN
 security-level 100
 ip address 192.168.254.2 255.255.255.252
!
interface Management0/0
 shutdown
 nameif management
 security-level 100
 no ip address
 ospf cost 10
 management-only
!
boot system disk0:/asa825-k8.bin
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns domain-lookup outside
dns domain-lookup inside
dns server-group DefaultDNS
 name-server 10.0.1.252
 domain-name AngelOak.local
same-security-traffic permit intra-interface
object-group network vpn-172.16.0.1
 network-object 172.16.0.0 255.255.255.0
object-group protocol TCPUDP
 protocol-object udp
 protocol-object tcp
object-group network NETWORK_OBJ_10.0.1.0_24
object-group protocol DM_INLINE_PROTOCOL_1
 protocol-object udp
 protocol-object tcp
object-group protocol DM_INLINE_PROTOCOL_2
 protocol-object udp
 protocol-object tcp
object-group protocol DM_INLINE_PROTOCOL_3
 protocol-object ip
access-list outside_access_in extended permit ip any any
access-list outside_access_in remark Allow Windows Tracert
access-list outside_access_in remark Allow Windows Tracert
access-list outside_access_in extended permit icmp any any time-exceeded inactive
access-list outside_access_in extended permit ip any 10.0.1.0 255.255.255.0
access-list outside_access_in extended permit ip 10.0.1.0 255.255.255.0 10.0.100.128 255.255.255.128
access-list outside_access_in extended permit ip 10.0.1.0 255.255.255.0 any
access-list outside_access_in extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list inside_access_in extended permit ip any any
access-list inside_access_in extended permit ip 10.0.100.128 255.255.255.128 any
access-list inside_access_in extended permit ip 10.0.1.0 255.255.255.0 any
access-list outside_nat0_outbound extended permit ip 10.0.1.0 255.255.255.0 any
access-list outside_nat0_outbound extended permit ip 10.0.1.0 255.255.255.0 10.0.100.128 255.255.255.128
access-list outside_nat0_outbound extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list angelaok_splitTunnelAcl standard permit 10.0.1.0 255.255.255.0
access-list angelaok_splitTunnelAcl standard permit 10.0.100.128 255.255.255.128
access-list inside_nat0_outbound extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list ACL extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list ACL extended permit object-group DM_INLINE_PROTOCOL_3 10.0.1.0 255.255.255.0 10.0.100.128 255.255.255.128
access-list acci extended permit tcp 10.0.1.0 255.255.255.0 any eq ssh
access-list acci extended permit tcp 10.0.100.128 255.255.255.128 any eq ssh
access-list exempt extended permit ip any 10.0.1.0 255.255.255.0
access-list exempt extended permit ip any 10.0.100.128 255.255.255.128
access-list tcp_bypass extended permit tcp 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
pager lines 24
logging enable
logging buffer-size 200000
logging monitor debugging
logging buffered debugging
logging history emergencies
logging asdm debugging
logging class ip history emergencies
mtu outside 1500
mtu inside 1500
mtu MPLSVPN 1500
mtu management 1500
icmp unreachable rate-limit 10 burst-size 5
icmp permit any inside
asdm image disk0:/asdm-714.bin
no asdm history enable
arp timeout 14400
nat-control
global (outside) 1 interface
global (inside) 1 interface
nat (outside) 0 access-list exempt
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 0.0.0.0 0.0.0.0
nat (MPLSVPN) 0 access-list inside_nat0_outbound
static (inside,outside) tcp interface 3786 10.0.1.247 ssh netmask 255.255.255.255
static (inside,inside) 10.0.2.0 10.0.2.0 netmask 255.255.255.0
static (inside,inside) 10.0.1.0 10.0.1.0 netmask 255.255.255.0
static (inside,outside) 192.168.1.1 10.0.1.1 netmask 255.255.255.255
static (inside,inside) 10.0.100.128 10.0.100.128 netmask 255.255.255.128
access-group outside_access_in in interface outside
access-group inside_access_in_1 in interface inside control-plane
access-group inside_access_in in interface inside
!
router rip
!
route outside 0.0.0.0 0.0.0.0 192.168.1.1 1
route inside 10.0.100.128 255.255.255.128 10.0.1.218 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
 webvpn
  svc ask none default svc
aaa-server radius-group protocol radius
aaa-server radius-group (inside) host 10.0.1.252
 key *****
 radius-common-pw *****
aaa authentication ssh console LOCAL
aaa authentication enable console LOCAL
aaa authentication http console LOCAL
aaa authorization command LOCAL
http server enable
http 10.0.1.0 255.255.255.0 inside
http 192.168.1.0 255.255.255.0 management
http 172.16.1.0 255.255.255.255 outside
http 172.16.0.0 255.255.255.0 outside
http redirect inside 80
no vpn-addr-assign aaa
no vpn-addr-assign dhcp
vpn-addr-assign local reuse-delay 10
telnet timeout 5
ssh 172.16.1.0 255.255.255.0 outside
ssh 172.16.0.0 255.255.255.0 outside
ssh 10.0.1.0 255.255.255.0 inside
ssh timeout 30
console timeout 0
management-access inside
threat-detection basic-threat
threat-detection statistics port
threat-detection statistics protocol
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
 enable outside
 svc image disk0:/anyconnect-dart-win-2.5.6005-k9.pkg 1
 svc profiles remote disk0:/remote_server.xml
 svc profiles remote_Server disk0:/remote_server.xml
 svc enable
!
class-map inspection_default
 match default-inspection-traffic
class-map tcp_bypass
 match access-list ACL
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map
  inspect ftp
  inspect h323 h225
  inspect h323 ras
  inspect rsh
  inspect rtsp
  inspect sqlnet
  inspect skinny
  inspect sunrpc
  inspect xdmcp
  inspect netbios
  inspect tftp
  inspect ip-options
 class class-default
  set connection decrement-ttl
policy-map tcp_bypass_policy
 class tcp_bypass
  set connection advanced-options tcp-state-bypass
!
service-policy global_policy global
service-policy tcp_bypass_policy interface inside
prompt hostname context
no call-home reporting anonymous
call-home
 profile CiscoTAC-1
  no active
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:35fab68b6ee9dbd46cbe4f308ed9eda1
: end

[Adam White]

Where does the 10.0.100. network default route to? 

If this is the corporate switch then the following is happening:

Traffic comes from your hosts, doesnt hit the firewall, because the corp switch routes directly to the DNS server. 

The DNS server replies by sending traffic to its default gateway, which is the firewall, it sees a problem with the state connection 

What doesnt make sense is that other traffic works. The firewall should drop everything because it doesnt see the initial connection / syn etc

What do the firewall logs say for the DNS traffic?

Can you provide the default gateway for the hosts on 10.0.100 network

Maybe the switch configs as well

[Adam White]

Why dont you also just create a subinterface on the ASA on the new vlan, dot1q that from the corporate switch and default to the ASA on the 10.0.100.128/25 network as well?

Then you have no same interface problems and its much cleaner as well?

[RednaxelaJR]

Adam,

Thanks for replying. The responses to your inquiries are as follows: all of the hosts on the 10.0.100 network connect to the noncorp switch, which has a default route sending all of that traffic to the corp switch, which acts as the gateway for that network at 10.0.100.129. Here's where it gets funky. The corp switch has a default route sending all traffic to the ASA at 10.0.1.1. The ASA has a default route stating that all traffic destined for the 10.0.100.128/25 network is sent to the corp switch at 10.0.1.218(the same network the raffic was sent to the ASA in). Once that traffic is sent back to the corp switch on the 10.0.1 network, it sees the directly connected network of 10.0.100.128/25 and routes it back to the noncorp switch and to the host connected to it at the IP.

Essentially, though asymmetrical routing was a concern at first, all traffic on both 10.0.1 and 10.0.100 sides hit the ASA and shouldn't be dropped there...this is proven whtn ICMP and TCP requests from one network are successful to the other. Especially perplexing is that DNS is generally a UDP process, so asymmetrical routing shouldn't affect it anyway.

Firewall logs don't show much except lines like the below:

%ASA-2-106007: Deny inbound UDP from 10.0.100.252/60908 to 10.0.20.252/53 due to DNS Query
%ASA-6-302015: Built inbound UDP connection 16681579 for inside:10.0.1.251/53 (10.0.1.251/53) to inside:10.0.100.252/60908 (10.0.100.252/60908)
%ASA-6-302015: Built inbound UDP connection 16681580 for inside:10.0.1.252/53 (10.0.1.252/53) to inside:10.0.100.252/60908 (10.0.100.252/60908)

The default gateway for all hosts on the 10.0.100 network is the corp switch, which is directly connected to the ASA who is the default gateway for the 10.0.1. network. Below you will find the switch configs for the corp and noncorp switch. Again, all hosts on the 10.0.1 network have no issues, and all hosts on the 10.0.100 network have no issues EXCEPT being unable to resolve DNS to the DNS server at 10.0.1.252:

CORP:

CORP#
sho run
Building configuration...

Current configuration : 11050 bytes
!
! Last configuration change at 13:54:26 UTC Mon Apr 20 2015 by admin
! NVRAM config last updated at 13:54:48 UTC Mon Apr 20 2015 by admin
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname CORP
!
boot-start-marker
boot-end-marker
!
!
enable secret 5
!
username admin privilege 15
no aaa new-model
switch 1 provision ws-c3750x-24
switch 2 provision ws-c3750x-24
system mtu routing 1500
ip routing
!
!
!
!
!

spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
!
!
!
!
!
interface GigabitEthernet1/0/22
 switchport access vlan 200
 switchport mode access
 switchport voice vlan 200
 spanning-tree portfast
!
interface GigabitEthernet1/0/23
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 100
 switchport mode trunk
!
interface GigabitEthernet1/0/24
 switchport access vlan 1000
 switchport mode access
!
interface GigabitEthernet2/0/22
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 100,120-122,192
 switchport mode trunk
!
interface GigabitEthernet2/0/23
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 100
 switchport mode trunk
!
interface GigabitEthernet2/0/24
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 100
 switchport mode trunk
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan100
 description DATA VLAN
 ip address 10.0.1.218 255.255.255.0
 ip helper-address 10.0.1.252
!
interface Vlan120
 ip address 10.20.0.1 255.255.255.0
!
interface Vlan121
 ip address 10.20.1.1 255.255.255.0
!
interface Vlan192
 ip address 192.168.2.1 255.255.255.0
!
interface Vlan200
 description VOICE VLAN
 ip address 10.0.2.1 255.255.255.0
 ip helper-address 10.0.1.252
!
interface Vlan1000
 ip address 10.0.100.129 255.255.255.128
 ip helper-address 10.0.1.252
!
ip default-gateway 10.0.1.1
!
ip http server
ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 10.0.1.1
!
logging esm config
!
snmp-server group AOCUser v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF7F
snmp-server group AOCGroup v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.F
snmp-server group AOCGroup v3 priv read AOCView write AOCView
snmp-server view AOCView internet included
snmp-server location AOC HQ
snmp-server contact it@angeloakcapital.com
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps stpx root-inconsistency loop-inconsistency
snmp-server host 10.0.1.244 version 3 auth AOCUser
!
!
line con 0
 logging synchronous
line vty 0 4
 password 7 1511021F0725
 login local
 length 0
 transport input ssh
line vty 5 15
 password 7 045802150C2E
 login local
 transport input ssh
!
ntp server 64.90.182.55
ntp server 98.175.203.200 prefer
end

NONCORP:

NONCORP#sho run
Building configuration...

Current configuration : 6617 bytes
!
! Last configuration change at 09:32:15 edt Mon Apr 20 2015 by admin
! NVRAM config last updated at 09:33:35 edt Mon Apr 20 2015 by admin
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname NONCORP
!
!
username admin privilege 15
no aaa new-model
clock timezone est -5
clock summer-time edt recurring
ip subnet-zero
ip routing
!
!
!
!
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
interface GigabitEthernet0/45
 switchport access vlan 1000
 switchport mode access
!
interface GigabitEthernet0/46
 switchport access vlan 1000
 switchport mode access
!
interface GigabitEthernet0/47
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1000
 switchport mode trunk
!
interface GigabitEthernet0/48
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1000
 switchport mode trunk
!
interface GigabitEthernet0/49
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1000
 switchport mode trunk
!
interface GigabitEthernet0/50
 description Uplink to Access04
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1000
 switchport mode trunk
!
interface GigabitEthernet0/51
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1000
 switchport mode trunk
!
interface GigabitEthernet0/52
 switchport trunk encapsulation dot1q
 switchport trunk native vlan 1000
 switchport mode trunk
!
interface Vlan1
 no ip address
 shutdown
!
interface Vlan1000
 ip address 10.0.100.130 255.255.255.128
!
ip default-gateway 10.0.100.129
ip classless
ip route 0.0.0.0 0.0.0.0 10.0.100.129
ip http server
ip http secure-server
!
!
snmp-server group AOCGroup v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.F
snmp-server group AOCGroup v3 priv read AOCView write AOCView
snmp-server view AOCView internet included
snmp-server location AOC HQ
snmp-server contact it@angeloakcapital.com
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps stpx root-inconsistency loop-inconsistency
snmp-server host 10.0.1.244 version 3 auth AOCUser
!
control-plane
!
!
line con 0
line vty 0 4
 login local
line vty 5 15
 no login
!
ntp clock-period 36029007
ntp server 64.90.182.55
ntp server 98.175.203.200 prefer
!
end

[Adam White · Accepted solution]

What i dont understand, is once traffic gets to the corp switch, why does it send it to the firewall first? 

It has an interface on Vlan100 and should just do inter-vlan routing? 

I.E traffic comes from non corp, routes to corp, Corp switch locally switches the packet to the DNS server. 

I would still strongly advise that you use the sub-interface to do this properly instead of hair-pinning traffic.

Alternatively, if you dont need the traffic to go through the ASA just route it locally.

Lastly, I dont really understand why its going to the ASA in the first place, unless I misunderstand the topology.Can you provide a traceroute?

The last thing i can suggest to try, which is a complete shot in the dark which i actually hate doing, is remove the DNS from the inspection on the ASA. 

[RednaxelaJR]

That second suggestion regarding the subinterface is actually a good idea, if I had thought of that first I probably would have done it...as it stands, because everything except DNS is working I don't want to redo the entire way it is routing but if I can't get DNS to come around I may end up doing that. We are in a production environment but this 10.0.100 network is only being used as a storage network so I can mess with it a bit if necessary