04-20-2015 02:56 PM - edited 03-07-2019 11:38 PM
Hello experts, first time poster here. I have quite a perplexing problem that I have been banging my head against the wall about for awhile now. Short explanation: I have a corporate network running on data VLAN 100 in the IP range 10.0.1.0/24, with the gateway 10.0.1.1 being the ASA5510 and a layer 3 switch behind it doing the routing with an IP of 10.0.1.218 (don't ask about the random IP, that's not important right now).
Recently, I have configured another layer 3 switch with the VLAN 1000 and IP range 10.0.100.128/25. The second switch (we'll refer to it as noncorp) is assigned the IP 10.0.100.130/25 with the gateway 10.0.100.129 configured on the corporate core switch. Connectivity between the 10.0.1.0 and 10.0.100.128 networks is working just fine, hosts on one network can ping, ssh into, and connect with hosts on the other and vice versa. There's quite a bit of NAT exempting and static routing going on at the ASA to accomplish that, but it all works.
My problem is the corporate DNS server is at IP 10.0.1.252. Hosts on the 10.0.100.128 network are unable to resolve DNS using this DNS server and I cannot for the life of me figure out why. Hosts on 10.0.100.128/25 can reach any external or internal address on either network via IP, but name resolution is not working at ALL. In the DNS server itself, I see none of these hosts being picked up, though I can ping from a host to the DNS server and vice versa. I've been messing with same-securirty-traffic, static NAT, ACLs...nothing seems to work. At first I thought it was a routing issue, but again ICMP and traces show that getting to the server is just fine, it's getting DNS from the server that is not working, and only for hosts in the 10.0.100.128/25 network. I think it may be a DNS server configuration issue, but it could still be a routing issue either in the ASA or the corp & noncorp switches as well.
Oh yes, the noncorp switch is connected to the corp switch via a gigabit ethernet port configured as an access port on both switches, in the VLAN 1000 which exists on both switches. The corp switch is still doing all inter-VLAN routing for traffic coming from or to the noncorp network. Any advice or suggestions at all would be appreciated! Please find a truncated view of my ASA below, and let me know if switch configs are needed for this as well:
# sho run
: Saved
:
ASA Version 8.2(5)
!
hostname
domain-name .local
enable
passwd
no names
!
interface Ethernet0/0
nameif outside
security-level 0
ip address 192.168.1.2 255.255.255.0
ospf cost 10
!
interface Ethernet0/1
shutdown
no nameif
no security-level
no ip address
!
interface Ethernet0/2
nameif inside
security-level 100
ip address 10.0.1.1 255.255.255.0
ospf cost 10
!
interface Ethernet0/3
nameif MPLSVPN
security-level 100
ip address 192.168.254.2 255.255.255.252
!
interface Management0/0
shutdown
nameif management
security-level 100
no ip address
ospf cost 10
management-only
!
boot system disk0:/asa825-k8.bin
ftp mode passive
clock timezone EST -5
clock summer-time EDT recurring
dns domain-lookup outside
dns domain-lookup inside
dns server-group DefaultDNS
name-server 10.0.1.252
domain-name AngelOak.local
same-security-traffic permit intra-interface
object-group network vpn-172.16.0.1
network-object 172.16.0.0 255.255.255.0
object-group protocol TCPUDP
protocol-object udp
protocol-object tcp
object-group network NETWORK_OBJ_10.0.1.0_24
object-group protocol DM_INLINE_PROTOCOL_1
protocol-object udp
protocol-object tcp
object-group protocol DM_INLINE_PROTOCOL_2
protocol-object udp
protocol-object tcp
object-group protocol DM_INLINE_PROTOCOL_3
protocol-object ip
access-list outside_access_in extended permit ip any any
access-list outside_access_in remark Allow Windows Tracert
access-list outside_access_in remark Allow Windows Tracert
access-list outside_access_in extended permit icmp any any time-exceeded inactive
access-list outside_access_in extended permit ip any 10.0.1.0 255.255.255.0
access-list outside_access_in extended permit ip 10.0.1.0 255.255.255.0 10.0.100.128 255.255.255.128
access-list outside_access_in extended permit ip 10.0.1.0 255.255.255.0 any
access-list outside_access_in extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list inside_access_in extended permit ip any any
access-list inside_access_in extended permit ip 10.0.100.128 255.255.255.128 any
access-list inside_access_in extended permit ip 10.0.1.0 255.255.255.0 any
access-list outside_nat0_outbound extended permit ip 10.0.1.0 255.255.255.0 any
access-list outside_nat0_outbound extended permit ip 10.0.1.0 255.255.255.0 10.0.100.128 255.255.255.128
access-list outside_nat0_outbound extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list angelaok_splitTunnelAcl standard permit 10.0.1.0 255.255.255.0
access-list angelaok_splitTunnelAcl standard permit 10.0.100.128 255.255.255.128
access-list inside_nat0_outbound extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list ACL extended permit ip 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
access-list ACL extended permit object-group DM_INLINE_PROTOCOL_3 10.0.1.0 255.255.255.0 10.0.100.128 255.255.255.128
access-list acci extended permit tcp 10.0.1.0 255.255.255.0 any eq ssh
access-list acci extended permit tcp 10.0.100.128 255.255.255.128 any eq ssh
access-list exempt extended permit ip any 10.0.1.0 255.255.255.0
access-list exempt extended permit ip any 10.0.100.128 255.255.255.128
access-list tcp_bypass extended permit tcp 10.0.100.128 255.255.255.128 10.0.1.0 255.255.255.0
pager lines 24
logging enable
logging buffer-size 200000
logging monitor debugging
logging buffered debugging
logging history emergencies
logging asdm debugging
logging class ip history emergencies
mtu outside 1500
mtu inside 1500
mtu MPLSVPN 1500
mtu management 1500
icmp unreachable rate-limit 10 burst-size 5
icmp permit any inside
asdm image disk0:/asdm-714.bin
no asdm history enable
arp timeout 14400
nat-control
global (outside) 1 interface
global (inside) 1 interface
nat (outside) 0 access-list exempt
nat (inside) 0 access-list inside_nat0_outbound
nat (inside) 1 0.0.0.0 0.0.0.0
nat (MPLSVPN) 0 access-list inside_nat0_outbound
static (inside,outside) tcp interface 3786 10.0.1.247 ssh netmask 255.255.255.255
static (inside,inside) 10.0.2.0 10.0.2.0 netmask 255.255.255.0
static (inside,inside) 10.0.1.0 10.0.1.0 netmask 255.255.255.0
static (inside,outside) 192.168.1.1 10.0.1.1 netmask 255.255.255.255
static (inside,inside) 10.0.100.128 10.0.100.128 netmask 255.255.255.128
access-group outside_access_in in interface outside
access-group inside_access_in_1 in interface inside control-plane
access-group inside_access_in in interface inside
!
router rip
!
route outside 0.0.0.0 0.0.0.0 192.168.1.1 1
route inside 10.0.100.128 255.255.255.128 10.0.1.218 1
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
timeout floating-conn 0:00:00
dynamic-access-policy-record DfltAccessPolicy
webvpn
svc ask none default svc
aaa-server radius-group protocol radius
aaa-server radius-group (inside) host 10.0.1.252
key *****
radius-common-pw *****
aaa authentication ssh console LOCAL
aaa authentication enable console LOCAL
aaa authentication http console LOCAL
aaa authorization command LOCAL
http server enable
http 10.0.1.0 255.255.255.0 inside
http 192.168.1.0 255.255.255.0 management
http 172.16.1.0 255.255.255.255 outside
http 172.16.0.0 255.255.255.0 outside
http redirect inside 80
no vpn-addr-assign aaa
no vpn-addr-assign dhcp
vpn-addr-assign local reuse-delay 10
telnet timeout 5
ssh 172.16.1.0 255.255.255.0 outside
ssh 172.16.0.0 255.255.255.0 outside
ssh 10.0.1.0 255.255.255.0 inside
ssh timeout 30
console timeout 0
management-access inside
threat-detection basic-threat
threat-detection statistics port
threat-detection statistics protocol
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
webvpn
enable outside
svc image disk0:/anyconnect-dart-win-2.5.6005-k9.pkg 1
svc profiles remote disk0:/remote_server.xml
svc profiles remote_Server disk0:/remote_server.xml
svc enable
!
class-map inspection_default
match default-inspection-traffic
class-map tcp_bypass
match access-list ACL
!
!
policy-map type inspect dns preset_dns_map
parameters
message-length maximum 512
policy-map global_policy
class inspection_default
inspect dns preset_dns_map
inspect ftp
inspect h323 h225
inspect h323 ras
inspect rsh
inspect rtsp
inspect sqlnet
inspect skinny
inspect sunrpc
inspect xdmcp
inspect netbios
inspect tftp
inspect ip-options
class class-default
set connection decrement-ttl
policy-map tcp_bypass_policy
class tcp_bypass
set connection advanced-options tcp-state-bypass
!
service-policy global_policy global
service-policy tcp_bypass_policy interface inside
prompt hostname context
no call-home reporting anonymous
call-home
profile CiscoTAC-1
no active
destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
destination address email callhome@cisco.com
destination transport-method http
subscribe-to-alert-group diagnostic
subscribe-to-alert-group environment
subscribe-to-alert-group inventory periodic monthly
subscribe-to-alert-group configuration periodic monthly
subscribe-to-alert-group telemetry periodic daily
Cryptochecksum:35fab68b6ee9dbd46cbe4f308ed9eda1
: end
Solved! Go to Solution.
04-21-2015 01:28 PM
What i dont understand, is once traffic gets to the corp switch, why does it send it to the firewall first?
It has an interface on Vlan100 and should just do inter-vlan routing?
I.E traffic comes from non corp, routes to corp, Corp switch locally switches the packet to the DNS server.
I would still strongly advise that you use the sub-interface to do this properly instead of hair-pinning traffic.
Alternatively, if you dont need the traffic to go through the ASA just route it locally.
Lastly, I dont really understand why its going to the ASA in the first place, unless I misunderstand the topology.Can you provide a traceroute?
The last thing i can suggest to try, which is a complete shot in the dark which i actually hate doing, is remove the DNS from the inspection on the ASA.
04-20-2015 04:21 PM
Where does the 10.0.100. network default route to?
If this is the corporate switch then the following is happening:
Traffic comes from your hosts, doesnt hit the firewall, because the corp switch routes directly to the DNS server.
The DNS server replies by sending traffic to its default gateway, which is the firewall, it sees a problem with the state connection
What doesnt make sense is that other traffic works. The firewall should drop everything because it doesnt see the initial connection / syn etc
What do the firewall logs say for the DNS traffic?
Can you provide the default gateway for the hosts on 10.0.100 network
Maybe the switch configs as well
04-20-2015 04:25 PM
Why dont you also just create a subinterface on the ASA on the new vlan, dot1q that from the corporate switch and default to the ASA on the 10.0.100.128/25 network as well?
Then you have no same interface problems and its much cleaner as well?
04-21-2015 06:25 AM
Adam,
Thanks for replying. The responses to your inquiries are as follows: all of the hosts on the 10.0.100 network connect to the noncorp switch, which has a default route sending all of that traffic to the corp switch, which acts as the gateway for that network at 10.0.100.129. Here's where it gets funky. The corp switch has a default route sending all traffic to the ASA at 10.0.1.1. The ASA has a default route stating that all traffic destined for the 10.0.100.128/25 network is sent to the corp switch at 10.0.1.218(the same network the raffic was sent to the ASA in). Once that traffic is sent back to the corp switch on the 10.0.1 network, it sees the directly connected network of 10.0.100.128/25 and routes it back to the noncorp switch and to the host connected to it at the IP.
Essentially, though asymmetrical routing was a concern at first, all traffic on both 10.0.1 and 10.0.100 sides hit the ASA and shouldn't be dropped there...this is proven whtn ICMP and TCP requests from one network are successful to the other. Especially perplexing is that DNS is generally a UDP process, so asymmetrical routing shouldn't affect it anyway.
Firewall logs don't show much except lines like the below:
%ASA-2-106007: Deny inbound UDP from 10.0.100.252/60908 to 10.0.20.252/53 due to DNS Query
%ASA-6-302015: Built inbound UDP connection 16681579 for inside:10.0.1.251/53 (10.0.1.251/53) to inside:10.0.100.252/60908 (10.0.100.252/60908)
%ASA-6-302015: Built inbound UDP connection 16681580 for inside:10.0.1.252/53 (10.0.1.252/53) to inside:10.0.100.252/60908 (10.0.100.252/60908)
The default gateway for all hosts on the 10.0.100 network is the corp switch, which is directly connected to the ASA who is the default gateway for the 10.0.1. network. Below you will find the switch configs for the corp and noncorp switch. Again, all hosts on the 10.0.1 network have no issues, and all hosts on the 10.0.100 network have no issues EXCEPT being unable to resolve DNS to the DNS server at 10.0.1.252:
CORP:
CORP#
sho run
Building configuration...
Current configuration : 11050 bytes
!
! Last configuration change at 13:54:26 UTC Mon Apr 20 2015 by admin
! NVRAM config last updated at 13:54:48 UTC Mon Apr 20 2015 by admin
!
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname CORP
!
boot-start-marker
boot-end-marker
!
!
enable secret 5
!
username admin privilege 15
no aaa new-model
switch 1 provision ws-c3750x-24
switch 2 provision ws-c3750x-24
system mtu routing 1500
ip routing
!
!
!
!
!
spanning-tree mode pvst
spanning-tree extend system-id
!
!
!
!
vlan internal allocation policy ascending
!
!
!
!
!
!
!
!
interface GigabitEthernet1/0/22
switchport access vlan 200
switchport mode access
switchport voice vlan 200
spanning-tree portfast
!
interface GigabitEthernet1/0/23
switchport trunk encapsulation dot1q
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet1/0/24
switchport access vlan 1000
switchport mode access
!
interface GigabitEthernet2/0/22
switchport trunk encapsulation dot1q
switchport trunk allowed vlan 100,120-122,192
switchport mode trunk
!
interface GigabitEthernet2/0/23
switchport trunk encapsulation dot1q
switchport trunk native vlan 100
switchport mode trunk
!
interface GigabitEthernet2/0/24
switchport trunk encapsulation dot1q
switchport trunk native vlan 100
switchport mode trunk
!
interface Vlan1
no ip address
shutdown
!
interface Vlan100
description DATA VLAN
ip address 10.0.1.218 255.255.255.0
ip helper-address 10.0.1.252
!
interface Vlan120
ip address 10.20.0.1 255.255.255.0
!
interface Vlan121
ip address 10.20.1.1 255.255.255.0
!
interface Vlan192
ip address 192.168.2.1 255.255.255.0
!
interface Vlan200
description VOICE VLAN
ip address 10.0.2.1 255.255.255.0
ip helper-address 10.0.1.252
!
interface Vlan1000
ip address 10.0.100.129 255.255.255.128
ip helper-address 10.0.1.252
!
ip default-gateway 10.0.1.1
!
ip http server
ip http secure-server
!
ip route 0.0.0.0 0.0.0.0 10.0.1.1
!
logging esm config
!
snmp-server group AOCUser v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF.FFFFFFFF7F
snmp-server group AOCGroup v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.F
snmp-server group AOCGroup v3 priv read AOCView write AOCView
snmp-server view AOCView internet included
snmp-server location AOC HQ
snmp-server contact it@angeloakcapital.com
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps stpx root-inconsistency loop-inconsistency
snmp-server host 10.0.1.244 version 3 auth AOCUser
!
!
line con 0
logging synchronous
line vty 0 4
password 7 1511021F0725
login local
length 0
transport input ssh
line vty 5 15
password 7 045802150C2E
login local
transport input ssh
!
ntp server 64.90.182.55
ntp server 98.175.203.200 prefer
end
NONCORP:
NONCORP#sho run
Building configuration...
Current configuration : 6617 bytes
!
! Last configuration change at 09:32:15 edt Mon Apr 20 2015 by admin
! NVRAM config last updated at 09:33:35 edt Mon Apr 20 2015 by admin
!
version 12.2
no service pad
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
!
hostname NONCORP
!
!
username admin privilege 15
no aaa new-model
clock timezone est -5
clock summer-time edt recurring
ip subnet-zero
ip routing
!
!
!
!
!
!
no file verify auto
spanning-tree mode pvst
spanning-tree extend system-id
!
vlan internal allocation policy ascending
!
interface GigabitEthernet0/45
switchport access vlan 1000
switchport mode access
!
interface GigabitEthernet0/46
switchport access vlan 1000
switchport mode access
!
interface GigabitEthernet0/47
switchport trunk encapsulation dot1q
switchport trunk native vlan 1000
switchport mode trunk
!
interface GigabitEthernet0/48
switchport trunk encapsulation dot1q
switchport trunk native vlan 1000
switchport mode trunk
!
interface GigabitEthernet0/49
switchport trunk encapsulation dot1q
switchport trunk native vlan 1000
switchport mode trunk
!
interface GigabitEthernet0/50
description Uplink to Access04
switchport trunk encapsulation dot1q
switchport trunk native vlan 1000
switchport mode trunk
!
interface GigabitEthernet0/51
switchport trunk encapsulation dot1q
switchport trunk native vlan 1000
switchport mode trunk
!
interface GigabitEthernet0/52
switchport trunk encapsulation dot1q
switchport trunk native vlan 1000
switchport mode trunk
!
interface Vlan1
no ip address
shutdown
!
interface Vlan1000
ip address 10.0.100.130 255.255.255.128
!
ip default-gateway 10.0.100.129
ip classless
ip route 0.0.0.0 0.0.0.0 10.0.100.129
ip http server
ip http secure-server
!
!
snmp-server group AOCGroup v3 auth notify *tv.FFFFFFFF.FFFFFFFF.FFFFFFFF.F
snmp-server group AOCGroup v3 priv read AOCView write AOCView
snmp-server view AOCView internet included
snmp-server location AOC HQ
snmp-server contact it@angeloakcapital.com
snmp-server enable traps snmp authentication linkdown linkup coldstart warmstart
snmp-server enable traps stpx root-inconsistency loop-inconsistency
snmp-server host 10.0.1.244 version 3 auth AOCUser
!
control-plane
!
!
line con 0
line vty 0 4
login local
line vty 5 15
no login
!
ntp clock-period 36029007
ntp server 64.90.182.55
ntp server 98.175.203.200 prefer
!
end
04-21-2015 01:28 PM
What i dont understand, is once traffic gets to the corp switch, why does it send it to the firewall first?
It has an interface on Vlan100 and should just do inter-vlan routing?
I.E traffic comes from non corp, routes to corp, Corp switch locally switches the packet to the DNS server.
I would still strongly advise that you use the sub-interface to do this properly instead of hair-pinning traffic.
Alternatively, if you dont need the traffic to go through the ASA just route it locally.
Lastly, I dont really understand why its going to the ASA in the first place, unless I misunderstand the topology.Can you provide a traceroute?
The last thing i can suggest to try, which is a complete shot in the dark which i actually hate doing, is remove the DNS from the inspection on the ASA.
04-21-2015 06:28 AM
That second suggestion regarding the subinterface is actually a good idea, if I had thought of that first I probably would have done it...as it stands, because everything except DNS is working I don't want to redo the entire way it is routing but if I can't get DNS to come around I may end up doing that. We are in a production environment but this 10.0.100 network is only being used as a storage network so I can mess with it a bit if necessary
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide