Cisco ISR900 - DNS query issue from client machine

[orochi_yagami]

Hi, 

I've setup an ISR900 router in DC which come with two internet links. Each internet link come with it's own /29 subnet. Both the WAN interfaces are connected to these two internet link with primary outgoing point to G4 with a static default route and secondary to G5 with static default an higher AD. 

The client machine are groups of VM's behind this WAN router. The WAN router can queries internet IP and hostname with no issue. DNS servers are public DNS provided by local ISP. 

The client machine are configured with the public DNS. The issue is, whenever the client machine initiated the DNS queries (e.g.: web browsing), there isn't any query replied received. i ran the packet capture on the client machine and confirmed this. The client machine able to ping to the public internet DNS and from the traceroute the traffic are routed correctly. In other words, the the ACL and the NAT are worked fine. 

DNS tracert

ACL to allow virtual machine to internet. The [443] hit count are increase when i replaced the URL with IP address. I can also SSH any host in internet which is IP based. 

Configuration:

track 1 ip sla 1 reachability

!
ip dns server
ip nat inside source route-map NAT_INET1 interface GigabitEthernet4 overload
ip nat inside source route-map NAT_INET2 interface GigabitEthernet5 overload
ip route 0.0.0.0 0.0.0.0 XXX.XXX.XXX.XXX track 1
ip route 0.0.0.0 0.0.0.0 YYY.YYY.YYY.YYY 10
!

ip access-list extended TO-INET
permit ip 10.110.130.0 0.0.1.255 any
!
ip sla 1
icmp-echo XXX.XXX.XXX.XXX source-ip XXX.XXX.XXX.XX2
ip sla schedule 1 life forever start-time now
!
route-map NAT_INET2 permit 10
match ip address TO-INET
match interface GigabitEthernet5
!
route-map NAT_INET1 permit 10
match ip address TO-INET
match interface GigabitEthernet4
!
!

Btw, i'm running the zone-based FW for the access control. 

TIA 

[Georg Pauwen]

Hello,

--> Btw, i'm running the zone-based FW for the access control

Post the full running configuration of the router that has the zone based firewall...

[orochi_yagami]

HI Georg,

Below the ZB FW configuration:

class-map type inspect match-any INSIDE-TO-OUTSIDE-CLASS
match access-group name INSIDE-TO-OUTSIDE
class-map type inspect match-any OUTSIDE-TO-INSIDE-CLASS
match access-group name OUTSIDE-TO-INSIDE
class-map type inspect match-any INSIDE-TO-DMZ-CLASS
match access-group name INSIDE-TO-DMZ
class-map type inspect match-any DMZ-TO-OUTSIDE-CLASS
match access-group name DMZ-TO-OUTSIDE
!
policy-map type inspect INSIDE-TO-DMZ-POLICY
class type inspect INSIDE-TO-DMZ-CLASS
inspect
class class-default
drop log
policy-map type inspect INSIDE-TO-OUTSIDE-POLICY
class type inspect INSIDE-TO-OUTSIDE-CLASS
inspect
class class-default
drop log
policy-map type inspect DMZ-TO-OUTSIDE-POLICY
class type inspect DMZ-TO-OUTSIDE-CLASS
inspect
class class-default
drop log
policy-map type inspect OUTSIDE-TO-INSIDE-POLICY
class type inspect OUTSIDE-TO-INSIDE-CLASS
inspect
class class-default
drop
!
zone security INSIDE
zone security OUTSIDE
zone security DMZ
zone-pair security IN-TO-OUT source INSIDE destination OUTSIDE
service-policy type inspect INSIDE-TO-OUTSIDE-POLICY
zone-pair security IN-TO-DMZ source INSIDE destination DMZ
service-policy type inspect INSIDE-TO-DMZ-POLICY
zone-pair security DMZ-TO-OUT source DMZ destination OUTSIDE
service-policy type inspect DMZ-TO-OUTSIDE-POLICY
zone-pair security OUT-TO-IN source OUTSIDE destination INSIDE
service-policy type inspect OUTSIDE-TO-INSIDE-POLICY

ip access-list extended DMZ-TO-OUTSIDE
remark Rules to permit DMZ to Internet
permit tcp 10.110.132.0 0.0.0.255 any eq www log
permit tcp 10.110.132.0 0.0.0.255 any eq 443 log
permit icmp 10.110.132.0 0.0.0.255 any log
permit tcp 10.110.132.0 0.0.0.255 any eq 22 log
remark Rules to permit DMZ to Internet
permit tcp 10.110.132.0 0.0.0.255 eq domain host 165.21.83.88 log
permit udp 10.110.132.0 0.0.0.255 eq domain host 165.21.83.88 log
permit tcp 10.110.132.0 0.0.0.255 eq domain host 165.21.100.88 log
permit udp 10.110.132.0 0.0.0.255 eq domain host 165.21.100.88 log
ip access-list extended INSIDE-TO-DMZ
remark Rules to permit Inside to DMZ
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq www log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq 443 log
permit icmp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq 22 log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq 3389 log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq telnet log
ip access-list extended INSIDE-TO-OUTSIDE
permit tcp 10.110.130.0 0.0.1.255 any eq www
permit tcp 10.110.130.0 0.0.1.255 any eq 443
permit tcp 10.110.130.0 0.0.1.255 any eq 22
permit icmp 10.110.130.0 0.0.1.255 any
permit tcp 10.110.130.0 0.0.1.255 any eq 5938
permit udp 10.110.130.0 0.0.1.255 any eq 5938
permit tcp 10.110.130.0 0.0.1.255 host 8.8.8.8 eq domain
permit udp 10.110.130.0 0.0.1.255 host 8.8.8.8 eq domain
permit tcp 10.110.130.0 0.0.1.255 host 165.21.100.88 eq domain
permit udp 10.110.130.0 0.0.1.255 host 165.21.100.88 eq domain
permit tcp 10.110.130.0 0.0.1.255 host 165.21.83.88 eq domain
permit udp 10.110.130.0 0.0.1.255 host 165.21.83.88 eq domain
permit udp 10.110.130.0 0.0.1.255 any eq ntp
ip access-list extended Infrastructure_Access
permit tcp 10.110.130.0 0.0.1.255 any eq 22 log
ip access-list extended OUTSIDE-TO-INSIDE
permit tcp any 10.110.131.0 0.0.0.255 eq www
permit tcp any 10.110.131.0 0.0.0.255 eq 443
permit tcp host 165.21.83.88 10.110.131.0 0.0.0.255 eq domain
permit udp host 165.21.83.88 10.110.131.0 0.0.0.255 eq domain
permit udp host 165.21.83.88 eq domain 10.110.131.0 0.0.0.255
ip access-list extended TO-INET
permit ip 10.110.130.0 0.0.1.255 any

TIA

[Georg Pauwen]

Hello,

the reason I was asking for the full config is that I wanted to see if you have any access lists applied to interfaces that are also part of security zone. From the partial config you posted it looks like there are. Access lists are applied before any security policy. Try and temporarily disable any access list applied directly to an interface (and if possible, post the output of 'show run')...

[Georg Pauwen]

Hello,

add the two lines below your ACL:

ip access-list extended INSIDE-TO-OUTSIDE
--> permit tcp 10.110.130.0 0.0.1.255 eq domain host 8.8.8.8
--> permit udp 10.110.130.0 0.0.1.255 eq domain host 8.8.8.8

[orochi_yagami]

Hi Georg,

Yup, i've added the two ACL as suggested, doesn't seem to have any DNS query traffic matched on it.

Btw, i've attached full configuration for your reference. I consider the Google DNS as tertiary DNS server since there two other DNS from ISP available locally which is 83.88 and 100.88. These two will be the primary and secondary. 

Also, as you suggested earlier, i removed the zone-member security inside and zone-member security outside from the Vl131 (Gateway, Inside)  and Gi4 (Internet, Outside) to totally remove the zone-based control, still the VM workstation unable to get the response from DNS. 

[balaji.bandi]

From the Router or from the device are you able to resolve those DNS Queries using Other than your DNS Servers ?

What is the status if you shutdown one of the ISP link and test it, what is the status if you bypass the router and connect ISP directly to the device and test (this is for testing).

[orochi_yagami]

Hi Balaji,

The DNS server which i mentioned above are all public DNS servers such as google and from the ISP, not the private internal DNS server. Below is the DNS lookup result from the WAN router, looking good. Also, im able to do the nslookup if the connect the ISP link directly to my laptop, so no issue from the ISP side. 

XXXXX01#ping www.cisco.com
Translating "www.cisco.com"...domain server (8.8.8.8) [OK]

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 23.15.104.32, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/2 ms
XXXXX01#traceroute 8.8.8.8
Type escape sequence to abort.
Tracing the route to dns.google (8.8.8.8)
VRF info: (vrf in name/id, vrf out name/id)
1 XXX.XXX.XXX.XXX 0 msec 0 msec 2 msec
2 10.11.34.141 0 msec 0 msec 2 msec
3 10.15.61.202 2 msec 0 msec 2 msec
4 142.250.169.57 2 msec 0 msec 2 msec
5 * * *
6 dns.google (8.8.8.8) 0 msec 2 msec 0 msec

If i shutdown either of the link, the result are same where the VM's don't get any DNS replied

[paul driver]

Hello

Try the follwoing:
class-map type inspect match-any INSIDE-TO-OUTSIDE-CLASS
match protocol dns
match protocol http
match protocol https
match protocol ntp
no match access-group name INSIDE-TO-OUTSIDE

no zone-pair security OUT-TO-IN source OUTSIDE destination INSIDE

[orochi_yagami]

Hi Paul,

I tried the suggestion above to replace the ACL matching with protocol, unfortunately there is no difference. 

TIA

[Georg Pauwen]

Hello,

what are you trying to accomplish, to have the router function as the DNS server ? In that case, you would need to add a self zone, but rather than doing that, try to make the changes marked in bold (in order to test if that resolves your client issue):

version 15.9
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime msec localtime show-timezone
service timestamps log datetime msec localtime show-timezone
service password-encryption
service compress-config
service sequence-numbers
!
hostname XXXXX01
!
boot-start-marker
boot system flash:c900-universalk9-mz.SPA.159-3.M2.bin
boot-end-marker
!
logging buffered 64000
!
aaa new-model
!
aaa session-id common
!
no ip source-route
!
--> no ip domain name abc.com
--> no ip name-server XXX.XXX.83.88
--> no ip name-server XXX.XXX.100.88
--> no ip name-server 8.8.8.8
ip cef
no ipv6 cef
!
multilink bundle-name authenticated
!
spanning-tree portfast bpduguard
vtp domain abc
vtp mode transparent
!
redundancy
!
vlan 130
name INSIDE-VPN
!
vlan 131
name INSIDE-VM-MGMT
!
vlan 132
name DMZ-VM
!
vlan 134
name MANAGEMENT
!
track 1 ip sla 1 reachability
!
class-map type inspect match-any INSIDE-TO-OUTSIDE-CLASS
match protocol dns
match protocol http
match protocol https
match protocol ntp
class-map type inspect match-any OUTSIDE-TO-INSIDE-CLASS
match access-group name OUTSIDE-TO-INSIDE
class-map type inspect match-any INSIDE-TO-DMZ-CLASS
match access-group name INSIDE-TO-DMZ
class-map type inspect match-any DMZ-TO-OUTSIDE-CLASS
match access-group name DMZ-TO-OUTSIDE
!
policy-map type inspect INSIDE-TO-DMZ-POLICY
class type inspect INSIDE-TO-DMZ-CLASS
inspect
class class-default
drop log
policy-map type inspect INSIDE-TO-OUTSIDE-POLICY
class type inspect INSIDE-TO-OUTSIDE-CLASS
inspect
class class-default
drop log
policy-map type inspect DMZ-TO-OUTSIDE-POLICY
class type inspect DMZ-TO-OUTSIDE-CLASS
inspect
class class-default
drop log
policy-map type inspect OUTSIDE-TO-INSIDE-POLICY
class type inspect OUTSIDE-TO-INSIDE-CLASS
inspect
class class-default
drop
!
zone security INSIDE
zone security OUTSIDE
zone security DMZ
zone-pair security IN-TO-OUT source INSIDE destination OUTSIDE
service-policy type inspect INSIDE-TO-OUTSIDE-POLICY
zone-pair security IN-TO-DMZ source INSIDE destination DMZ
service-policy type inspect INSIDE-TO-DMZ-POLICY
zone-pair security DMZ-TO-OUT source DMZ destination OUTSIDE
service-policy type inspect DMZ-TO-OUTSIDE-POLICY
!
interface Loopback130
description *** INSIDE-VPN-GW ***
ip address 10.110.130.1 255.255.255.0
no ip unreachables
zone-member security INSIDE
no ip route-cache
!
interface GigabitEthernet0
description |-- HP DL360P iLO --|
switchport access vlan 134
switchport mode access
no ip address
zone-member security INSIDE
!
interface GigabitEthernet1
description |-- Hypervisor ESXi VMNIC3 --|
switchport access vlan 134
switchport mode access
no ip address
zone-member security INSIDE
!
interface GigabitEthernet2
description |-- INSIDE-ZONE VMNIC1 --|
switchport mode trunk
no ip address
zone-member security INSIDE
!
interface GigabitEthernet3
description |-- DMZ-ZONE VMNIC0 --|
switchport access vlan 132
switchport mode access
no ip address
zone-member security DMZ
!
interface GigabitEthernet4
ip address XXX.XXX.XXX.XXX 255.255.255.248
ip nat outside
ip virtual-reassembly in
zone-member security OUTSIDE
duplex auto
speed auto
!
interface GigabitEthernet5
ip address YYY.YYY.YYY.YYY 255.255.255.248
ip nat outside
ip virtual-reassembly in
zone-member security OUTSIDE
duplex auto
speed auto
!
interface Virtual-Template1
ip unnumbered GigabitEthernet4
ip nat inside
ip virtual-reassembly in
zone-member security INSIDE
!
interface Vlan1
no ip address
shutdown
!
interface Vlan131
description *** INSIDE-VM-MGMT ***
ip address 10.110.131.1 255.255.255.0
no ip redirects
no ip unreachables
ip nat inside
ip virtual-reassembly in
zone-member security INSIDE
no ip route-cache
!
interface Vlan132
description *** DMZ-ZONE ***
ip address 10.110.132.1 255.255.255.0
no ip unreachables
ip nat inside
ip virtual-reassembly in
zone-member security DMZ
no ip route-cache
!
interface Vlan134
description *** XXXXX01-MGMT ***
ip address 10.110.134.1 255.255.255.0
no ip redirects
zone-member security INSIDE
no ip route-cache
!
ip forward-protocol nd
no ip http server
no ip http secure-server
!
--> no ip dns server
ip nat inside source route-map NAT_INET1 interface GigabitEthernet4 overload
ip nat inside source route-map NAT_INET2 interface GigabitEthernet5 overload
ip route 0.0.0.0 0.0.0.0 XXX.XXX.XXX.XXX track 1
ip route 0.0.0.0 0.0.0.0 YYY.YYY.YYY.YYY 10
!
ip access-list extended DMZ-TO-OUTSIDE
remark Rules to permit DMZ to Internet
permit tcp 10.110.132.0 0.0.0.255 any eq www log
permit tcp 10.110.132.0 0.0.0.255 any eq 443 log
permit icmp 10.110.132.0 0.0.0.255 any log
permit tcp 10.110.132.0 0.0.0.255 any eq 22 log
remark Rules to permit DMZ to Internet
permit tcp 10.110.132.0 0.0.0.255 eq domain host 165.21.83.88 log
permit udp 10.110.132.0 0.0.0.255 eq domain host 165.21.83.88 log
permit tcp 10.110.132.0 0.0.0.255 eq domain host 165.21.100.88 log
permit udp 10.110.132.0 0.0.0.255 eq domain host 165.21.100.88 log
ip access-list extended INSIDE-TO-DMZ
remark Rules to permit Inside to DMZ
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq www log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq 443 log
permit icmp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq 22 log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq 3389 log
permit tcp 10.110.130.0 0.0.1.255 10.110.132.0 0.0.0.255 eq telnet log
ip access-list extended INSIDE-TO-OUTSIDE
permit tcp 10.110.130.0 0.0.1.255 any eq www
permit tcp 10.110.130.0 0.0.1.255 any eq 443
permit tcp 10.110.130.0 0.0.1.255 any eq 22
permit icmp 10.110.130.0 0.0.1.255 any
permit tcp 10.110.130.0 0.0.1.255 any eq 5938
permit udp 10.110.130.0 0.0.1.255 any eq 5938
permit tcp 10.110.130.0 0.0.1.255 host 8.8.8.8 eq domain
permit udp 10.110.130.0 0.0.1.255 host 8.8.8.8 eq domain
permit tcp 10.110.130.0 0.0.1.255 host 165.21.100.88 eq domain
permit udp 10.110.130.0 0.0.1.255 host 165.21.100.88 eq domain
permit tcp 10.110.130.0 0.0.1.255 host 165.21.83.88 eq domain
permit udp 10.110.130.0 0.0.1.255 host 165.21.83.88 eq domain
permit udp 10.110.130.0 0.0.1.255 any eq ntp
ip access-list extended Infrastructure_Access
permit tcp 10.110.130.0 0.0.1.255 any eq 22 log
ip access-list extended OUTSIDE-TO-INSIDE
permit tcp any 10.110.131.0 0.0.0.255 eq www
permit tcp any 10.110.131.0 0.0.0.255 eq 443
permit tcp host 165.21.83.88 10.110.131.0 0.0.0.255 eq domain
permit udp host 165.21.83.88 10.110.131.0 0.0.0.255 eq domain
permit udp host 165.21.83.88 eq domain 10.110.131.0 0.0.0.255
ip access-list extended TO-INET
permit ip 10.110.130.0 0.0.1.255 any
!
ip sla 1
icmp-echo 103.200.217.209 source-interface GigabitEthernet4
ip sla schedule 1 life forever start-time now
!
route-map NAT_INET2 permit 10
match ip address TO-INET
match interface GigabitEthernet5
!
route-map NAT_INET1 permit 10
match ip address TO-INET
match interface GigabitEthernet4
!
control-plane
!
end

[orochi_yagami]

Hello Georg,

Still the same result after removed the bold configuration that you highlighted, the VM machine unable to get the query resolved. I've also flush the dns configuration on the VM machine but no luck. 

TIA. 

[Georg Pauwen]

Hello,

which port on the router connects to the hosts ? I would think it is this one ?

interface GigabitEthernet2
description |-- INSIDE-ZONE VMNIC1 --|
switchport mode trunk
no ip address
zone-member security INSIDE

[orochi_yagami]

Hello Georg,

Yes, that is the port where the "INSIDE" VM's connected to. The default gateway for the VM subnet is Vl131, so the VM's reach the default gateway and ping to the public DNS via this port.

TIA.

[Georg Pauwen]

Hello,

the reason I am asking is because interface GigabitEthernet2 is configured as a trunk. What other Vlans are connected to that trunk ?

If it is (and it should) be only Vlan 131, configure that interface as an access port:

interface GigabitEthernet2
description |-- INSIDE-ZONE VMNIC1 --|
--> switchport mode access
--> switchport access vlan 131
no ip address
zone-member security INSIDE

or leave it as a trunk, but remove the zone membership (since not all Vlans are in the same 'INSIDE' security zone:

interface GigabitEthernet2
description |-- INSIDE-ZONE VMNIC1 --|
switchport mode trunk
no ip address
--> no zone-member security INSIDE

Also, what eaxctly are the VM machines ?