IP NAT INSIDE VLAN Hostname Resolution Issue

[epcomworld]

I am not sure if using IP NAT inside static or destination could fix this issue.  We are having issues with clients within the VLAN using hostname resolution to access applications all on the SAME static IP from being able to access the pages from within the vlan, but works fine over the external LAN for all external global users.  NSlookup shows the public IP Address when checking the hostname, but the VLAN machines are unable to access it - it's not translating back to the internal address etc.

ip nat inside source route-map FE0_General_Failover interface FastEthernet0 overload
ip nat inside source route-map GE8_General_Failover interface GigabitEthernet8 overload
ip nat inside source static tcp 192.168.0.200 81 55.55.55.55 81 extendable
ip nat inside source static tcp 192.168.0.201 443 55.55.55.55 443 extendable
ip nat inside source static tcp 192.168.0.202 8040 55.55.55.55 8040 extendable
ip nat inside source static tcp 192.168.0.202 8041 55.55.55.55 8041 extendable
ip nat inside source static tcp 192.168.0.203 8250 55.55.55.55 8250 extendable (static IP has bee updated in the above example).

route-map GE8_General_Failover permit 1
match ip address 1
match interface GigabitEthernet8
!
route-map FE0_General_Failover permit 1
match ip address 1
match interface FastEthernet0

Basically - we have ports opened so applications and services can be accessed via hostname external.  An example of this is hr.mydomain.com.  All external users have no problem access the server.  However, the users in the VLAN are the ones that are not able to access the server via hostname automatically.  I know that with the DNS Server from Windows Server I can add this hostname and point it to the private IP within the VLAN.  However, not all servers are on the DNS server within the VLAN and some use public DNS Servers such as 4.2.2.2 and 8.8.8.8.  The problem here is when one of these stations goes to look up the domain it gets the public IP Address on GE8 and then it says connection timeout and the connection never works.

Why is the Cisco router for the NAT parameters above not taking the command from the public IP address and sending into back into the internal one.

Previously these servers were running from a cheap D-Link environemtn and it had no problem allowing the VLAN users to connect to the servers via the hostname vs. static IP Address - it knew - why isn't the Cisco knowing.

I have read various DNS doctoring technics, however, I cannot seem to get the router to work.

I just want a client within the VLAN using a public DNS Server - when it returns the public static IP Address for it to just work.  This seems to work correctly when all applications ahve their own static IP address, however, we don't have enough access from the ISP to allow for multi statics per application server, etc.  Obivously we will look into doing this, but why did the cheap D-Link make it so simple and the Cisco ISR make it so hard.

What is the work around to get this to work?

Keep me posted if you have any ideas.  Thank you.

Device: Cisco 891f-K9

[epcomworld]

Hello Peter @Peter Paluch,

Thank you for the response.  There is still an open service request number on this: SR683629902 - I'm awaiting to here from the Cisco team on it - we worked on it and they had to take the case to the lab and I have not heard anything back yet.

You are correct I would like a server such as 192.168.0.200 with public DNS Server when they are looking to reach the hostname assigned to server 192.168.0.201 to load from within the VLAN.  It only works externally.  The only IP Address assigned to the server currently is just the private IP.  So when you state that one of the following alternatives would need to be true the one that is true is this:

"The servers are configured only with their private IP addresses. (Yes) For packets from inside hosts targeted to the public server IP addresses, the router would translate the destination IP address from the public IP address to the private IP address. However, the router would also need to translate the source IP address of these packets to a fictious IP range to cause the servers to respond to clients through the router and not directly; if the source IP addresses were not changed, the communication would fail because the clients would attempt to talk to the servers using their public IPs but would be receiving responses from their private IPs, and would drop them as unexpected. Essentially, this would require a double NAT. Based on the nature of NAT on Cisco IOS-based devices, this alternative would be cumbersome and unintuitive to configure, and I would personally recommend avoiding it at all if possible."

Now since the configuration for that is not simple - I have no problem going with the first option.  The only thing here is that the: you state the servers would need to be configured with their appropriate public IP addresses as additional IP addresses they have, on top of their existing private IP addresses.

In the scenario here the servers all have the SAME public IP Address - hence why they all use different ports.  I'm not sure how I could add the public IP address to the servers as secondary without more managed switching devices, etc.  Maybe the double NAT is the solution.  If I had more IPs I would just use the 1:1 NAT option and it would work, but for this case that is not an option.

Yesterday before your recommendation came in I tried to reconfigure the NAT with just IP NAT enable commands.  Looking at the form at this link I thought this could be the issue based on the debug ip nat commands that were received:

https://supportforums.cisco.com/t5/lan-switching-and-routing/port-forwarding-issue/td-p/1446759

Regardless it did not work, but I want you to see the configuration that was applied:

I updated the configuration to the following:

! Last configuration change at 14:33:45 EST Mon Dec 18 2017
version 15.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
!
hostname Router
!
boot-start-marker
boot-end-marker
!
aqm-register-fnf
!
logging buffered 51200 warnings
!
no aaa new-model
clock timezone EST -5 0
!
!
!
!
!
!


!
ip dhcp excluded-address 192.168.0.1 192.168.0.99
ip dhcp excluded-address 192.168.0.200 192.168.0.254
!
ip dhcp pool internal
 network 192.168.0.0 255.255.255.0
 dns-server 192.168.0.213 4.2.2.2 71.243.0.12 68.237.161.12
 domain-name epcomworld.local
 default-router 192.168.0.1
 lease 14
!
!
!
ip domain name router.testrouter.local
ip name-server 4.2.2.2
ip cef
no ipv6 cef
!
!
!
!
!
multilink bundle-name authenticated
!
!
!
!
!
!
!
license udi pid C891F-K9
!
!
!
!
!
!
!
ip ssh version 2
!
class-map match-all Voice
 match ip dscp ef
!
policy-map EGNVoice
 class Voice
  priority 25000
!
!
!
!
!
!
!
!
!
!
!
interface BRI0
 no ip address
 encapsulation hdlc
 shutdown
 isdn termination multidrop
!
interface FastEthernet0
 no ip address
 duplex auto
 speed auto
!
interface GigabitEthernet0
 no ip address
!
interface GigabitEthernet1
 no ip address
!
interface GigabitEthernet2
 no ip address
!
interface GigabitEthernet3
 no ip address
!
interface GigabitEthernet4
 no ip address
!
interface GigabitEthernet5
 no ip address
!
interface GigabitEthernet6
 no ip address
!
interface GigabitEthernet7
 no ip address
!
interface GigabitEthernet8
 ip address 96.253.18.131 255.255.255.0 secondary
 ip address 96.253.18.130 255.255.255.0
 ip nat enable
 ip virtual-reassembly in
 duplex auto
 speed auto
 service-policy output EGNVoice
!
interface Vlan1
 ip address 192.168.0.1 255.255.252.0
 ip nat enable
 ip virtual-reassembly in
!
interface Async3
 no ip address
 encapsulation slip
!
ip default-gateway 96.253.18.1
ip forward-protocol nd
no ip http server
no ip http secure-server
!
!
ip nat source static tcp 192.168.0.202 8040 interface GigabitEthernet8 8040
ip nat source static tcp 192.168.0.202 8041 interface GigabitEthernet8 8041
ip nat source static tcp 192.168.0.251 3389 interface GigabitEthernet8 48526
ip nat source static tcp 192.168.0.200 81 interface GigabitEthernet8 81
ip nat source static tcp 192.168.0.201 443 interface GigabitEthernet8 443
ip nat source static tcp 192.168.0.203 8250 interface GigabitEthernet8 8250
ip nat source static tcp 192.168.0.216 48520 interface GigabitEthernet8 48520
ip nat source static tcp 192.168.0.217 48521 interface GigabitEthernet8 48521
ip nat source static tcp 192.168.0.218 48522 interface GigabitEthernet8 48522
ip nat source list 100 interface GigabitEthernet8 overload (I tried with access list 1 too and that did not work either) --> Nat Traffic did work, but the internal servers resolving to hostname via public DNS would not load sites.
ip nat source static tcp 192.168.0.250 3389 interface GigabitEthernet8 48525
ip nat source static tcp 192.168.0.251 80 interface GigabitEthernet8 80
ip nat inside source static tcp 192.168.0.202 8040 interface GigabitEthernet8 8040
ip nat inside source static tcp 192.168.0.202 8041 interface GigabitEthernet8 8041
ip nat inside source static tcp 192.168.0.251 3389 interface GigabitEthernet8 48526
ip nat inside source static tcp 192.168.0.200 81 interface GigabitEthernet8 81
ip nat inside source static tcp 192.168.0.201 443 interface GigabitEthernet8 443
ip nat inside source static tcp 192.168.0.203 8250 interface GigabitEthernet8 8250
ip nat inside source static tcp 192.168.0.216 48520 interface GigabitEthernet8 48520
ip nat inside source static tcp 192.168.0.217 48521 interface GigabitEthernet8 48521
ip nat inside source static tcp 192.168.0.218 48522 interface GigabitEthernet8 48522
ip nat inside source static tcp 192.168.0.250 3389 interface GigabitEthernet8 48525
ip nat inside source static tcp 192.168.0.251 80 interface GigabitEthernet8 80
ip nat inside source route-map FE0_General_Failover interface FastEthernet0 overload
ip nat inside source route-map GE8_General_Failover interface GigabitEthernet8 overload
ip nat inside source static tcp 192.168.0.250 3389 96.253.18.130 48525 extendable
ip nat inside source static tcp 192.168.0.251 3389 96.253.18.130 48526 extendable
ip route 0.0.0.0 0.0.0.0 96.253.18.1
!
!
route-map GE8_General_Failover permit 1
 match ip address 100
 match interface GigabitEthernet8
!
route-map FE0_General_Failover permit 1
 match ip address 100
 match interface FastEthernet0
!
access-list 1 permit 192.168.0.0 0.0.3.255
access-list 100 deny   ip 192.168.0.0 0.0.3.255 host 96.253.18.130
access-list 100 deny   ip 192.168.0.0 0.0.3.255 host 96.253.18.131
access-list 100 permit ip 192.168.0.0 0.0.3.255 any
!
control-plane
!
!
!
mgcp behavior rsip-range tgcp-only
mgcp behavior comedia-role none
mgcp behavior comedia-check-media-src disable
mgcp behavior comedia-sdp-force disable
!
mgcp profile default
!
!
!
!
banner login ^CAuthorized Personnel Only!  Please enter username and password.  All attempts on this device will be logged and recorded.  Contact Epcom Global Networks for support and service! ^C
!
line con 0
 logging synchronous
 login
 no modem enable
 transport preferred none
line aux 0
 login
 transport preferred none
line 3
 modem InOut
 speed 115200
 flowcontrol hardware
line vty 0 4
 privilege level 15
 login local
 transport preferred none
 transport input ssh
!
scheduler allocate 20000 1000
!

I will now re-code the router with the configuration you have suggested.  It is so interesting to me that the D-Link Router does this fine as this was the environment these devices / servers were used in and within that VLAN with the open ports to the various servers you could use hostname resolution and it did work.  I never realized / knew this would be so complex to enable on the Cisco ISR platform.

Please let me know your continued thoughts - many thanks.

[Peter Paluch]

Hello,

You are very much welcome!

I do not contest the fact that configuring this type of NAT on Cisco routers is complex - I agree that it is anything but simple. Still, if you start breaking down the necessary steps that need to take place for such communication to be working (internal client talking to internal server using the server's public IP), you will find out that there are many non-intuitive and non-trivial operations needed in background, and the real question is what does the D-Link device exactly do, and how does it fill in the blanks, as the process I've explained in my previous response needs some design choices to be made, and it looks like the D-Link does not allow you to influence them at all. In this case, flexibility truly comes at the price of complexity.

you state the servers would need to be configured with their appropriate public IP addresses as additional IP addresses they have, on top of their existing private IP addresses.

For the PBR approach, yes.

I'm not sure how I could add the public IP address to the servers as secondary without more managed switching devices, etc.

Current server operating systems will certainly allow you to add secondary IP addresses to their configuration. You do not need more managed switching device or anything in addition in your network infrastructure itself. If you are concerned about the fact that this would constitute an IP address conflict, here comes the second part of what I indicated: Ideally, you should add the public IP address to a different interface on the server than the NIC connected to 192.168.0.0/22 itself, thereby preventing the servers from ever discovering they all share the same public IP. If it was Linux, I would add the public IP address to the lo (loopback) interface; if it was Windows, I would install the Microsoft Loopback Adapter, and have the IP address configured there.

Either way, there is no need for extra network equipment to have the servers share the same public IP.

I suspect that the approach through ip nat enable failed because the configuration was not specific enough to set apart the client->server and server->client flows, and likely did not translate the source and destination addresses in both directions properly. Once again, the problem is this: If the clients talk to a public IP, they expect to receive a response from the public IP back. If the servers respond from their private IP, the response needs to go through the router to modify the private server's IP back to the public IP - however, in order for this to happen, the clients' IP addresses themselves have to be overwritten to a source apparently reachable through the router, otherwise the responses from the servers to the clients will bypass the router entirely and go to clients directly.

Please feel welcome to ask further!

Best regards,
Peter

[epcomworld]

Thanks Peter,

I'm trying it right now with the loop back and the PBR Routing.  Could you provide an example of the Double NAT scenario you had referenced earlier - I would like to see what that type of configuration would look like on the system.

Continue to keep me posted and I will provide an update shortly.  Thank you for your support.

[Peter Paluch]

Hello,

My apologies for the delay - got busy during the last couple of days.

I've checked the case notes for the TAC SR 683629902 and it looks like the engineer handling your TAC case also came up with a solution though it was not specified closer. Can you perhaps comment on the details? Also, considering the PBR approach I've suggested, and the solution suggested by TAC, would you still want me to come up with the double-NAT approach?

Thanks!

Best regards,
Peter

[epcomworld]

Hi Peter,

No apologies needed on the delay.  Just early this morning we were working on the TAC SR 683629902 case and the solution provided didn't really address the requirements.  We were able to configure that on our own.

Then I showed them the forum post you provided and they said yes that is one way it could be done.

Literally we programmed it with them, but because it was so late / early in the morning depending on how you look at it, when we tried to test I needed to review the Microsoft Loopback configuration on the Windows Server.

For example I added the loopback interface - configured the public IP address on it and that is where I stopped.  I probably need to start either Internet sharing or bridging the NICs to get that to work so it will work with the router - unless just having the adapter there with the static IP on it is enough.

I advised them I would e-mail them.

I would honestly still be curious to see what the double NAT configuration will look like.  I will have physical access to the site again this is when I will test everything out and see if I can make the site function as needed based on the requirements.

Any additional insight or thoughts you have please share as I will be on site in a couple hours to review this further.  Many thanks.

[Peter Paluch]

Hello,

Constructing the double-NAT will take me some time - I am not sure if I'll be able to complete it today but I will do my best.

As for the Microsoft Loopback - can you please describe the issues you have had with it? From your last post, I only vaguely understand that it did not work as expected, but I need to know more. Either way, you should not need to enable bridging, ICS or anything of the sort - it should be entirely sufficient to have the Microsoft Virtual Looback NIC installed and configured with an IP address and netmask. I am not sure about the Windows Firewall, though - is there an option of temporarily disabling it on one test host and see if it makes any difference? Also, when working with a particular Windows server that has the Loopback virtual interface already configured, can this server ping itself using its own public IP?

Best regards,
Peter

[Francesco Molino]

Hi

I don't want to interfere in your discussion but just wanted to be sure that I understood your issue.

From inside host when trying to access a server with its public IP it's not working, am I right?

Here is my config that works fine:

NAT-RTR#sh run | i nat
ip nat source list 100 interface GigabitEthernet0/1 overload
ip nat source static tcp 192.168.0.200 2222 96.253.18.130 2222 extendable
NAT-RTR#sh ip int bri | i 0/1
GigabitEthernet0/1 96.253.18.130 YES NVRAM up up

access-list 100 permit ip 192.168.0.0 0.0.3.255 any

NAT-RTR#sh ip nat nvi translations
Pro Source global Source local Destin local Destin global
tcp 96.253.18.130:2222 192.168.0.200:2222 --- ---

I've 2 linux machines: 192.168.0.201 (linux1) and 192.168.0.200

From linux1 when I ping linux2, it's resolved with its public IP:

srv1:~$ ping linux2
PING linux2 (96.253.18.130) 56(84) bytes of data.
64 bytes from linux2 (96.253.18.130): icmp_seq=1 ttl=255 time=1.49 ms
64 bytes from linux2 (96.253.18.130): icmp_seq=2 ttl=255 time=1.05 ms
64 bytes from linux2 (96.253.18.130): icmp_seq=3 ttl=255 time=0.985 ms
64 bytes from linux2 (96.253.18.130): icmp_seq=4 ttl=255 time=1.01 ms
64 bytes from linux2 (96.253.18.130): icmp_seq=5 ttl=255 time=1.03 ms
64 bytes from linux2 (96.253.18.130): icmp_seq=6 ttl=255 time=1.62 ms

When I do a ssh on port 2222, I get redirected correctly on my linux2 machine.

Check the tcpdump on linux1 and linux2.

Did I reproduce your exact issue? What am i missing?

[epcomworld]

Hi Francesco,

Let us say that we have four servers:

192.168.0.200 - production1.domain.com - Open Port 81

192.168.0.201 - production2.domain.com - Open Port 443

192.168.0.202 - production3.domain.com - Open Port 8040

192.168.0.203 - production4.domain.com - Open Port 8250

In the VLAN of 1022 potential hosts: 192.168.0.1 - 192.168.3.254 - let us say we have a virtual desktop or computer at:

192.168.0.250 - Normal Computer, this computer has the following static IP Address:

192.168.0.250

255.255.255.0

192.168.0.1

DNS 1: 4.2.2.2 - or whatever the ISP has issued

DNS 2: 8.8.8.8 - or whatever the ISP has issued

If you open a browser and type in production1.domain.com:81 --> it fails to load from within the VLAN unless the DNS Server is the actual router with the DNS Server enabled and the IP hosts the router can use to resolve it or if the etc host file or domain controller with DNS server roll running is the primary DNS for the workstation.

This is not possible in this application the DNS Servers have to be public ones.

When you try to visit the various hostnames above from teh VLAN it fails, but works fine externally.

If you do an nslookup on the desktop (192.168.0.250) for production1.domain.com it returns the public IP Address of GE8.  It responds to pings, but the router doesn't let it automatically resolve back in with the hostname or even if I type in 96.253.18.130:8040 into the browser.

I know this is a very odd case, but this is the requirements for the site that computers within the subnet with non-private DNS Servers are able to access any server via hostname that maybe hosted from within that VLAN.

At 10:15 EST - I will try various configurations on the router and immediatley post back to this page.

I hope I explained it better.  Please let me know.

[Francesco Molino]

OK gotcha,

Did you read my post? We are in the same situation right? 

Have you done some tcp dump to check what's going on?

[epcomworld]

Yes I read your post and I am looking at my configuration right now.  Looks to be the exact same concept.

The configuration should work I agree, but the NAT is not allowing the traffic back in - in my case.  I included some of the debug ip nat detail above.

Did you also use a route map in your test configuration and I'm assuming the normal GE 0/1 ip nat outside and the ip nat inside for VLAN 1.

Let me change some of the rules and add the any to the access list and see what happens - I will report back in just about 1 hour.  We will get this wrapped up tonight.

[Francesco Molino]

OK. Just share some screenshot of tcpdump when testing