Upgrading router from 2811 (IOS 12.4) to 4331 (IOS 15.2).

[BenEvans77]

Hello all. I hope you will help me with this as I've put more time in to this project than I care to admit. Please? The last big problem that I resolved was that in IOS 12.4 "established" connections were allowed automatically in the ACL for the outside interface. In 15.2 you have to specify an permit statement in the ACL to allow established connections so if you are on a host, device or nat pool trying to open up a web page it would allow the connection to come back in on whatever port it was going out on. But I digress. My current issue is this.

First the configs are for the most part identical between the two devices. The problem is that from any host on the inside network on the 2811 if I tracert or ping mail.mycompany.com it resolves to the private IP and not the public IP. On the tracert I don't even hit the router as a hop as my first hop is the host of the private ip. If I go in to the server room and swap the ethernet cables from the 2811 interfaces to the 4331 interfaces and try to do the same thing I end up going to the public IP. So the first hop is the internal interface, second hop is a third party IP address (Meaning it's not my outside interface or a public IP I own. It would be the IP my outside interface communicates with.) Third hop times out and the 4th hop resolves to the correct "Public" IP but I get the various DNS names pointing at that public IP. Like mail.mycompany.com or mail.myothercompany.com or vpn.myotherothercompany.com etc... even though I pinged or tracert something different than the hostname returned.

The problem with this is that we've been running on the 2811 for almost 10 years now. And in that time we have various programs that run on various servers or client computers that reference hostnames or public IP's. Like our mailserver for example. Which is all well and good. But on the 4331 Router it routes it out and thus either a external IP or Natpool IP is used as the source IP when it comes back in, creating all sorts of problems for us.

So I'm wondering if anyone has any ideas what has changed or where I might look to find some answers on this. Because I am not finding it and I've spent hours searching and reading. If you need the configs or more information let me know. It would not take long for me to strip out sensitive information from them and post them here. 

Thanks!

[Jon Marshall]

The problem is that from any host on the inside network on the 2811 if I tracert or ping mail.mycompany.com it resolves to the private IP and not the public IP. On the tracert I don't even hit the router as a hop as my first hop is the host of the private ip. If I go in to the server room and swap the ethernet cables from the 2811 interfaces to the 4331 interfaces and try to do the same thing I end up going to the public IP.

I'm not sure I follow.

If the mail server has a private IP and it resolves to the private IP when you ping with the 2811 that is a DNS issue not a router issue.

Unless you are doing some form of NAT on the 2811 to make this happen ?

Are you saying the DNS name of mail.company.com resolves to a public IP but your 2811 is then translating it back to a private IP for your client whereas the 4331 isn't ?

Jon

[BenEvans77]

Jon, thank you so much for your reply, I'll try to explain my situation better. First I'm running either the 2811 or the 4331. As the 4331 will replace the 2811. There are no other router devices or layer 3 devices in this process and all hardware and configurations are identical. The only thing that is changing is the router being the 2811 or the 4331.

On the 2811 we have a mail/web server that is NAT'ed. From any computer or server on the inside network if I do a tracert or ping to the hostname it resolves to the private ip. Forexample:

ping mail.mycompany.com resolves to 192.168.1.4

tracert mail.mycompany.com routes to 192.168.1.4 as the first hop.

now if I switch to the 4331 and do that from the same computer I get this.

ping mail.mycompany.com resolves to 204.204.204.4

tracert mail.mycompany.com routes to

192.168.1.1 (LAN IP of the router)

204.204.10.1 

* * * Request Timed Out. (I think I need to add a rule to the ACL for external interface inbound access-group, as this resolves on the 2811 when I tracert to the IP (not the hostname))

204.204.204.4

We have applications and processes that point to mail.mycompany.com that run from inside the network that are either taking a NAT'ed IP or using the NAT Pool on the 4331. This is problematic for us as we have rules defined on the mail server of what IP sources we allow to send or relay. 

The ACL's are identical on both devices, other than me adding the permit for any, public ip range, for established connections. Other than that one addition, they are identical. Interfaces are identical, IP route statements are identical, IP Name-Servers are identical. IP NAT is identical.

Does that help? I could post the configs if you want to see them. But the ACL is rather long and there are some settings in the 2811 that were configured once upon a time for certain interfaces that no longer exist like inspection and class mapping or something that I did not carry over since they are not used. I have had 2 personal friends of mine who are versed in older cisco technology (IOS 10-12) look at the configs and not find any particular reason for my issue. And when I say looked, I mean we spent 4-5 hours going over the config and stripping it down and rebuilding it, running bare minimum config, etc... trying to identify and resolve some of the issues we are seeing between IOS 12.4 and IOS 15.2 in migrating the config.

I hope this helps, thanks!

[Jon Marshall]

So just so I understand the issue fully.

What entry does the DNS server have for mail.my company.com.

Is it a public IP or a private IP.

And where is the DNS server in relation to your clients ?

If the DNS server responds with a public IP it sounds as though you have configuration that is doing a DNS rewrite when it sends the response back to the client.

And this isn't working on the 4331.

But before we start looking at it in detail i need to make sure I understand the problem.

Jon

[BenEvans77]

Thanks for the reply Jon.

Unfortunately I'm not sure what to tell you. On the internal DNS servers, the mx record does not exist for the domain. In reverse lookup the old hostname exists though and resolves to the IP.  So mail.mycompany.com does not exist on the internal DNS. but mail.mycompanyold.com does in the reverse lookup only.

It is something I should fix and I will tonight as well as test the other records to if tracert functions the same for them on the 4331 even though they do exist on the internal DNS server.

Still, isn't it odd that the 2811 handles this differently than the 4331? I mean nothing else is changing other than router hardware and cisco IOS. Still I suppose if something is configured wrong/incomplete, then it's configured wrong and should resolved despite this inconsistency between routers. The responsibility of the DNS Servers as well as domain registration is shared, and I don't really deal with that when things change, so I'm not the best person to talk to about this sort of thing.

Thoughts? 

Thanks!

[Jon Marshall]

If the DNS record does not exist on the internal DNS server then it sounds like what is happening is that with the 2811 your client is requesting a DNS name to IP lookup and the DNS server is returning a public IP which your router is then rewriting to a private IP for your client.

Your 4331 doesn't appear to be doing this.

Your clients may be pointing to the internal DNS but it may be configured to forward any unknown requests to an external DNS server which it would do for your mail server if your internal DNS server doesn't have a record which it sounds like it doesn't.

It would helpful to see the configurations of both routers.

Jon

[BenEvans77]

Thanks for your Response Jon.

Give me a bout an hour to clean up each config. Then I'll attach them in a zip.

[BenEvans77]

Jon here are the configs with no ACL's. I stand corrected on the IP Name Servers. They are different, though I think I might have changed that in trying to troubleshoot the new config on the 4331. To test though I went in to the 2811 and changed them to the internal DNS servers we have, flushed my dnscache and did a tracert to the hostname of the mail server and it went straight to the Private IP on the first hop.

If you think there is a problem in the ACL please let me know and I can confirm or try your recommendation tonight.

Thanks again for all your help! Also the PW for the archive is the same as my username, (case sensitive).

[Jon Marshall]

Ben

Bit of confusion :-)

In your first post you said the mail server was 192.168.1.4 but where is that in relation to the interfaces on your routers ie. your internal LAN is 192.168.8.0/22 and each router only has a default route pointing to a public IP next hop and a route for another internal network via a tunnel interface.

There is no mention of 192.168.1.4 anywhere. Did you mean 192.168.11.4 ?

Secondly the "ip name-server .." commands on your routers are nothing to do with client resolutions unless your routers are acting as DNS servers and they aren't.

Those commands are simply for the router to be able to resolve it's own queries eg. if you were on the router and did "telnet <hostname>" the router could resolve the hostname.

So what hands out DHCP IPs to your clients and what DNS servers are set there ie. when you do an "ipconfig /all" on your client what are the DNS servers ?

Jon

[BenEvans77]

Yeah, when I first mentioned the IP's I was masking the real ones. The real are in the config I gave you in the nat translations. You can identify the mail server by looking for a port 25 and 703 in the NAT.

DHCP is done by Windows Server 2008R2, that same server also does DNS which points at google DNS for primary and secondary and then the ISP DNS as redundant. My ipconfig will show 2 internal dns servers.

As I said I really don't know why there is a difference in the behavior of the two routers for our devices on the internal network. Nothing else changes other than me removing the ethernet cables from the old router and plugging them in to the new router. When I do that it seems the new router (4331) wants to route the traffic to the outside interface where as the old router (2811) for some reason resolves to the private IP as the first hop.

My IP Config statement returns

IP : 192.168.8.240 (this is a DHCP Assigned Reservation)

Subnet : 255.255.252.0

Gateway 192.168.11.1

DNS 192.168.10.4

DNS 192.168.10.3

Thanks again for your response. I hope this answers your questions, and I apologize before for citing invalid IP's in my examples before you had the configs. 

[Jon Marshall · Accepted solution]

Ben

It may be the was IOS XE does things slightly differently than IOS but that is just a guess.

If your internal DNS serves do not contain a record for mail.mycompany.com then they forward to an external DNS server. This contains the public IP of that name and sends it back. The router intercepts this and because of your static NAT statement it goes into the DNS response and rewrites the public IP to be the private IP of your mail server.

This is what the 2811 is doing for you.

The 4331 on the other hand doesn't appear to be doing the rewrite so instead your client receives the public IP and so has to go out to the internet.

You can easily verify this by using nslookup on your PC and debug it and you should see the reply contents coming back from the clients DNS query.

There are a couple of things to say -

1) in your configuration you have as you say specified the ports for the mail server in your static NAT statement.  But then the very next entry is a static statement for the entire private IP to public IP mapping.

Now as far as I know, DNS rewrite does not work with port mappings so I assume it must be using the one to one mapping.

It is unlikely but perhaps the NAT ordering is different on the 4331. So try removing the specific port mapping entries and see if that makes a difference.

It shouldn't affect any connectivity because like I say you have a full one to one mapping in there anyway for the same IPs.

2) It's difficult to say if other parts of the configuration are playing a part but again, as far as I know, DNS rewrite is not dependant on acls ie. either DNS is allowed or it isn't.

3) Finally you have an internal DNS. If you want internal clients to use the internal IP then why do you not simply have an internal record for it pointing to the private IP ?

It may not be that simple but if all apps are using hostnames then you can simply have an internal entry for the mail server.

Jon

[BenEvans77]

Thanks for the reply Jon. I think you are correct that the DNS Doctoring / DNS Rewrite is not working like it did in IOS 12.4 and that in combination with the lack of a hostname record on the DNS server is sending the traffic outside. I'm going to spend some time today researching to make sure I understand this correctly and identify the necessity of IP Nat entries with port translations.

Thanks for all your help. I'm going to mark your answer as correct.