topic NAT/routing issue from one subinterface to another in Network Security

NAT/routing issue from one subinterface to another

WStoffel1 — Tue, 12 Mar 2019 00:25:48 GMT

I have a customer that sits behind an ASA on Int g0/1.143 (sec-level set to 100).

I have another customer that sits behind the same firewall on Int g0/1.156 (sec level = 100 as well).

The default gateway for each customer is the ip address on the above interfaces, 192.168.143.254 and 192.168.156.254 respectively.

The customer on 1.156 has a web server that's up and alive on the internet, natted to a public address of 72.46.x.x.

Customers on 192.168.143.0/24 cannot get to the website. When i tracert from 192.168.143.10 i can see the router that sits on the outside interface of the ASA then it fails, so it's being sent out the default gateway of the ASA.

What am i missing here?

Thanks as always....

NAT/routing issue from one subinterface to another

varrao — Wed, 21 Nov 2012 01:58:39 GMT

Hi,

What you might be missing is a nat command and also the command:

same-security-traffic permit inter-interface

If you can share your configuration or atleast the interface configuration, I can suggest you more specific configuration for it.

Thanks,
Varun Rao
Security Team,
Cisco TAC

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 02:45:29 GMT

Kind of hard to show the whole config, it's enormous, and i have customers behind it. Same security traffic is enabled inter and intra.

But the two interfaces are:

interface GigabitEthernet0/1.143

vlan 143

nameif Amherst

security-level 100

ip address 192.168.143.254 255.255.255.0

interface GigabitEthernet0/1.156

vlan 156

nameif Franklin

security-level 100

ip address 192.168.156.254 255.255.255.0

same-security-traffic permit inter-interface

same-security-traffic permit intra-interface

I thought it was just similar to another issue I had help resolving last week...i thought i just needed a static nat:

static (Amherst,Franklin) 192.168.143.0 192.168.143.0 netmask 255.255.255.0

But that didn't work.

Re: NAT/routing issue from one subinterface to another

Jouni Forss — Wed, 21 Nov 2012 07:22:16 GMT

Hi,

Are you trying to reach the site from behind the ASA (on the LAN) using the public NATed IP of the server?

I guess you are trying to test connections or reach it using DNS name?

Make sure you have a "dns" parameter in the "static" command for server you are trying to reach.

Example might be

static (inside,outside) 1.2.3.4 10.10.10.10 netmask 255.255.255.255 dns

This will make it possible to reach the server using the public DNS name from the local LAN. What the ASA will do with the "dns" parameter is that it will see the DNS query go out and the reply come in. It will then modify the DNS reply sent to the client by changing the resolved IP address to the local LAN address of the server.

Though for this to work you will have to be using public DNS server or the server has to be somewhere behind the ASA so that the clients DNS query will be seen by the ASA. If the DNS server is in the same LAN as the client ASA wont see the DNS query.

You cant simply use the public NAT IP from behind the ASA to connect to the server OR you would possibly have to make NAT configurations that would also affect your access rules between those interfaces.

I might have understood you incorrectly but it seemed like an issue I just described above.

- Jouni

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 12:56:07 GMT

Are you trying to reach the site from behind the ASA (on the LAN) using the public NATed IP of the server?

Yes

I guess you are trying to test connections or reach it using DNS name?

Yes, dns name. I'm trying to get from one inside interface, to a web page on another inside interface.

This is actually eerily similar to another post I have going currently, but I'm having trouble making the leap on the dns resolution/natting problem. So please bear with me.

My Amherst subinterface above is a complete /24 lan. Meaning it's a customer's office Active Directory domain....and the domain controlller, 192.168.143.10, which of course is all of the clients DNS server. Client dhcp addresses are 192.168.143.100-200.

The web server sits on the Franklin interface, 192.168.156.20 and has a public nat:

static (Franklin,outside) 74.11.x.x 192.168.156.20 netmask 255.255.255.255

What you're saying above is this nat should read:

static (Franklin,outside) 74.11.x.x 192.168.156.20 netmask 255.255.255.255 dns

However, since the dns query would never leave the 192.168.143.0 network, it's not going to work. Correct?

Re: NAT/routing issue from one subinterface to another

Jouni Forss — Wed, 21 Nov 2012 14:09:42 GMT

Hi,

When we have run into this problem the most common solution has been to do configuration change on the actual LAN DNS server.

I guess one option is also to do a Static NAT between the 2 LAN interfaces. In other words NAT the Web server LAN address to the public IP address towards the other LAN. In this case you have to take into consideration that all traffic from the LAN initiating the connection have to use the new public Static NAT IP address. It could therefore affect some LAN - to - LAN traffic (not VPN but actual LAN to LAN through the same ASA )

Example command would be (with made up IPs and names)

Static NAT to outside

static (lan1,outside) 1.2.3.4 10.10.10.10 netmask 255.255.255.255 dns

Static NAT between LANs

static (lan1,lan2) 1.2.3.4 10.10.10.10 netmask 255.255.255.255

I'm not sure would a Static Policy NAT be even better. To my understanding it would be done in the following way

access-list WEBSERVER-POLICY-NAT permit tcp host 10.10.10.10 www 10.10.20.0 255.255.255.0

static (lan1,lan2) 1.2.3.4 access-list WEBSERVER-POLICY-NAT

Where:

10.10.20.0/24 is the other LAN

I guess this should work too BUT I can't say for sure as I'm just writing this off my head. The above Static Policy NAT should only apply for the web/http traffic with the Web server, other traffic would use the actual IP address of the server when connecting from the other LAN.

I can't test this at the moment with an actual ASA so I can't confirm if the above configuration is possible or if it would work. The first option I mentioned should work but as I said it might cause problems between the LANs if the Web server has some other services towards the other LAN.

- Jouni

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 14:35:47 GMT

While I digest this, and thanks for all your help by the way, I'm wondering if your first thought isn't the right way to go on this:

When we have run into this problem the most common solution has been to do configuration change on the actual LAN DNS server.

what sort of creativity are you thinking there?

I'm thinking you mean an A record for the web server using the private IP, then an access list to allow HTTP from nameif Amherst->nameif Frankin?

NAT/routing issue from one subinterface to another

Jouni Forss — Wed, 21 Nov 2012 15:51:08 GMT

Hi,

I'm not sure how the change is configured as I don't handle any kind of IT in my work or anything related to server other than use them

The basic idea is just to do the same on the actual DNS server that would have been done on the ASA with the "dns" parameter. Basically a separate configuration that would tell any host/client requesting the IP address with DNS query to contact the local IP address and NOT the public IP address.

This would atleast eliminate any need to modify the ASA configurations by doing special rules there. I always aim to keep the firewall NAT configurations simple so they don't come back to haunt me later.

But I guess the DNS configuration on the DNS server might be a same sort of special setup that might cause problem while troubleshooting if the configuration isnt documented well etc.

- Jouni

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 16:18:46 GMT

Gotcha. I'm a jack of all trades master of none

I might investigate the local dns server a bit more, as I break something there it affects a 20 person office. I mess up the ASA, dozens of different customers feel the pain. And apparently no one knows what a lab is so I'm kind of stuck doing this right in production.

I can add an entry, even as a test a host file entry on a workstation, and a route now that i think about it, pointing to the Franklin web server at 192.168.156.20 instead of the public....and if i lowered the security level on Franklin then i should be able to traverse Amherst to Franklin....off the top of my head, still might need to work that out a little better..thanks though!

NAT/routing issue from one subinterface to another

Jouni Forss — Wed, 21 Nov 2012 16:26:15 GMT

Hi,

I feel your pain I'm kinda cautious testing things on our ASAs or FWSMs (A major screw up or run in with a bug in a software can affect up to 200 separate firewalls, though I guess that would be pretty rare occurence) Usually have an ASA5505 at hand though and some test Security Contexts.

I totally forgot to mention about testing this by changing configuration on the a single clients host file.

The "dns" parameter you can naturally also test (if you see any reason for it) by changing some test computers DNS server to public DNS for the tests.

Please let us know if get this settled and please mark the question answered/rate the answer if you feel its been of help. Always keeps the motivation up for everyone answering on these forums.

- Jouni

NAT/routing issue from one subinterface to another

Jouni Forss — Wed, 21 Nov 2012 16:30:02 GMT

Oh and by the way,

I think most ASAs (atleast models 5510 and above) should have available 2 Security Contexts for your use. So if you want to totally overhaul your ASA setup and get 2 virtual firewalls (one for production and one for tests) if would be possible.

It does naturally take some time/work and changing from single firewall to multiple mode (virtual Security Contexts) will eliminate some features from the ASA depending on your used software. One major example is loosing support for Client VPN. (L2L VPN work only with newest software)

- Jouni

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 17:48:10 GMT

Interesting that you mention that. It's a 5540 in single context mode, but I've noticed there were two contexts available and never paid much attention to it. But the ASA has a secondary and since it's single context i just always thought it was Active/Standby. I didnt set any of this up, i just support it now. But last night it failed over and now trying to figure out why. Then i notice it's Active/Active. But it can't be since Active/Active is only supported in multiple context mode. So i haven't quite made sense of that. But then i see the 2 security contexts you mention and it made me laugh. These forums far outweigh any book i've bought. The main reason we havent been multiple context is the tunnels, we have several dozen L2L. But multiple contexts make much more sense for this environment, considering we're only at 8.04, i didn't even know vpn was supported in newer code. Now completely off topic from where i started, but thanks again for the enlightenment!

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 17:54:48 GMT

Yes thank you, I have a bit of testing to do, fortunately with Thanksgiving i have a few days of downtime to work around. I'll post back my results, but yes i need to keep people such as yourself motivated, since i often find myself in unchartered waters. Love the challenge, but to easy to break things!

NAT/routing issue from one subinterface to another

n_schloemer — Wed, 21 Nov 2012 19:23:18 GMT

I would have to agree with JouniForss. Since the firewall is pushing the web trafic out the external interface and then attempting to hairpin back to the .156 interface, more than likely the ASA is going to natively look at the tcp syn-ack process and deny it. Do you have any debugs of the session traffic which you could post? Either way, I would suggest two different options. First could probably be the easiest for future troubleshooting or maintenance and that would be to add an A record on the local DNS server for the .143 subnet so it resolves to the .156 internal IP address of the web server. This is going to force the ASA to recognize the locally connected route of .156 as the destination and route the traffic accordingly. Along with this you will need to add the appropriate ACL's and NAT exempt statement to allow the routing. Something like this:

access-list AMHERST_ACCESS_IN ext permit tcp 192.168.146.0 255.255.255.0 192.168.156.0 255.255.255.0 eq http

nat (AMHERST,FRANKLIN) source static AMHERST_SUBNET AMHERST_SUBNET destintation static FRANKLIN_SUBNET FRANKLIN_SUBNET

The other option would be to add a class-map which matches the source traffic of Amherst to the destination of the webserver and add the policy-map which will allow tcp-state-bypass. You could add this rule to the default system service-policy or the interface. Below I have added a link that speaks about tcp-state-bypass.

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/conns_tcpstatebypass.pdf

NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 21:03:42 GMT

quick question, how would you debug the session traffic?

Re: NAT/routing issue from one subinterface to another

Jouni Forss — Wed, 21 Nov 2012 21:28:30 GMT

Hi,

If you can make the DNS change on the local LAN DNS server to point to the private IP address of the server instead of the public this should be almost enough.

Ofcourse you need to allow the traffic on the access-list in the source interface (unless you have some wider rule already allowing that)

The NAT setup between the interfaces depends on your ASA software. In a 8.3 and newer software you wouldnt need any NAT configurations for this traffic as you wont be doing any NAT and the traffic will be using the client/server actual IP address.

In software 8.2 and below there is a thing called "nat-control" which determines if a NAT is always required or never required. The default setting, if I remember right, is that the "nat-control" is not enabled. (Its not visible in the configuration in this case). Then again if "nat-control" is enabled it will show in the configuration just above the "global" configuration lines. I havent had to deal with this in ages so I might have gotten it completely the wrong way (which setting was default that is)

If you want to capture traffic for a connection you are testing you can do it either at the test computer itself, on the server, in the switches you might have or on the ASA (which I have used alot in recent months to get familiar with using it for troubleshooting)

Now if we presume that you want to capture only HTTP connections towards the server using (and return traffic) the ASA, you could do following configurations. I also presume you are connecting using the local IP of the server (even though it should work if access rules are OK and there is not some other NAT configuration in the way)

access-list WEB-SERVER-CAPTURE permit tcp host host

capture WEB-SERVER-CAPTURE type raw-data packet-length 1522 access-list WEB-SERVER-CAPTURE interface Amherst buffer 33500000 circular-buffer

You can check the amount of data captured by the above configuration by using command "show capture". The capture wont show in the "show run" output of ASA. Only with "show capture" command. If the ASA happens to boot the capture will be removed also.

"buffer" parameter sets the amount of memory used for storing data. I'm using almost the maximum amount ASA allows per capture.

"circular-buffer" tells the ASA to overwrite the old data if the memory amount set with "buffer" is exceeded. Otherwise the capture will stop when buffer is full. Access-list and Capture name dont have to match but I usually do it that way.

You can show the capture data with "show capture "

Though I would suggest using Wireshark software for opening the capture file. For this you need to copy the file from ASA to your computer with TFTP

copy /pcap capture:WEB-SERVER-CAPTURE tftp://x.x.x.x/filename.pcap

You can also use Wireshark for capturing traffic on the computer you install it on ofcourse (I guess you might be familiar with this already, personally I havent used it for a long time even if it is a very basic tool for networking)

To remove the capture use the command "no capture ". Notice though that this command will also delete the captured data from the ASA.

If you dont want to go quite so deep you can simple watch the ASA logs through ASDM, ASA CLI or better yet Syslog server. Logging level on ASA would need to be atleast "informational" so that you can see the messages related to connection forming and closing.

- Jouni

Re: NAT/routing issue from one subinterface to another

WStoffel1 — Wed, 21 Nov 2012 22:49:28 GMT

I was thinking wireshark and packet captures would be a last resort, maybe i'm at that point then as nothing seems to be working. But i just got a little hope...by the way i'm running asa804-k8.bin

I've made no dns changes locally yet but when i'm on a server on the Amherst lan, 192.168.143.10, and i try to ping the web server on the Franklin lan, 192.168.156.10, the requests time out.

i added, based on your above

Static NAT to outside

static (lan1,outside) 1.2.3.4 10.10.10.10 netmask 255.255.255.255 dns

static (Franklin,outside) 74.11.x.x 192.168.156.10 netmask 255.255.255.255 dns

Static NAT between LANs

static (lan1,lan2) 1.2.3.4 10.10.10.10 netmask 255.255.255.255

static (Franklin,amherst) 74.11.x.x 192.168.156.10 netmask 255.255.255.255

I have 2 constant pings running now and i see log file entries:

For the local addressing:

Nov 21 2012 17:30:18: %ASA-3-305006: portmap translation creation failed for icmp src amherst:192.168.143.10 dst Franklin:192.168.156.10 (type 8, code 0)

Likewise the ping to the Public url yields:

Nov 21 2012 17:31:34: %ASA-3-305006: portmap translation creation failed for icmp src amherst:192.168.143.10 dst Franklin:74.11.x.x (type 8, code 0)

I'm wondering if your lan1 and lan2 are matching up with my amherst and Franklin lans, respectively. Or do i have things backwards?

NAT/routing issue from one subinterface to another

Jouni Forss — Thu, 22 Nov 2012 06:32:01 GMT

Hi,

Seems to be some problem with NAT configurations then.

We still havent seen your configurations though so I can't say anything for sure.

What I would suggest would be to leave the server to outside Static NAT with the DNS parameter added and remove the Static NAT server to other lan to return to a normal situation.

I would first try the DNS changes before trying anything else. At the least use a test computer in the LAN where you need to connect from and use a public DNS server and see if you can connect then to the server by using DNS name.

You can also post the ASA configurations and remove the sensitive information if there is some.

Cisco explanation for your log message ID

305006

Error Message    %ASA-3-305006: {outbound static|identity|portmap|regular) translation 
creation failed for protocol src interface_name:source_address/source_port 
[(idfw_user)] dst interface_name:dest_address/dest_port [(idfw_user)]

Explanation A protocol (UDP, TCP, or ICMP) failed to create a translation through the ASA. The ASA does not allow packets through that are destined for network or broadcast addresses. The ASA provides this checking for addresses that are explicitly identified with static commands. For inbound traffic, the ASA denies translations for an IP address identified as a network or broadcast address.

The ASA does not apply PAT to all ICMP message types; it only applies PAT ICMP echo and echo-reply packets (types 8 and 0). Specifically, only ICMP echo or echo-reply packets create a PAT translation. As a result, when the other ICMP messages types are dropped, this message is generated.

The ASA uses the global IP address and mask from configured static commands to differentiate regular IP addresses from network or broadcast IP addresses. If the global IP address is a valid network address with a matching network mask, then the ASA does not create a translation for network or broadcast IP addresses with inbound packets.

For example:


static (inside,outside) 10.2.2.128 10.1.1.128 netmask 255.255.255.128


 
 
 
 The ASA responds to global address 10.2.2.128 as a network address and  to 10.2.2.255 as the broadcast address. Without an existing translation,  the ASA denies inbound packets destined for 10.2.2.128 or 10.2.2.255,  and logs this message. 

 
 When the suspected IP address is a host IP address, configure a separate static command with a host mask in front of the subnet static command (the first match rule for static commands). The following static commands cause the ASA to respond to 10.2.2.128 as a host address: 

 

static (inside,outside) 10.2.2.128 10.2.2.128 netmask 255.255.255.255 
static (inside,outside) 10.2.2.128 10.2.2.128 netmask 255.255.255.128


 
 
 
 The translation may be created by traffic started from the inside host  with the IP address in question. Because the ASA views a network or  broadcast IP address as a host IP address with an overlapped subnet  static configuration, the network address translation for both static commands must be the same. 

 
Recommended Action    None required.

- Jouni

Re: NAT/routing issue from one subinterface to another

WStoffel1 — Fri, 23 Nov 2012 15:43:28 GMT

Ok, posting my config is going to take some work, it's 1900 lines long and there's a lot of stuff i can't post publicly, but I'm afraid to make changes sense i may change something relevant. However i will work on that today.

But i had an interesting thing i just noticed. I got back on that server, having not changed anything since i left it Wednesday, and went to ping that website, it's coming back as the local private address. I can still get to the website externally so the web site is still up...meaning it's still using the public address elsewhere.

So something in this translation is working? The server is point to 4.2.2.2 for dns. And I have to say it's very odd to see a windows server responding back from a public DNS server with a private address.

Re: NAT/routing issue from one subinterface to another

WStoffel1 — Fri, 23 Nov 2012 16:41:10 GMT

Ok actually that wasn't so bad to go through. I removed all of my other customers subinterfaces and access lists, as well as the crypto and I think it's ok. Any questions on anything i changed just let me know. Also note, i left the real vlan and private subnet for Amherst and Franklin this time, so this is the actual config. I'm trying to get hosts on 192.168.133.0 to the webserver at 192.168.146.10. The webservers public address is network-object host x.x.122.45

Thanks again for all the help.