topic ASA 5510 subinterfaces+NAT nightmare in Network Security

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Tue, 12 Mar 2019 00:33:48 GMT

Hi guys,

We have an ASA 5510 firewall and in our DMZ we have a server listening on port 443. For this server, we already have a public IP address on the ISP' router. All the traffic to this IP address on port 443 is natted to the ASA's outside interface on port 443. So far, everything is working fine.

We'd like to add another server listening on this same port (443). To do so, we have obtained from our ISP a second public IP address for the second server. As the two servers must be listening on the same port, we have configured a subinterface on the outside interface of the firewall and the ISP has done so as well. We can ping each other on this VLAN, so there is no connectivity issue between us. The ISP's router is natting (and patting) the traffic on the second public IP address to the IP address on the subinterface. We have added an access rule to allow incoming traffic on this subinterface and a NAT rule to send this traffic to the server in the DMZ on port 443.

But despite the access rule, https traffic on the subinterface is being denied by the implicit rule on this interface as shown in the logs below

Why is the explicit rule on the interface is not being applied? I'm hopeless, I need your help

ASA 5510 subinterfaces+NAT nightmare

Jouni Forss — Thu, 06 Dec 2012 17:00:00 GMT

Hi,

Can you check that the source and destination interfaces "security-level" settings isnt identical. Think I just had something like this I was wondering as I had copy/pasted one interface configuration without changing the "security-level"

If they are try adding commands line

same-security-traffic permit intra-interface

same-security-traffic permit inter-interface

or simply change the "security-level" to not match.

- Jouni

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Thu, 06 Dec 2012 22:33:42 GMT

Hi Jouni,

Thank you very much for your quick reply. As it's night here in Europe, I'll try this solution tomorrow and let you know if it works. Thanks again for your consideration

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Fri, 07 Dec 2012 08:36:26 GMT

Hi Jouni,

I have changed the security level on the subinterface to a different value from those of the physical outside interface and the DMZ but the traffic is still being denied. In fact, traffic is being dropped as it hits the subinterface from the ISP's router. It's never routed from one of the ASA's interfaces to another.

Has anyone come across this kind of issue?

ASA 5510 subinterfaces+NAT nightmare

Jouni Forss — Fri, 07 Dec 2012 09:45:21 GMT

Hi,

The Syslog message description from Cisco says the following

710003

Error Message    %ASA-3-710003: {TCP|UDP} access denied by ACL from 
source_IP/source_port to interface_name:dest_IP/service

Explanation The ASA denied an attempt to connect to the interface service. For example, the ASA received an SNMP request from an unauthorized SNMP management station. If this message appears frequently, it can indicate an attack.

For example:

%ASA-3-710003: UDP access denied by ACL from 95.1.1.14/5000 to outside:95.1.1.13/1005

Recommended Action Use the show run http, show run ssh, or show run telnet commands to verify that the ASA is configured to permit the service access from the host or network.

I'm not totally sure but usually when you see a Deny log message that doesnt specify any ACL thats blocking the connection attempt the problem is usually related to some other ASA configuration.

Also I find it quite strange that when the ISP has given you another public IP address that you would have to configure a new subinterface for it on ASA. So I'm kinda doubtful of that thing alone already. Add to that the fact that the ISP is doing some sort of port forwarding NAT before your ASA. Seems quite complicated scenario for 2 public IPs.

Without knowing the actual ASA configuration and the ISP setup I'm not sure how much I can really help with this situation.

- Jouni

ASA 5510 subinterfaces+NAT nightmare

Jouni Forss — Fri, 07 Dec 2012 13:08:52 GMT

Hi,

Never seen a setup like this But guess theres first time for everything

If I looked correctly the below configuraitons lines should be the ones you are referring to regarding to the NAT

First one would be the one thats working
Second would be the one you are trying to get working

static (Internal,Outside) tcp interface https Host_MAIL01 https netmask 255.255.255.255

static (Outside_3,DMZ) Host_E2CWeb  access-list Outside_3_nat_static

The first one you have done as basic Portforward configuration using the "Outside" interface IP. The second one is a Static Policy NAT which to my understanding is supposed to NAT traffic coming from certain outside host to the Outside_3 interface IP. The source host would be NATed to a IP address which is part of the DMZ network.

Is there a specific reason you have not done the configuration the same way as the last Port Forward configuration for port TCP/443?

I mean like

static (DMZ,Outside_3) tcp interface https https netmask 255.255.255.255

Still to my eye this setup is looking a really complex thing to manage when the ISP could simply change the setup so that you can actually use the public IPs directly on your firewall and avoid all the special setups.

What kind of device is there in front of the firewall at the moment?

- Jouni

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Fri, 07 Dec 2012 13:44:25 GMT

Hi,

Thanks again for your quick reply. In fact the configuration was done through the ASDM and I followed what's in the doc to yield this configuration. I can change the NAT rule to have exactly the same thing as on the physical interface but I am not sure it'll change anything as this is a nat rule and it comes far bahind the access rule, which is the rule being applied here. Our current contract with the ISP only allow us the use of some services on their public IP addresses, not all the services. So they can only forward those services to us. I don't think I have the power to change this, unfortunately. But when you look at the ACLs on the the Outside and Outside_3 interfaces, what's wrong with that? Why is the traffic to Outside_3 not being granted access?

ASA 5510 subinterfaces+NAT nightmare

Jouni Forss — Fri, 07 Dec 2012 14:49:57 GMT

Hi,

Your current new Static Policy NAT is configured to do the following

When the host 81.247.191.241 connects to host 192.168.3.2 (interface) --> Translate host IP 81.247.191.241 to host IP 172.16.2.5

I would first try to change the NAT configuration to match the way the previous one is configured and try again with the same ACL rules

So maybe try to remove the current NAT rule and replace it with the below one and test connections

static (DMZ,Outside_3) tcp interface https https netmask 255.255.255.255

You could also test it with the "packet-tracer" command in CLI and copy the output here

packet-tracer input Outside_3 tcp 1.1.1.1 1025 192.168.3.2 443

If it has no effect you can just return back to the old one.

- Jouni

ASA 5510 subinterfaces+NAT nightmare

Jouni Forss — Fri, 07 Dec 2012 15:31:37 GMT

Hi,

Regarding the log message, Cisco document says the following

305013

Error Message    %ASA-5-305013: Asymmetric NAT rules matched for forward and reverse 
flows; Connection protocol src interface_name:source_address/source_port dst 
interface_name:dst_address/dst_port denied due to NAT reverse path failure.

Explanation An attempt to connect to a mapped host using its actual address was rejected.

Recommended Action When not on the same interface as the host using NAT, use the mapped address instead of the actual address to connect to the host. In addition, enable the inspect command if the application embeds the IP address.

The above explanation/recomendation text lead to believe that the connect was attempted using the actual IP address of the server and not the mapped/NAT address. Hmm....

And just to confirm something

Is your situation the following?

IPS is using 2 public IP addresses on their network devices
Both of the 2 public IP addresses have been configured the forward the port TCP/443 in the following way
- x.x.x.x/443 -> 192.168.0.2/443
- y.y.y.y/443 -> 192.168.3.2/443

If the above is true I would perhaps even try something else (but you might have to save the current configuration or atleast take not what you have changed if you want to revert back)

What I would suggest trying (if the above situation regarding the ISP is correct)

Remove all the Subinterfaces
- Ethernet0/0.1
- Ethernet0/0.2
- Ethernet0/0.3
Configure a new Port Forward in the following way
- static (DMZ,Outside) tcp 192.168.0.3 443 netmask 255.255.255.255
Configure the "Outside" interface ACL to allow connections to the IP address 192.168.0.3
Ask your ISP to forward the new public IP addresses port TCP/443 to the new IP address configured in the NAT (192.168.0.3)
Change any other configurations that might be needed between yours and the ISPs devices

- Jouni

ASA 5510 subinterfaces+NAT nightmare

Muhammed Safwan — Fri, 07 Dec 2012 15:52:56 GMT

Aziza,

Seems to be asymmetric nat is due to below statements.

access-list DMZ_nat0_outbound extended permit ip host Host_E2CWeb any

nat (DMZ) 0 access-list DMZ_nat0_outbound

Any chance to remove the above ACL temperory and give a try?

With Regards,

Safwan

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Fri, 07 Dec 2012 15:54:40 GMT

Hi Jouni,

I guess we're close to the solution. Yes we'd like to achieve this situation :

x.x.x.x/443 -> 192.168.0.2/443
y.y.y.y/443 -> 192.168.3.2/443

If it works, then we'll add other services on the other subinterfaces. So we can leave those, as there is no configuration on them. The ISP techs have done the appropriate changes because traffic is hitting the right interface on the right port.

I'm on holiday on Monday. I'll try this on Tuesday and let you know.

It's very kind of you helping me sort out this issue.

ASA 5510 subinterfaces+NAT nightmare

Jouni Forss — Fri, 07 Dec 2012 15:57:00 GMT

Hi,

The NAT0 that Safwan mentioned would seem to be the cause

Didnt even notice it myself. Better try that first

- Jouni

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Fri, 07 Dec 2012 16:00:51 GMT

Hi guys,

I'll give it a try on Tuesday and get back to you.

You're so helpful, thanks for your effort

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Tue, 11 Dec 2012 13:46:37 GMT

Hi guys,

It works now. It was a routing problem. We needed to add another static route on the interface Outside_3 to allow outgoing traffic to be routed on the appropriate interface. Thank you so much for your help, you are my heros

ASA 5510 subinterfaces+NAT nightmare

Peter Koltl — Tue, 11 Dec 2012 22:09:43 GMT

I'm afraid your problems are still just starting due to bad design and I don't think this will be a working solution. What are your routes at present?

Suppose a client from 81.247.191.241 or any other public address sends requests to your webserver. Then it sends traffic to Host_Spamfilter or Host_SFTP. How does the firewall route the return traffic? The routing is destination-based (slightly changed in 8.3 by route-lookup command) so the packets will choose between Outside and Outside_3 based on the routing table. Unless the ISP router handles this by NATing the client source addresses per interface, this causes asymmetric traffic, packet drops or connection failures.

The simple design without contradictions would be using a single outside interface utilizing multiple outside private addresses from the same subnet (192.168.0.x) and having a static PAT for each published service as JouniForss suggested.

ASA 5510 subinterfaces+NAT nightmare

Peter Koltl — Tue, 11 Dec 2012 22:32:16 GMT

Actually, this is my assumption as I haven't encountered a similar setup yet. However, ASA 8.3 documentation states that

"ASA uses the NAT configuration to determine the egress interface. (8.3(1) through 8.4(1)) "

So the interface the translated packet will use to leave the firewall will be determined by your NAT rule destination interface, and NOT THE ROUTING TABLE.

But the 8.2 docs don't mention this topic, the ASA may use a similar method. This is what may be saving you even with this setup and cause things to work. You will need extreme caution with the route-lookup keyword when you switch to a newer software version. I would still suggest migrating to the simple setup.

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Wed, 12 Dec 2012 10:22:02 GMT

Hi Peter,

Thank you for your post. In fact, it was the solution. The outgoing traffic is routed according to the routing table. The ISP router is the default gateway for all the outgoing routes. So, if the traffic is for Host_SFTP or Host_Spamfilter, it's routed back according to the route configured for the appropriate interface, namely Outside for Host_SFTP and Host_Spamfilter. On the Outside interface, there's a default route with outgoing interface Outside and next hop 192.168.0.3 (the ISP's router side of the physical link). And for Host_E2CWeb, on the Outside_3, there's a default route with next hop 192.168.2.1, the ISP's router side of the subinterface. Everything is working so I think this is a possible correct design. Maybe there are others but this one is working fine. The goal by configuring subinterfaces was to separate different traffics on different VLANs. I went even further by configuring subinterfaces for traffic bound to Host_SFTP and Host_Spamfilter. I can access open services on those servers from the inside and from the outside and no others services.

Thank you to all of you for your contributions, you are just wonderful!

ASA 5510 subinterfaces+NAT nightmare

Peter Koltl — Wed, 12 Dec 2012 22:50:33 GMT

OK, I'm glad that it works but don't think that either default route belong to Host_Spamfilter or Host_E2CWeb. Just copy here your default routes: they contain next hop and interface but no referrals to inside zones or servers. Routes are for packets and they don't contain server-based dependencies. ASA routing engine does NOT match the reply packets to leave the box on the same outside interface where the requests came. It is simply destination-based. That is, an outbound request packet (outgoing HTTP request from your server or desktop) may have problems on which route to choose. I still assume the undocumented NAT preference over routing table lets the outgoing reply packets to find their way which I regard as sort of luck.

ASA 5510 subinterfaces+NAT nightmare

Leader1980 — Wed, 12 Dec 2012 23:22:40 GMT

There's no reference to inside zones. What I meant is that each service has a dedicated outside subinterface for the incoming traffic and the outgoing traffic refers to the subinterface as the egress interface and the ISP's router subinterface as the next hop. There is no matter of luck, because if it just doesn't work if I don't correctly configure default routes per subinterfaces. Subinterfaces are on different VLANs so the don't "see" each others. I don't see another way to configure default routes, apart from doing it by subinterface