ASA 5510 subinterfaces+NAT nightmare

Leader1980 · ‎12-06-2012

Hi guys,

We have an ASA 5510 firewall and in our DMZ we have a server listening on port 443. For this server, we already have a public IP address on the ISP' router. All the traffic to this IP address on port 443 is natted to the ASA's outside interface on port 443. So far, everything is working fine.

We'd like to add another server listening on this same port (443). To do so, we have obtained from our ISP a second public IP address for the second server. As the two servers must be listening on the same port, we have configured a subinterface on the outside interface of the firewall and the ISP has done so as well. We can ping each other on this VLAN, so there is no connectivity issue between us. The ISP's router is natting (and patting) the traffic on the second public IP address to the IP address on the subinterface. We have added an access rule to allow incoming traffic on this subinterface and a NAT rule to send this traffic to the server in the DMZ on port 443.

But despite the access rule, https traffic on the subinterface is being denied by the implicit rule on this interface as shown in the logs below

Why is the explicit rule on the interface is not being applied? I'm hopeless, I need your help

Jouni Forss · ‎12-06-2012

Hi,

Can you check that the source and destination interfaces "security-level" settings isnt identical. Think I just had something like this I was wondering as I had copy/pasted one interface configuration without changing the "security-level"

If they are try adding commands line

same-security-traffic permit intra-interface

same-security-traffic permit inter-interface

or simply change the "security-level" to not match.

- Jouni

Leader1980 · ‎12-06-2012

Hi Jouni,

Thank you very much for your quick reply. As it's night here in Europe, I'll try this solution tomorrow and let you know if it works. Thanks again for your consideration

Leader1980 · ‎12-07-2012

Hi Jouni,

I have changed the security level on the subinterface to a different value from those of the physical outside interface and the DMZ but the traffic is still being denied. In fact, traffic is being dropped as it hits the subinterface from the ISP's router. It's never routed from one of the ASA's interfaces to another.

Has anyone come across this kind of issue?

Jouni Forss · ‎12-07-2012

Hi,

The Syslog message description from Cisco says the following

710003

Error Message    %ASA-3-710003: {TCP|UDP} access denied by ACL from 
source_IP/source_port to interface_name:dest_IP/service

Explanation The ASA denied an attempt to connect to the interface service. For example, the ASA received an SNMP request from an unauthorized SNMP management station. If this message appears frequently, it can indicate an attack.

For example:

%ASA-3-710003: UDP access denied by ACL from 95.1.1.14/5000 to outside:95.1.1.13/1005

Recommended Action Use the show run http, show run ssh, or show run telnet commands to verify that the ASA is configured to permit the service access from the host or network.

I'm not totally sure but usually when you see a Deny log message that doesnt specify any ACL thats blocking the connection attempt the problem is usually related to some other ASA configuration.

Also I find it quite strange that when the ISP has given you another public IP address that you would have to configure a new subinterface for it on ASA. So I'm kinda doubtful of that thing alone already. Add to that the fact that the ISP is doing some sort of port forwarding NAT before your ASA. Seems quite complicated scenario for 2 public IPs.

Without knowing the actual ASA configuration and the ISP setup I'm not sure how much I can really help with this situation.

- Jouni

Jouni Forss · ‎12-07-2012

Hi,

Never seen a setup like this But guess theres first time for everything

If I looked correctly the below configuraitons lines should be the ones you are referring to regarding to the NAT

First one would be the one thats working
Second would be the one you are trying to get working

static (Internal,Outside) tcp interface https Host_MAIL01 https netmask 255.255.255.255

static (Outside_3,DMZ) Host_E2CWeb  access-list Outside_3_nat_static

The first one you have done as basic Portforward configuration using the "Outside" interface IP. The second one is a Static Policy NAT which to my understanding is supposed to NAT traffic coming from certain outside host to the Outside_3 interface IP. The source host would be NATed to a IP address which is part of the DMZ network.

Is there a specific reason you have not done the configuration the same way as the last Port Forward configuration for port TCP/443?

I mean like

static (DMZ,Outside_3) tcp interface https https netmask 255.255.255.255

Still to my eye this setup is looking a really complex thing to manage when the ISP could simply change the setup so that you can actually use the public IPs directly on your firewall and avoid all the special setups.

What kind of device is there in front of the firewall at the moment?

- Jouni

Leader1980 · ‎12-07-2012

Hi,

Thanks again for your quick reply. In fact the configuration was done through the ASDM and I followed what's in the doc to yield this configuration. I can change the NAT rule to have exactly the same thing as on the physical interface but I am not sure it'll change anything as this is a nat rule and it comes far bahind the access rule, which is the rule being applied here. Our current contract with the ISP only allow us the use of some services on their public IP addresses, not all the services. So they can only forward those services to us. I don't think I have the power to change this, unfortunately. But when you look at the ACLs on the the Outside and Outside_3 interfaces, what's wrong with that? Why is the traffic to Outside_3 not being granted access?

Jouni Forss · ‎12-07-2012

Hi,

Your current new Static Policy NAT is configured to do the following

When the host 81.247.191.241 connects to host 192.168.3.2 (interface) --> Translate host IP 81.247.191.241 to host IP 172.16.2.5

I would first try to change the NAT configuration to match the way the previous one is configured and try again with the same ACL rules

So maybe try to remove the current NAT rule and replace it with the below one and test connections

static (DMZ,Outside_3) tcp interface https https netmask 255.255.255.255

You could also test it with the "packet-tracer" command in CLI and copy the output here

packet-tracer input Outside_3 tcp 1.1.1.1 1025 192.168.3.2 443

If it has no effect you can just return back to the old one.

- Jouni

Jouni Forss · ‎12-07-2012

Hi,

Regarding the log message, Cisco document says the following

305013

Error Message    %ASA-5-305013: Asymmetric NAT rules matched for forward and reverse 
flows; Connection protocol src interface_name:source_address/source_port dst 
interface_name:dst_address/dst_port denied due to NAT reverse path failure.

Explanation An attempt to connect to a mapped host using its actual address was rejected.

Recommended Action When not on the same interface as the host using NAT, use the mapped address instead of the actual address to connect to the host. In addition, enable the inspect command if the application embeds the IP address.

The above explanation/recomendation text lead to believe that the connect was attempted using the actual IP address of the server and not the mapped/NAT address. Hmm....

And just to confirm something

Is your situation the following?

IPS is using 2 public IP addresses on their network devices
Both of the 2 public IP addresses have been configured the forward the port TCP/443 in the following way
- x.x.x.x/443 -> 192.168.0.2/443
- y.y.y.y/443 -> 192.168.3.2/443

If the above is true I would perhaps even try something else (but you might have to save the current configuration or atleast take not what you have changed if you want to revert back)

What I would suggest trying (if the above situation regarding the ISP is correct)

Remove all the Subinterfaces
- Ethernet0/0.1
- Ethernet0/0.2
- Ethernet0/0.3
Configure a new Port Forward in the following way
- static (DMZ,Outside) tcp 192.168.0.3 443 netmask 255.255.255.255
Configure the "Outside" interface ACL to allow connections to the IP address 192.168.0.3
Ask your ISP to forward the new public IP addresses port TCP/443 to the new IP address configured in the NAT (192.168.0.3)
Change any other configurations that might be needed between yours and the ISPs devices

- Jouni

Leader1980 · ‎12-07-2012

Hi Jouni,

I guess we're close to the solution. Yes we'd like to achieve this situation :

x.x.x.x/443 -> 192.168.0.2/443
y.y.y.y/443 -> 192.168.3.2/443

If it works, then we'll add other services on the other subinterfaces. So we can leave those, as there is no configuration on them. The ISP techs have done the appropriate changes because traffic is hitting the right interface on the right port.

I'm on holiday on Monday. I'll try this on Tuesday and let you know.

It's very kind of you helping me sort out this issue.

Muhammed Safwan · ‎12-07-2012

Aziza,

Seems to be asymmetric nat is due to below statements.

access-list DMZ_nat0_outbound extended permit ip host Host_E2CWeb any

nat (DMZ) 0 access-list DMZ_nat0_outbound

Any chance to remove the above ACL temperory and give a try?

With Regards,

Safwan

Jouni Forss · ‎12-07-2012

Hi,

The NAT0 that Safwan mentioned would seem to be the cause

Didnt even notice it myself. Better try that first

- Jouni

Leader1980 · ‎12-07-2012

Hi guys,

I'll give it a try on Tuesday and get back to you.

You're so helpful, thanks for your effort

Leader1980 · ‎12-11-2012

Hi guys,

It works now. It was a routing problem. We needed to add another static route on the interface Outside_3 to allow outgoing traffic to be routed on the appropriate interface. Thank you so much for your help, you are my heros

Peter Koltl · ‎12-11-2012

I'm afraid your problems are still just starting due to bad design and I don't think this will be a working solution. What are your routes at present?

Suppose a client from 81.247.191.241 or any other public address sends requests to your webserver. Then it sends traffic to Host_Spamfilter or Host_SFTP. How does the firewall route the return traffic? The routing is destination-based (slightly changed in 8.3 by route-lookup command) so the packets will choose between Outside and Outside_3 based on the routing table. Unless the ISP router handles this by NATing the client source addresses per interface, this causes asymmetric traffic, packet drops or connection failures.

The simple design without contradictions would be using a single outside interface utilizing multiple outside private addresses from the same subnet (192.168.0.x) and having a static PAT for each published service as JouniForss suggested.