NAT issue on 8.4 but it was working fine on 8.2

[Hemant Sajwan]

Hi, I have following question:

We recently upgraded a firewall from 8.2(2) to 8.4(5). We have following NATs (not all shown below):

nat (Transit,Public) source static any any no-proxy-arp route-lookup

|

Some other NATs

|

object network obj-172.22.10.80

nat (DMZ_CastIron-MGMT,Public) static 206.16.250.202

|

Some other NATs

|

The problem that started happening after the upgrade was that people on the outside could not access 206.16.250.202 anymore. After troubleshooting, it was found that following NAT was getting hit:

nat (Transit,Public) source static any any no-proxy-arp route-lookup

Because of the above NAT, ASA was not translating 206.16.250.202 to 172.22.10.80 and was rather trying to send it back out through the Public interface (U-turning). Packet-tracer confirmed this:

ASA#packet-tracer input Public tcp 182.50.76.10 2345 206.16.250.202 443

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   0.0.0.0         0.0.0.0         Public

Phase: 2

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in   0.0.0.0         0.0.0.0         Public

Phase: 3

Type: ACCESS-LIST

Subtype:

Result: DROP

Config:

Implicit Rule

Additional Information:

Result:

input-interface: Public

input-status: up

input-line-status: up

output-interface: Public

output-status: up

output-line-status: up

Action: drop

Drop-reason: (acl-drop) Flow is denied by configured rule

The following line also confirmed it:

ASA(config)# nat (Transit,Public) 4 source static any any no-proxy-arp route-lookup

WARNING: This rule will match all incoming traffic on interface 'Public'.

Use 'unidirectional' option to apply the rule for outgoing traffic only.

To resolve the issue, we had to remove following NAT:

nat (Transit,Public) source static any any no-proxy-arp route-lookup

Thinking about it, it does look correct and logical that the above line was causing the destination IP address to NOT get NATed (because of any any) and ASA was performing a route lookup for 206.16.250.202 because of which it was being routed back out through the Public interface (and getting dropped there).

However, my question is why this problem did not happen when the ASA was on 8.2? On 8.2, the equivalent lines were:

nat (Transit) 0 access-list Transit_nat0_outbound

access-list Transit_nat0_outbound extended permit ip any any

static (DMZ_CastIron-MGMT,Public) 206.16.250.202 172.22.10.80

So, on 8.2, the static for 206.16.250.202 was getting hit and the nat-exempt never caused any issues. Any idea why this would work fine on 8.2 but not on 8.4?

[varrao]

Hi Hemant,

How are you doing!!!!!

A few reasons that I can see in here:

1. In the 8.2 nat exempt, it was for traffic from your transit interface to a lower security interface and unidirectional.

2. The static was for DMZ and Pubilc interface traffic only and a very specific one.

3. In post 8.3, the nat exempt got converted to a destination nat which started redirecting all the traffic from the WAN back to the default gateway. Since the manual nats are bidirectional (if unidirectional not used) as compared to nat exempt in 8.2 which is unidirectional.

So the issue lies with the 8.2 nat exempt, which had a very open ACL, whicvh when converted to 8.3 started causing a problem, since manual nat is always hit before the auto nat.

Thanks,
Varun Rao

[Jouni Forss]

Hi Varun

Just a quick question.

Could the above problematic configuration be replaced with the following command?

nat (Transit,Public) source static any any no-proxy-arp

And if YES then would it make any difference to the setup. (Not having the route-lookup) Would it still cause problems for the Static NAT under the "object network" ?

- Jouni

[Hemant Sajwan]

Varun, I am fine...thanks..

NAT-exempt has been bidirectional since PIX 6.x days and is so in 8.2 also. It is never unidirectional in any scenario.

So, my question is why it was not causing any issues in 8.2 but only in 8.4. The behaviour that it is showing in 8.4 is understandable and correct (i.e. packet coming from outside destined for 206.16.250.202  hitting the rule

"nat (Transit,Public) source static any any no-proxy-arp route-lookup", not translating 206.16.250.202, performing route-lookup for it and trying to send it out through Public interface). But my question is why this configuration was working in 8.2? In 8.2, why static was translating 206.16.250.202 to 172.22.10.80 instead of matching the nat-exempt rule first?

If anything is not clear, do let me know.

[Jouni Forss]

Hi,

Usually I think the "packet-tracer" should define which NAT rule the traffic is hitting if its hitting any NAT rule. If not hitting any configured rule there should be no NAT Phase in the "packet-tracer"

Could you share with us all the NAT configurations before the Object Network NAT configurations. With "show run nat" they would be the first to show before you start seeing lines with the "object network"

- Jouni

[Haitham Jaradat]

The problematic NAT command is working as expected, keep in mind that NAT statements are read in two directions, when traffic is coming from transit to public it will keep the source as it is that's not a problem and I believe that is why you put that statement in the first place.

However when traffic is being initiated from the public interface this is now a destination translation statement, and destination translation takes precedence in egress interface determination over route lookup, hence this statement will cause the destination IP not to be translated and also if you don't have "route-lookup" at the end of the statement it will cause all traffic to be sent through the transit interface.

You may want to put the keyword "unidirectional" at the end

Sent from Cisco Technical Support iPhone App

[Hemant Sajwan]

Haitham, what you said is absolutely correct. The behaviour exhibited in 8.4 is correct (i.e. packet coming from outside destined for 206.16.250.202  hitting the rule "nat  (Transit,Public) source static any any no-proxy-arp route-lookup", not  translating 206.16.250.202, performing route-lookup for it and trying to  send it out through Public interface). But then why static NAT in 8.2 was coming into picture to translate public IP to private IP whereas it is not happening in 8.4? Why, in 8.2, nat-exempt acl of any any was not taking precedence over static NAT?

If I am not explaining anything well, do let me know.

[Haitham Jaradat]

NAT0 in 8.2 is applied to the inside interface and is unidirectional (i.e. only traffic coming from the inside interface will match it) traffic coming from the outside interface will not match but if you you do a packet tracer it will be matched during the NAT Reverse Path verification checks.

In 8.4 identity NAT is applied bi-directionaly, hence the issue is happening, if you apply the unidirectional command it will become more like NAT0 in 8.2.

[Hemant Sajwan]

Haitham,

Are you saying nat-exempt (i.e. nat 0 with ACL) is undirectional only in 8.2 or in all codes before 8.3?

Here's a line from 8.2 command reference saying that it is bidirectional:

NAT exemption (nat 0 access-list command)—NAT exemption allows both translated and remote hosts to initiate connections

http://www.cisco.com/en/US/docs/security/asa/asa82/command/reference/no.html#wp1756533

And here's a line from 8.2 configuration guide saying the same:

NAT exemption exempts addresses from translation and allows both translated and remote hosts to initiate connections.

http://www.cisco.com/en/US/docs/security/asa/asa82/configuration/guide/nat_bypassing.html#wp1080803

[Jouni Forss]

Hi Hemant,

Following up on my original question,

Would it be possible to see your Section 1 NAT (Twice NAT , Manual NAT, etc whatever they are called nowadays ) configurations so I can go through them. I would be interested to go through this thing. Might even learn something new again.

I personally always avoid writing rules that are not specific and always write the ver 8.2 rules by hand to the 8.3 and after to make absolutely sure that everything works.

On a side note I also wonder the term "birectional" and "unidirectional" in this situation. To my understanding your 8.2 version of the command makes it possible to reach any IP address behind the Transit interface from behind any interface from any source network and this should work no matter which side is initiating the connection.

- Jouni

[Hemant Sajwan]

Hi Jouni,

Here are all the NATs:

nat (Transit,Public) source static any any no-proxy-arp route-lookup

nat (DMZ_ABC,Public) source static any any destination static R_Network R_Network no-proxy-arp route-lookup

nat (DMZ_ABC,Transit) source static any any destination static R_Network R_Network no-proxy-arp route-lookup

nat (DMZ_CastIron-MGMT,any) source static any any destination static R_Network R_Network no-proxy-arp route-lookup

!

object network 172.22.10.10

nat (DMZ_ABC,Public) static 206.16.250.196

object network obj-172.22.10.20

nat (DMZ_ABC,Public) static 206.16.250.200

object network obj-172.22.10.21

nat (DMZ_ABC,Public) static 206.16.250.201

object network 172.22.10.70

nat (DMZ_CastIron-MGMT,Public) static 206.16.250.197

object network obj-172.22.10.80

nat (DMZ_CastIron-MGMT,Public) static 206.16.250.202

object network obj-172.22.10.81

nat (DMZ_CastIron-MGMT,Public) static 206.16.250.203

=========

On a side note I also wonder the term "birectional" and "unidirectional"  in this situation. To my understanding your 8.2 version of the command  makes it possible to reach any IP address behind the Transit interface  from behind any interface from any source network and this should work  no matter which side is initiating the connection.

=========

  -- Yes, you are correct above. However, I am wondering how the statics in 8.2 were working fine (because they shouldn't have, given the "any any" nat-exempt ACL). In 8.4, the migrated statics stopped working because of "nat (Transit,Public) source static any any no-proxy-arp route-lookup".

[Jouni Forss]

Hi,

Thank you for sharing the configuration.

As you say it does seem that the 8.4 configuration you mentioned would be the only possible rule that could be causing your problems. I think I will still Lab this on my home ASA perhaps later today.

I'm still a bit confused to all the NAT changes/additions that have changed during the 8.4(1) - 8.4(5). I currently have for example 8.4(3) which lack some options that you have in 8.4(5) (To my knowledge atleast)

What I was thinking to try with this setup would be the following

• To my understanding when you compare 8.4(1 or 2) and 8.4(5) there should be a major change between them as to how you let the NAT operate in deciding the ingress and eggress interfaces. Since you have the "route-lookup" I would imagine that routing takes precedence of the source and destination interfaces configured in the "nat (source,destination)" portion of the NAT configuration.
• Because of the above, I was wondering if it would be possible to change the NAT command to NOT include the "route-lookup" and therefore let the NAT decide to which interfaces the NAT applies to. In this case I think it should mean that it will only apply to traffic with Transit and Public either as source or destination and thefore not handle the traffic as it did after the migration. (I will have to check if it actually is possible to disable the route-lookup)

I'm not 100% sure of the above before I lab it but I think I will to get some clarification to this myself also.

As to why this was not a problem with 8.2 and is now problem with 8.4 I think it just simply related to the major NAT operation changes. I do know that many people have faced problems after automatic migration of NAT rules between 8.2 and 8.4. They have also found out that there is a lot of "useless" NAT configuration after the migration.

Only thing at the moment before any of my own tests I would have to guess that the "route-lookup" parameter in the migrated NAT configuration is the reason why same traffic now applies differently in your new software. Routing table is deciding the ingress/eggress interfaces instead of the actual NAT source/destination ports.

So take what I said above with the consideration that I might be completely wrong.

I will get back to you when I have had the chance to Lab this a bit and cram the information to my head

EDIT: Naturally if you have the chance to test this yourself, let us know were you able to change the configuration and did it have any effect on the situation.

- Jouni

[Hemant Sajwan]

Jouni,

What you are saying about route-lookup keyword is correct. Because of route-lookup keyword, the ASA performs a route lookup for the destination i.e. 206.16.250.202 and tries to send it out through the public interface (where it gets dropped). However, even if we remove the route-lookup keyword, I don't think this would resolve the issue because now ASA would determine the destination interface based on NAT and not on the basis of route lookup. So, now, it would try to send the packet to the Transit interface (because "nat (Transit,Public) source static any any no-proxy-arp" would still match everything). So, the migrated statics below it would still not be matched.

[Jouni Forss]

Hi,

So I had a go with this setup (a bit simplified ofcourse) and tested connections with different configurations.

For some reason I got different type of "packet-tracer" outputs but nevertheless the situation was the same that you ran into.

So heres the things I witnessed

• The configuration "route-lookup" wether enabled or disabled doesnt change the situation
  • ASA gives warning message in both cases about the problem this NAT configuration will cause
• Problem could be avoided by configuring the "unidirectional" in the end of the NAT configuration
  • As the Cisco employees have stated here
• Also it would seem to me that the other options is to change the NAT configuration abit by using "object-group" to define all the networks behind the Transit interface. Wether this is possible in your case I cant say
  • After configuring a "object-group network TRANSIT-NETWORKS" with all the networks routed towards that interfaces and the actual connected interface if needed, the configuration could look like this
    • nat (Transit,Public) source static TRANSIT-NETWORKS TRANSIT-NETWORKS no-proxy-arp route-lookup

I guess this answer didnt really explain why this is happening after the automatic migration of the NAT configurations (when the NAT configurations and their operation is supposed to be "identical"). But as I said I wouldnt trust the ASA to do the configurations for me both for the reasons that it can cause problems and that it creates quite confusing CLI format of the configurations in some certain situations.

Hopefully the information has been helpfull in someway atleast.

- Jouni

[Hemant Sajwan]

Thanks a lot Jouni for devoting your time for this. Whatever you said is correct and are some possible workarounds. However, the question still remains - How the nat-exempt ACL of any any was not causing this issue in 8.2 but it broke the migrated statics after upgrading to 8.4? May be, I will open a case with TAC to further look into it.