ASA Can perform 'policy NAT' but VPNs not work

tholmes · ‎01-30-2014

Hello,

I've got a tough one, so any help would be greatly appreciated.

I'm implementing a new network for hosting customers. Each new customer will have their own public routable subnet.

The ASA5525X firewall needs to have multiple default static routes to the Internet via each public next hop IP address

route hostcust01-outside 0.0.0.0 0.0.0.0 195.119.10.14 1

route hostcust02-outside 0.0.0.0 0.0.0.0 195.119.10.22 2 (I've hidden the real IP addresses)

ah the ASA doesn't support policy based routing I hear you say, but it does if these NAT statements are added

object network hostcust01-outside-PAT

host 195.119.10.13

object network hostcust01-outside-PAT

host 195.119.10.21

object network hostcust01-inside

subnet 10.1.201.0 255.255.255.0

object network hostcust02-inside

subnet 10.1.202.0 255.255.255.0

nat (hostcust01-outside,hostcust01-inside) after-auto source static any any destination static hostcust01-outside-PAT hostcust01-inside

nat (hostcust02-outside,hostcust02-inside) after-auto source static any any destination static hostcust02-outside-PAT hostcust02-inside

This works very well, packets from 10.1.201.x egress via host01-outside interface , and packets from 10.1.202.X egress using host02-outside interface

It performs a kind of inverse NAT translation, outside,inside, to ensure the correct egress interface is used in the NAT table and overrides any route lookup (I think)

The problem comes when adding a VPN identify NAT

nat (hostcust02-inside,hostcust02-outside) source static host-cust02-inside host-cust02-inside destination static remote_192.168.101.0_24 remote_192.168.101.0_24 no-proxy-arp route-lookup

This VPN identity NAT is higher in the list than the other after-autos but the ASA doesn't want to use it, instead uses that other NAT

Packet tracer shows this

NGD-HE-FW1# packet-tracer input hostcust02-inside icmp 10.1.202.10 1 8 192.168.101.10 detail

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (hostcust02-outside,hostcust02-inside) after-auto source static any any destination static hostcust02-outside-PAT hostcust02-inside-nets

Additional Information:

NAT divert to egress interface hostcust02-outside

Untranslate 192.168.101.10/0 to 192.168.101.10/0

(CAN I STOP IT USING THIS NAT EVEN THOUGH ITS THE LOWEST ONE IN THE NAT LIST ???)

Phase: 2

Type: ACCESS-LIST

Subtype: log

Result: ALLOW

Config:

access-group hostcust02-inside in interface hostcust02-inside

access-list hostcust02-inside extended permit ip any any

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff379c9ed0, priority=13, domain=permit, deny=false

hits=3390, user_data=0x7fff2fee9d80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=hostcust02-inside, output_ifc=any

Phase: 3

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff379e64d0, priority=0, domain=inspect-ip-options, deny=true

hits=3429, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0

src ip/id=0.0.0.0, mask=0.0.0.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=hostcust02-inside, output_ifc=any

Phase: 4

Type: INSPECT

Subtype: np-inspect

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff379e5ef0, priority=66, domain=inspect-icmp-error, deny=false

hits=1557, user_data=0x7fff379b6db0, cs_id=0x0, use_real_addr, flags=0x0, protocol=1

src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0

dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, dscp=0x0

input_ifc=hostcust02-inside, output_ifc=any

Phase: 5

Type: DEBUG-ICMP

Subtype:

Result: ALLOW

Config:

Additional Information:

Forward Flow based lookup yields rule:

in id=0x7fff3813f510, priority=13, domain=debug-icmp-trace, deny=false

hits=1557, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=1

src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0

dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, dscp=0x0

input_ifc=hostcust02-inside, output_ifc=any

Phase: 6

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (hostcust02-inside,hostcust02-outside) source static host-cust02-internal-10.1.202.0 host-cust02-internal-10.1.202.0 destination static cust-02-remote01_192.168.101.0_24 cust-02-remote01_192.168.101.0_24 no-proxy-arp route-lookup

Additional Information:

Static translate 10.1.202.10/0 to 195.119.10.13/0 ........................(PAT USED BUT NOT WANTED FOR VPN)

Forward Flow based lookup yields rule:

in id=0x7fff388b7230, priority=6, domain=nat, deny=false

hits=1, user_data=0x7fff38001e20, cs_id=0x0, use_real_addr, flags=0x0, protocol=0

src ip/id=10.1.202.0, mask=255.255.255.0, port=0

dst ip/id=192.168.101.0, mask=255.255.255.0, port=0, dscp=0x0

input_ifc=hostcust02-inside, output_ifc=hostcust02-outside

Phase: 7

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (hostcust02-outside,hostcust02-inside) after-auto source static any any destination static hostcust02-outside-PAT hostcust02-inside-nets

Additional Information:

Forward Flow based lookup yields rule:

out id=0x7fff37925ce0, priority=6, domain=nat-reverse, deny=false

hits=12, user_data=0x7fff3841fc00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0

src ip/id=10.1.202.0, mask=255.255.255.0, port=0

dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, dscp=0x0

input_ifc=hostcust02-inside, output_ifc=hostcust02-outside

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 12124, packet dispatched to next module

Module information for forward flow ...

snp_fp_tracer_drop

snp_fp_inspect_ip_options

snp_fp_translate

snp_fp_dbg_icmp

snp_fp_adjacency

snp_fp_fragment

snp_ifc_stat

Module information for reverse flow ...

Result:

input-interface: hostcust02-inside

input-status: up

input-line-status: up

output-interface: hostcust02-outside ............ CORRECT INTERFACE USED BUT NO VPN TUNNEL OR EVEN CRYPTO ACL USED !!

output-status: up

output-line-status: up

Action: allow

Any ideas, I didn't design it, I'm only trying to get it to work.

Regards Tony

Jouni Forss · ‎01-30-2014

Hi,

There was a similiar question on the Firewall section but was related to having 3 ISP links and forwarding different LAN networks out different ISPs.

So to copy/paste some of the things I posted there then I guess you could try changing the existing NAT configurations a bit and seeing if that changes how the configurations match.

object network ANY-0.0.0.0-1

subnet 0.0.0.0 128.0.0.0

object network ANY-128.0.0.0-1

subnet 128.0.0.0 128.0.0.0

object-group network ALL

network-object object ANY-0.0.0.0-1

network-object object ANY-128.0.0.0-1

nat (hostcust01-inside,hostcust01-outside) after-auto source dynamic hostcust01-inside hostcust01-outside-PAT destination static ALL ALL

nat (hostcust02-inside,hostcust02-outside) after-auto source dynamic hostcust02-inside hostcust02-outside-PAT destination static ALL ALL

And use the same NAT0 configuration for VPN.

I am not sure if your current environment is in production use yet. The above configuration essentially says that any traffic from those source networks will get Dynamic PAT to the specified public IP address and forwarded through the specified external interface no matter what the destination IP address is. Higher priority NAT configurations should override these though.

The discussion I just posted can be found here:

https://supportforums.cisco.com/thread/2264406?tstart=0

- Jouni

tholmes · ‎01-30-2014

Hi Jouni,

Many thanks for the speedy reply, alas I had initially tried this method but it conflicted with whatever IP addresses I used for the LAN failover interfaces, when deploying active/standby ASAs

It displays an error saying "NAT policy clashed with 1.1.1.1 on failover interface, and can not be downloaded", or something. I guess its related to the ALL object.

I think the guys who did this were not using an HA pair

Thanks though, much appreciated

Regards Tony

Jouni Forss · ‎01-30-2014

Hi,

I have actually not tried this with a Failover pair. Good to know of this limitation. Or I am not sure if its good thing but atleast I know it now

I am not sure I can see any way around the problem with the Failover at the moment. Especially when you cant exclude a network in the configuration which in some cases (to my understanding) was possible with the old NAT format using ACLs.

- Jouni

tholmes · ‎01-30-2014

Yes, thanks again Jouni

I tried policy NAT and changing the source address so they were out of the inside range, if you follow, but that didn't work either.

I may open TAC and see if they have any ideas.

I can see the customer getting annoyed if he can't offer VPN capability to his hosted customers!!!!

Cheers Tony

Jouni Forss · ‎01-30-2014

Hmm,

I wonder why the ASA doesnt accept this. I am not sure why the Failover link would be an issue as to my understanding that is "from the box" and "to the box" traffic.

Though I was thinking and was wondering if the following could be a solution.

As the Failover link not a network that any other device in the network needs to know about how about if you configure the Failover link as 1.0.0.1 255.255.255.252 for example

Then configure this

object-group network ALL

network-object 2.0.0.0 254.0.0.0

network-object 4.0.0.0 252.0.0.0

network-object 8.0.0.0 248.0.0.0

network-object 16.0.0.0 240.0.0.0

network-object 32.0.0.0 224.0.0.0

network-object 64.0.0.0 192.0.0.0

network-object 128.0.0.0 128.0.0.0

and use it in the NAT configurations.

Naturally could be even more specific to really narrow it down to EVERYTHING but the Failover link subnet.

Let me know if this makes any difference.

Hope this helps

- Jouni

tholmes · ‎01-31-2014

Good Morning Jouni (morning here in the UK, not sure where you are mate)

Thanks again for your input though

I tried adding the policy NAT you suggested but limited it to the customer 2 NAT only, as I need the customer 1 NAT untouched as this is my remote access in.

The NAT didn't work when I added it in, traffic defaulted to the first egress interface used for customer 1. :-/

The network is in semi-production, the hosted customer 2 being a little tolerant of the odd outage.

So I'm back to this, which works for general Internet 'policy' NAT but not identity NAT (nonat for VPN)

nat (hostcust01-outside,hostcust01-inside) after-auto source static any any destination static hostcust01-outside-PAT hostcust01-inside

nat (hostcust02-outside,hostcust02-inside) after-auto source static any any destination static hostcust02-outside-PAT hostcust02-inside

BTW - for info, I'm using IOS 8.6.(1).2 I found version 9.1 did not support it at all !!

Cheers Tony

Jouni Forss · ‎01-31-2014

Hi,

So you are using one of the first software levels for the new ASA5500-X series. Don't really have much expirience of its behaviour since we have several ASA5585-X which were released before the new ASA5500-X Series so they are running the latest 8.4 software levels.

I have also only used 9.x series software only for personal testing purposes. Mainly to answer questions here.

There has been some bugs related to the NAT where the ASA simply doesnt follow the NAT order at all.

I decided to do a quick test on my home ASA5505 running 8.4(5)

What I noticed is that for some reason unknown to me if I configured the Dynamic PAT for the specific external interface with the "object-group network ALL" in Section 3 Manual NAT it simply would not follow the NAT configuration at all. It for example didnt match the destination address UN-NAT that it should do but actually decided to follow the routing table.

I just decided to move the rule to Section 1 Manual NAT and the behaviour changed.

To my understanding the only difference with Section 1 and Section 3 Manual NAT is just that the matching order is different but the operation otherwise the same. This seems to indicate something else or perhaps a bug.

So my very simple home ASA NAT configuration at the monent

object network LAN

subnet 10.0.0.0 255.255.255.0

object network VPN-POOL

subnet 10.0.1.0 255.255.255.0

object network HOST

host 10.0.0.123

object-group network ALL

network-object 2.0.0.0 254.0.0.0

network-object 4.0.0.0 252.0.0.0

network-object 8.0.0.0 248.0.0.0

network-object 16.0.0.0 240.0.0.0

network-object 32.0.0.0 224.0.0.0

network-object 64.0.0.0 192.0.0.0

network-object 128.0.0.0 128.0.0.0

object-group network LAN-NETWORK

network-object 10.0.0.0 255.255.255.0

Lets first look at the NAT configuration I had originally to match yours. Not completely though as I have configure the NAT from LAN to WAN rather than WAN to LAN which might have some effect also on the behaviour.

nat (LAN,WAN) source static LAN LAN destination static VPN-POOL VPN-POOL

!

nat (LAN,WLAN) after-auto source dynamic HOST interface destination static ALL ALL

nat (any,WAN) after-auto source dynamic LAN-NETWORK interface

As you can see I have the NAT configuration for VPN Client first and then I have the special NAT configuration though for testing purposes I simply try to divert single hosts all traffic towards my WLAN interface while it could as well be some other WAN interface.

Now if I run a "packet-tracer" with the source address of HOST and destination to 8.8.8.8 for example this happens

ASA(config)# packet-tracer input LAN tcp 10.0.0.123 12345 8.8.8.8 80

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 WAN

Phase: 4

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (any,WAN) after-auto source dynamic LAN-NETWORK interface

Additional Information:

Dynamic translate 10.0.0.123/12345 to x.x.x.x/12345

So as you can see no UN-NAT Phase, just a route lookup and translation matches to the default Dynamic PAT I have configured even though the NAT configuration is clearly first in Section 3

Now I change the NAT configuration

nat (LAN,WAN) source static LAN LAN destination static VPN-POOL VPN-POOL

nat (LAN,WLAN) source dynamic HOST interface destination static ALL ALL

!

nat (any,WAN) after-auto source dynamic LAN-NETWORK interface

Now again the same "packet-tracer" test

ASA(config)# packet-tracer input LAN tcp 10.0.0.123 12345 8.8.8.8 80

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (LAN,WLAN) source dynamic HOST interface destination static ALL ALL

Additional Information:

NAT divert to egress interface WLAN

Untranslate 8.8.8.8/80 to 8.8.8.8/80

Phase: 4

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (LAN,WLAN) source dynamic HOST interface destination static ALL ALL

Additional Information:

Dynamic translate 10.0.0.123/12345 to 10.0.255.1/12345

As you can see the test first matches the NAT for the destination address. It eventually reaches also the translation for the Dynamic PAT (that in this case translates to my WLAN interface IP address)

If I finally also test the VPN NAT configuration it seems to match just fine before the above NAT configuration just like its supposed to as its at the top

ASA(config)# packet-tracer input LAN tcp 10.0.0.123 12345 10.0.1.100 80

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (LAN,WAN) source static LAN LAN destination static VPN-POOL VPN-POOL

Additional Information:

NAT divert to egress interface WAN

Untranslate 10.0.1.100/80 to 10.0.1.100/80

Phase: 4

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (LAN,WAN) source static LAN LAN destination static VPN-POOL VPN-POOL

Additional Information:

Static translate 10.0.0.123/12345 to 10.0.0.123/12345

Hope this helps

EDIT: I am from Finland so its around 2hours ahead of your time.

- Jouni

tholmes · ‎01-31-2014

HI Jouni,

wow - you're fast with you configuration!

I think I follow you but one of my main problems seesms to be that the firewall will not conform to this rule

nat (LAN,WLAN) source dynamic HOST interface destination static ALL ALL

(or my variation of it)

using this method results in the FW using the first route out cust 1's external, not customer 2

I think I might downgrade the IOS to 8.4.2 to see if that helps

Many thanks though, much apprecaited

Regards Tony

Jouni Forss · ‎01-31-2014

Hi,

For the new ASA5500-X Series (excluding ASA5585-X models) the lowest software level is 8.6 so you wont be able to downgrade to 8.4(2) and I would NOT suggest that software as there were some changes with regards to the NAT and ARP.

Here is link to the Compatability matrix

http://www.cisco.com/en/US/docs/security/asa/compatibility/asamatrx.html#wp42231

Would it be possible to see your NAT/Interface/Routing configurations? Naturally could share them through Private message or something if you dont want to post something here. Would be interested to test this out.

I do have a ASA5515-X at home so might be able to test with exactly same software level also.

- Jouni

tholmes · ‎01-31-2014

Hi Jouni,

That would be very good of you, please can you Email me on

tholmes@cistek-sol.com

i'll response to you as I wouldn't ask you to give out your Email address publicly

Cheers Tony

Jouni Forss · ‎01-31-2014

Hi,

Posted the following in the other discussion in the Firewall section but it applies to your situation.

In short, I was unable to get the ASA5515-X to follow the NAT rules on the latest versions of each major/minor software level (8.6 , 9.0 and 9.1). I finally tested a software level that worked on the original ASA5500 Series and is compatible with the ASA5500-X Series and it worked.

------------------------------------------------------------------------------------------------------------------------

Hi,

Just spend some time testing some of the software levels supported by the new ASA5500-X Series (as they dont support 8.4)

It seems that 8.6(1) , 9.0(4) and 9.1(4) all completely ingnore the NAT configuration when determining the egress interface. The ASA simply performs a Route Lookup even though the document states that it should not.

I remember trying this with 9.1(1) even on the original ASA5500 Series and booted up that software and it worked just fine.

So Cisco must have changed something.

I really getting fed up these changes to the ASA operation especially when they are undocumented changes.

Either I am missing something or there are simply softwares that act totally different compared to the documentation.

Here is example from the very latest documentation

Determining the Egress Interface
#
When the ASA receives traffic for a mapped address, the ASA unstranslates the destination address according to the NAT rule, and then it sends the packet on to the real address. The ASA determines the egress interface for the packet in the following ways:
#
•Transparent mode—The ASA determines the egress interface for the real address by using the NAT rule; you must specify the source and destination interfaces as part of the NAT rule.
#
•Routed mode—The ASA determines the egress interface in one of the following ways:
#
–You configure the interface in the NAT rule—The ASA uses the NAT rule to determine the egress interface. However, you have the option to always use a route lookup instead. In certain scenarios, a route lookup override is required; for example, see the "NAT and VPN Management Access" section.
#
–You do not configure the interface in the NAT rule—The ASA uses a route lookup to determine the egress interface.

The above section bolded above to my understanding states that if you have a translation for the destination address and if you specify the interface in the "nat" command then NAT should determine the egress interface NOT the routing table.

Obviously this seems to be dependant on the software level for some reason

Command Reference also states this

If you specify an optional interface, then the ASA uses the NAT configuration to determine the egress interface.

- Jouni

tholmes · ‎01-31-2014

Hello Jouni,

I gotta say a big thank you for your efforts, I had a look at your room there and it looks like you run your own OPs centre!

I'm going to to take this back to Cisco, it appears to be something that once worked on the 5500 non-X which now does not work on their recommended replacement models.

I'll let you know how it goes, massive thanks again for you assistance

Best Regards Tony

Jouni Forss · ‎01-31-2014

Hi,

The egress interface selection seemed to work on the 9.1(1) software atleast even on the ASA5515-X. So it could be gotten working with that software I presume if you had to get it working right away.

I havent yet tested the 9.1(2) or 9.1(3) but on the 9.1(4) it did not work.

Naturally its not really a option in the long run to be tied to a certain software level just because something only works on the software level.

I would be interested to hear if you get any clear information about this.

Glad if this helped. Feel free to rate any answer if any of them helped. I guess I didn't really answer the question 100%.

I might look further into this and do some additional test later but now its probably time to take a small break

- Jouni

tholmes · ‎01-31-2014

Of course mate, many thanks, I'd buy you a beer but Finland is a long way for a night out, your English is superb, if you don't mind me saying so, have you sent some time in the UK or US?

I will try 9.1.1 as you've taken the time to too, I also tried 9.1.4 and 9.0.4 which knocked my access off.

I was about to set up a VPN from the 4500X VSS pair behind the ASA and statically NAT through it, but my head isn't up to it right now

I'll rate you with pleasure, is there an 11/10?

Cheers Tony