Cisco ASA 9.1(1) nat issue

Irakli Shavgulidze · ‎02-14-2013

Hello,

We have ASA 5520 and we have upgraded OS from 8.4 to 9.1.1.

We have two outside interfaces OUTSIDE1 and OUTSIDE2, and we have configured Twice Dynamic PATs for some sessions.

ASA is getting dynamic default route(eigrp) from Routers in this interfaces (OUTSIDE1 and OUTSIDE2)

OUTSIDE1 has better DF route. Twice NATs of OUTSIDE1 are upper then the same TWICE NATs of OUTSIDE2.

In 8.4 OS when OUTSIDE1 is going down, for example, via "shutdown" command, NAT rule which was translating into OUTSIDE inteface is moving to another NAT rule and is translated to OUTSIDE2.

After upgrade to 9.1.1 OS and shutting down of OUTSIDE, interface nat is tranlating into OUTSIDE and is not moving to OUTSIDE2, though this interface is in shutdown state.

Is there any workaround about this issue?

Julio Carvajal · ‎02-21-2013

Hello,

Does the route changes when the outside interface goes down?

Do you see the change on the routing table now pointing to the other EIGRP router ( the backup one?

Is there a way you could get a capture while the issue is happening with the trace option to see the packet-tracer as well?

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

Irakli Shavgulidze · ‎02-22-2013

Hello,

Interface goes down, route is changed and pointing to OUTISDE2 but connections can't be established and

packet-tracer is showing that this connection uses OUTSIDE1 NAT rules though OUTSIDE1 is down.

Julio Carvajal · ‎02-22-2013

Hello,

Interesting, definetly not expected at all,

Is there a way you could share the following output while having issues:

show nat divert-table

Regards

Julio Carvajal
Senior Network Security and Core Specialist
CCIE #42930, 2xCCNP, JNCIP-SEC

plotniku7 · ‎03-12-2013

Hi,

i have the same problem, but on IOS 8.4(5). i t was not on earlier versions - 8.4.4(1).

i have many pair of 5510, not sure if i want to move to 9.1.1 version, i am waiing for more stable versions.

please help me with the nat divert feature (or bug )

sh nat divert-table interface inside163

id=0xad8c0270, domain=twice-nat section=1 ignore=no

type=static, hits=0, flags=0x9, protocol=0

src ip/id=192.168.119.96, mask=255.255.255.224 port=0-0

dst ip/id=10.109.136.237, mask=255.255.255.255 port=0-0

input_ifc=inside163, output_ifc=outside2

also the nat configuration i have is dynamic - not static n the first section)

nat (inside163, outside2) source dynamic ob1 ob2 destination static ob3 ob3

ciscosafecomputing · ‎04-09-2013

I'm having the same issue on 9.1(1) Did anyone manage to find a solution to this issue?

Jouni Forss · ‎04-09-2013

Hi,

To be honest I am not entirely sure how the NAT is supposed to work in the new software anymore. Not that I dont know the configuration format but the actual configurations seem to act differently to how the release notes mention.

By far the most visible problem has been the fact that determining the eggress interface for traffic doesnt seem to follow the rules that Cisco have stated.

In some situations with certain configurations the NAT should decide the destination interface for example. However "packet-tracer" tells the opposite by showing a route-lookup that shouldnt happen.

In this case the situation seems to be the opposite. It seems route lookup is not done if the NAT is not being applied. Though to be honest, in any of the above post I have not seen any NAT configurations mentioned so its very hard to try and reproduce the problems you might be having.

Please share the configurations you are using (to the extent that is possible) so other people can confirm the situation. I might possibly approach Cisco through a TAC case because to be honest there have been several oddities regarding the NAT.

- Jouni

ciscosafecomputing · ‎04-09-2013

Hi JouniForss,

I'm currently trying with Cisco TAC, but this one has them completely stump and hoping the community would be able to shed some light on the issue.

They have confirm my rules are correct and things should work as i expect them to, but unfortunately every work around currently will not redirect the traffic through the second external interface.

Currently I'm trying something different. I'm attempting to get all traffic on subnet 10.0.x.x/16 to the first outside interface (which currently works), then all traffic on 10.9.0.x/24 on the second outside interface. but as you can see from the packet tracer, it goes out of outside interface.

To achive what I'm trying I did the following

object service TCP

service tcp destination range 0 65535

object service UDP

service udp destination range 0 65535

object network NET_INSIDE

subnet 10.9.0.0 255.255.255.0

nat (inside,outside_2) 1 source dynamic NET_INSIDE interface service UDP UDP

nat (inside,outside_2) 2 source dynamic NET_INSIDE interface service TCP TCP

Obviously these NAT rules need to be at the top, to ensure that all UDP & TCP traffic goes out of the secondard outside interface, of course doing this meants that there is a limitation of this rule set.

I would like to pass IPSEC and ICMP traffic through this interface, but I get the following error msg:

ERROR: real service object includes protocol that doesnt match TCP or UDP.

When I try:

nat (inside,outside_2) 3 source dynamic NET_INSIDE interface service IPSEC IPSEC

If your using a secondard line as a backup, you could just make these inactive until you need them. Unfortunately I'm not able to make these dynamic as of yet.

Any insight would be helpful

Akshay Dubey · ‎04-09-2013

Hi Chris,

Going back to the original issue. Traffic must pass through 1 ISP and when it fails it should be routed to the other one.

Topology:

10.9.0.0/16 ------ ASA (outside_1) ------- ISP1

|

|____ (outside_2)------ ISP2

object network NET_INSIDE

subnet 10.9.0.0 255.255.255.0

object network NET_INSIDE-1

subnet 10.9.0.0 255.255.255.0

object network NET_INSIDE

nat (inside,outside_1) dynamic interface

object network NET_INSIDE-1

nat (inside,outside_2) dynamic interface

route outside_1 0 0

route outside_2 0 0 10

I verified it and it appears to work. However with manual NAT it is not working.

Hope this helps.

-Akshay

ciscosafecomputing · ‎04-10-2013

Hi Akshay,

Cisco TAC & myself have attempted the above and I can confirm that this does not work.

I've even had Cisco TAC take a copy of my config an have it ran within there labs on there debug kit and it still wouldn't work in the way Cisco believed it would.

They have put this down to how the NAT engine now works within version 9.1(1)

Currently I'm using my connections as 10.x.x.x/16 > outside_1 & 10.9.0.0/24 > outside_2 as detailed above.

Any insight on how to work around to get IPSEC and ICMP traffic from 10.9.0.0/24 through outside_2 would be apprechiated.

Chris

Akshay Dubey · ‎04-10-2013

Hi Chris,

Routing a part of the traffic through one ISP and rest through other ISP may not be possible as source based routing is not supported. The route tables can be differentiated if you are using multiple contexts. Not sure if you want to implement that.

I tested the original scenario in my lab and found two workarounds:

First workaround is to use the NAT rules from outside to inside. But this would also translate any traffic coming fro outside to the external interface.

nat (out2,inside) source static any any destination static interface obj-1.1.1.0

nat (out1,inside) source static any any destination static interface obj-1.1.1.0

Another one is to use the auto NAT. Below is the lab result with auto NAT:

ciscoasa(config-if)# sh ip addr

System IP Addresses:

Interface Name IP address Subnet mask Method

Ethernet0/0 inside 1.1.1.1 255.255.255.0 CONFIG

Ethernet0/1 out1 20.106.36.22 255.255.255.0 manual

Management0/0 out2 10.106.36.22 255.255.255.0 CONFIG

ciscoasa(config)# sh run nat

!

object network obj-1.1.1.0

nat (inside,out2) dynamic interface

object network obj-1.1.1.0-1

nat (inside,out1) dynamic interface

ciscoasa(config-if)# sh run route

route out2 0.0.0.0 0.0.0.0 10.106.36.1 1 <<<<, preferred

route out1 0.0.0.0 0.0.0.0 20.106.36.1 10

show route

Gateway of last resort is 10.106.36.1 to network 0.0.0.0

C 1.1.1.0 255.255.255.0 is directly connected, inside

C 20.106.36.0 255.255.255.0 is directly connected, out1

C 10.106.36.0 255.255.255.0 is directly connected, out2

S* 0.0.0.0 0.0.0.0 [1/0] via 10.106.36.1, out2 <<<<<<<<<

ciscoasa(config-if)# packet-tracer input inside icmp 1.1.1.5 8 0 4.2.2.2

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 out2

Phase: 2

Type: NAT

Subtype:

Result: ALLOW

Config:

object network obj-1.1.1.0

nat (inside,out2) dynamic interface <<<<<<<<<<<

Additional Information:

Dynamic translate 1.1.1.5/0 to 10.106.36.22/41806

After shutting down the out2 interface:

Phase: 1

Type: ROUTE-LOOKUP

Subtype: input

Result: ALLOW

Config:

Additional Information:

in 0.0.0.0 0.0.0.0 out1

Phase: 2

Type: NAT

Subtype:

Result: ALLOW

Config:

object network obj-1.1.1.0-1

nat (inside,out1) dynamic interface <<<<<<<<<<<<<<

Additional Information:

Dynamic translate 1.1.1.5/0 to 20.106.36.22/11779

I tested this on asa9.1.1

ciscoasa(config-if)# sh ver | i asa

System image file is "disk0:/asa911-k8.bin"

Hope this helps.

Regards,

Akshay

Jouni Forss · ‎04-12-2013

Hi Chris and others,

So I booted one of my test ASA5520 to software 9.1(1) and did some testing with regards to using NAT configuration to determine the eggress interface of the traffic.

I tried a few different setups and I managed to get one of them working. While a couple of tested configurations always resulted in the ASA doing a route lookup the last one seemed to follow the NAT configuration definitions and not the route lookup/routing table

So my lab setup is the following

Single ASA5520 9.1(1)
2 WAN links
1 LAN link

What I tried

Forward traffic to different WAN link depending on the LAN host IP address
Host 10.0.0.30 out of WAN-1 for all destination networks
Host 10.0.0.200 out of WAN-2 for all destination networks

I guess this configuration in particular doesnt help with the orignal posters situation but it does seem to help in a situation where you want to control the DUAL WAN link usage depending on the source IP address of the LAN host.

Here are the example NAT/Interface/route configurations and "packet-tracer" output

interface GigabitEthernet0/0

description Primary ISP

nameif WAN-1

security-level 0

ip address 192.168.101.2 255.255.255.0

!

interface GigabitEthernet0/1

description Secondary ISP

nameif WAN-2

security-level 0

ip address 192.168.102.2 255.255.255.0

!

interface GigabitEthernet0/2

description LAN

nameif LAN

security-level 100

ip address 10.0.20.2 255.255.255.0

route WAN-1 0.0.0.0 0.0.0.0 192.168.101.1 1

route WAN-2 0.0.0.0 0.0.0.0 192.168.102.1 254

route LAN 10.0.0.0 255.255.255.0 10.0.20.1 1

object network LAN-SOURCE-1

host 10.0.0.30

object network LAN-SOURCE-2

host 10.0.0.200

object network ANY-0.0.0.0-1

subnet 0.0.0.0 128.0.0.0

object network ANY-128.0.0.0-1

subnet 128.0.0.0 128.0.0.0

object-group network ALL

network-object object ANY-0.0.0.0-1

network-object object ANY-128.0.0.0-1

nat (LAN,WAN-1) source static LAN-SOURCE-1 LAN-SOURCE-1 destination static ALL ALL

nat (LAN,WAN-2) source static LAN-SOURCE-2 LAN-SOURCE-2 destination static ALL ALL

ASA(config)# packet-tracer input LAN tcp 10.0.0.30 12345 1.1.1.1 80

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (LAN,WAN-1) source static LAN-SOURCE-1 LAN-SOURCE-1 destination static ALL ALL

Additional Information:

NAT divert to egress interface WAN-1

Untranslate 1.1.1.1/80 to 1.1.1.1/80

Phase: 2

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (LAN,WAN-1) source static LAN-SOURCE-1 LAN-SOURCE-1 destination static ALL ALL

Additional Information:

Static translate 10.0.0.30/12345 to 10.0.0.30/12345

Phase: 3

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 5

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (LAN,WAN-1) source static LAN-SOURCE-1 LAN-SOURCE-1 destination static ALL ALL

Additional Information:

Phase: 6

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 13, packet dispatched to next module

Result:

input-interface: LAN

input-status: up

input-line-status: up

output-interface: WAN-1

output-status: up

output-line-status: up

Action: allow

ASA(config)# packet-tracer input LAN tcp 10.0.0.200 12345 1.1.1.1 80

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (LAN,WAN-2) source static LAN-SOURCE-2 LAN-SOURCE-2 destination static ALL ALL

Additional Information:

NAT divert to egress interface WAN-2

Untranslate 1.1.1.1/80 to 1.1.1.1/80

Phase: 2

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (LAN,WAN-2) source static LAN-SOURCE-2 LAN-SOURCE-2 destination static ALL ALL

Additional Information:

Static translate 10.0.0.200/12345 to 10.0.0.200/12345

Phase: 3

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 5

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (LAN,WAN-2) source static LAN-SOURCE-2 LAN-SOURCE-2 destination static ALL ALL

Additional Information:

Phase: 6

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 14, packet dispatched to next module

Result:

input-interface: LAN

input-status: up

input-line-status: up

output-interface: WAN-2

output-status: up

output-line-status: up

Action: allow

This also works with setting the NAT to Dynamic Policy PAT instead of the above Identity NAT / NAT Exempt

nat (LAN,WAN-1) source dynamic LAN-SOURCE-1 interface destination static ALL ALL

nat (LAN,WAN-2) source dynamic LAN-SOURCE-2 interface destination static ALL ALL

ASA(config)# packet-tracer input LAN tcp 10.0.0.30 12345 1.1.1.1 80

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (LAN,WAN-1) source dynamic LAN-SOURCE-1 interface destination static ALL ALL

Additional Information:

NAT divert to egress interface WAN-1

Untranslate 1.1.1.1/80 to 1.1.1.1/80

Phase: 2

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (LAN,WAN-1) source dynamic LAN-SOURCE-1 interface destination static ALL ALL

Additional Information:

Dynamic translate 10.0.0.30/12345 to 192.168.101.2/12345

Phase: 3

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 5

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (LAN,WAN-1) source dynamic LAN-SOURCE-1 interface destination static ALL ALL

Additional Information:

Phase: 6

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 15, packet dispatched to next module

Result:

input-interface: LAN

input-status: up

input-line-status: up

output-interface: WAN-1

output-status: up

output-line-status: up

Action: allow

ASA(config)# packet-tracer input LAN tcp 10.0.0.200 12345 1.1.1.1 80

Phase: 1

Type: UN-NAT

Subtype: static

Result: ALLOW

Config:

nat (LAN,WAN-2) source dynamic LAN-SOURCE-2 interface destination static ALL ALL

Additional Information:

NAT divert to egress interface WAN-2

Untranslate 1.1.1.1/80 to 1.1.1.1/80

Phase: 2

Type: NAT

Subtype:

Result: ALLOW

Config:

nat (LAN,WAN-2) source dynamic LAN-SOURCE-2 interface destination static ALL ALL

Additional Information:

Dynamic translate 10.0.0.200/12345 to 192.168.102.2/12345

Phase: 3

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 4

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 5

Type: NAT

Subtype: rpf-check

Result: ALLOW

Config:

nat (LAN,WAN-2) source dynamic LAN-SOURCE-2 interface destination static ALL ALL

Additional Information:

Phase: 6

Type: NAT

Subtype: per-session

Result: ALLOW

Config:

Additional Information:

Phase: 7

Type: IP-OPTIONS

Subtype:

Result: ALLOW

Config:

Additional Information:

Phase: 8

Type: FLOW-CREATION

Subtype:

Result: ALLOW

Config:

Additional Information:

New flow created with id 16, packet dispatched to next module

Result:

input-interface: LAN

input-status: up

input-line-status: up

output-interface: WAN-2

output-status: up

output-line-status: up

Action: allow

Hopefully this helps someone. Please if so

- Jouni

Derek Au · ‎05-03-2013

Hi Jouni,

I have been trying to get 9.1(1) NAT to divert to an egress interface other than the one that is currently for the default route with no success and I was about ready to give up and work around the problem by putting a router in front of the ASA and configuring PBR in the router to have it correctly decide the egress interface for WAN-1 vs. WAN-2. I got the same result as Ashkay. In Ashkay's example, he can only get the egress interface to change to the less preferred default route after shutting down the preferred WAN interface. That's great for a WAN failover scenario, but what we really want to do here is to duplicate what Policy Based Routing (PBR) can do, and that is to select the egress interface based on the source IP address of a flow, while both WAN interfaces are up.

As your packet tracer output shows, you did it!! Here is why your posted config made all the difference for me. The way I was trying to do it is shown below (shown in the context of your config example which I am using as the reference config):

object network any_0.0.0.0

subnet 0.0.0.0 0.0.0.0

! This doesn't work !

nat (LAN,WAN-2) source static LAN-SOURCE-2 LAN-SOURCE-2 destination static any_0.0.0.0 any

When I had the above nat in my ASA, packet tracer showed that Phase 1 was a route lookup, which leads to the egress interface being WAN-1 (since that is the interface for the default route in the route table) which is not the desired result.

After I changed the destination NAT to use the network object-group ALL (as you constructed it in your config) like this:

! Changing the destination translation makes it work!

nat (LAN,WAN-2) source static LAN-SOURCE-2 LAN-SOURCE-2 destination static ALL ALL

Now packet tracer shows that Phase 1 is NAT diverting the egress interface to WAN-2. Perfect. This just goes to demonstrate that for ASA NAT rules, matching against "any" network (i.e. 0.0.0.0 0.0.0.0) is not the same as matching against ALL networks (i.e. 0.0.0.0 128.0.0.0 plus 128.0.0.0 128.0.0.0).

Thanks for sharing this! You solved my headache and now I don't have to put a router in front of my ASA to solve the egress problem with IOS Policy Based Routing.

Regards,

Derek

Jouni Forss · ‎05-03-2013

Hi,

Glad it helped

You can always rate the answer if you felt it was helpfull

- Jouni