upgrade to V9 breaks access to local network due to VPN acl

[tato386]

I have two secure (level 100) networks on my ASA which I permit traffic to each other using "enable traffic between interfaces with same security level".  One of the networks is part of a class B space but I have it subnetted down to a /29 subnet. The other is a class A that I have subnetted down to a /24.  I also have an IPSec tunnel defined with an ACL that matches the entire class B from which the /29 is carved out of.

Before upgrade to V9.1.2 I was running V8.4 and all was fine. However, after the upgrade the communication between the /29 and the /24 broke.  I did some troubleshooting and saw that traffic from the /24 with destination of the /29 was being put in the tunnel instead of being delivered out the local interface.  Removing the class B ACL from the cryptomap gets traffic going again.  So it looks like the ASA V9.1.2 now evaluates cryptomap ACL _before_ checking the local route table.  Removing the class B from the ACL confirmed that this what is happening.

There are litterally dozens of subnets that I need to reach over the VPN.  The class B entry on the ACL took care of this nicely.  I don't want to enter all of them manually.  So how do i fix this?

Is there a way I can get the ASA to evaluate traffic like it did in V8.4? Is there a way to add a "deny" in the network group object so that I can keep the class B and several other networks in the VPN ACL and block my local /29 from being included in the tunnel?

Thanks,

Diego

[Jouni Forss · Accepted solution]

Hi,

Never heard of this kind of problem.

The ASA should handle Route Lookup, interface ACLs, NAT configurations etc before doing anything related to VPN.

I would sharing the NAT configurations and Crypto ACL and also taking some "packet-tracer" command output testing the traffic that is not working correctly. The output should tell us what rules the traffic is hitting.

To my understanding there was some new problems with NAT when the 9.x softwares were introduced which we not present in the 8.4 to my understanding. Perhaps this is somehow related to one of them. There was for example some bugs which caused the NAT configurations to be matched in the wrong order which could possibly cause problems.

Command format for the "packet-tracer" is

packet-tracer input

- Jouni

[tato386]

Looks like you were right about the NAT being related to this.  The reason the traffic was being placed in tunnel was due to different NAT behaviour and not due to crypto ACL coming in front of local route lookup.

In addition to what what I mentioned in my first post, I also had PAT setup whereby IPs from the /24 would be NATed to a local IP from the /29 subnet and then sent into the tunnel.  The PAT command also used a /16 that covers both a local subnet and the subnets on the tunnel.  So it looks like now PAT is evaluated before local routes and due to the PAT the traffic was being placed in the tunnel.

I was using the same network object group to define both the crypto ACLs and the PAT.  What I did was to create a second network group for PAT that did not include the /16 and things started to work.  For now I am OK because the PAT does not need to access all the remote subnets that the local /29 needs.  However, this might not be true for long and I can see where I might again need to add the /16 to the PAT.

Is there a way to evaluate local route table before doing any NAT/PAT?  Or maybe I will need to setup an identity NAT for traffic between the /24 and /29 local subnets and place that in front of the existing PAT statement that translates the /24 to a /29.

Thanks,

Diego

[Jouni Forss]

Hi,

There should be a "route-lookup" parameter that you can enter at the end of the Twice NAT / Manual NAT configurations. But to be honest there has been some wierd behaviour related to this very specific parameter which has caused a lot of headache for some people.

So far I have not run into a situation with NAT that couldnt be corrected somehow. I am not sure how you have configured your NAT configurations but the way I use to configure NAT configurations I rarely run into problems with NAT ordering.

If for example my ASA had the following NAT configurations

• NAT0 for L2L VPN
• Static NAT for server
• Dynamic PAT for multiple local networks

Then I would configure them this way while at the same time utilizing all NAT Sections of the new software (Sections 1 - 3)

NAT0

object network LOCAL

subnet 10.10.10.0 255.255.255.0

object network REMOTE

subnet 192.168.10.0 255.255.255.0

nat (inside,outside) source static LOCAL LOCAL destination static REMOTE REMOTE

STATIC NAT

object network STATIC

host 172.16.10.10

nat (dmz,outside) static 1.1.1.1

DYNAMIC PAT

object-group network DEFAULT-PAT-SOURCE

network-object 10.10.10.0 255.255.255.0

network-object 172.16.10.0 255.255.255.0

nat (any,outside) after-auto source dynamic DEFAULT-PAT-SOURCE interface

With the above configurations

• NAT0 is always matched first and neither of the 2 other NAT configurations will ever override it
• Static NAT is only overriden when there is L2L VPN related traffic that uses the NAT0 configuration (could be changed if wanted). Also the Static NAT would always get applied before Dynamic PAT
• Dynamic PAT is overriden in the case of L2L VPN connection traffic and with regards to the host using Static NAT. Other hosts behind "inside" and "dmz" will use the Dynamic PAT translation towards "outside"

In addition to the above NAT configurations, if I very to presume that no other NAT configurations exists then "inside" and "dmz" could communicate between eachother with their original IP addresses and they would NOT need any NAT configurations.

I am not sure if your problem was due to bug or simply the ordering of the NAT configurations. Though if it worked with some 8.4 software then there might be planned changes involved or possibly some bug. Can you share the exact 8.4(x) version number you had before the change.

- Jouni

[tato386]

The route-lookup seems like it would fix my problem but is not available with a PAT statement which is the statement with the problem.  I will try to explain a little better if you care to hear the boring details.

Subnet A is a /24 which is the primary subnet that all my users are on.  On a second interface I have a subnet B which is a /29 and is part of a large organization which uses many /24 and /16 networks.   Subnet B needs access via VPN to its parent company.  Additionaly, users from Subnet A need to access subnet B primarily but also need to use many other subnets from the large org occasionally.   The caveat is that users from subnet A users must be NATed to an IP on subnet B to access the large org since our subnet A is not allowed on the large org network.

What I did was to create a network object group that included all the subnets of the large org.  This group includes /24, /32, /30 and /16 subnets of the large org.  My subnet B is a /29 and is a subnet of one of the /16 in this group.  I used this same object group in two locations.  I used it in the crypto map for the VPN and I also used it in PAT statement that allows my subnet A to be NATed to subnet B so it can use remote subnets available thru VPN.

This was working fine with 8.4.4.1.  Even though my subnet B was technically included in the PAT and cryptomap ACLs my subnet A users had no problems using subnet B and the VPN.

When I upgraded  to 9.1.2 users on subnet A instanly complained about not being able to get to subnet B.  I saw the traffic being sent to VPN and hastily posted that crypto maps where being evaluated before local route table.  After your post I realized that subnet A was being PATed first and then being placed in the VPN even though the destination network (subnet B) is local and in the route table.

So my quick fix was to create a second network object group for the PAT that does not include the /16 subnet that encompasses my subnet B /29. I left the original network object group for the VPN ACL intact, no changes.  However by removing the /16 I have also removed PAT access to  many other networks that my subnet A guys need.

So I guess the true fix here is to add an identity NAT for subnet A to subnet B traffic that is before the PAT statement.  I guess that is no big deal and maybe I should have had it from the begining.  Nevertheless it wasn't needed before 9.1.2 so defintely some changes there in the order of processing if you ask me.

Thanks for all your help and if you have any other ideas I am always happy to listen/read them.

Thanks again,

Diego

[Jouni Forss]

Hi,

If you want I could always take a look at the configurations. If you dont want to show them here you can always send them as a private message through my profile.

If I understood you correctly you essentially have the following

• Subnet A which is your LAN
• Subnet B which is some small network on the edge of your network and the other organizations network
• Subnet B is also used as a NAT addresses for Subnet A to Subnet B traffic
• Subnet B is also a source address for a L2L VPN connection to another site of the other organization

If this is the case then it would seem wierd to me that traffic from Subnet A would be suddenly forwarded to the L2L VPN instead of being NATed to the NAT IP address of Subnet B while connecting to the other organizations local network.

To have a better idea of the setup it would be easier to see the configurations.

I can remember some bug related to NAT configurations which was caused by using same "object-group" in multiple NAT statements. In this case the workaround suggestion from Cisco was to make separate "object-group" for each use even if the contents were identical. Though to be honest that is the way I usually configure things since I dont want to potentially link an "object-group" to multiple things and cause problems in the longer run.

- Jouni

[tato386]

Thanks for the offer.  I took you up on it and sent you a PM with before and after configs.

Rgds,

Diego

[Jouni Forss]

Hi,

I will have a look at the configurations tomorrow (well today actually) as its already 1am here

Will get back to you here when I've looked through the configurations.

- Jouni

[tato386]

Jouni!  You are doing me a great favor!  No hurry at all.  Whenever you get a chance.

Thanks,

Diego