Site to Site IPSec NAT reverse path failure

[it]

I am getting the following error on my ASA 5505 Feb 14 2013       10:17:14      305013      10.2.0.82      389                   Asymmetric NAT rules matched for forward and reverse flows;  Connection for tcp src outside:10.1.0.141/14830 dst inside:10.2.0.82/389  denied due to NAT reverse path failure. Showing that my LDAP traffic is  not crossing properly.

I know that this means my NAT Policies are set up incorrectly. I am looking for some opinions on how to best set up  the NAT for the VPN traffic. 10.1.0.0/22 being the Main site (ASA 5510)  and 10.2.0.0/22 being the remote site. The errors are showing up on the  remote site (ASA 5505).

Here is a look at my config on the ASA 5505. Also the results of show Nat is list at the end.

ASA Version 8.2(2)

interface Vlan10

description Internet facing interface

nameif outside

security-level 0

ip address dhcp

!

interface Vlan20

description Internal Network facing interface

nameif inside

security-level 100

ip address 10.2.0.1 255.255.252.0

!

interface Vlan30

description For the DMZ

<--- More --->

no forward interface Vlan20

nameif dmz

security-level 25

no ip address

!

interface Ethernet0/0

description outside interface

switchport access vlan 10

!

interface Ethernet0/1

description inside interface

switchport access vlan 20

!

interface Ethernet0/2

description dmz interface

switchport access vlan 30

shutdown

dns domain-lookup outside

dns domain-lookup inside

dns server-group DefaultDNS

domain-name kensington.org

object-group network DM_INLINE_NETWORK_1

network-object 10.1.0.0 255.255.252.0

network-object 172.16.2.0 255.255.255.0

access-list match-icmp-acl remark Match all ICMP traffic

access-list match-icmp-acl extended permit icmp any any inactive

access-list match-client-udp-acl remark Match all Client based UDP traffic

access-list match-client-udp-acl extended permit udp any any inactive

access-list match-client-tcp-acl remark Match all Client based TCP traffic

access-list match-client-tcp-acl extended permit tcp any any inactive

access-list inside_access_in remark Allow anything

access-list inside_access_in extended permit ip 10.2.0.0 255.255.252.0 any

access-list inside_access_in remark Clean up rule for logging

access-list inside_access_in extended deny ip any any

<--- More --->

access-list outside_access_in remark Clean up rule for logging

access-list outside_access_in extended deny ip any any inactive

access-list outside_access_in extended permit ip host 10.2.0.1 any inactive

access-list outside_cryptomap extended permit ip 10.2.0.0 255.255.252.0 object-group DM_INLINE_NETWORK_1

access-list outside_cryptomap extended permit ip 10.2.0.0 255.255.252.0 10.1.0.0 255.255.252.0

access-list outside_cryptomap extended permit ip 10.2.0.0 255.255.252.0 10.8.0.0 255.255.255.0

access-list outside_cryptomap extended permit ip 10.2.0.0 255.255.252.0 10.100.0.0 255.255.252.0

access-list outside_cryptomap extended permit ip 10.2.0.0 255.255.252.0 any

access-list outside_cryptomap extended permit ip host *Remote Site Public IP* host *Main Site Public IP*

access-list jrsvpn-splittun-acl standard permit 10.1.0.0 255.255.252.0

access-list jrsvpn-splittun-acl standard permit 10.7.0.0 255.255.255.0

access-list jrsvpn-splittun-acl standard permit 10.2.0.0 255.255.252.0

access-list jrsvpn-splittun-acl standard permit 10.8.0.0 255.255.255.0

access-list jrsvpn-splittun-acl standard permit 172.16.0.0 255.255.252.0

pager lines 24

logging enable

logging timestamp

logging buffer-size 65535

logging asdm-buffer-size 512

logging monitor debugging

logging buffered debugging

logging trap notifications

logging asdm notifications

logging queue 2048

<--- More --->

mtu outside 1500

mtu inside 1500

mtu dmz 1500

ip verify reverse-path interface outside

ip verify reverse-path interface inside

ip verify reverse-path interface dmz

ip audit name attack attack action alarm

ip audit name info info action alarm

ip audit interface outside info

ip audit interface outside attack

ip audit interface inside info

ip audit interface inside attack

ip audit interface dmz info

ip audit interface dmz attack

icmp unreachable rate-limit 1 burst-size 1

asdm image disk0:/asdm-623.bin

asdm history enable

arp timeout 14400

global (outside) 10 interface

nat (inside) 10 10.2.0.0 255.255.252.0

access-group outside_access_in in interface outside

access-group inside_access_in in interface inside

route outside 0.0.0.0 0.0.0.0 *ISP Gateway* 1

timeout xlate 3:00:00

timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02

timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00

timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00

timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute

timeout tcp-proxy-reassembly 0:01:00

And Here is the Show NAT

Orion-FW1# show nat

NAT policies on Interface inside:

  match ip inside 10.2.0.0 255.255.252.0 outside any

    dynamic translation to pool 10 (166.150.232.106 [Interface PAT])

    translate_hits = 105177, untranslate_hits = 11034

  match ip inside 10.2.0.0 255.255.252.0 inside any

    dynamic translation to pool 10 (No matching global)

    translate_hits = 0, untranslate_hits = 0

  match ip inside 10.2.0.0 255.255.252.0 dmz any

    dynamic translation to pool 10 (No matching global)

    translate_hits = 0, untranslate_hits = 0

  match ip inside 10.2.0.0 255.255.252.0 _internal_loopback any

    dynamic translation to pool 10 (No matching global)

    translate_hits = 0, untranslate_hits = 0

Any help would be greatly appreciated!                            

[Jouni Forss]

Hi,

Yes, it does seem that everything is hitting the correct rule and there should be nothing stopping either side from initiating connections through the L2L VPN connection.

What I have found out through the times I have troubleshooted ASAs is that they sometimes give log message that simply make no sense at all or they dont in any way hint what the problem could be.

In this case I can't see a reason why in the world the MAIN SITE LAN network would be getting NATed. This is also because the L2L VPN configuration specifically states the source addresses that are even forwarded to the L2L VPN. If the MAIN SITE ASA was NATing the LAN address before the VPN they wouldnt match the L2L VPN rules anymore and shouldnt even get forwarded to the L2L VPN.

Also one thing that comes to my mind is that you say that the RDP connections prompt you for the credentials. This should already mean that there is a totally formed connection between the REMOTE and MAIN SITE. The reason why you are getting a black screen is not something I can answer with any knowledge. Same goes probably for everything related to IT.

ACTUALLY, I only now noticed (or payed attention to) the fact that your other sites "outside" interface IP address is gotten with DHCP? This is pretty problematic for a connection that is "static" in nature. Has your ISP set the DHCP service so that it gives the same public IP always to your ASA because of its MAC address? Or what provides the public IP address for the REMOTE SITE?

The NAT configuration you mention above was only mean for the 172.16.2.0/24 network to be able to connect to the REMOTE SITE. It shouldnt help at all with the connection between these 10.x.0.0/22 networks.

- Jouni

[it]

Hey Jouni,

That all makes sense, thank you so much for clearing everything up and allowing me be able to get a road map in my mind of how all the traffic is flowing. The only question I have left for you is that when I remove the Dynamic NAT (Global Policy) My RDP connections, and all other 2 two way data starts working again. Why do you think that would be? It truly seems that all originating traffic is fine, but any replies seems to get blocked or mistranslated.

I can try removing it again and see if my issues get resloved...just to check my theory. Once we got web traffic flowing again with the commands you gave me then the RDP, and server communication started having problems again. Mostly timeouts (since like you said obviously they can get information to each other, my fear is that the Cisco Intrusion detection system is stopping certain traffic)

-Mike

[Jouni Forss]

Hi,

Truly something wierd. I can't at the moment wrap my head around this problem. Its even harder to troubleshoot naturally when I can't monitor connections and troubleshoot them myself.

Its almost starting to sound like some bug.

Are you truly saying that removing the following commands solves the L2L VPN traffic problem?

no global (outside) 10 interface

no nat (inside) 10 10.2.0.0 255.255.252.0

If this is true then I wonder if the inclusion of the your ASAs "outside" interface IP addresses would have something to do with the problem.

I would personally try removing them from the configuration, then maybe clearing the L2L VPN connection and trying connections again WITH the above Dynamic PAT configured.

MAIN SITE

no access-list outside_cryptomap_2 extended permit ip host *Main Site Public IP* host *Remote Site Public IP*

REMOTE SITE

no access-list outside_cryptomap extended permit ip host *Remote Site Public IP* host *Main Site Public IP*

This is just pure guessing since I have never run into such a situation. Mostly this guess is due to the problem being somehow tied to the public IP address NAT configuration. Ofcourse there is always the possibility that I might be missing something and not checking something since I'm not the one controlling the devices.

- Jouni

[it]

Hey Jouni,

Are you truly saying that removing the following commands solves the L2L VPN traffic problem?

no global (outside) 10 interface

no nat (inside) 10 10.2.0.0 255.255.252.0

Yes this is true, I ran the command just now and now instantly all my RDP connections work perectly and active directory is replicating again. So weird, right?!

I completely understand this is a weird issue, and hopefully I am presenting everything in the right fashion. You have helped me narrow this issue down so much.

Now that we know running no nat (inside) 10 10.2.0.0 255.255.252.0

I will try you suggestion right now, I have nothing to loose.

Thanks again!

-Mike

[it]

Since I was runnig out of a time and my hunch is that IDS is being a pain (for obvious reasons since outside (VPN- 10.1.0.0) traffic being seen as 10.2.0.1. I ran

No inspect ICMP

No inspect ICMP error

Leaving the nat (inside) 10 10.2.0.0 255.255.252.0

Everyting works, the all external web traffic works perfectly, and all our traffic is crossing the VPN tunnel perfectly.

For now I will take a break until next week when I can start figuring out the best solution as this is just a temporary one.

Please let me know what you think. Thanks again Jouni for all your help on this I am so eager to learn and this is just accelerating that process.

-Mike

[Jouni Forss]

Hi,

Great that it works now.

What I personally still wonder about this is that is the applications/connections used between the site somehow dependant on using ICMP first to test connectivity and only then attempt the actual TCP/UDP connections or why would "inspect icmp" and "inspect icmp error" (or rather the block they cause) prevent the actual traffic between the sites?

Only time so far that I have witnessed this was with a certain hospitals medical equipment. Local IT were attempting to get it working. It would form a connection through a secure route to certain servers. However the remote end had not opened ICMP so we never saw a TCP connection form as its initiation required for the ICMP to first pass and receive an echo reply back.

Also one thing I already wondered before was that how is it that your REMOTE SITE ASA gets it IP address with DHCP? Has the ISP set their DHCP configuration so that your current ASA will always receive the same IP address? I know we at some point used to have those kind of setups on our ISP side but not anymore, atleast what I know of.

Well main thing everything is working Personally I always want to understand the actual problem and the cause of it. And in this case I still arent really sure.

- Jouni

[it]

Hey,

The Remote Site ASA obtains it address from a Craddlepoint MBR 1400 (We are currently using an LTE wireless connection until we get our MPLS from AT&T up) which is set up to use IP Address Pass-thru which turns it into a dumb router that just hands off the puiblic ip address which is statically set by Verizon.

I am still curious as well to what the main problem is as our Main Site has ICMP and ICMP Error inspect on and isn't having a problem. So it still remains to be a problem with the Remote Site specifically. I still feel that the this log error

Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src outside:10.2.0.1 dst inside:10.2.0.82 (type 3, code 4) denied due to NAT reverse path failure

Is the key to figuring out why ICMP is being filtered. First of all the source address is from the outside with what seems to be a NAT'd adress of 10.2.0.1 and Secondly it is denied due to NAT Reverse path failure. (Maybe this cause the intrusion detection system to stop traffic thinking this is an attempt to break into the inside network)

It does sound similar to your experience with the hospital but I had no idea simple protocols like RDP, LDAP, and RPC needed to have ICMP first pass before receiving an echo reply back. 

Let me know if this seem like I am just going in circles on my line of thought.

-Mike