Solved: denied due to NAT reverse path failure for second internal network

RichardBeach46587 · ‎07-15-2021

Hello all,

We trying to connect a second internal network to our existing site to site VPN.

The VPN works for the initial two networks, but when trying to add the second network at site B we get the following error:

Asymmetric NAT rules matched for forward and reverse flows; Connection for icmp src outside:172.16.1.94 dst

Lab:192.168.2.15 (type 8, code 0) denied due to NAT reverse path failure

Site B: 1.2.3.100, Internal network: 192.168.1.0

Site A: 1.2.3.200, internal network: 172.16.1.0

Site B network to add is 192.168.2.0

Seems from different threads that I have seen that we are missing a NAT statement of some sort but I am not sure the statement we need.

Let me know any shows that you would like me to run and I can post them up for you, rather than posting the whole config here.

Thanks

Rob Ingram · ‎07-15-2021

@RichardBeach46587

The 192.168.2.0/24 network is not connected to the "inside" network, it's connected to "lab". So therefore the nat rule won't apply

object network LAB
 subnet 192.168.2.0 255.255.255.0
!
nat (lab,outside) source static LAB LAB destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp

The difference above is the source interface is "nat (lab,.....".

View solution in original post

Rob Ingram · ‎07-15-2021

@RichardBeach46587

You probable just need a NAT exemption rule, to ensure that traffic is not unintentially translated. Run packet-tracer from the CLI, this will confirm what NAT rule is currently being matched, provide the output for review.

Example:-

packet-tracer input INSIDE icmp <src ip> 8 0 <dst ip>

A NAT Exemption rule would look like this:-

nat (INSIDE,OUTSIDE) source static LAN LAN destination static REMOTE REMOTE no-proxy-arp

Just replace the objects LAN and REMOTE to reflect the network objects representing your networks.

RichardBeach46587 · ‎07-15-2021

Hello Rob,

Thanks for the reply.

I ran the packet trace command. Hopefully I ran it the way you were asking. In here you will see that there is some sort of NAT statement like you suggested but it doesn't seem to be in the configuration you gave. I didn't set these rules up, so I am not 100% sure what they were doing here...? It appears that the config they used is something like this

nat (INSIDE,OUTSIDE) source static LAN LAN destination static LAN LAN no-proxy-arp

Because DM_INLINE_NETWORK_1 is just an object for the two internal networks.

Result of the command: "packet-tracer input INSIDE icmp 192.168.2.15 8 0 172.16.1.94"

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 1.2.3.217 using egress ifc outside

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static DM_INLINE_NETWORK_1 DM_INLINE_NETWORK_1 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface outside
Untranslate 172.16.1.94/0 to 172.16.1.94/0

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,outside) source static DM_INLINE_NETWORK_1 DM_INLINE_NETWORK_1 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp route-lookup
Additional Information:
Static translate 192.168.2.15/0 to 192.168.2.15/0

Phase: 4
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 5
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static DM_INLINE_NETWORK_1 DM_INLINE_NETWORK_1 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp route-lookup
Additional Information:

Phase: 10
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 11
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 14
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 15
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 167349, packet dispatched to next module

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Rob Ingram · ‎07-15-2021

@RichardBeach46587

Ok, the output from that packet-tracer command confirms it is working ok.

Is it actually working now?

When do you see the initial error message?

Has the crypro ACL been amending to include the new network?

Please provide the output of "show crypto ipsec sa"

Do you control both firewall devices?

RichardBeach46587 · ‎07-15-2021

No, not working. This is output before I have made any changes. I have not made any as of yet.

but here is the output of the command.

Site A can send to 192.168.2.0 but never receives, site B receives from 172.16.1.0 but never sends.

Result of the command: "show crypto ipsec sa"

interface: outside
Crypto map tag: outside_map, seq num: 1, local addr: 1.2.3.100

access-list outside_cryptomap extended permit ip 192.168.1.0 255.255.255.0 172.16.1.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0)
current_peer: 1.2.3.200

#pkts encaps: 52011, #pkts encrypt: 52011, #pkts digest: 52011
#pkts decaps: 44549, #pkts decrypt: 44549, #pkts verify: 44549
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 52011, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: 1.2.3.100/500, remote crypto endpt.: 1.2.3.200/500
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: A033F2A0
current inbound spi : 2DA50B3C

inbound esp sas:
spi: 0x2DA50B3C (765791036)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 243879936, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4190126/16425)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xA033F2A0 (2687759008)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 243879936, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4230334/16425)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: outside_map, seq num: 1, local addr: 1.2.3.100

access-list outside_cryptomap extended permit ip 192.168.2.0 255.255.255.0 172.16.1.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0)
current_peer: 1.2.3.200

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 902, #pkts decrypt: 902, #pkts verify: 902
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: 1.2.3.100/500, remote crypto endpt.: 1.2.3.200/500
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: ECB0C018
current inbound spi : 864E9E1A

inbound esp sas:
spi: 0x864E9E1A (2253299226)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 243879936, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4193227/24286)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0xECB0C018 (3971006488)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 243879936, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4147200/24286)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Rob Ingram · ‎07-15-2021

@RichardBeach46587

So this output above is from Site B right?

Is the 192.168.2.0/24 network directly connected? If not can the ASA connect to the network (check routing)

Provide the following from Site B - "show route", "show nat detail" and "show run".

RichardBeach46587 · ‎07-15-2021

Yes this is from site B, and 192.168.2.0 is directly connected.

Site A appears to be configured correctly, not seeing any errors there.

Result of the command: "show route"

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
Gateway of last resort is 1.2.3.217 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 1.2.3.217, outside
C 1.2.3.0 255.255.255.0 is directly connected, outside
L 1.2.3.100 255.255.255.255 is directly connected, outside
C 192.168.1.0 255.255.255.0 is directly connected, inside
L 192.168.1.1 255.255.255.255 is directly connected, inside
C 192.168.2.0 255.255.255.0 is directly connected, Lab
L 192.168.2.1 255.255.255.255 is directly connected, Lab
C 192.168.3.0 255.255.255.0 is directly connected, Wifi
L 192.168.3.1 255.255.255.255 is directly connected, Wifi

Result of the command: "show nat detail"

Manual NAT Policies (Section 1)
1 (inside) to (outside) source static DM_INLINE_NETWORK_1 DM_INLINE_NETWORK_1 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp route-lookup
translate_hits = 7100, untranslate_hits = 7110
Source - Origin: 192.168.1.0/24, 192.168.2.0/24, Translated: 192.168.1.0/24, 192.168.2.0/24
Destination - Origin: 172.16.1.0/24, Translated: 172.16.1.0/24

Auto NAT Policies (Section 2)
1 (any) to (outside) source dynamic obj_any interface
translate_hits = 148738, untranslate_hits = 1081
Source - Origin: 0.0.0.0/0, Translated: 1.2.3.100/24

Manual NAT Policies (Section 3)
1 (inside) to (outside) source dynamic any interface
translate_hits = 0, untranslate_hits = 0
Source - Origin: 0.0.0.0/0, Translated: 1.2.3.100/24
2 (Elk) to (outside) source dynamic any interface
translate_hits = 0, untranslate_hits = 0
Source - Origin: 0.0.0.0/0, Translated: 1.2.3.100/24
3 (Wifi) to (outside) source dynamic any interface
translate_hits = 0, untranslate_hits = 0
Source - Origin: 0.0.0.0/0, Translated: 1.2.3.100/24

Rob Ingram · ‎07-15-2021

@RichardBeach46587

The 192.168.2.0/24 network is not connected to the "inside" network, it's connected to "lab". So therefore the nat rule won't apply

object network LAB
 subnet 192.168.2.0 255.255.255.0
!
nat (lab,outside) source static LAB LAB destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp

The difference above is the source interface is "nat (lab,.....".

RichardBeach46587 · ‎07-15-2021

Thanks Rob,

Are you suggesting that I add this statement?

nat (lab,outside) source static LAB LAB destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp

Rob Ingram · ‎07-15-2021

@RichardBeach46587 Yes I am.

Also amend the existing NAT exemption rule to take out the 192.168.2.0/24 network.

RichardBeach46587 · ‎07-16-2021

I added the statements this morning and the error: denied due to NAT reverse path failure seems to no longer be showing up.

The traffic still is not passing but I am not seeing any logs indicating as to why. I have a message out to the group to verify that I can ping something on that network to bring the tunnel up, maybe my pings are getting blocked by a Windows Firewall somewhere.

Let me know if you would like to see anything that may shed some light here...

I greatly appreciated the help.

Rob Ingram · ‎07-16-2021

@RichardBeach46587

Run packet-tracer to a destination device (any IP) and confirm the result (allow/drop)

Run the command "show crypto ipsec sa" and confirm encaps|decaps counters are increasing.

Take a packet capture from traffic to/from remote network.

Run a ping sweep across the remote network range, something should respond.

RichardBeach46587 · ‎07-16-2021

I ran this packet tracer, but not sure if I can it with the right parameters.

Not sure if I should run it from inside or lab interface, this was from lab

packet-tracer input Lab icmp 172.16.1.94 8 0 192.168.2.15

With these results:

Result of the command: "packet-tracer input Lab icmp 172.16.1.94 8 0 192.168.2.15"

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 192.168.2.15 using egress ifc Lab

Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config:
Implicit Rule
Additional Information:

Result:
input-interface: Lab
input-status: up
input-line-status: up
output-interface: Lab
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Result of the command: "show crypto ipsec sa" - Still no encaps for the VPN in question as bolded below

interface: outside
Crypto map tag: outside_map, seq num: 1, local addr: 1.2.3.100

access-list outside_cryptomap extended permit ip 192.168.1.0 255.255.255.0 172.16.1.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.1.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0)
current_peer: 1.2.3.200

#pkts encaps: 114361, #pkts encrypt: 114361, #pkts digest: 114361
#pkts decaps: 92253, #pkts decrypt: 92253, #pkts verify: 92253
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 114361, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: 1.2.3.100/500, remote crypto endpt.: 1.2.3.200/500
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 704431B2
current inbound spi : 7370B351

inbound esp sas:
spi: 0x7370B351 (1936765777)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 129183744, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (3910546/17463)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0x704431B2 (1883517362)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 129183744, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4222723/17463)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

Crypto map tag: outside_map, seq num: 1, local addr: 1.2.3.100

access-list outside_cryptomap extended permit ip 192.168.2.0 255.255.255.0 172.16.1.0 255.255.255.0
local ident (addr/mask/prot/port): (192.168.2.0/255.255.255.0/0/0)
remote ident (addr/mask/prot/port): (172.16.1.0/255.255.255.0/0/0)
current_peer: 1.2.3.200

#pkts encaps: 0, #pkts encrypt: 0, #pkts digest: 0
#pkts decaps: 2265, #pkts decrypt: 2265, #pkts verify: 2265
#pkts compressed: 0, #pkts decompressed: 0
#pkts not compressed: 0, #pkts comp failed: 0, #pkts decomp failed: 0
#pre-frag successes: 0, #pre-frag failures: 0, #fragments created: 0
#PMTUs sent: 0, #PMTUs rcvd: 0, #decapsulated frgs needing reassembly: 0
#TFC rcvd: 0, #TFC sent: 0
#Valid ICMP Errors rcvd: 0, #Invalid ICMP Errors rcvd: 0
#send errors: 0, #recv errors: 0

local crypto endpt.: 1.2.3.100/500, remote crypto endpt.: 1.2.3.200/500
path mtu 1500, ipsec overhead 74(44), media mtu 1500
PMTU time remaining (sec): 0, DF policy: copy-df
ICMP error validation: disabled, TFC packets: disabled
current outbound spi: 9CE177B5
current inbound spi : 6870E8A1

inbound esp sas:
spi: 0x6870E8A1 (1752230049)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 129183744, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4193147/17464)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0xFFFFFFFF 0xFFFFFFFF
outbound esp sas:
spi: 0x9CE177B5 (2632021941)
SA State: active
transform: esp-aes-256 esp-sha-hmac no compression
in use settings ={L2L, Tunnel, IKEv2, }
slot: 0, conn_id: 129183744, crypto-map: outside_map
sa timing: remaining key lifetime (kB/sec): (4239360/17464)
IV size: 16 bytes
replay detection support: Y
Anti replay bitmap:
0x00000000 0x00000001

RichardBeach46587 · ‎07-16-2021

A few more packet traces:

Result of the command: "packet-tracer input Lab icmp 192.168.2.15 8 0 172.16.1.94"

Phase: 1
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (Lab,outside) source static NETWORK_OBJ_192.168.2.0_24 NETWORK_OBJ_192.168.2.0_24 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp
Additional Information:
NAT divert to egress interface outside
Untranslate 172.16.194/0 to 172.16.1.0/0

Phase: 3
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (Lab,outside) source static NETWORK_OBJ_192.168.2.0_24 NETWORK_OBJ_192.168.2.0_24 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp
Additional Information:
Static translate 192.168.2.15/0 to 192.168.2.15/0

Phase: 4
Type: ACCESS-LIST
Subtype:
Result: ALLOW
Config:
Implicit Rule
Additional Information:

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 6
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
match default-inspection-traffic
policy-map global_policy
class inspection_default
inspect icmp
service-policy global_policy global
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 9
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (Lab,outside) source static NETWORK_OBJ_192.168.2.0_24 NETWORK_OBJ_192.168.2.0_24 destination static NETWORK_OBJ_172.16.1.0_24 NETWORK_OBJ_172.16.1.0_24 no-proxy-arp
Additional Information:

Phase: 11
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 12
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:

Phase: 13
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 14
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:

Phase: 15
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:

Phase: 16
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 356780, packet dispatched to next module

Result:
input-interface: Elk
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Rob Ingram · ‎07-16-2021

Fine thats working, compare the ipsec sa on that same firewall

Run packet tracer on the other firewall and compare it's ipsec sa, confirm the packet.

Generate some real traffic, capture some packets.