Asymmetric NAT rules matched for forward and reverse flows

[collinsjl]

We have a 5516 that we Client VPN into using AnyConnect then we are trying to connect to items in a site to site VPN over to a 5506's at 2 other sites.  Anyhow I get the following when I try to connect to the device.  But Packet-tracer says it will go fine.

%ASA-5-305013: Asymmetric NAT rules matched for forward and reverse
flows; Connection protocol src interface_name:source_address/source_port [(idfw_user)] dst interface_name:dst_address/dst_port [(idfw_user)] denied due to
NAT reverse path failure.
An attempt to connect to a mapped host using its actual address was rejected.

Here is the packet-tracer

10.123.15.X is the AnyConnect Network

10.245.10.X is the distant end network on the 5506 via the site to site VPN

XYXYJV/pri/act# packet-tracer input inside tcp 10.123.15.48 80 10.245.10.1 80 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static XYXYInc_BlahBlahBla_Networks XYXYInc_BlahBlahBla_Networks destination static XYXYInc_NV_DC_Mgmnt_Networks XYXYInc_NV_DC_Mgmnt_Networks
Additional Information:
NAT divert to egress interface outside
Untranslate 10.245.10.1/80 to 10.245.10.1/80

Phase: 2
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,outside) source static XYXYInc_BlahBlahBla_Networks XYXYInc_BlahBlahBla_Networks destination static XYXYInc_NV_DC_Mgmnt_Networks XYXYInc_NV_DC_Mgmnt_Networks
Additional Information:
Static translate 10.123.15.48/80 to 10.123.15.48/80
Forward Flow based lookup yields rule:
in id=0x2aaad9a47fc0, priority=6, domain=nat, deny=false
hits=1, user_data=0x2aaada979d00, cs_id=0x0, flags=0x0, protocol=0
src ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any
dst ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=outside

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad7354eb0, priority=0, domain=nat-per-session, deny=false
hits=165485789, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad7c7a8b0, priority=0, domain=inspect-ip-options, deny=true
hits=162426692, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 5
Type: FOVER
Subtype: standby-update
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x2aaad7f1b3f0, priority=21, domain=lu, deny=true
hits=16243518, user_data=0x0, cs_id=0x0, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=80, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 6
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x2aaad93a7cc0, priority=70, domain=encrypt, deny=false
hits=7, user_data=0xdb553e4, cs_id=0x2aaadabf1c00, reverse, flags=0x0, protocol=0
src ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any
dst ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=outside

Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static XYXYInc_BlahBlahBla_Networks XYXYInc_BlahBlahBla_Networks destination static XYXYInc_NV_DC_Mgmnt_Networks XYXYInc_NV_DC_Mgmnt_Networks
Additional Information:
Forward Flow based lookup yields rule:
out id=0x2aaada0246a0, priority=6, domain=nat-reverse, deny=false
hits=2, user_data=0x2aaadaa13390, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any
dst ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=outside

Phase: 8
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x2aaada343c90, priority=0, domain=user-statistics, deny=false
hits=169452707, user_data=0x2aaad9d50aa0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=outside

Phase: 9
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaadb12b120, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=6, user_data=0xf95815c, cs_id=0x2aaadabf1c00, reverse, flags=0x0, protocol=0
src ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 10
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaad7354eb0, priority=0, domain=nat-per-session, deny=false
hits=165485791, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 11
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x2aaad7e8f5b0, priority=0, domain=inspect-ip-options, deny=true
hits=163501909, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 12
Type: USER-STATISTICS
Subtype: user-statistics
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
out id=0x2aaada344c30, priority=0, domain=user-statistics, deny=false
hits=160484343, user_data=0x2aaad9d50aa0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=inside

Phase: 13
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 194990492, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_ipsec_tunnel_flow
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Yes we do have the

same-security-traffic permit inter-interface
same-security-traffic permit intra-interface

The networks are defined in the interesting traffic in the site to site including the AnyConnect network. 

Here is the distant end packet-tracer:

XYXY-NV-Mgmnt-ASA# packet-tracer input inside tcp 10.245.10.1 80 10.123.15.48 80 detailed

Phase: 1
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,outside) source static XYXY_NV_DC_Networks XYXY_NV_DC_Networks destination static XYXY_BlahBlahBla_Networks XYXY_BlahBlahBla_Networks
Additional Information:
NAT divert to egress interface outside
Untranslate 10.123.15.48/80 to 10.123.15.48/80

Phase: 2
Type: NAT
Subtype:
Result: ALLOW
Config:
nat (inside,outside) source static XYXY_NV_DC_Networks XYXY_NV_DC_Networks destination static XYXY_BlahBlahBla_Networks XYXY_BlahBlahBla_Networks
Additional Information:
Static translate 10.245.10.1/80 to 10.245.10.1/80
Forward Flow based lookup yields rule:
in id=0x7fafe5c29d50, priority=6, domain=nat, deny=false
hits=5, user_data=0x7fafe49647c0, cs_id=0x0, flags=0x0, protocol=0
src ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=outside

Phase: 3
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fafe3234210, priority=0, domain=nat-per-session, deny=false
hits=22330360, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 4
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
in id=0x7fafe3f4cb70, priority=0, domain=inspect-ip-options, deny=true
hits=47155499, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=any

Phase: 5
Type: VPN
Subtype: encrypt
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fafe5060840, priority=70, domain=encrypt, deny=false
hits=6, user_data=0x345c54, cs_id=0x7fafe3fb9ec0, reverse, flags=0x0, protocol=0
src ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=outside

Phase: 6
Type: ACCESS-LIST
Subtype: filter-aaa
Result: ALLOW
Config:
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fafe54fb570, priority=13, domain=filter-aaa, deny=false
hits=2, user_data=0x7fafdc8ab180, filter_id=0x3(NashvillePOD_BlahBlahBla), protocol=0
src ip=0.0.0.0, mask=0.0.0.0, port=0
dst ip=10.123.15.48, mask=255.255.255.254, port=0

Phase: 7
Type: NAT
Subtype: rpf-check
Result: ALLOW
Config:
nat (inside,outside) source static XYXY_NV_DC_Networks XYXY_NV_DC_Networks destination static XYXY_BlahBlahBla_Networks XYXY_BlahBlahBla_Networks
Additional Information:
Forward Flow based lookup yields rule:
out id=0x7fafe49b23b0, priority=6, domain=nat-reverse, deny=false
hits=2, user_data=0x7fafdcb59db0, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
src ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any
dst ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any, dscp=0x0
input_ifc=inside, output_ifc=outside

Phase: 8
Type: VPN
Subtype: ipsec-tunnel-flow
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fafe505fc10, priority=70, domain=ipsec-tunnel-flow, deny=false
hits=2, user_data=0x34a8ac, cs_id=0x7fafe3fb9ec0, reverse, flags=0x0, protocol=0
src ip/id=10.123.15.48, mask=255.255.255.254, port=0, tag=any
dst ip/id=10.245.10.0, mask=255.255.255.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 9
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fafe3234210, priority=0, domain=nat-per-session, deny=false
hits=22330362, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=6
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=any, output_ifc=any

Phase: 10
Type: IP-OPTIONS
Subtype:
Result: ALLOW
Config:
Additional Information:
Reverse Flow based lookup yields rule:
in id=0x7fafe3eea4d0, priority=0, domain=inspect-ip-options, deny=true
hits=47192300, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any
dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=any, dscp=0x0
input_ifc=outside, output_ifc=any

Phase: 11
Type: FLOW-CREATION
Subtype:
Result: ALLOW
Config:
Additional Information:
New flow created with id 53516190, packet dispatched to next module
Module information for forward flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_tcp_normalizer
snp_fp_translate
snp_fp_adjacency
snp_fp_encrypt
snp_fp_fragment
snp_ifc_stat

Module information for reverse flow ...
snp_fp_tracer_drop
snp_fp_inspect_ip_options
snp_fp_ipsec_tunnel_flow
snp_fp_translate
snp_fp_tcp_normalizer
snp_fp_adjacency
snp_fp_fragment
snp_ifc_stat

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: allow

Any Suggestions?