Can't route traffic between two site-to-site VPN Tunnels

[EM1]

We've got an ASA-5525-X in the office here, and we have a VPC in both AWS and Google Cloud Compute. The office has the IP range 10.1.0.0/16, Amazon is 10.2.0.0/16 and Google is 10.3.0.0/16. We have tunnels from the office to both cloud providers. Both the VPN tunnels work fine from individually, We can send traffic between the office and Amazon, and between the office and Google fine. However if we attempt to send traffic from Google to Amazon, nothing happens; and if we try to send traffic from Amazon to Google, the Amazon VPN stops working and needs 'clear ipsec sa peer x.x.x.x'  to reinitialise. Both Amazon and Google are set to route the entire 10.0.0.0/8 over the VPN tunnel.

This is almost certainly down to me doing something very stupid, as I'm fairly new to using Cisco devices, and their VPN set up is somewhat different to the Junipers I was using before; but if someone can spot what it is I've done wrong I'd be very grateful. I've attached a sanitised copy of the runnng config, if any other information would be useful, let me know and I'll provide it.

[Francesco Molino]

Hi

Sorry for my typo if any, I'm replying through my iPhone.

Your Google crypto map is passing traffic over tunnel only betwen 10.3.0.0/16 to 10.1.0.0/16 where Amazon is on subnet 10.2.0.0/16.
You'll need to add your Amazon subnet in your crypto acl for Google.

For Amazon, every traffic is coming back to your asa as you set it up with 0.0.0.0, is that normal? Just a simple question for curiosity

[EM1]

Hi, thanks for the help, I thought I'd reset all of those back for uploading the config, I need better vlan names. My config at the time I noticed the problem did have the correct cryptomaps, (The GCP map was changed to prevent the issue whereby pinging between s2s tunnels causes the tunnel to drop, and the amazon one was changed to 0.0.0.0 as a test because it was recommended in an Amazon document somewhere.) I've reverted those changes, and forwarded the 10.0.0.0/8 across the tunnel in both cases.

Now that everything is reverted, the problem continues to exist, relevant updated config bits:

access-list AMZN_Cryptomap_ACL extended permit ip object obj-all-internal object obj-amzn 
access-list AMZN_Cryptomap_ACL extended permit ip object obj-amzn object obj-all-internal 
access-list GCP_Cryptomap_ACL extended permit ip object obj-all-internal object obj-gcp 
access-list GCP_Cryptomap_ACL extended permit ip object obj-gcp object obj-all-internal 

Removed line:

nat (Outside_Catalyst,Outside_Catalyst) source dynamic obj-amzn interface

The only thing that's changed now, is that the tunnel drop has switched sides: now if I ping gcp from aws, nothing happens, but if I ping AWS from GCP the Office-GCP tunnel stops sending traffic, and needs a 'clear ipsec' to start working again.

[Francesco Molino]

Can you do a test with packet-tracer? Do you know how to use? Otherwise I'll give you the command.

Can you share the confoing of all your routers involved in that design!

[EM1]

I was under the impression that the packet tracer didn't work for inbound connections over s2s links as it doesn't handle encryption properly, but if thats not true, here they are:

Google > Office (working):

packet-tracer input Outside_Internet icmp 10.3.0.6 8 0 10.1.1.26 deta$

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in   10.1.1.0        255.255.255.0   Inside_trust_3

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,Outside_Internet) source static All_Internal_10 All_Internal_10 destination static obj-gcp obj-gcp no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Inside_trust_3
Untranslate 10.1.1.26/0 to 10.1.1.26/0

Phase: 3
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any    
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e2a670, priority=7, domain=conn-set, deny=false
	hits=318754, user_data=0x7fff33e1ef80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 4
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (any,Outside_Internet) source static All_Internal_10 All_Internal_10 destination static obj-gcp obj-gcp no-proxy-arp route-lookup
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff348ea800, priority=6, domain=nat, deny=false
	hits=40573, user_data=0x7fff32f2f140, cs_id=0x0, flags=0x0, protocol=0
	src ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0
        dst ip/id=10.1.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32047fd0, priority=0, domain=nat-per-session, deny=true
	hits=32114825, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=any, output_ifc=any

Phase: 6
Type: ACCESS-LIST
Subtype: 
Result: DROP
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32efa030, priority=0, domain=permit, deny=true
	hits=2354590, user_data=0x9, cs_id=0x0, use_real_addr, flags=0x1000, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Result:
input-interface: Outside_Internet
input-status: up
input-line-status: up
output-interface: Inside_trust_3
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Amazon to Office (working):

packet-tracer input Outside_Internet icmp 10.2.103.95 8 0 10.1.1.26 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in   10.1.1.0        255.255.255.0   Inside_trust_3

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (any,Outside_Internet) source static All_Internal_10 All_Internal_10 destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Inside_trust_3
Untranslate 10.1.1.26/0 to 10.1.1.26/0

Phase: 3
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any    
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e2a670, priority=7, domain=conn-set, deny=false
	hits=319242, user_data=0x7fff33e1ef80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 4
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (any,Outside_Internet) source static All_Internal_10 All_Internal_10 destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff342f27b0, priority=6, domain=nat, deny=false
	hits=110080, user_data=0x7fff34068290, cs_id=0x0, flags=0x0, protocol=0
	src ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0
        dst ip/id=10.1.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 5
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32047fd0, priority=0, domain=nat-per-session, deny=true
	hits=32121883, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=any, output_ifc=any

Phase: 6
Type: ACCESS-LIST
Subtype: 
Result: DROP
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32efa030, priority=0, domain=permit, deny=true
	hits=2356229, user_data=0x9, cs_id=0x0, use_real_addr, flags=0x1000, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Result:
input-interface: Outside_Internet
input-status: up
input-line-status: up
output-interface: Inside_trust_3
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Amazon to Google (NOT working)

packet-tracer input Outside_Internet icmp 10.2.103.95 8 0 10.3.0.7 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in   10.3.0.0        255.255.0.0     via xxxxx, Outside_Internet

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Outside_Internet
Untranslate 10.3.0.7/0 to 10.3.0.7/0

Phase: 3
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any    
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e2a670, priority=7, domain=conn-set, deny=false
	hits=319402, user_data=0x7fff33e1ef80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 4
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
Static translate 10.2.103.95/0 to 10.2.103.95/0
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f94a30, priority=6, domain=nat, deny=false
	hits=5, user_data=0x7fff33c4d0b0, cs_id=0x0, flags=0x0, protocol=0
        src ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=Outside_Internet

Phase: 5
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f90b70, priority=3, domain=permit, deny=false
	hits=143479, user_data=0x0, cs_id=0x0, flags=0x4000, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=Outside_Internet

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32047fd0, priority=0, domain=nat-per-session, deny=true
	hits=32124156, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f01f00, priority=0, domain=inspect-ip-options, deny=true
	hits=79691676, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:       
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e0f2c0, priority=70, domain=inspect-icmp, deny=false
	hits=6867, user_data=0x7fff33a6f770, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
	src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e1cd30, priority=70, domain=inspect-icmp-error, deny=false
	hits=6867, user_data=0x7fff33e12290, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
        src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff348bad70, priority=70, domain=ipsec-tunnel-flow, deny=false
	hits=205, user_data=0x37f4c94, cs_id=0x7fff32503ff0, reverse, flags=0x0, protocol=0
	src ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Result:
input-interface: Outside_Internet
input-status: up
input-line-status: up
output-interface: Outside_Internet
output-status: up
output-line-status: up
Action: drop  
Drop-reason: (acl-drop) Flow is denied by configured rule

Google to Amazon (Not working)

packet-tracer input Outside_Internet icmp 10.3.0.7 8 0 10.2.103.95 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in   10.2.0.0        255.255.0.0     via xxxxx, Outside_Internet

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Outside_Internet
Untranslate 10.2.103.95/0 to 10.2.103.95/0

Phase: 3
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any    
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e2a670, priority=7, domain=conn-set, deny=false
	hits=320703, user_data=0x7fff33e1ef80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 4
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
Static translate 10.3.0.7/0 to 10.3.0.7/0
 Forward Flow based lookup yields rule:
 in  id=0x7fff3439ac50, priority=6, domain=nat, deny=false
	hits=120, user_data=0x7fff34066cd0, cs_id=0x0, flags=0x0, protocol=0
        src ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=Outside_Internet

Phase: 5
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f90b70, priority=3, domain=permit, deny=false
	hits=143574, user_data=0x0, cs_id=0x0, flags=0x4000, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=Outside_Internet

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32047fd0, priority=0, domain=nat-per-session, deny=true
	hits=32146638, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f01f00, priority=0, domain=inspect-ip-options, deny=true
	hits=79734189, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:       
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e0f2c0, priority=70, domain=inspect-icmp, deny=false
	hits=6868, user_data=0x7fff33a6f770, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
	src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e1cd30, priority=70, domain=inspect-icmp-error, deny=false
	hits=6868, user_data=0x7fff33e12290, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
        src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33d62070, priority=70, domain=ipsec-tunnel-flow, deny=false
	hits=4231, user_data=0x0, cs_id=0x7fff32503ff0, reverse, flags=0x0, protocol=0
	src ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0
	dst ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Result:
input-interface: Outside_Internet
input-status: up
input-line-status: up
output-interface: Outside_Internet
output-status: up
output-line-status: up
Action: drop  
Drop-reason: (acl-drop) Flow is denied by configured rule

The attached ASA is the only router config. The endpoints on Google and Amazon are setup through their webui and have limited configuration options.

Thanks!

[Francesco Molino]

Hi

Can you make sure your nat(outside,outside) is at the first position?

[EM1]

I moved that rule up to the top, no change in outcome.

Thanks

[Francesco Molino]

This is an issue with crypto acl or acl on your ASA.
Can you attach the config in text format please to have an updated one.

Based on the config I saw at the beginning, you have vpn-filter applied. On amazon you authorized any to amazon subnet and vice versa. However, on google, only internal LAN to google and vice versa. The amazon subnet isn't authorized. Can you modify it?

[EM1]

I've attached the latest config here, which contains all the suggested changes, including adding amazon to the google cryptomap.

[Francesco Molino]

I don't see your acl on your outside interface to allow that traffic.

Can you add that and test please:

access-list outside extended permit ip object obj-amzn object obj-gcp 

access-list outside extended permit ip object obj-gcp object obj-amzn

access-group outside in interface Outside_Internet

Can you also the packet-tracer please?

[EM1]

I've added those lines, but nothing has changed, no traffic flows either way, and pinging from google>amazon drops the google tunnel. Output of the packet tracer:

AWS > Google

packet-tracer input Outside_Internet icmp 10.2.103.95 8 0 10.3.0.7 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in   10.3.0.0        255.255.0.0     via <external IP Address>, Outside_Internet

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Outside_Internet
Untranslate 10.3.0.7/0 to 10.3.0.7/0

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside in interface Outside_Internet
access-list outside extended permit ip object obj-amzn object obj-gcp 
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff339d6f40, priority=13, domain=permit, deny=false
	hits=0, user_data=0x7fff2b576d80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
	src ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 4
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e2a670, priority=7, domain=conn-set, deny=false
	hits=683881, user_data=0x7fff33e1ef80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 5
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
Static translate 10.2.103.95/0 to 10.2.103.95/0
 Forward Flow based lookup yields rule:
 in  id=0x7fff33b18630, priority=6, domain=nat, deny=false
	hits=4, user_data=0x7fff347de6b0, cs_id=0x0, flags=0x0, protocol=0
	src ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=Outside_Internet

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32047fd0, priority=0, domain=nat-per-session, deny=true
	hits=37701204, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f01f00, priority=0, domain=inspect-ip-options, deny=true
	hits=86356172, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW 
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e0f2c0, priority=70, domain=inspect-icmp, deny=false
	hits=9037, user_data=0x7fff33a6f770, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
	src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e1cd30, priority=70, domain=inspect-icmp-error, deny=false
        hits=9037, user_data=0x7fff33e12290, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
	src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff343407e0, priority=70, domain=ipsec-tunnel-flow, deny=false
	hits=3, user_data=0x0, cs_id=0x7fff33924990, reverse, flags=0x0, protocol=0
	src ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0
	dst ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Result:
input-interface: Outside_Internet
input-status: up
input-line-status: up
output-interface: Outside_Internet
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

Google to AWS:

Prowler# packet-tracer input Outside_Internet icmp 10.3.0.7 8 0 10.2.103.95 detailed

Phase: 1
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
in   10.2.0.0        255.255.0.0     via <external IP Address>, Outside_Internet

Phase: 2
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface Outside_Internet
Untranslate 10.2.103.95/0 to 10.2.103.95/0

Phase: 3
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group outside in interface Outside_Internet
access-list outside extended permit ip object obj-gcp object obj-amzn 
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33d5eb30, priority=13, domain=permit, deny=false
	hits=0, user_data=0x7fff2b576b00, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
	src ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 4
Type: CONN-SETTINGS
Subtype: 
Result: ALLOW
Config:
class-map class-default
 match any
policy-map global_policy
 class class-default
  set connection decrement-ttl
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e2a670, priority=7, domain=conn-set, deny=false
	hits=683904, user_data=0x7fff33e1ef80, cs_id=0x0, use_real_addr, flags=0x0, protocol=0
        src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 5
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (Outside_Internet,Outside_Internet) source static obj-gcp obj-gcp destination static obj-amzn obj-amzn no-proxy-arp route-lookup
Additional Information:
Static translate 10.3.0.7/0 to 10.3.0.7/0
 Forward Flow based lookup yields rule:
 in  id=0x7fff341c0730, priority=6, domain=nat, deny=false
	hits=2, user_data=0x7fff32fb4b60, cs_id=0x0, flags=0x0, protocol=0
	src ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.2.0.0, mask=255.255.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=Outside_Internet

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32047fd0, priority=0, domain=nat-per-session, deny=true
	hits=37705600, user_data=0x0, cs_id=0x0, reverse, use_real_addr, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=any, output_ifc=any

Phase: 7
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32f01f00, priority=0, domain=inspect-ip-options, deny=true
	hits=86361902, user_data=0x0, cs_id=0x0, reverse, flags=0x0, protocol=0
	src ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW 
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e0f2c0, priority=70, domain=inspect-icmp, deny=false
	hits=9038, user_data=0x7fff33a6f770, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
	src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff33e1cd30, priority=70, domain=inspect-icmp-error, deny=false
        hits=9038, user_data=0x7fff33e12290, cs_id=0x0, use_real_addr, flags=0x0, protocol=1
	src ip/id=0.0.0.0, mask=0.0.0.0, icmp-type=0, tag=0
	dst ip/id=0.0.0.0, mask=0.0.0.0, icmp-code=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow
Result: DROP
Config:
Additional Information:
 Forward Flow based lookup yields rule:
 in  id=0x7fff32fbdfc0, priority=70, domain=ipsec-tunnel-flow, deny=false
	hits=203, user_data=0x3b3c1c4, cs_id=0x7fff33924990, reverse, flags=0x0, protocol=0
	src ip/id=10.3.0.0, mask=255.255.0.0, port=0, tag=0
	dst ip/id=10.0.0.0, mask=255.0.0.0, port=0, tag=0, dscp=0x0
	input_ifc=Outside_Internet, output_ifc=any

Result:
input-interface: Outside_Internet
input-status: up
input-line-status: up
output-interface: Outside_Internet
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

[Francesco Molino]

Hi

Can you follow this doc and share outputs.

Can you attach the full config of the 3 asas (remove what's confidential) , I'll try to reproduce your issue.

It looks like it's not taking the L2L.

I'm in est timezone, are you available end of afternoon to do a troubleshooting session?