No Return traffic for new connection over existing Site-to-Site VPN tunnel

[S.U.H.E.L]

VPN tunnel is set up between an ASAv30 on AWS and ASA5545-X on-premise. Recently, a new connection was added which is having issues, the flow is as follows:
AWS Server (172.18.1.7) -> AWS ASAv30 -> ASA5545-X -> Physical Server (172.16.100.5)

Following ACL and NAT statements were added:

AWS (ASAv30)

access-list VPN extended permit ip host 172.18.1.7 host 172.16.100.5

nat (inside,outside) source static OBJ_172.18.1.7 OBJ_172.18.1.7 destination static OBJ_172.16.100.5 OBJ_172.16.100.5 no-proxy-arp

ON-PREMISE (ASA5545-X)

access-list VPN  extended permit ip host 172.16.100.5 host 172.18.1.7

nat (inside,outside) 20 source static OBJ_172.16.100.5 OBJ_172.16.100.5 destination static OBJ_172.18.1.7 OBJ_172.18.1.7

When ping is initiated from the AWS server as a source and capture are taken, the following is the output:

ASAv30# sh cap mycap

43 packets captured

1: 16:09:06.504398 172.18.1.7 > 172.16.100.5: icmp: echo request
2: 16:09:07.528323 172.18.1.7 > 172.16.100.5: icmp: echo request
3: 16:09:08.552995 172.18.1.7 > 172.16.100.5: icmp: echo request
4: 16:09:09.575974 172.18.1.7 > 172.16.100.5: icmp: echo request
5: 16:09:10.600463 172.18.1.7 > 172.16.100.5: icmp: echo request
6: 16:09:11.624067 172.18.1.7 > 172.16.100.5: icmp: echo request
7: 16:09:12.647839 172.18.1.7 > 172.16.100.5: icmp: echo request
8: 16:09:13.673091 172.18.1.7 > 172.16.100.5: icmp: echo request
9: 16:09:14.696237 172.18.1.7 > 172.16.100.5: icmp: echo request
10: 16:09:15.720284 172.18.1.7 > 172.16.100.5: icmp: echo request

Below capture from destination firewall

ASA5545-X# sh cap mycap

46 packets captured

1: 16:09:25.417230 172.18.1.7 > 172.16.100.5: icmp: echo request
2: 16:09:25.417565 172.16.100.5 > 172.18.1.7: icmp: echo reply
3: 16:09:26.440574 172.18.1.7 > 172.16.100.5: icmp: echo request
4: 16:09:26.440910 172.16.100.5 > 172.18.1.7: icmp: echo reply
5: 16:09:27.465521 172.18.1.7 > 172.16.100.5: icmp: echo request
6: 16:09:27.465780 172.16.100.5 > 172.18.1.7: icmp: echo reply
7: 16:09:28.489140 172.18.1.7 > 172.16.100.5: icmp: echo request
8: 16:09:28.489369 172.16.100.5 > 172.18.1.7: icmp: echo reply
9: 16:09:29.514957 172.18.1.7 > 172.16.100.5: icmp: echo request
10: 16:09:29.515155 172.16.100.5 > 172.18.1.7: icmp: echo reply

Traffic travels over both firewalls, returns to the ASA5545-X but does not return to ASAv30.

When ping is initiated from the Physical server as a source and capture are taken, only ICMP request packets are observed on ASA5545-X but there are logs on ASAv30. 

This means that traffic for some reason is not flowing from ASA5545-X to ASAv30. We need to know if it is blocked at ASA5545-X or ASAv30.

When a packet-capture is run on either firewall, the flow is successful.

[Rob Ingram]

Hi,

Can you run a packet-tracer and provide the output for review.

From the ASAv30 run packet-tracer, e.g.

packet-tracer input inside icmp 172.18.1.7 8 0 172.16.100.5

From the ASA5545X run packet-tracer, e.g.

packet-tracer input inside icmp 172.16.100.5 8 0 172.18.1.7

You could also run capture ASPDROP type asp-drop all at the same you run the ping and observe the output of show capture ASPDROP upload the output for review.

HTH

[S.U.H.E.L]

Please find attached output as requested

[Rob Ingram]

The packet-tracer output looks ok. However from your ASAv30 aspdrop output, we can see the echo reply is dropped.

20473: 13:33:03.365261 172.16.100.5 > 172.18.1.7: icmp: echo reply Drop-reason: (no-adjacency) No valid adjacency

What ASA version are you running?

[S.U.H.E.L]

ASAv30 running the following version:
Cisco Adaptive Security Appliance Software Version 9.9(2)1
Firepower Extensible Operating System Version 2.3(1.84)
Device Manager Version 7.9(2)

ASA 5545-X running the following version:
Cisco Adaptive Security Appliance Software Version 9.8(2)
Firepower Extensible Operating System Version 2.2(2.52)
Device Manager Version 7.8(2)

[Rob Ingram]

Name: no-adjacency
No valid adjacency:
This counter is incremented when the security appliance has tried to obtian an adjacency and could not obtain mac-address for next hop. The packet is dropped.

Recommendation:
Configure a capture for this drop reason and check if a host with specified destination address exists on connected network or is routable from the device.

[S.U.H.E.L]

Could you help me with the command to configure packet capture with that specific reason, please?

That IP is reachable from the firewall, both (the firewall and server) are part of the same VPC on AWS and can communicate with each other locally.

[vsurresh]

Hello.
I am pretty sure you checked this already but just to make sure, do you have any NACL configured on the VPC for on-prem subnet? (NACL is stateless) Have you tried configuring VPC flow logs to make sure that AWS is not dropping any traffic?

If I understand it correctly if you initiate a connection from AWS to on prem-server, the echo request arrives on-prem and an echo reply is sent out but AWS side is not receiving them, right?

[S.U.H.E.L]

Hello vsurresh,

Yes, you understood correctly. Let me elaborate more to make sure we are on the same page.

Echo-request : Source (A server on AWS - 172.18.1.7) --> AWS firewall (ASAv30)   [successful]

Echo-request: AWS firewall (ASAv30) --> On-Premise Firewall (ASA5545X) [successful]

Echo-request: On-Premise Firewall (ASA5545X) --> Destination (A server on-premise - 172.16.100.5) [successful]

Echo-reply: Destination (A server on-premise - 172.16.100.5) --> On-Premise Firewall (ASA5545X) [successful]

Echo-reply: On-Premise Firewall (ASA5545X) --> AWS firewall (ASAv30) [drops after reaching ASAv30]

[vsurresh]

Make sense. I tend to you ASDM for live traffic troubleshooting. Are you able to see on ASDM live logs why the traffic is getting dropped? Just double-check the VPC flow logs to make sure AWS is not the culprit here.

[S.U.H.E.L]

Well, ASDM does not work for some reason and I can only use CLI. Could you help me with the steps to troubleshoot and check VPC flow logs? I strongly suspect AWS to be the culprit here.

There is no config for NACL and its left at default with permit any any.

The existing connection which works is part of a different subnet and the new connection which is having an issue is a different subnet. However, both subnets are associated with the same VPC.

[vsurresh]

Setting up a VPC flow log is very easy, please check out my blog here: https://packetswitch.co.uk/aws-vpc-flow-logs/

Can you please share the AWS VPC diagram? I presume you use a different ASAv interface for the new subnet right? Did you disable source/destination check for the new interface which is very important?

[S.U.H.E.L]

The same interface is used. An additional interface wasn't configured for the new subnet. The ASA interface and new server are both in different subnets. I do not see the server IP 172.18.1.7 in the ASA's ARP table. I tried adding it manually but it still did not work. 

[vsurresh]

Hi,

I double-checked your outputs attached on the earlier message and looks good to me. I am pretty sure ASAs are configured properly.
Have you checked NACLs, SGs and Flow logs on AWS?

[S.U.H.E.L · Accepted solution]

Hi @vsurresh,

The issue is resolved now. Since these servers were on a different subnet from the ASA subnet, a route had to be configured on the firewall to reach them.

Thanks for your help. Thanks @Rob Ingram for pointing out that the packet drop could be due to a missing route.