Need help understanding packet-tracer on ASA with Anyconnect and ipsec-tunnel-flow drop

[BrianFranklin1984]

We use Anyconnect to access our network from remote locations.  I am trying to ping from my laptop (172.16.95.181) that is connected via Anyconnect to a device (192.168.3.200) that is on the inside network.  This is not working.   Looking at the packet trace, I see it is saying it is being dropped here:  

Phase: 10
Type: VPN
Subtype: ipsec-tunnel-flow

What I can't figure out is why. I know packet tracer sometimes is a little weird when it comes to testing packets in/out VPN, so maybe it is not really telling me where the error is.

I have tried a capture but I don't see any packets at all when I ping from my laptop to the device.

# capture myping interface internet_outside real-time match icmp host 172.16.95.181 host 192.168.3.200

However, if I watch the hitcnt on the ACL, it increments

access-list internet_outside_access_in line 1 extended permit icmp object-group ANYCONNECT_VPN_POOL any (hitcnt=6528) 0x536e52c1 

If I run a capture on the inside interface looking for all ICMP from the device, I see the request come in, but no echo replies (notice the inside_access_in has permit icmp any any at the very top)

# capture myping interface inside real-time match icmp host 192.168.3.200 any 

Warning: using this option with a slow console connection may
         result in an excessive amount of non-displayed packets
         due to performance limitations.

Use ctrl-c to terminate real-time capture


   1: 15:56:52.416253       802.1Q vlan#2 P0 192.168.3.200 > 172.16.95.181 icmp: echo request 
   2: 15:56:57.039899       802.1Q vlan#2 P0 192.168.3.200 > 172.16.95.181 icmp: echo request 
   3: 15:57:02.036115       802.1Q vlan#2 P0 192.168.3.200 > 172.16.95.181 icmp: echo request 
   4: 15:57:07.045407       802.1Q vlan#2 P0 192.168.3.200 > 172.16.95.181 icmp: echo request 

I have included the entire packet trace and sanitized configuration as attachments.  Any help would be greatly appreciated...

[Francesco Molino]

Hi

Can you run the packet tracker the other way around like
packet-tracer input inside icmp 192.168.3.200 8 0 172.16.95.181

And share the output?

[BrianFranklin1984]

Here you go...

# packet-tracer input inside icmp 192.168.3.200 8 0 172.16.95.181
WARNING: 5 sec waittime expire start 6363473, end 6363474,flags 0, trace 0x0000000008cdae77/0x0000000008cdae77

Phase: 1
Type: ACCESS-LIST
Subtype: 
Result: ALLOW
Config:
Implicit Rule
Additional Information:
MAC Access list

Phase: 2
Type: ROUTE-LOOKUP
Subtype: Resolve Egress Interface
Result: ALLOW
Config:
Additional Information:
found next-hop 172.16.95.181 using egress ifc  internet_outside

Phase: 3
Type: UN-NAT
Subtype: static
Result: ALLOW
Config:
nat (inside,internet_outside) source static any any destination static NETWORK_OBJ_172.16.95.0_24 NETWORK_OBJ_172.16.95.0_24 no-proxy-arp route-lookup
Additional Information:
NAT divert to egress interface internet_outside
Untranslate 172.16.95.181/0 to 172.16.95.181/0

Phase: 4
Type: ACCESS-LIST
Subtype: log
Result: ALLOW
Config:
access-group inside_access_in in interface inside
access-list inside_access_in extended permit icmp any any 
Additional Information:

Phase: 5
Type: NAT
Subtype: 
Result: ALLOW
Config:
nat (inside,internet_outside) source static any any destination static NETWORK_OBJ_172.16.95.0_24 NETWORK_OBJ_172.16.95.0_24 no-proxy-arp route-lookup
Additional Information:
Static translate 192.168.3.200/0 to 192.168.3.200/0

Phase: 6
Type: NAT
Subtype: per-session
Result: ALLOW
Config:
Additional Information:

Phase: 7
Type: IP-OPTIONS
Subtype: 
Result: ALLOW
Config:
Additional Information:

Phase: 8
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
class-map inspection_default
 match default-inspection-traffic
policy-map global_policy
 class inspection_default
  inspect icmp 
service-policy global_policy global
Additional Information:
              
Phase: 9
Type: INSPECT
Subtype: np-inspect
Result: ALLOW
Config:
Additional Information:

Phase: 10
Type: VPN
Subtype: encrypt
Result: DROP
Config:
Additional Information:

Result:
input-interface: inside
input-status: up
input-line-status: up
output-interface: internet_outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by configured rule

[Francesco Molino]

I put your config sample into my lab asa and everything works as expected.
Can you share your full config?

From your inside machine, can you try a real ping to your anyconnect client?
If you capture on your ASA on the inside, do you see the echo and echo-reply?

Can you share the output of the command: sh vpn-sessiondb anyconnect filter name xxxx
--> Replace xxxx by your vpn username

[BrianFranklin1984]

95% of the config was already in the file.  I left out the very specific stuff that will not get hit in the ACLs (TCP ports, etc)

The data file has the output you requested and also a few more captures

I really appreciate the help trying to figure this out...

[Francesco Molino]

Can you PM me directly to organize a troubleshooting session together and we will put here in the post our findings.
I'm missing something but with your exact config everything works just fine on my side.

Just to make sure, because it's obvious but I just want we're on the same page. The host you're trying to ping connected on VPN has no firewall blocking icmp?
Can you adjust your acl and try a tcp connection for example instead of ICMP?

[BrianFranklin1984]

Hi Francesco,

  I haven't forgotten about this, we just have a bit of another fire I am dealing with at the moment. 

To answer your question, there is another line in the ACL that allows RDP to the inside host from the VPN and it is working fine.  I also tried pinging the host from a different subnet, but that does not go through the FW and it is able to ping fine, therefore there is no FW on the host that should be blocking the ping...

Once this fire settles I will send you a PM and we can troubleshoot further.  It will probably be next week before I get back to you...

I appreciate the help!

-Brian

[Supermantech]

packet tracer with VPN only works with the "detailed encrypted" command argument,  and that only works on ASA version 9.9 and above.   I have not tried it myself,  but I see others have.