IPSEC vpn IKEv1 - some remote subnet are not able to reach HQ server

[AirSail]

Hello ASA Gurus, 

Running into a crazy issue here, we have a VPN concentrator (ASA5516), used a concentrator for more than 100 remote sites (3rf party companies that want access to a specific app) 

recently we noticed an issue, to explain correctly below an exemple

remote site X ( remote Subnet A, B, C) |Meraki Firewall|---<IPSEC VPN IKEV1> --->|ASA55160| local host (Z). 

phase 1 and phase 2 are ok, the customer is able to reach Z from subnets A and C, but he recently noticed that subnet B isn't able to reach anymore to host Z, 

no changes have been made, the main concern that this behavior is now affection other VPN, 

Can someone suggest a diagnostic approach here? I checked #show cryp ipsec sa peer <remote peer> from ASA5516, I can see different SAs for subnets A and C with encry/decry increments except for B, 

NOTE: host Z can reach out to all subnets, A/B/C. which is weird, 

Waiting for your feedback folks, 

[MHM Cisco World]

points to check 
1- are you config any PBR in interface B ?
2- are you sure you exclude the A/B/C/Z from the dynamic NAT ??

[AirSail]

1- Can you please help me w/ the show cmd to check an interface PBR ? 

2- I have A/B/C belong to a network-group, that network-group name is called in the NAT exemption line (inside, outside) static/static for the source(host Z), static/static for the destination (remote site network-group)

and yes, interface Z is exempted from NAT, as I said we have subne A and C working just fine host Z 

[MHM Cisco World]

I correct my comment check it again
thanks 

[AirSail]

@MHM Cisco World  - updated above  

[MHM Cisco World]

OK, 
to be sure that no nat add above the exception NAT lead to this case in ASA, please do packet tracer in ASA 

for PBR only check the interface of lan  B, did you see any policy-route ??

[AirSail]

Hello @MHM

yes, totally Agree, I did packet trace and I can clearly see that the right NAT is matched,  

show run inter, inside interface isn't showing any policy-route, 

I did quite a few searches, and the topics I see are talking about VPN issues with multiple subnets, which are similar to my case but non of the resolutions fits, 

I'm suspecting the remote site is not presenting that subnet (child SA) correctly, may be a wrong mask or something, what do you think? 

what is the best and most accurate debugging I have to turn on and what are the right parameters to look at during debugs analytics? 

[MHM Cisco World]

but you run IKEv1 not IKEv2. 
can you share packet-tracer 

[AirSail]

Hello @MHM

Yes, I have Ikev1 VPN

for Packet-trace it may content some sensitive name/data, 

but I Can confirm that I went through the output of the packet trace in both directions I can see always the right NAT is matched, 

[MHM Cisco World]

show crypto ipsec sa <<- share this if you can 

[AirSail]

Attached doc, with the necessary comments, Thank you MHM 

[MHM Cisco World]

[AirSail]

to be honest, seems I have a hard time understanding all meaning of that diagram, 

I just drafted something in paint see below how it my case look like

if you could comment on it it will be great, 

in the meantime, what are the best IPsec debugging levels I have to enable to get detailed info while troubleshooting? 

[MHM Cisco World]

I prefer make debug last option it hard to CPU. 
any way now, if the Z can reach all subnet can you confirm that the encrypt and decrypt count increase in ASA IPSec SA?

[AirSail · Accepted solution]

the fix was the subnet size of subnet B/20 (affected subnet) is bigger than what is configured in the ASA as a remote subnet B/22. 

by aligning subnets from both ends, the subnet was able to communicate.