Active IPSEC tunnel - ping request go through but response is dropped

[Micccc4]

Hi Everyone, 

We are trying to set up an IPSEC tunnel between ASA and FTD. Nothing strange and normally it works w/o any issues. But in this case it does not..

IPSEC tunnel is established and, in this example being tested with ping traffic (tried other type of flow with the same result):

- ping request from ASA side is being sent over the IPSEC and received on FTD side

- on FTD side it reaches the target IP and the respond is being sent from the target device (verified with packet capture)

- however this response is not taken into the tunnel and seems to dropped, and at the same time the ipsec '#send errors' volume increases

- encryption domains from both peers have been verified and they are identical

My understanding is that since the ipsec parameter #send errors increases, then the response is actually trying to be routed into the tunnel. Which means tha ACLs, NAT Exempt and Encryption domain are OK. 

#debug crypto ipsec, show this error:

IPSEC ERROR: Invalid PF_Key DELETE - sadb_by_spi inbound parameters

And here are the IPSEC monitoring statistics showing just increasing volume on #send errors...

If you ever experienced such problem, or know what the debug error message indicates then thanks in advance if you share it here

[MHM Cisco World]

can I see ASA IPsec config ?
I think you missing add pfs under phaseII 
MHM

[tvotna]

Did any IPSec tunnel ever work on this FTD? Is it Firepower 3100? Did you try "reload"? Can you bring the tunnel down, then "clear asp drop", "clear counters", "clear flow-offload-ipsec statistics", enable "debug crypto ikev2 protocol 4", "debug crypto ikev2 platform 4", "debug crypto ipsec 255" (if you have other tunnels on the box, "debug crypto condition ..." needs to be enabled first by peer IP), initiate the tunnel again and provide "show asp drop", "show counters", "show flow-offload-ipsec statistics", "show crypto ikev2 sa detail", "show crypto ipsec sa detail" and complete debug output just to begin with?

What is unusual is that the number of encrypts doubles the number of encaps. I've never seen this.

[Micccc4]

Thanks to both of you for quick replies.

- I have requested config from the other side and should get it by tomorrow.

- indeed @tvotna - this is the new FP3110 and it's the first and, for the time being, the only one IPSEC tunnel on it. Is there any known issues with FP3100 and ipsec? 

- I have also noted the doubled volume if encrypted packets - unusual.. I thought that maybe the packets are being resent??

I am planning to continue troubleshooting tomorrow and friday - I will update you with the findings.

[MHM Cisco World]

Do you use ikev1 or ikev2?

If ikev2 then check pfs and group.

Sometimes one side accept group but in real the traffic drop due to mismatch.

Share all debug you get tomorrow as text let me check it

MHM

[tvotna]

@Micccc4, FP3100 has unique IPSec features, e.g. IPSec flow offload, but I'm not aware of specific bugs for this platform. IPSec itself doesn't resend dropped packets, so I have no idea how it is possible that the number of encrypts doubles the number of encaps. Can this be somehow caused by IPSec flow offload?

@MHM Cisco World, I don't quite understand how PFS group mismatch can lead to issues like this. In case of PFS group mismatch initial child SA can be created successfully, because IKE_AUTH uses configured IKEv2 D-H group, rather than PFS D-H group, but CREATE_CHILD_SA should always fail due to group mismatch. This means that user traffic can only be dropped during IPSec rekey, as CREATE_CHILD_SA is used to rekey IPSec SA. (I'm not aware of cases when CREATE_CHILD_SA is used to create initial IPSec SA, although they might exist. But in this case child SA wouldn't be created, although on the attached screenshot we can clearly see that IPSec SA is created and is active).