ASA VPN site-to-site tunnel failed during Phase 2

[ken.attong]

Hello, Thank you in advance! I'm trying to transition from a Cisco 2900 router that is currently setup with a site-to-site VPN to a Checkpoint firewall, with a new Cisco ASA 5500-x firewall running 9.12.x.  I've established that the tunnel has completed IKEV1 Phase 1, and I get a MM Active status briefly, but it then fails during the Phase 2 step.  I ran the following debug command to see what is going on:

ciscoasa# debug crypto isa 127

Results:

--------------------------------------------------------------------------------------------------

ciscoasa# May 26 21:13:15 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Sending keep-alive of type DPD R-U-THERE (seq number
0x462ba58c)
May 26 21:13:15 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:15 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:15 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=eb03acb0) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:15 [IKEv1]IKE Receiver: Packet received on 10.10.10.200:500 from 10.10.10.20:500
May 26 21:13:15 [IKEv1]IP = 10.10.10.20, IKE_DECODE RECEIVED Message (msgid=f2831aaf) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:15 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing hash payload
May 26 21:13:15 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing notify payload
May 26 21:13:15 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Received keep-alive of type DPD R-U-THERE-ACK (seq number
0x462ba58c)
May 26 21:13:18 [IKEv1 DEBUG]Pitcher: received a key acquire message, spi 0x0
May 26 21:13:18 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, IKE Initiator: New Phase 2, Intf outside, IKE Peer 10.10.10.20 local
Proxy Address 192.168.1.0, remote Proxy Address 192.168.5.0, Crypto map (outside_map)
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Oakley begin quick mode
May 26 21:13:18 [IKEv1 DECODE]Group = 10.10.10.20, IP = 10.10.10.20, IKE Initiator starting QM: msg id = f230b881
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE got SPI from key engine: SPI = 0xb9bacb99
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, oakley constucting quick mode
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing IPSec SA payload
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing IPSec nonce payload
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing proxy ID
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Transmitting Proxy Id:
Local subnet: 192.168.1.0 mask 255.255.255.0 Protocol 0 Port 0
Remote subnet: 192.168.5.0 Mask 255.255.255.0 Protocol 0 Port 0
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:18 [IKEv1 DECODE]Group = 10.10.10.20, IP = 10.10.10.20, IKE Initiator sending 1st QM pkt: msg id = f230b881
May 26 21:13:18 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=f230b881) with payloads : HDR + HASH (8) + SA (1) + NONCE
(10) + ID (5) + ID (5) + NONE (0) total length : 172
May 26 21:13:18 [IKEv1]IKE Receiver: Packet received on 10.10.10.200:500 from 10.10.10.20:500
May 26 21:13:18 [IKEv1]IP = 10.10.10.20, IKE_DECODE RECEIVED Message (msgid=b7c57be5) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 80
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing hash payload
May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing notify payload
May 26 21:13:18 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Received non-routine Notify message: No proposal chosen (14)
May 26 21:13:25 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Sending keep-alive of type DPD R-U-THERE (seq number 0x462ba58d)
May 26 21:13:25 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:25 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:25 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=17e7cfed) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:25 [IKEv1]IKE Receiver: Packet received on 10.10.10.200:500 from 10.10.10.20:500
May 26 21:13:25 [IKEv1]IP = 10.10.10.20, IKE_DECODE RECEIVED Message (msgid=eca2d477) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:25 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing hash payload
May 26 21:13:25 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing notify payload
May 26 21:13:25 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Received keep-alive of type DPD R-U-THERE-ACK (seq number
0x462ba58d)
May 26 21:13:31 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, QM FSM error (P2 struct &0x00002aaab886ab70, mess id 0xcaa2dab2)!
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE QM Initiator FSM error history (struct &0x00002aaab886ab70)
<state>, <event>: QM_DONE, EV_ERROR-->QM_WAIT_MSG2, EV_TIMEOUT-->QM_WAIT_MSG2, NullEvent-->QM_SND_MSG1, EV_SND_MSG-->QM_SND_MSG1,
EV_START_TMR-->QM_SND_MSG1, EV_RESEND_MSG-->QM_WAIT_MSG2, EV_TIMEOUT-->QM_WAIT_MSG2, NullEvent
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, sending delete/delete with reason message
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing IPSec delete payload
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:31 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=24f6d6ce) with payloads : HDR + HASH (8) + DELETE (12) +
NONE (0) total length : 68
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE Deleting SA: Remote Proxy 192.168.6.0, Local Proxy
192.168.1.0
May 26 21:13:31 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Removing peer from correlator table failed, no match!
May 26 21:13:31 [IKEv1 DEBUG]Pitcher: received key delete msg, spi 0x82a293e5
May 26 21:13:35 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Sending keep-alive of type DPD R-U-THERE (seq number 0x462ba58e)
May 26 21:13:35 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:35 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:35 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=a7a92e8) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:35 [IKEv1]IKE Receiver: Packet received on 10.10.10.200:500 from 10.10.10.20:500
May 26 21:13:35 [IKEv1]IP = 10.10.10.20, IKE_DECODE RECEIVED Message (msgid=a33992a8) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:35 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing hash payload
May 26 21:13:35 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing notify payload
May 26 21:13:35 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Received keep-alive of type DPD R-U-THERE-ACK (seq number
0x462ba58e)
May 26 21:13:45 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Sending keep-alive of type DPD R-U-THERE (seq number 0x462ba58f)
May 26 21:13:45 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:45 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:45 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=6d2f3313) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:45 [IKEv1]IKE Receiver: Packet received on 10.10.10.200:500 from 10.10.10.20:500
May 26 21:13:45 [IKEv1]IP = 10.10.10.20, IKE_DECODE RECEIVED Message (msgid=a4f20f61) with payloads : HDR + HASH (8) + NOTIFY (11) +
NONE (0) total length : 84
May 26 21:13:45 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing hash payload
May 26 21:13:45 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, processing notify payload
May 26 21:13:45 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Received keep-alive of type DPD R-U-THERE-ACK (seq number
0x462ba58f)
May 26 21:13:50 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, QM FSM error (P2 struct &0x00002aaac28bb580, mess id 0xf230b881)!
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE QM Initiator FSM error history (struct &0x00002aaac28bb580)
<state>, <event>: QM_DONE, EV_ERROR-->QM_WAIT_MSG2, EV_TIMEOUT-->QM_WAIT_MSG2, NullEvent-->QM_SND_MSG1, EV_SND_MSG-->QM_SND_MSG1,
EV_START_TMR-->QM_SND_MSG1, EV_RESEND_MSG-->QM_WAIT_MSG2, EV_TIMEOUT-->QM_WAIT_MSG2, NullEvent
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, sending delete/delete with reason message
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing IPSec delete payload
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:50 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=cac4e539) with payloads : HDR + HASH (8) + DELETE (12) +
NONE (0) total length : 68
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE Deleting SA: Remote Proxy 192.168.5.0, Local Proxy
192.168.1.0
May 26 21:13:50 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Removing peer from correlator table failed, no match!
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE SA MM:116d659e rcv'd Terminate: state MM_ACTIVE flags
0x0000c062, refcnt 1, tuncnt 0
May 26 21:13:50 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Remove from IKEv1 Tunnel Table succeeded for SA with logicalId 77824
May 26 21:13:50 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Remove from IKEv1 MIB Table succeeded for SA with logical ID 77824
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE SA MM:116d659e terminating: flags 0x0100c022, refcnt 0,
tuncnt 0
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, sending delete/delete with reason message
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing blank hash payload
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing IKE delete payload
May 26 21:13:50 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, constructing qm hash payload
May 26 21:13:50 [IKEv1]IP = 10.10.10.20, IKE_DECODE SENDING Message (msgid=657fed81) with payloads : HDR + HASH (8) + DELETE (12) +
NONE (0) total length : 80
May 26 21:13:50 [IKEv1 DEBUG]Pitcher: received key delete msg, spi 0xb9bacb99
May 26 21:13:50 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Session is being torn down. Reason: Lost Service
May 26 21:13:50 [IKEv1]Ignoring msg to mark SA with dsID 77824 dead because SA deleted
May 26 21:13:50 [IKEv1]IKE Receiver: Packet received on 10.10.10.200:500 from 10.10.10.20:500
May 26 21:13:50 [IKEv1]IP = 10.10.10.20, Received encrypted packet with no matching SA, dropping

------------------------------------------------------------------------------

I've tried a number of settings to best match what was setup on the Cisco 2900, this is the closet I've been able to achieve over the past 2 weeks.   Researching forum for possible answers has brought me here.

Thank you Cisco Community, keep up the good work!  Stay Safe!

[cmarva]

in my experience, general rule of thumb is:

phase 1 fail - check PSK

phase 2 fail - check traffic select

from your debugs:

May 26 21:13:18 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, Transmitting Proxy Id:
Local subnet: 192.168.1.0 mask 255.255.255.0 Protocol 0 Port 0
Remote subnet: 192.168.5.0 Mask 255.255.255.0 Protocol 0 Port 0

>snip

NONE (0) total length : 68
May 26 21:13:31 [IKEv1 DEBUG]Group = 10.10.10.20, IP = 10.10.10.20, IKE Deleting SA: Remote Proxy 192.168.6.0, Local Proxy
192.168.1.0

so it seems like a traffic select mismatch, i would start there.

[ken.attong]

Hi cmarva,  Thank you for the response & insight.  I'm checking our NAT rules & ACLs to see if something is out of place, haven't come across any obvious issues. 

Just going over your snip sections from my debug listing, I've been trying to figure out this line :

May 26 21:13:31 [IKEv1]Group = 10.10.10.20, IP = 10.10.10.20, Removing peer from correlator table failed, no match!

Do you know  what this means by "Removing peer from correlator table failed, no match!"

Thanks again!