Site-to-site VPN failes to establish phase 1 IKEv1 or IKEv2

Mate.Barna · ‎02-08-2023

Hi All,

Out of the blue one of our customer's S2S VPN has gone down and doesn't come up since. We have hundreds of other tunnels working just fine.

The remote end - 999.999.999.25, peer named "WIBBLE-F2F" - says they have done no changes, and we didn't either - the usual network phenomenon!

Initially, we tried changing phase 1 and 2 details and policy order on the local ASA (111.111.111.133), ran multiple debugs and packet traces and now we started using IKEv1 to no avail. Also checked traceroutes, access rules etc.

Debug is attached below for both IKEv2 and IKEv1. Configuration for IKEv1 is also attached. The device isn't behind NAT.

Does anyone have some pointers for local troubleshooting? I've asked the remote end to do as such, but that will take a while and I would like to cover all I can.

The remote end firewall is unknown, they won't tell me due to security. Our local end is a Cisco 5585 running ASA 9.12(4)35

MHM Cisco World · ‎02-08-2023

tunnel-group 999.999.999.25 ipsec-attributes
 ikev1 pre-shared-key *****
 ikev2 remote-authentication pre-shared-key ***** <<- you use IKEv1 and use pre-shared key IKEv2, correct this and check again
 ikev2 local-authentication pre-shared-key *****

Mate.Barna · ‎02-08-2023

Hi!

The ikev1 PSK is also specified above there, so thought this shouldn't affect it when switching between IKEv1 / IKEv2 during troubleshooting.

I have now removed the ikev2 psk specific lines from the ipsec-attributes bit, reset all connections and am still getting the exact same output in debug and ASDM logs (see attached).

MHM Cisco World · ‎02-08-2023

first we will use IKEv1,
so you share config of tunnel-group and group-policy
can you share the config of
IKEv1 policy
IPsec IKEv1 transform
and crypto map ?

Mate.Barna · ‎02-08-2023

@MHM Cisco World wrote:
first we will use IKEv1,
so you share config of tunnel-group and group-policy
can you share the config of
IKEv1 policy
IPsec IKEv1 transform
and crypto map ?

Hi MHM,

Very good point, see below:

Crypto Map

crypto map vpnmap 37 match address N3_cryptomap_5
crypto map vpnmap 37 set pfs group5
crypto map vpnmap 37 set peer WIBBLE-F2F
crypto map vpnmap 37 set ikev1 transform-set AES256
crypto map vpnmap 37 set security-association lifetime seconds 3600

IKEv1 Policy Set - Negotiated; remote site set equal to policy 4

crypto ikev1 policy 1
authentication pre-share
encryption aes-256
hash sha
group 2
lifetime 28800
crypto ikev1 policy 2
authentication pre-share
encryption aes-256
hash sha
group 2
lifetime 86400
crypto ikev1 policy 3
authentication pre-share
encryption aes
hash sha
group 2
lifetime 3600
crypto ikev1 policy 4
authentication pre-share
encryption aes-256
hash sha
group 5
lifetime 28800
crypto ikev1 policy 5
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 28800
crypto ikev1 policy 6
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 28800
crypto ikev1 policy 7
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
crypto ikev1 policy 8
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400
crypto ikev1 policy 9
authentication pre-share
encryption aes
hash sha
group 2
lifetime 28800
crypto ikev1 policy 10
authentication pre-share
encryption aes-256
hash sha
group 5
lifetime 86400

IKEv1 IPsec Transform-Set

crypto ipsec ikev1 transform-set AES256 esp-aes-256 esp-sha-hmac

MHM Cisco World · ‎02-08-2023

crypto map vpnmap 37 match address N3_cryptomap_5
crypto map vpnmap 37 set pfs group5 <<- remove group5 if you can
crypto map vpnmap 37 set peer WIBBLE-F2F <<- I know you use IP->name in your config but it better use direct IP instead of name
crypto map vpnmap 37 set ikev1 transform-set AES256
crypto map vpnmap 37 set security-association lifetime seconds 3600

Mate.Barna · ‎02-08-2023

@MHM Cisco World wrote:
crypto map vpnmap 37 match address N3_cryptomap_5
crypto map vpnmap 37 set pfs group5 <<- remove group5 if you can
crypto map vpnmap 37 set peer WIBBLE-F2F <<- I know you use IP->name in your config but it better use direct IP instead of name
crypto map vpnmap 37 set ikev1 transform-set AES256
crypto map vpnmap 37 set security-association lifetime seconds 3600

Hi MHM,

Thanks for the suggestion, removed pfs and naming for this peer, still getting duplicate packet errors. I have a feeling the issue might just be on the other side

Rob Ingram · ‎02-08-2023

@Mate.Barna if the VPN was previously working then randomly changing settings is unlikely to resolve the issue.

It seems like a communication issue from the debug provided, can you ping the remote peer IP address from your device?

Is there another firewall or ACL in the path that could be blocking the IKE/ESP communcation? Take a packet capture (filter the peer IP address) attempt to establish the tunnel and check the output in the capture (upload here).

Mate.Barna · ‎02-08-2023

@Rob Ingram wrote:
@Mate.Barnaif the VPN was previously working then randomly changing settings is unlikely to resolve the issue.
It seems like a communication issue from the debug provided, can you ping the remote peer IP address from your device?
Is there another firewall or ACL in the path that could be blocking the IKE/ESP communcation? Take a packet capture (filter the peer IP address) attempt to establish the tunnel and check the output in the capture (upload here).

Hi Rob,

Thanks for the input, I figured it was worth a try at least!
The only thing between our firewall and theirs on our end is our provider, traceroute confirmed that packets do arrive on the remote network. I can also see responses coming in just fine on UDP 500.
It's also important to note this tunnel isn't routed via the internet, but a leased line on HSCN, a private network for the NHS in UK, so ping will mostly be denied on firewalls connecting to this network.

The packet capture looks as attached - I'm unable to upload this raw as I'm hiding IP addresses for security purposes, anything .133 is our local firewall, .25 is the remote end; let me know if you need to see any layer-specific information other than the example snippet below.

Rob Ingram · ‎02-08-2023

@Mate.Barna I'm very familar with HSCN/N3 network.

Looks like there is bi-directional traffic, so there is a response at least. I do notice there are 2 x established SPIs, can you send me that packet capture file please (send DM) so I can have a closer look.

If you run "show crypto ikev1 sa" / "show crypto isakmp sa" are there any IKEv1 SA for this peer?

A full debug using "debug crypto ikev1 127" and "debug crypto ipsec 127" would be helpful.

https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113574-tg-asa-ipsec-ike-debugs-main-00.html

Mate.Barna · ‎02-08-2023

@Rob Ingram wrote:
@Mate.BarnaI'm very familar with HSCN/N3 network.
Looks like there is bi-directional traffic, so there is a response at least. I do notice there are 2 x established SPIs, can you send me that packet capture file please (send DM) so I can have a closer look.
If you run "show crypto ikev1 sa" / "show crypto isakmp sa" are there any IKEv1 SA for this peer?
A full debug using "debug crypto ikev1 127" and "debug crypto ipsec 127" would be helpful.
https://www.cisco.com/c/en/us/support/docs/security/asa-5500-x-series-next-generation-firewalls/113574-tg-asa-ipsec-ike-debugs-main-00.html

Hi Rob,

crypto sa output shows MM_WAIT_MSG3, which collaborates with what we're seeing in the debugs (see below). I have omitted ipsec rule lookups from it as they were irrelevant for the traffic selectors in question (we are failing on phase 1).

I'm unable to send you the pcap in a DM, unfortunately.

sh crypto isakmp sa / sh crypto ikev1 sa:
71 IKE Peer: 999.999.999.25
Type : user Role : responder
Rekey : no State : MM_WAIT_MSG3

Debug:
5585X-SYSTEM/N3# Feb 08 16:44:06 [IKEv1]IP = 999.999.999.25, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 168
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, processing SA payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, Oakley proposal is acceptable
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, Received DPD VID
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, Received Fragmentation VID
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, Received Fragmentation VID
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, IKE Peer included IKE fragmentation capability flags: Main Mode: True Aggressive Mode: True
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, processing IKE SA payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, IKE SA Proposal # 1, Transform # 1 acceptable Matches global IKE entry # 14
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, constructing ISAKMP SA payload
Feb 08 16:44:06 [IKEv1 DEBUG]IP = 999.999.999.25, constructing Fragmentation VID + extended capabilities payload
Feb 08 16:44:06 [IKEv1]IP = 999.999.999.25, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:44:07 [IKEv1 DEBUG]IP = 999.999.999.25, IKE MM Responder FSM error history (struct &0x00007fc772d87280) <state>, <event>: MM_DONE, EV_ERROR-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent-->MM_SND_MSG2, EV_SND_MSG-->MM_SND_MSG2, EV_START_TMR-->MM_SND_MSG2, EV_RESEND_MSG-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent
Feb 08 16:44:07 [IKEv1 DEBUG]IP = 999.999.999.25, IKE SA MM:d6970f26 terminating: flags 0x01000002, refcnt 0, tuncnt 0
Feb 08 16:44:07 [IKEv1 DEBUG]IP = 999.999.999.25, sending delete/delete with reason message
Feb 08 16:44:10 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.
Feb 08 16:44:14 [IKEv1]IP = 999.999.999.25, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:44:16 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.
Feb 08 16:44:37 [IKEv1]IP = 999.999.999.25, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 168
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, processing SA payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, Oakley proposal is acceptable
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, Received DPD VID
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, Received Fragmentation VID
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, Received Fragmentation VID
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, IKE Peer included IKE fragmentation capability flags: Main Mode: True Aggressive Mode: True
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, processing IKE SA payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, IKE SA Proposal # 1, Transform # 1 acceptable Matches global IKE entry # 14
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, constructing ISAKMP SA payload
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, constructing Fragmentation VID + extended capabilities payload
Feb 08 16:44:38 [IKEv1]IP = 999.999.999.25, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, IKE MM Responder FSM error history (struct &0x00007fc772d3bdf0) <state>, <event>: MM_DONE, EV_ERROR-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent-->MM_SND_MSG2, EV_SND_MSG-->MM_SND_MSG2, EV_START_TMR-->MM_SND_MSG2, EV_RESEND_MSG-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, IKE SA MM:1058bc19 terminating: flags 0x01000002, refcnt 0, tuncnt 0
Feb 08 16:44:38 [IKEv1 DEBUG]IP = 999.999.999.25, sending delete/delete with reason message
Feb 08 16:44:41 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.
Feb 08 16:44:46 [IKEv1]IP = 999.999.999.25, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:44:47 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.
Feb 08 16:44:54 [IKEv1]IP = 999.999.999.25, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:44:59 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.
Feb 08 16:45:02 [IKEv1]IP = 999.999.999.25, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:45:09 [IKEv1]IP = 999.999.999.25, IKE_DECODE RECEIVED Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + VENDOR (13) + VENDOR (13) + VENDOR (13) + NONE (0) total length : 168
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, processing SA payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, Oakley proposal is acceptable
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, Received DPD VID
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, Received Fragmentation VID
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, Received Fragmentation VID
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, IKE Peer included IKE fragmentation capability flags: Main Mode: True Aggressive Mode: True
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, processing VID payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, processing IKE SA payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, IKE SA Proposal # 1, Transform # 1 acceptable Matches global IKE entry # 14
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, constructing ISAKMP SA payload
Feb 08 16:45:09 [IKEv1 DEBUG]IP = 999.999.999.25, constructing Fragmentation VID + extended capabilities payload
Feb 08 16:45:09 [IKEv1]IP = 999.999.999.25, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:45:10 [IKEv1 DEBUG]IP = 999.999.999.25, IKE MM Responder FSM error history (struct &0x00007fc782c35c40) <state>, <event>: MM_DONE, EV_ERROR-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent-->MM_SND_MSG2, EV_SND_MSG-->MM_SND_MSG2, EV_START_TMR-->MM_SND_MSG2, EV_RESEND_MSG-->MM_WAIT_MSG3, EV_TIMEOUT-->MM_WAIT_MSG3, NullEvent
Feb 08 16:45:10 [IKEv1 DEBUG]IP = 999.999.999.25, IKE SA MM:ecbb4152 terminating: flags 0x01000002, refcnt 0, tuncnt 0
Feb 08 16:45:10 [IKEv1 DEBUG]IP = 999.999.999.25, sending delete/delete with reason message
Feb 08 16:45:12 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.
Feb 08 16:45:17 [IKEv1]IP = 999.999.999.25, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 108
Feb 08 16:45:18 [IKEv1]IP = 999.999.999.25, Duplicate first packet detected. Ignoring packet.

MHM Cisco World · ‎02-08-2023

https://www.petenetlive.com/KB/Article/0001531

check this link, I think it solution for your case

in other words, the other side using ACL that deny your ASA reply to ISAKMP, and hence other side send again, here you see the duplicate.
so check if other side using ACL IN direction deny ISKAMP UDP 500 port.

Mate.Barna · ‎02-08-2023

@MHM Cisco World wrote:
https://www.petenetlive.com/KB/Article/0001531
check this link, I think it solution for your case

in other words, the other side using ACL that deny your ASA reply to ISAKMP, and hence other side send again, here you see the duplicate.
so check if other side using ACL IN direction deny ISKAMP UDP 500 port.

Thanks, I have advised the remote side to have a look.

Mate.Barna · ‎02-09-2023

Hi MHM!

Remote side confirmed that nothing is blocking traffic and they can also see bidirectional movement on port 500.
They are using a Fortigate firewall, and seeing the exact same retransmission debugs I do.

We had an hour-long session today, moving between policies, IKEv1-2, NATing etc. to no avail, unfortunately (fresh debug included)

We have agreed to both go back to our network suppliers if they can see anything along the route that would cause this as nothing obvious seems to come to mind for any of us.

Sheraz.Salim · ‎02-12-2023

how you get on with your vpn issue. I had a look into your provided debug. there seem to be a an issue with remote side with IKEV1.

For the IKEV2 I noted in your debug logs "Auth exchange failed" is showing up. if PSK is correct try to lower your phase1 and phase2 setting. and give it a go. if possible doing a remote session (do a screen share and check if they applying the correct setting). as Auth exchange failed could lead to wrong psk but sometime if the key is correct is kind of grey area to tshoot. in my experience lowering down the vpn setting fix the issue.

I used to worked for NHS Lancashire Teaching Hospital. at my time they were upgrading the N3 network to HSCN.

please do not forget to rate.