Strongswan IPSEC VPN with Cisco 7201 sudden failure

[Deltarune]

Hello,
We are encountering a very annoying problem with our IPSEC IKEv1 connection between a cloud server with Strongswan and a Cisco 7201 VPN endpoint, the connection is stuck in the "Connecting" status on the server side.
The IPSEC configuration functioned without problems for more than a year, and last month the same symptoms manifested, a full reboot on our side and a “clear crypto session” si “clear crypto sa peer” on the endpoint side did not help.
The connection reestablished itself after about 12h, and functioned without problems since then. 8 hours ago the connection dropped, and again, an ipsec stop / ipsec start on our side and a “clear crypto session” si “clear crypto sa peer” on the endpoint side did not help.
The only noticeable thing is that the ping between our two sites varies between 5-180ms and a tracepath shows that the route is asymmetric.

Configs and logs below, the server has the public IP Server_IP, the VPN endpoint has the Endpoint_IP, and offers the VPN_Subnet.

Cisco 7201: Version 15.0(1)M5


crypto map CLIENT 830 ipsec-isakmp

description Company

set peer Server_IP

set transform-set ts_company

match address vpn-company-ipsec2


#sh ip access-lists vpn-company-ipsec2

Extended IP access list vpn-company-ipsec2

10 permit ip VPN_Subnet 0.0.0.255 host Server_IP (378222656 matches)

sh crypto ipsec transform-set ts_company

Transform set ts_company: { esp-256-aes esp-sha-hmac  }

   will negotiate = { Tunnel,  },

 

#crypto ipsec transform-set ts_company esp-aes 256 esp-sha-hmac

ipsec.conf:

conn %default
ikelifetime=3h
rekeymargin=3m
keyingtries=1
keyexchange=ikev1
conn server-to-endpoint
authby=secret
left=Server_IP
leftfirewall=yes
leftid=Server_IP
right=Endpoint_IP
rightsubnet=VPN_Subnet/24
rightid=Endpoint_IP
auto=route
dpdtimeout=60
dpddelay=30
dpdaction=clear
esp=aes256-sha1!
ike=aes256-sha1-modp1024!

Endpoint side logs:

VPN_GPRS_7201_1#sh crypto isakmp sa | i Server_IP

Server_IP Endpoint_IP MM_NO_STATE 0 ACTIVE

Server_IP Endpoint_IP MM_NO_STATE 0 ACTIVE (deleted)

Endpoint_IP Server_IP MM_NO_STATE 13522 ACTIVE (deleted)

.Sep 23 2021 15:59:10.667: ISAKMP:(0): retransmitting phase 1 MM_NO_STATE...

.Sep 23 2021 15:59:10.667: ISAKMP:(0): Phase 1 negotiation failed with DPD active; deleting IKE/IPSec SAs

.Sep 23 2021 15:59:10.667: ISAKMP:(0):received initial contact, deleting SA

.Sep 23 2021 15:59:10.667: ISAKMP:(0):peer does not do paranoid keepalives.


.Sep 23 2021 15:59:10.667: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (R) MM_SA_SETUP (peer Server_IP)

.Sep 23 2021 15:59:10.667: ISAKMP:(0):peer does not do paranoid keepalives.


.Sep 23 2021 15:59:10.667: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer Server_IP)

.Sep 23 2021 15:59:10.667: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (R) MM_SA_SETUP (peer Server_IP)

.Sep 23 2021 15:59:10.667: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL

.Sep 23 2021 15:59:10.667: ISAKMP:(0):Old State = IKE_R_MM2 New State = IKE_DEST_SA


.Sep 23 2021 15:59:10.667: ISAKMP:(0):deleting SA reason "Death by retransmission P1" state (I) MM_NO_STATE (peer Server_IP)

.Sep 23 2021 15:59:10.667: ISAKMP:(0):deleting node 1461866681 error FALSE reason "IKE deleted"

.Sep 23 2021 15:59:10.667: ISAKMP:(0):deleting node -1191377405 error FALSE reason "IKE deleted"

.Sep 23 2021 15:59:10.667: ISAKMP:(0):Input = IKE_MESG_INTERNAL, IKE_PHASE1_DEL

.Sep 23 2021 15:59:10.667: ISAKMP:(0):Old State = IKE_I_MM1 New State = IKE_DEST_SA

Server side logs:

root@server:~# ipsec status
Routed Connections:
server-to-endpoint{1}:  ROUTED, TUNNEL, reqid 1
server-to-endpoint{1}:   Server_IP/32 === VPN_Subnet/24
Security Associations (0 up, 2 connecting):
   (unnamed)[44]: CONNECTING, Server_IP[%any]...Endpoint_IP[%any]
server-to-endpoint[43]: CONNECTING, Server_IP[%any]...Endpoint_IP[%any]

Sep 23 17:37:57 server charon[3970138]: 07[KNL] creating acquire job for policy Server_IP/32[udp/59740] === Device_IP/32[udp/1025] >
Sep 23 17:37:57 server charon[3970138]: 07[IKE] initiating Main Mode IKE_SA server-to-endpoint[41] to Endpoint_IP
Sep 23 17:37:57 server charon[3970138]: 07[IKE] initiating Main Mode IKE_SA server-to-endpoint[41] to Endpoint_IP
Sep 23 17:37:57 server charon[3970138]: 07[ENC] generating ID_PROT request 0 [ SA V V V V V ]
Sep 23 17:37:57 server charon[3970138]: 07[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:38:01 server charon[3970138]: 09[IKE] sending retransmit 1 of request message ID 0, seq 1
Sep 23 17:38:01 server charon[3970138]: 09[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:38:08 server charon[3970138]: 12[IKE] sending retransmit 2 of request message ID 0, seq 1
Sep 23 17:38:08 server charon[3970138]: 12[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:38:21 server charon[3970138]: 08[IKE] sending retransmit 3 of request message ID 0, seq 1
Sep 23 17:38:21 server charon[3970138]: 08[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:38:44 server charon[3970138]: 05[IKE] sending retransmit 4 of request message ID 0, seq 1
Sep 23 17:38:44 server charon[3970138]: 05[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:39:21 server charon[3970138]: 05[NET] received packet: from Endpoint_IP[500] to Server_IP[500] (996 bytes)
Sep 23 17:39:21 server charon[3970138]: 05[ENC] parsed ID_PROT request 0 [ SA V V V V ]
Sep 23 17:39:21 server charon[3970138]: 05[IKE] received NAT-T (RFC 3947) vendor ID
Sep 23 17:39:21 server charon[3970138]: 05[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Sep 23 17:39:21 server charon[3970138]: 05[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Sep 23 17:39:21 server charon[3970138]: 05[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Sep 23 17:39:21 server charon[3970138]: 05[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:39:21 server charon[3970138]: 05[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:39:21 server charon[3970138]: 05[CFG] selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Sep 23 17:39:21 server charon[3970138]: 05[ENC] generating ID_PROT response 0 [ SA V V V ]
Sep 23 17:39:21 server charon[3970138]: 05[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (140 bytes)
Sep 23 17:39:26 server charon[3970138]: 07[IKE] sending retransmit 5 of request message ID 0, seq 1
Sep 23 17:39:26 server charon[3970138]: 07[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:39:51 server charon[3970138]: 10[JOB] deleting half open IKE_SA with Endpoint_IP after timeout
Sep 23 17:40:42 server charon[3970138]: 08[KNL] creating delete job for CHILD_SA ESP/0x00000000/Endpoint_IP
Sep 23 17:40:42 server charon[3970138]: 08[JOB] CHILD_SA ESP/0x00000000/Endpoint_IP not found for delete
Sep 23 17:40:42 server charon[3970138]: 14[IKE] giving up after 5 retransmits
Sep 23 17:40:42 server charon[3970138]: 14[IKE] establishing IKE_SA failed, peer not responding
Sep 23 17:40:42 server charon[3970138]: 01[KNL] creating acquire job for policy Server_IP/32[udp/35926] === Another_Device/32[udp/1025>
Sep 23 17:40:42 server charon[3970138]: 01[IKE] initiating Main Mode IKE_SA server-to-endpoint[43] to Endpoint_IP
Sep 23 17:40:42 server charon[3970138]: 01[IKE] initiating Main Mode IKE_SA server-to-endpoint[43] to Endpoint_IP
Sep 23 17:40:42 server charon[3970138]: 01[ENC] generating ID_PROT request 0 [ SA V V V V V ]
Sep 23 17:40:42 server charon[3970138]: 01[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:40:46 server charon[3970138]: 06[IKE] sending retransmit 1 of request message ID 0, seq 1
Sep 23 17:40:46 server charon[3970138]: 06[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:40:53 server charon[3970138]: 12[IKE] sending retransmit 2 of request message ID 0, seq 1
Sep 23 17:40:53 server charon[3970138]: 12[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:06 server charon[3970138]: 10[IKE] sending retransmit 3 of request message ID 0, seq 1
Sep 23 17:41:06 server charon[3970138]: 10[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server charon[3970138]: 14[NET] received packet: from Endpoint_IP[500] to Server_IP[500] (996 bytes)
Sep 23 17:41:24 server charon[3970138]: 14[ENC] parsed ID_PROT request 0 [ SA V V V V ]
Sep 23 17:41:24 server ipsec[3970138]: 12[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 09[IKE] sending retransmit 2 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 09[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 13[IKE] sending retransmit 3 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 13[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] sending retransmit 4 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 14[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] sending retransmit 5 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 05[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 05[KNL] creating delete job for CHILD_SA ESP/0x00000000/Endpoint_IP
Sep 23 17:41:24 server ipsec[3970138]: 05[JOB] CHILD_SA ESP/0x00000000/Endpoint_IP not found for delete
Sep 23 17:41:24 server ipsec[3970138]: 16[KNL] creating delete job for CHILD_SA ESP/0x00000000/Endpoint_IP
Sep 23 17:41:24 server ipsec[3970138]: 16[JOB] CHILD_SA ESP/0x00000000/Endpoint_IP not found for delete
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] giving up after 5 retransmits
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] establishing IKE_SA failed, peer not responding
Sep 23 17:41:24 server ipsec[3970138]: 07[KNL] creating acquire job for policy Server_IP/32[udp/59740] === Another_Device/32[udp/1025] w>
Sep 23 17:41:24 server ipsec[3970138]: 07[IKE] initiating Main Mode IKE_SA server-to-endpoint[41] to Endpoint_IP
Sep 23 17:41:24 server ipsec[3970138]: 07[ENC] generating ID_PROT request 0 [ SA V V V V V ]
Sep 23 17:41:24 server ipsec[3970138]: 07[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 09[IKE] sending retransmit 1 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 09[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 12[IKE] sending retransmit 2 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 12[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 08[IKE] sending retransmit 3 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 08[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] sending retransmit 4 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 05[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 05[NET] received packet: from Endpoint_IP[500] to Server_IP[500] (996 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 05[ENC] parsed ID_PROT request 0 [ SA V V V V ]
Sep 23 17:41:24 server charon[3970138]: 14[IKE] received NAT-T (RFC 3947) vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] received NAT-T (RFC 3947) vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 05[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:41:24 server ipsec[3970138]: 05[CFG] selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Sep 23 17:41:24 server ipsec[3970138]: 05[ENC] generating ID_PROT response 0 [ SA V V V ]
Sep 23 17:41:24 server ipsec[3970138]: 05[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (140 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 07[IKE] sending retransmit 5 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 07[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 10[JOB] deleting half open IKE_SA with Endpoint_IP after timeout
Sep 23 17:41:24 server ipsec[3970138]: 08[KNL] creating delete job for CHILD_SA ESP/0x00000000/Endpoint_IP
Sep 23 17:41:24 server ipsec[3970138]: 08[JOB] CHILD_SA ESP/0x00000000/Endpoint_IP not found for delete
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] giving up after 5 retransmits
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] establishing IKE_SA failed, peer not responding
Sep 23 17:41:24 server ipsec[3970138]: 01[KNL] creating acquire job for policy Server_IP/32[udp/35926] === Another_Device/32[udp/1025]>
Sep 23 17:41:24 server ipsec[3970138]: 01[IKE] initiating Main Mode IKE_SA server-to-endpoint[43] to Endpoint_IP
Sep 23 17:41:24 server ipsec[3970138]: 01[ENC] generating ID_PROT request 0 [ SA V V V V V ]
Sep 23 17:41:24 server ipsec[3970138]: 01[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 06[IKE] sending retransmit 1 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 06[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 12[IKE] sending retransmit 2 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 12[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 10[IKE] sending retransmit 3 of request message ID 0, seq 1
Sep 23 17:41:24 server ipsec[3970138]: 10[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 14[NET] received packet: from Endpoint_IP[500] to Server_IP[500] (996 bytes)
Sep 23 17:41:24 server ipsec[3970138]: 14[ENC] parsed ID_PROT request 0 [ SA V V V V ]
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] received NAT-T (RFC 3947) vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Sep 23 17:41:24 server charon[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Sep 23 17:41:24 server ipsec[3970138]: 14[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:41:24 server charon[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Sep 23 17:41:24 server charon[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Sep 23 17:41:24 server charon[3970138]: 14[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:41:24 server charon[3970138]: 14[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:41:24 server charon[3970138]: 14[CFG] selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Sep 23 17:41:24 server charon[3970138]: 14[ENC] generating ID_PROT response 0 [ SA V V V ]
Sep 23 17:41:24 server charon[3970138]: 14[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (140 bytes)
Sep 23 17:41:29 server charon[3970138]: 05[IKE] sending retransmit 4 of request message ID 0, seq 1
Sep 23 17:41:29 server charon[3970138]: 05[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:41:54 server charon[3970138]: 16[JOB] deleting half open IKE_SA with Endpoint_IP after timeout
Sep 23 17:42:11 server charon[3970138]: 14[IKE] sending retransmit 5 of request message ID 0, seq 1
Sep 23 17:42:11 server charon[3970138]: 14[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (180 bytes)
Sep 23 17:42:15 server charon[3970138]: 14[NET] received packet: from Endpoint_IP[500] to Server_IP[500] (996 bytes)
Sep 23 17:42:15 server charon[3970138]: 14[ENC] parsed ID_PROT request 0 [ SA V V V V ]
Sep 23 17:42:15 server charon[3970138]: 14[IKE] received NAT-T (RFC 3947) vendor ID
Sep 23 17:42:15 server charon[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-07 vendor ID
Sep 23 17:42:15 server charon[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-03 vendor ID
Sep 23 17:42:15 server charon[3970138]: 14[IKE] received draft-ietf-ipsec-nat-t-ike-02\n vendor ID
Sep 23 17:42:15 server charon[3970138]: 14[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:42:15 server charon[3970138]: 14[IKE] Endpoint_IP is initiating a Main Mode IKE_SA
Sep 23 17:42:15 server charon[3970138]: 14[CFG] selected proposal: IKE:AES_CBC_256/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_1024
Sep 23 17:42:15 server charon[3970138]: 14[ENC] generating ID_PROT response 0 [ SA V V V ]
Sep 23 17:42:15 server charon[3970138]: 14[NET] sending packet: from Server_IP[500] to Endpoint_IP[500] (140 bytes)
Sep 23 17:42:25 server charon[3970138]: 11[NET] received packet: from Endpoint_IP[500] to Server_IP[500] (996 bytes)

[Deltarune · Accepted solution]

A reboot of our server and a “clear crypto session”, “clear crypto sa peer” on the router did not help, however the connection reestablished after a change of the remote endpoint IP and in the ACL on the router side, followed by a change back.

I see that there are no publically available changelogs for Cisco 7201, version 15.0(1)M5, perhaps there was some IPSEC bug solved in later versions.