FlexVPN IKEv2 tunnel between 892FSP and CSR1000v dropping

[mhmservice]

Hi all

Im having a problem with my FlexVPN deployment that im struggling to troubleshoot,

The IKEv2 tunnel to the hubs are dropping approx every 10-15 mins causing a short outage

I have 2x CSR1000v in Azure as the hubs and 1x 892FSP in the site as a spoke

Hub router config is as follows:

crypto ikev2 profile default
 description *** FLEXVPN TO BRANCHES ***
 match identity remote any
 authentication remote pre-share key XXXXXXX
 authentication local pre-share key XXXXX
 aaa authorization group psk list default default
 virtual-template 1 mode auto

crypto ipsec profile default
 set security-association lifetime kilobytes disable
 set security-association lifetime seconds 86400
 set ikev2-profile default

!
crypto ikev2 nat keepalive 30
crypto ikev2 dpd 10 2 periodic
crypto ikev2 fragmentation mtu 1300

Branch router config is as follows:

interface Tunnel1
 description *** TUNNEL TO HUB 1 ***
 ip address negotiated
 ip mtu 1400
 ip nhrp network-id 1
 ip nhrp shortcut virtual-template 1
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet8
 tunnel destination 1.1.1.1
 tunnel vrf FVRF
 tunnel protection ipsec profile default
end

interface Tunnel2
 description *** TUNNEL TO HUB 2 ***
 ip address negotiated
 ip mtu 1400
 ip nhrp network-id 1
 ip nhrp shortcut virtual-template 1
 ip tcp adjust-mss 1360
 tunnel source GigabitEthernet8
 tunnel destination 2.2.2.2
 tunnel vrf FVRF
 tunnel protection ipsec profile default
end

crypto ikev2 profile default
 description *** FLEXVPN TO HUB ***
 match fvrf FVRF
 match identity remote any
 authentication remote pre-share key XXXXXXXXXXXXXX
 authentication local pre-share key XXXXXXXXXXXXX
 aaa authorization group psk list default default
 virtual-template 1

crypto ipsec profile default
 set security-association lifetime kilobytes disable
 set security-association lifetime seconds 86400
 set ikev2-profile default

crypto ikev2 nat keepalive 30
crypto ikev2 dpd 10 2 periodic
crypto ikev2 fragmentation mtu 1300

The last commands I introduced in an attempt to fix the problem but it didn't help

I can see that the hub connections are dropping but the spoke-to-spoke connections are fine (Based on sh crypto ikev2 sa:

BRANCH#sh crypto ikev2 sa
 IPv4 Crypto IKEv2  SA

Tunnel-id Local                 Remote                fvrf/ivrf            Status
4         BRANCH/500      BRANCH/500      FVRF/none            READY
      Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/42950 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
9         BRANCH/4500     BRANCH/4500   FVRF/none            READY
      Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/42222 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
7         LOCAL_PUBLIC_IP/500      BRANCH/500    FVRF/none            READY
      Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/42950 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
3         LOCAL_PUBLIC_IP/4500     HUB2/4500   FVRF/none            READY
      Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/342 sec

Tunnel-id Local                 Remote                fvrf/ivrf            Status
1         LOCAL_PUBLIC_IP/4500     HUB1/4500     FVRF/none            READY
      Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK
      Life/Active Time: 86400/616 sec

I can see from the logs on the branch router the following, but it doesnt give me much info (debug crypto ikev2 and debug crypto ipsec):

49921: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Received Packet [From HUB:4500/To BRANCH:4500/VRF i1:f1]
49922: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 0
49923: IKEv2 INFORMATIONAL Exchange REQUEST
49924: Payload contents:
49925:  DELETE
49926: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Building packet for encryption.
49927: Payload contents:
49928:  DELETE
49929: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Sending Packet [To HUB:4500/From BRANCH:4500/VRF i1:f1]
49930: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 0
49931: IKEv2 INFORMATIONAL Exchange RESPONSE
49932: Payload contents:
49933:  ENCR
49934: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Process delete request from peer
49935: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Processing DELETE INFO message for IPsec SA [SPI: 0xFF2F1FE0]
49936: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Check for existing active SA
49937: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Received Packet [From HUB:4500/To BRANCH:4500/VRF i1:f1]
49938: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 1
49939: IKEv2 INFORMATIONAL Exchange REQUEST
49940: Payload contents:
49941:  DELETE NOTIFY(DELETE_REASON)
49942: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Building packet for encryption.
49943: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Sending Packet [To HUB:4500/From BRANCH:4500/VRF i1:f1]
49944: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 1
49945: IKEv2 INFORMATIONAL Exchange RESPONSE
49946: Payload contents:
49947:  ENCR
49948: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Process delete request from peer
49949: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Processing DELETE INFO message for IKEv2 SA [ISPI: 0xA01342DAA0F39361 RSPI: 0x47DE062E851C7196]
49950: Jul 12 11:15:17: IKEv2:Deleted IKEv2 route <HUB LOOPBACK IP> 255.255.255.255 via Tunnel1 in vrf global
49951: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Check for existing active SA
49952: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Delete all IKE SAs
49953: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Deleting SA
49954: Jul 12 11:15:17: IPSEC(key_engine): got a queue event with 1 KMI message(s)
49955: Jul 12 11:15:17: IDB is NULL : in crypto_ipsec_key_engine_delete_sas (), 5437
49956: Jul 12 11:15:17: IPSEC(key_engine_delete_sas): rec'd delete notify from ISAKMP
49957: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) still in use sa: 0x12E6036C
49958: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (key_engine_delete_sas) delete SA with spi 0x2D0E8CEA proto 50 for BRANCH
49959: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (delete_sa) deleting SA
49960: ,
49961:   (sa) sa_dest= BRANCH, sa_proto= 50,
49962:     sa_spi= 0x2D0E8CEA(755928298),
49963:     sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 158
49964:     sa_lifetime(k/sec)= (0/86400),
49965:   (identity) local= BRANCH:0, remote= HUB:0,
49966:     local_proxy= BRANCH/255.255.255.255/47/0,
49967:     remote_proxy= HUB/255.255.255.255/47/0
49968: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (delete_sa) deleting SA,
49969:   (sa) sa_dest= HUB, sa_proto= 50,
49970:     sa_spi= 0xFF2F1FE0(4281278432),
49971:     sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 157
49972:     sa_lifetime(k/sec)= (0/86400),
49973:   (identity) local= BRANCH:0, remote= HUB:0
49974: ,
49975:     local_proxy= BRANCH/255.255.255.255/47/0,
49976:     remote_proxy= HUB/255.255.255.255/47/0
49977: Jul 12 11:15:17: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS
49978: Jul 12 11:15:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to down
49979: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (ident_delete_notify_kmi) Failed to send KEY_ENG_DELETE_SAS
49980: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (ident_update_final_flow_stats) Collect Final Stats and update MIB
49981: IPSEC get IKMP peer index from peer 0x1255EE34 ikmp handle 0x0
49982: [ident_update_final_flow_stats] : Flow delete complete event received for flow id 0x1400009E,peer index 0

Any help to troubelshoot this in more detail appreciated