07-12-2019 03:28 AM - edited 02-21-2020 09:41 PM
Hi all
Im having a problem with my FlexVPN deployment that im struggling to troubleshoot,
The IKEv2 tunnel to the hubs are dropping approx every 10-15 mins causing a short outage
I have 2x CSR1000v in Azure as the hubs and 1x 892FSP in the site as a spoke
Hub router config is as follows:
crypto ikev2 profile default description *** FLEXVPN TO BRANCHES *** match identity remote any authentication remote pre-share key XXXXXXX authentication local pre-share key XXXXX aaa authorization group psk list default default virtual-template 1 mode auto crypto ipsec profile default set security-association lifetime kilobytes disable set security-association lifetime seconds 86400 set ikev2-profile default ! crypto ikev2 nat keepalive 30 crypto ikev2 dpd 10 2 periodic crypto ikev2 fragmentation mtu 1300
Branch router config is as follows:
interface Tunnel1 description *** TUNNEL TO HUB 1 *** ip address negotiated ip mtu 1400 ip nhrp network-id 1 ip nhrp shortcut virtual-template 1 ip tcp adjust-mss 1360 tunnel source GigabitEthernet8 tunnel destination 1.1.1.1 tunnel vrf FVRF tunnel protection ipsec profile default end interface Tunnel2 description *** TUNNEL TO HUB 2 *** ip address negotiated ip mtu 1400 ip nhrp network-id 1 ip nhrp shortcut virtual-template 1 ip tcp adjust-mss 1360 tunnel source GigabitEthernet8 tunnel destination 2.2.2.2 tunnel vrf FVRF tunnel protection ipsec profile default end crypto ikev2 profile default description *** FLEXVPN TO HUB *** match fvrf FVRF match identity remote any authentication remote pre-share key XXXXXXXXXXXXXX authentication local pre-share key XXXXXXXXXXXXX aaa authorization group psk list default default virtual-template 1 crypto ipsec profile default set security-association lifetime kilobytes disable set security-association lifetime seconds 86400 set ikev2-profile default crypto ikev2 nat keepalive 30 crypto ikev2 dpd 10 2 periodic crypto ikev2 fragmentation mtu 1300
The last commands I introduced in an attempt to fix the problem but it didn't help
I can see that the hub connections are dropping but the spoke-to-spoke connections are fine (Based on sh crypto ikev2 sa:
BRANCH#sh crypto ikev2 sa IPv4 Crypto IKEv2 SA Tunnel-id Local Remote fvrf/ivrf Status 4 BRANCH/500 BRANCH/500 FVRF/none READY Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/42950 sec Tunnel-id Local Remote fvrf/ivrf Status 9 BRANCH/4500 BRANCH/4500 FVRF/none READY Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/42222 sec Tunnel-id Local Remote fvrf/ivrf Status 7 LOCAL_PUBLIC_IP/500 BRANCH/500 FVRF/none READY Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/42950 sec Tunnel-id Local Remote fvrf/ivrf Status 3 LOCAL_PUBLIC_IP/4500 HUB2/4500 FVRF/none READY Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/342 sec Tunnel-id Local Remote fvrf/ivrf Status 1 LOCAL_PUBLIC_IP/4500 HUB1/4500 FVRF/none READY Encr: AES-CBC, keysize: 256, PRF: SHA512, Hash: SHA512, DH Grp:5, Auth sign: PSK, Auth verify: PSK Life/Active Time: 86400/616 sec
I can see from the logs on the branch router the following, but it doesnt give me much info (debug crypto ikev2 and debug crypto ipsec):
49921: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Received Packet [From HUB:4500/To BRANCH:4500/VRF i1:f1] 49922: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 0 49923: IKEv2 INFORMATIONAL Exchange REQUEST 49924: Payload contents: 49925: DELETE 49926: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Building packet for encryption. 49927: Payload contents: 49928: DELETE 49929: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Sending Packet [To HUB:4500/From BRANCH:4500/VRF i1:f1] 49930: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 0 49931: IKEv2 INFORMATIONAL Exchange RESPONSE 49932: Payload contents: 49933: ENCR 49934: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Process delete request from peer 49935: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Processing DELETE INFO message for IPsec SA [SPI: 0xFF2F1FE0] 49936: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Check for existing active SA 49937: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Received Packet [From HUB:4500/To BRANCH:4500/VRF i1:f1] 49938: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 1 49939: IKEv2 INFORMATIONAL Exchange REQUEST 49940: Payload contents: 49941: DELETE NOTIFY(DELETE_REASON) 49942: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Building packet for encryption. 49943: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Sending Packet [To HUB:4500/From BRANCH:4500/VRF i1:f1] 49944: Initiator SPI : A01342DAA0F39361 - Responder SPI : 47DE062E851C7196 Message id: 1 49945: IKEv2 INFORMATIONAL Exchange RESPONSE 49946: Payload contents: 49947: ENCR 49948: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Process delete request from peer 49949: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Processing DELETE INFO message for IKEv2 SA [ISPI: 0xA01342DAA0F39361 RSPI: 0x47DE062E851C7196] 49950: Jul 12 11:15:17: IKEv2:Deleted IKEv2 route <HUB LOOPBACK IP> 255.255.255.255 via Tunnel1 in vrf global 49951: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Check for existing active SA 49952: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Delete all IKE SAs 49953: Jul 12 11:15:17: IKEv2:(SESSION ID = 2,SA ID = 3):Deleting SA 49954: Jul 12 11:15:17: IPSEC(key_engine): got a queue event with 1 KMI message(s) 49955: Jul 12 11:15:17: IDB is NULL : in crypto_ipsec_key_engine_delete_sas (), 5437 49956: Jul 12 11:15:17: IPSEC(key_engine_delete_sas): rec'd delete notify from ISAKMP 49957: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) still in use sa: 0x12E6036C 49958: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (key_engine_delete_sas) delete SA with spi 0x2D0E8CEA proto 50 for BRANCH 49959: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (delete_sa) deleting SA 49960: , 49961: (sa) sa_dest= BRANCH, sa_proto= 50, 49962: sa_spi= 0x2D0E8CEA(755928298), 49963: sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 158 49964: sa_lifetime(k/sec)= (0/86400), 49965: (identity) local= BRANCH:0, remote= HUB:0, 49966: local_proxy= BRANCH/255.255.255.255/47/0, 49967: remote_proxy= HUB/255.255.255.255/47/0 49968: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (delete_sa) deleting SA, 49969: (sa) sa_dest= HUB, sa_proto= 50, 49970: sa_spi= 0xFF2F1FE0(4281278432), 49971: sa_trans= esp-aes esp-sha-hmac , sa_conn_id= 157 49972: sa_lifetime(k/sec)= (0/86400), 49973: (identity) local= BRANCH:0, remote= HUB:0 49974: , 49975: local_proxy= BRANCH/255.255.255.255/47/0, 49976: remote_proxy= HUB/255.255.255.255/47/0 49977: Jul 12 11:15:17: IPSEC(send_delete_notify_kmi): not sending KEY_ENGINE_DELETE_SAS 49978: Jul 12 11:15:17: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel1, changed state to down 49979: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (ident_delete_notify_kmi) Failed to send KEY_ENG_DELETE_SAS 49980: Jul 12 11:15:17: IPSEC:(SESSION ID = 2) (ident_update_final_flow_stats) Collect Final Stats and update MIB 49981: IPSEC get IKMP peer index from peer 0x1255EE34 ikmp handle 0x0 49982: [ident_update_final_flow_stats] : Flow delete complete event received for flow id 0x1400009E,peer index 0
Any help to troubelshoot this in more detail appreciated