ASA5515 - QM FSM error / failed to establish L2L SA when transferring large files

CrazyHorse019 · ‎01-15-2019

Hi All,

I was hoping someone could help me with a problem I’m having - a little clarification or advice would be much appreciated:

The Issue: I have set a new backup server and I want to back up data from one of our production servers to the backup one. The servers are connected by a site-to-site tunnel between DC A and DC B - both using Cisco ASA 5515’s. The way I am transferring the file is through innobackup which uses SSH (port 22). I was transferring a 5gb file when all of a sudden it got to 4.6gb and dropped the connection. I tried SCP which also uses port 22 and it did the same thing. Then I used netcat and did the transfer again on a raw port and it still failed. So after seeing nothing in the log files and running the above tests I ruled out it was a port or ufw configuration issue, so I looked to the router for answers. ( It is important to give you guys context ). So I ran the debugging tool on the ASA and attempted the transfer again and bingo! - some logs for me!

FW A logs
Group = <peer ip>, IP = <peer ip>, Removing Peer from correlator table failed, no match!
Group = <peer ip>, IP = <peer ip>, QM FSM error (P2 struct &0x00007fbda978db0, mess id 0xd36ff5ab)!
Group = <peer ip>, IP = <peer ip>, Removing Peer from correlator table failed, no match!
Group = <peer ip>, IP = <peer ip>, QM FSM error (P2 struct &0x00007fb9d8fde4a0, mess id 0xd36ff5ab)!

FW B logs 
Group = <peer ip>, IP = <peer ip>, QM FSM error (P2 struct &0x00007fff29852230, mess id 0xd36ff5ab)!
Tunnel Manager has failed to establish an L2L SA. All configured IKE versions failed to establish the tunnel. Map Tag= outside_map. Map Sequence Number = 1.
Group = <peer ip>, IP = <peer ip>, Removing Peer from correlator table failed, no match!
Group = <peer ip>, IP = <peer ip>, QM FSM error (P2 struct &0x00007fff29124500, mess id 0xb09bc855)!
Group = <peer ip>, IP = <peer ip>, Removing Peer from correlator table failed, no match!
Group = <peer ip>, IP = <peer ip>, QM FSM error (P2 struct &0x00007fff29124500, mess id 0xb09bc855)!
Group = <peer ip>, IP = <peer ip>, Removing Peer from correlator table failed, no match!kbs to gb
Group = <peer ip>, IP = <peer ip>, QM FSM error (P2 struct &0x00007fff29124500, mess id 0xb09bc855)!
Group = <peer ip>, IP = <peer ip>, Removing Peer from correlator table failed, no match!

So after looking this up on google i’ve come across two potential causes (there maybe more) - an ACL mismatch or a crypto map set security-association lifetime problem.

My Dilema - For the moment I have increased the crypto map set security-association lifetime to satisfy the size of the file transfer I require (and this works!) - but can’t help feel this is a bit of a hack to get around another issue. You see we have another DC that contains a backup and that is using the crypto map set security-association lifetime default and transfers files of comparable sizes just fine.

The reason I haven’t gone down changing the ACL yet is because there is already one FW A and FW B under the header ‘outside_crypto_map’ in the ACL Manager - is this ignored? Or do I need to add another entry? Both ACLs are configured as follows: Source: <internal network> Destination: <remote network> Service: IP Action: Allow

Thank you for your time!

Sheraz.Salim · ‎01-15-2019

My Dilema - For the moment I have increased the crypto map set security-association lifetime to satisfy the size of the file transfer I require (and this works!) - but can’t help feel this is a bit of a hack to get around another issue. You see we have another DC that contains a backup and that is using the crypto map set security-association lifetime default and transfers files of comparable sizes just fine.

what was security-association earlier. and what number you change to?

The reason I haven’t gone down changing the ACL yet is because there is already one FW A and FW B under the header ‘outside_crypto_map’ in the ACL Manager - is this ignored? Or do I need to add another entry? Both ACLs are configured as follows: Source: <internal network> Destination: <remote network> Service: IP Action: Allow

that fine nothing to change here. as the crypto_map is binded/married to ACL. where you have accurately define and its corrects Source: <internal network> Destination: <remote network> Service: IP Action: Allow

what are the nat rules and the acl on both side including with objects if you using. I had a smiliar issue in past and it was inconsistent nat and ip addresses for the interested ACLs.

please do not forget to rate.

CrazyHorse019 · ‎01-15-2019

Hi, thanks for your questions - here are my responses :)

what was security-association earlier. and what number you change to?

on firewall a the default was 4208000 and was changed to unlimited

on firewall b the default was 4608000 and was changed to its max of 2147483647 - this is because of the version this sits on it wont allow unlimited as a value

what are the nat rules and the acl on both side including with objects if you using?

firewall b - (backup)

object network my-inside-net
nat (inside,outside) dynamic interface

firewall a - (main)

nat (inside,any) source static network_internal network_internal destination static backup_internal backup_internal no-proxy-arp route-lookup

Sheraz.Salim · ‎01-15-2019

firewall b - (backup)

object network my-inside-net
nat (inside,outside) dynamic interface

firewall a - (main)

nat (inside,any) source static network_internal network_internal destination static backup_internal backup_internal no-proxy-arp route-lookup

how about other side of the firewall with nat rule and objects.

why you giving a nat(inside,any) ?? be more specific example nat(inside,outside). having any is ok but not a good practice

please do not forget to rate.

Marius Gunnerud · ‎01-15-2019

I agree that NAT statements should have specific interfaces defined, it is however changing "any" to a more specific interface will not solve the drop in VPN during file transfer.

--
Please remember to select a correct answer and rate helpful posts

Sheraz.Salim · ‎01-15-2019

with respect to @Marius Gunnerud I spot some thing so i made my point.

just notice in your @CrazyHorse019 config the debug logs QM FSM Error. The IPsec L2L VPN tunnel does not come up on the PIX firewall or ASA, and the QM FSM error message appears. One possible reason is the proxy identities, such as interesting traffic, access control list (ACL) or crypto ACL, do not match on both the ends.

please check if you have proper ACL in place (also check the ip address mirror each firewall)

please do not forget to rate.

Marius Gunnerud · ‎01-15-2019

P2 references Phase 2 in the ISAKMP process and often refers to a mismatched crypto ACL. But we are just guessing here as we do not know your configuration. If you could provide us with the full configuration of the ASAs at both ends of the VPN we will get a better idea of what the issue might be. Please remember to remove any public IPs, usernames and passwords from the configuration before you post it.

--
Please remember to select a correct answer and rate helpful posts

CrazyHorse019 · ‎01-17-2019

FW A (MAIN):

ASA Version 9.8(2) 
 
access-list outside_cryptomap_4 extended permit ip object network_internal object network_backup
 

nat (inside,outside) source static network_internal network_internal destination static Backup_Internal Backup_Internal no-proxy-arp route-

crypto ipsec ikev1 transform-set ESP-AES-SHA esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set transform-amzn esp-aes esp-sha-hmac 
crypto ipsec ikev2 ipsec-proposal DES
 protocol esp encryption des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal 3DES
 protocol esp encryption 3des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES
 protocol esp encryption aes
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES192
 protocol esp encryption aes-192
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES256
 protocol esp encryption aes-256
 protocol esp integrity sha-1 md5

crypto map outside_map 7 match address outside_cryptomap_4
crypto map outside_map 7 set pfs 
crypto map outside_map 7 set peer X.X.X.X
crypto map outside_map 7 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map 7 set security-association lifetime kilobytes unlimited
crypto map outside_map 65535 ipsec-isakmp dynamic SYSTEM_DEFAULT_CRYPTO_MAP
crypto map outside_map interface outside


crypto isakmp identity address 
crypto ikev2 policy 1
 encryption aes-256
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 10
 encryption aes-192
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 20
 encryption aes
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 30
 encryption 3des
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 40
 encryption des
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400

 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 201
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 28800

group-policy GroupPolicy_X.X.X.X internal
group-policy GroupPolicy_X.X.X.X attributes
 vpn-tunnel-protocol ikev1 

tunnel-group X.X.X.X type ipsec-l2l
tunnel-group X.X.X.X general-attributes
 default-group-policy GroupPolicy_X.X.X.X
tunnel-group X.X.X.X ipsec-attributes
 ikev1 pre-shared-key XXXXXXX
 ikev2 remote-authentication pre-shared-key XXXXXXX
 ikev2 local-authentication pre-shared-key XXXXXXX

FW B (BACKUP):

    ASA Version 9.1(1) 

access-list outside_cryptomap extended permit ip Internal_Network object Main_DC_Internal 

nat (inside,outside) source static Internal_Network Internal_Network destination static Main_DC_Internal Main_DC_Internal no-proxy-arp route-lookup

crypto ipsec ikev1 transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS esp-aes esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS esp-aes esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-128-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS esp-aes-192 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS esp-aes-192 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-192-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS esp-aes-256 esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS esp-aes-256 esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-AES-256-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS esp-3des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS esp-3des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-3DES-MD5-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-SHA esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5 esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS esp-des esp-sha-hmac 
crypto ipsec ikev1 transform-set ESP-DES-SHA-TRANS mode transport
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS esp-des esp-md5-hmac 
crypto ipsec ikev1 transform-set ESP-DES-MD5-TRANS mode transport
crypto ipsec ikev2 ipsec-proposal DES
 protocol esp encryption des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal 3DES
 protocol esp encryption 3des
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES
 protocol esp encryption aes
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES192
 protocol esp encryption aes-192
 protocol esp integrity sha-1 md5
crypto ipsec ikev2 ipsec-proposal AES256
 protocol esp encryption aes-256
 protocol esp integrity sha-1 md5
crypto ipsec security-association pmtu-aging infinite
crypto map outside_map 1 match address outside_cryptomap
crypto map outside_map 1 set pfs 
crypto map outside_map 1 set peer X.X.X.X 
crypto map outside_map 1 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map 1 set ikev2 ipsec-proposal DES 3DES AES AES192 AES256
crypto map outside_map interface outside
crypto ca trustpool policy
crypto ikev2 policy 1
 encryption aes-256
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 10
 encryption aes-192
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 20
 encryption aes
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 30
 encryption 3des
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 policy 40
 encryption des
 integrity sha
 group 5 2
 prf sha
 lifetime seconds 86400
crypto ikev2 enable outside
crypto ikev1 enable outside
crypto ikev1 policy 10
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 20
 authentication rsa-sig
 encryption aes-256
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 40
 authentication pre-share
 encryption aes-192
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 50
 authentication rsa-sig
 encryption aes-192
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 70
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 80
 authentication rsa-sig
 encryption aes
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 100
 authentication pre-share
 encryption aes-256
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 110
 authentication rsa-sig
 encryption 3des
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 130
 authentication pre-share
 encryption des
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 140
 authentication rsa-sig
 encryption des
 hash sha
 group 2
 lifetime 86400
crypto ikev1 policy 201
 authentication pre-share
 encryption aes
 hash sha
 group 2
 lifetime 28800

group-policy GroupPolicy_X.X.X.X internal
group-policy GroupPolicy_X.X.X.X attributes
 vpn-tunnel-protocol ikev1 ikev2 

tunnel-group X.X.X.X type ipsec-l2l
tunnel-group X.X.X.X general-attributes
 default-group-policy GroupPolicy_X.X.X.X
tunnel-group X.X.X.X ipsec-attributes
 ikev1 pre-shared-key XXXXXX
 ikev2 remote-authentication pre-shared-key XXXXXXXX
 ikev2 local-authentication pre-shared-key XXXXXXXX

hi, please see the above configuration - if there is anything further you would like to see please let me know :) Thanks for your help!

Sheraz.Salim · ‎01-17-2019

Firewall A MAIN

!

the crypto-map is not match.

nat (inside,outside) source static network_internal network_internal destination static Backup_Internal Backup_Internal no-proxy-arp route
access-list outside_cryptomap extended permit ip object network_internal object network_backup
access-list outside_cryptomap_1 extended permit ip object network_internal object network_backup

crypto map outside_map 7 match address outside_cryptomap_7

please do not forget to rate.

CrazyHorse019 · ‎01-17-2019

ah ok! - apologies, it seems I have copied the config for FW A wrong - so on our FW A we have the acl for this particular S2S VPN labelled as:

access-list outside_cryptomap_4 extended permit ip object network_internal object Backup_Internal

but the crypto-map profile for the same S2S VPN is priority 7:

crypto map outside_map 7 match address outside_cryptomap_4
crypto map outside_map 7 set pfs 
crypto map outside_map 7 set peer X.X.X.X 
crypto map outside_map 7 set ikev1 transform-set ESP-AES-128-SHA ESP-AES-128-MD5 ESP-AES-192-SHA ESP-AES-192-MD5 ESP-AES-256-SHA ESP-AES-256-MD5 ESP-3DES-SHA ESP-3DES-MD5 ESP-DES-SHA ESP-DES-MD5
crypto map outside_map 7 set security-association lifetime kilobytes unlimited

so are you saying that I have to make a new acl with the name outside_cryptomap_7 with a rule stating:

crypto map outside_map 7 match address outside_cryptomap_7

or is this now OK the way it is?

Marius Gunnerud · ‎01-17-2019

This is fine.

I do however see that you have mixed som IKEv2 config with IKEv1 in the same crypto map and tunnel-group. This IKEv2 config should be removed.

--
Please remember to select a correct answer and rate helpful posts

Sheraz.Salim · ‎01-17-2019

I can not find the doc i read you can run ASA on ikev1 and ikev2 at same time. no issue.

please do not forget to rate.

Marius Gunnerud · ‎01-17-2019

That is correct, IKEv2 is preferred and if it cannot establish a VPN with IKEv2 it will fall back to IKEv1. However, in the configuration in question only the Backup site crypto map and tunnel-group has IKEv2 configuration while Main site does not (for the VPN in question).

So for the sake of cleaning up configuration and eliminating possible interference removing IKEv2 configuration from Backup site crypto map outside_map 1 and tunnel-group x.x.x.x ipsec-attributes would be prudent.

--
Please remember to select a correct answer and rate helpful posts

Marius Gunnerud · ‎01-17-2019

Yes you can have IKEv1 and IKEv2 configured in the same crypto map / tunnel-group at the same time. IKEv2 is preferred but will fall back to IKEv1 if IKEv2 tunnel cannot be established. However, in the given configuration only Backup site has IKEv2 configuration for the given VPN while Main site does not have this configuration. So for the sake of cleaning up the configuration and eliminating possible interference I suggest removing the un-needed configuration.

--
Please remember to select a correct answer and rate helpful posts

Marius Gunnerud · ‎01-15-2019

What is the MTU size you have on your interfaces? I have seen a similar issue with routers where the resolution was to set the MTU size of the interfaces to 1360.

--
Please remember to select a correct answer and rate helpful posts