Phase 2 VPN Problem ("QM FSM error (P2 struct" Error Message)

[Kurt Lei]

Hi All,

In my production topology, remote site ASA5505 (static peer) is forming VPN with HQ Cisco 2921 router (dynamic peer). Reported from my client, he said the ASA can't be monitored by Solarwind since last week.

I found Phase 1 seems working normal but Phase 2 VPN can't be formed. I don't have any idea what's the problem is. Suppose my client didn't get any configuration change in this period.

Connection

Remote ASA inside (10.135.101.1) ---- HQ Solarwind (10.130.8.248)

ASA Error Log

3|Jul 10 2015 19:11:17|713042: IKE Initiator unable to find policy: Intf outside, Src: 10.135.101.1, Dst: 10.130.8.248

3|Jul 10 2015 22:28:02|713902: Group = XX.XX.XX.XX IP = XX.XX.XX.XX, QM FSM error (P2 struct &0xca9c9998, mess id 0xda8423a3)!
3|Jul 10 2015 22:28:02|713902: Group = XX.XX.XX.XX, IP = XX.XX.XX.XX, Removing peer from correlator table failed, no match!
5|Jul 10 2015 22:28:02|713041: Group = XX.XX.XX.XX, IP = XX.XX.XX.XX, IKE Initiator: New Phase 2, Intf outside, IKE Peer XX.XX.XX.XX  local Proxy Address 10.135.101.0, remote Proxy Address 10.130.8.248,  Crypto map (outside_map_1)

------------------------------------------

Remote ASA Config

-------------------------------------------

ASA Version 8.3(2) 
!
hostname ASA
names
name 10.133.2.37 SHCSM
name 10.130.8.248 Solarwind
!
interface Vlan1
 nameif inside
 security-level 100
 ip address 10.135.101.1 255.255.255.0 
!
interface Vlan2
 nameif outside
 security-level 0
 ip address dhcp setroute 
!
interface Ethernet0/0
 switchport access vlan 2
!
interface Ethernet0/1
!
interface Ethernet0/2
!
interface Ethernet0/3
!
interface Ethernet0/4
!
interface Ethernet0/5
!
interface Ethernet0/6
!
interface Ethernet0/7
!
boot system disk0:/asa832-k8.bin
ftp mode passive
same-security-traffic permit inter-interface
object network obj_any 
 subnet 0.0.0.0 0.0.0.0
object network all 
 subnet 0.0.0.0 0.0.0.0
object network obj-10.136.1.0 
 subnet 10.136.1.0 255.255.255.0
object network obj-10.135.1.0 
 subnet 10.135.1.0 255.255.255.0
object network obj-10.135.101.0 
 subnet 10.135.101.0 255.255.255.0
object network obj-10.130.196.0 
 subnet 10.130.196.0 255.255.255.0
object network obj-10.130.21.0 
 subnet 10.130.21.0 255.255.255.0
object network obj-10.130.1.0 
 subnet 10.130.1.0 255.255.255.0
object network obj-10.220.0.0 
 subnet 10.220.0.0 255.255.0.0
object network obj-10.130.237.0 
 subnet 10.130.237.0 255.255.255.0
object network obj-172.16.0.0 
 subnet 172.16.0.0 255.240.0.0
object network obj-10.130.96.0 
 subnet 10.130.96.0 255.255.255.0
object network obj-10.130.8.68 
 subnet 10.130.8.68 255.255.255.255
object network obj-10.130.20.0 
 subnet 10.130.20.0 255.255.255.0
object network obj-10.130.8.248 
 subnet 10.130.8.248 255.255.255.255
access-list inside_access_out extended permit ip any any 
access-list inside_access_out extended permit icmp any any 
access-list outside_cryptomap_50 extended permit ip 10.135.101.0 255.255.255.0 10.130.196.0 255.255.255.0 
access-list outside_cryptomap_50 extended permit ip 10.135.101.0 255.255.255.0 10.130.21.0 255.255.255.0 
access-list outside_cryptomap_50 extended permit ip 10.135.101.0 255.255.255.0 10.220.0.0 255.255.0.0 
access-list outside_cryptomap_50 extended permit ip 10.135.101.0 255.255.255.0 10.133.0.0 255.255.0.0 
access-list outside_cryptomap_30 extended permit ip 10.135.101.0 255.255.255.0 10.136.1.0 255.255.255.0 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.1.111 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.1.110 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.1.94 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.1.130 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 10.130.1.0 255.255.255.0 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 172.16.0.0 255.240.0.0 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 10.130.237.0 255.255.255.0 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 10.130.96.0 255.255.255.0 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.8.68 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 10.130.20.0 255.255.255.0 
access-list outside_cryptomap_60 extended permit ip host 10.135.101.1 host 10.133.2.37 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.8.67 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.8.69 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 host 10.130.8.248 
access-list outside_cryptomap_60 extended permit ip 10.135.101.0 255.255.255.0 10.130.8.240 255.255.255.240 
access-list outside_access_in extended permit icmp any any 
access-list outside_access_in extended permit ip any any 
access-list outside_cryptomap_80 extended permit ip 10.135.101.0 255.255.255.0 10.135.1.0 255.255.255.0 
pager lines 24
logging enable
logging trap informational
logging asdm informational
logging host inside Solarwind
mtu inside 1500
mtu outside 1500
icmp unreachable rate-limit 1 burst-size 1
asdm image disk0:/asdm-631.bin
no asdm history enable
arp timeout 14400
nat (inside,outside) source static obj-10.135.101.0 obj-10.135.101.0 destination static all all
!
object network obj_any
 nat (inside,outside) dynamic interface
object network all
 nat (inside,outside) dynamic interface
access-group inside_access_out out interface inside
access-group outside_access_in in interface outside
timeout xlate 3:00:00
timeout conn 1:00:00 half-closed 0:10:00 udp 0:02:00 icmp 0:00:02
timeout sunrpc 0:10:00 h323 0:05:00 h225 1:00:00 mgcp 0:05:00 mgcp-pat 0:05:00
timeout sip 0:30:00 sip_media 0:02:00 sip-invite 0:03:00 sip-disconnect 0:02:00
timeout sip-provisional-media 0:02:00 uauth 0:05:00 absolute
timeout tcp-proxy-reassembly 0:01:00
dynamic-access-policy-record DfltAccessPolicy
aaa authentication ssh console IAS LOCAL
aaa authentication enable console IAS LOCAL
aaa authentication telnet console IAS LOCAL
aaa authentication http console IAS LOCAL
http server enable
http 10.135.101.0 255.255.255.0 inside
http 0.0.0.0 0.0.0.0 outside
http 10.136.0.0 255.255.0.0 inside
http SHCSM 255.255.255.255 inside
snmp-server host inside 10.130.1.140 community XXX
snmp-server host inside 10.130.1.141 community XXX
snmp-server host inside 10.130.1.59 community XXX
snmp-server host inside 10.130.21.197 community XXX
snmp-server host inside Solarwind community XXX
snmp-server community XXX
snmp-server enable traps snmp authentication linkup linkdown coldstart
snmp-server enable traps syslog
snmp-server enable traps ipsec start stop
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac 
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac 
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto map outside_map_1 60 match address outside_cryptomap_60
crypto map outside_map_1 60 set peer X.X.X.X
crypto map outside_map_1 60 set transform-set ESP-AES-256-SHA
crypto map outside_map_1 interface outside
crypto isakmp enable outside
crypto isakmp policy 30
 authentication pre-share
 encryption 3des
 hash sha
 group 2
 lifetime 3600
telnet 10.135.101.0 255.255.255.0 inside
telnet 10.0.0.0 255.0.0.0 inside
telnet timeout 5
ssh 0.0.0.0 0.0.0.0 outside
ssh timeout 5
console timeout 0
management-access inside
dhcp-client client-id interface outside
dhcpd auto_config outside
!

threat-detection basic-threat
threat-detection statistics access-list
no threat-detection statistics tcp-intercept
ssl encryption des-sha1
webvpn
tunnel-group X.X.X.X type ipsec-l2l
tunnel-group X.X.X.X ipsec-attributes
 pre-shared-key xxx
!
class-map inspection_default
 match default-inspection-traffic
!
!
policy-map type inspect dns preset_dns_map
 parameters
  message-length maximum client auto
  message-length maximum 512
policy-map global_policy
 class inspection_default
  inspect dns preset_dns_map 
  inspect ftp 
  inspect h323 h225 
  inspect h323 ras 
  inspect rsh 
  inspect rtsp 
  inspect esmtp 
  inspect sqlnet 
  inspect skinny  
  inspect sunrpc 
  inspect xdmcp 
  inspect sip  
  inspect netbios 
  inspect tftp 
  inspect ip-options 
!
service-policy global_policy global
privilege cmd level 3 mode exec command perfmon
privilege cmd level 3 mode exec command ping
privilege cmd level 3 mode exec command who
privilege cmd level 3 mode exec command logging
privilege cmd level 3 mode exec command failover
privilege show level 5 mode exec command running-config
privilege show level 3 mode exec command reload
privilege show level 3 mode exec command mode
privilege show level 3 mode exec command firewall
privilege show level 3 mode exec command interface
privilege show level 3 mode exec command clock
privilege show level 3 mode exec command dns-hosts
privilege show level 3 mode exec command access-list
privilege show level 3 mode exec command logging
privilege show level 3 mode exec command ip
privilege show level 3 mode exec command failover
privilege show level 3 mode exec command asdm
privilege show level 3 mode exec command arp
privilege show level 3 mode exec command route
privilege show level 3 mode exec command ospf
privilege show level 3 mode exec command aaa-server
privilege show level 3 mode exec command aaa
privilege show level 3 mode exec command crypto
privilege show level 3 mode exec command vpn-sessiondb
privilege show level 3 mode exec command ssh
privilege show level 3 mode exec command dhcpd
privilege show level 3 mode exec command vpn
privilege show level 3 mode exec command blocks
privilege show level 3 mode exec command uauth
privilege show level 3 mode configure command interface
privilege show level 3 mode configure command clock
privilege show level 3 mode configure command access-list
privilege show level 3 mode configure command logging
privilege show level 3 mode configure command ip
privilege show level 3 mode configure command failover
privilege show level 5 mode configure command asdm
privilege show level 3 mode configure command arp
privilege show level 3 mode configure command route
privilege show level 3 mode configure command aaa-server
privilege show level 3 mode configure command aaa
privilege show level 3 mode configure command crypto
privilege show level 3 mode configure command ssh
privilege show level 3 mode configure command dhcpd
privilege show level 5 mode configure command privilege
privilege clear level 3 mode exec command dns-hosts
privilege clear level 3 mode exec command logging
privilege clear level 3 mode exec command arp
privilege clear level 3 mode exec command aaa-server
privilege clear level 3 mode exec command crypto
privilege cmd level 3 mode configure command failover
privilege clear level 3 mode configure command logging
privilege clear level 3 mode configure command arp
privilege clear level 3 mode configure command crypto
privilege clear level 3 mode configure command aaa-server
prompt hostname context 
call-home
 profile CiscoTAC-1
  no active
  destination address http https://tools.cisco.com/its/service/oddce/services/DDCEService
  destination address email callhome@cisco.com
  destination transport-method http
  subscribe-to-alert-group diagnostic
  subscribe-to-alert-group environment
  subscribe-to-alert-group inventory periodic monthly
  subscribe-to-alert-group configuration periodic monthly
  subscribe-to-alert-group telemetry periodic daily

------------------------------------------

HQ Cisco 2921 Config

-------------------------------------------

crypto isakmp key <removed> address 0.0.0.0 

!

crypto dynamic-map To-MYStore 100
 set transform-set ESP-3DES-SHA ESP-AES-256-SHA

!
crypto dynamic-map outside_dyn_map 10
 set transform-set ESP-AES-256-SHA 
 set pfs group2
crypto dynamic-map outside_dyn_map 20
 set transform-set dial-vpn 

!

crypto map outside_map 65534 ipsec-isakmp dynamic To-MYStore 

crypto map outside_map 65535 ipsec-isakmp dynamic outside_dyn_map 

[rvarelac]

Hi Kurt Lei  , 

I don't see the PFS enabled on the ASA site. 

crypto map outside_map_1 60 match address outside_cryptomap_60
crypto map outside_map_1 60 set peer X.X.X.X
crypto map outside_map_1 60 set transform-set ESP-AES-256-SHA
crypto map outside_map_1 interface outside

You should remove it from the router config an try again. 

crypto dynamic-map outside_dyn_map 10
 set transform-set ESP-AES-256-SHA 
 set pfs group2
crypto dynamic-map outside_dyn_map 20
 set transform-set dial-vpn 

Cheers,

[Kurt Lei]

Thanks Rvarelac,

Sorry to tell you my client don't allow me to change configuration for testing. I can just refer the configuration for troubleshooting.

For my understanding, remote site ASA side will form the VPN tunnel with the dynamic map To-MYStore because the priority is lower (i.e. 65534). Also it didn't enable the PFS. 

Moreover, the existing HQ VPN router is forming many remote site and they also didn't enable PFS. Guess it's not the root cause.