ASA5505: site to site VPN stopped working

[jasonchristophercobb]

We have 2 ASA's that connect to a 2811, but for some reason, the 2nd ASA wont connect anymore. Debuging ipsec or isakmp on the 2811 doesn't come up with any messages. 

External IP's still correct, and the sites can ping each other.

Only debug on ASA for crypto isakmp comes up with messages (ipsec doesn't give any messages).

ASDM says:

Removing peer from peer table failed, no match!

Error: Unable to remove PeerTblEntry

I found some info on the above error messages, but those links didn't quite help.

Below is fromt he debug on the ASA:

Sep 18 22:06:09 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:09 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:10 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:10 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:13 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:13 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:16 [IKEv1]: IP = 64.X.X.X, IKE_DECODE RESENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 148
Sep 18 22:06:17 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:17 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:18 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:18 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:20 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:20 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:24 [IKEv1 DEBUG]: IP = 64.X.X.X, IKE MM Initiator FSM error history (struct &0x42b0b10)  <state>, <event>:  MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY
Sep 18 22:06:24 [IKEv1 DEBUG]: IP = 64.X.X.X, IKE SA MM:f9f683c2 terminating:  flags 0x01000022, refcnt 0, tuncnt 0
Sep 18 22:06:24 [IKEv1 DEBUG]: IP = 64.X.X.X, sending delete/delete with reason message
Sep 18 22:06:24 [IKEv1]: IP = 64.X.X.X, Removing peer from peer table failed, no match!
Sep 18 22:06:24 [IKEv1]: IP = 64.X.X.X, Error: Unable to remove PeerTblEntry
Sep 18 22:06:24 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:24 [IKEv1]: IP = 64.X.X.X, IKE Initiator: New Phase 1, Intf inside, IKE Peer 64.X.X.X  local Proxy Address 192.168.27.0, remote Proxy Address 10.30.18.0,  Crypto map (outside_map)
Sep 18 22:06:24 [IKEv1 DEBUG]: IP = 64.X.X.X, constructing ISAKMP SA payload
Sep 18 22:06:24 [IKEv1 DEBUG]: IP = 64.X.X.X, constructing Fragmentation VID + extended capabilities payload
Sep 18 22:06:24 [IKEv1]: IP = 64.X.X.X, IKE_DECODE SENDING Message (msgid=0) with payloads : HDR + SA (1) + VENDOR (13) + NONE (0) total length : 148
Sep 18 22:06:25 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:25 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.
Sep 18 22:06:32 [IKEv1 DEBUG]: Pitcher: received a key acquire message, spi 0x0
Sep 18 22:06:32 [IKEv1]: IP = 64.X.X.X, Queuing KEY-ACQUIRE messages to be processed when P1 SA is complete.

I can post the configs if neeeded.

Thanks,

JAson

[Javier Portuguez]

Hi Jason,

According to the following line:

MM_DONE, EV_ERROR-->MM_WAIT_MSG2, EV_RETRY-->MM_WAIT_MSG2, EV_TIMEOUT-->MM_WAIT_MSG2, NullEvent-->MM_SND_MSG1, EV_SND_MSG-->MM_SND_MSG1, EV_START_TMR-->MM_SND_MSG1, EV_RESEND_MSG-->MM_WAIT_MSG2, EV_RETRY

It looks like the remote end is either not responding or misconfigured.

I would suggest to check on the VPN configuration of the remote end.

Thanks.

Portu. 

[jasonchristophercobb]

The log messages are coming from what I consider the remote end, an ASA5505. The main site is a 2811, and that isn't giving any messages

show debug
VPN:
  L2X protocol errors debugging is on

Cryptographic Subsystem:
  Crypto ISAKMP debugging is on
  Crypto ISAKMP Error debugging is on
  Crypto IPSEC debugging is on

but the other VPN (another ASA) is working and connected to the hq2811

Here is the config from the HQ2811

crypto isakmp policy 10

encr aes

authentication pre-share

group 5

!

crypto isakmp policy 20

encr 3des

hash md5

authentication pre-share

group 2

crypto isakmp key - address 99.21.X.X

crypto isakmp key - address 108.238.X.Y

!

!

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

crypto ipsec df-bit clear

!

crypto map SDM_CMAP_1 1 ipsec-isakmp

description Tunnel to 99.21.X.X

set peer 99.21.X.X

set transform-set ESP-AES-128-SHA

match address 103

crypto map SDM_CMAP_1 2 ipsec-isakmp

description Tunnel to 108.238.X.Y

set peer 108.238.X.Y

set transform-set ESP-AES-128-SHA

match address 105

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac

crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac

crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac

crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac

crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac

crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac

crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac

crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac

crypto ipsec df-bit clear-df outside

crypto map outside_map 2 match address outside_cryptomap

crypto map outside_map 2 set pfs

crypto map outside_map 2 set peer 64.0.Z.Z

crypto map outside_map 2 set transform-set ESP-AES-128-SHA

crypto map outside_map interface outside

crypto isakmp enable outside

crypto isakmp policy 10

authentication pre-share

encryption aes

hash sha

group 5

lifetime 86400

crypto isakmp policy 20

authentication pre-share

encryption 3des

hash md5

group 2

lifetime 86400

crypto isakmp policy 10
encr aes
authentication pre-share
group 5
!
crypto isakmp policy 20
encr 3des
hash md5
authentication pre-share
group 2
crypto isakmp key - address 99.21.X.X
crypto isakmp key - address 108.238.X.Y
!
!
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec df-bit clear
!
crypto map SDM_CMAP_1 1 ipsec-isakmp
description Tunnel to 99.21.X.X
set peer 99.21.X.X
set transform-set ESP-AES-128-SHA
match address 103
crypto map SDM_CMAP_1 2 ipsec-isakmp
description Tunnel to 108.238.X.Y
set peer 108.238.X.Y
set transform-set ESP-AES-128-SHA
match address 105

This is from the non VPNing ASA:

crypto ipsec transform-set ESP-AES-128-SHA esp-aes esp-sha-hmac
crypto ipsec transform-set ESP-AES-256-MD5 esp-aes-256 esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-DES-MD5 esp-des esp-md5-hmac
crypto ipsec transform-set ESP-AES-192-MD5 esp-aes-192 esp-md5-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-AES-256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec transform-set ESP-AES-192-SHA esp-aes-192 esp-sha-hmac
crypto ipsec transform-set ESP-AES-128-MD5 esp-aes esp-md5-hmac
crypto ipsec df-bit clear-df outside
crypto map outside_map 2 match address outside_cryptomap
crypto map outside_map 2 set pfs
crypto map outside_map 2 set peer 64.0.Z.Z
crypto map outside_map 2 set transform-set ESP-AES-128-SHA
crypto map outside_map interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption aes
hash sha
group 5
lifetime 86400
crypto isakmp policy 20
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 86400

[jasonchristophercobb]

With this log message, where is it getting the "remote Proxy Address 10.30.18.0" from?

Sep 18 22:06:24 [IKEv1]: IP = 64.X.X.X, IKE Initiator: New Phase 1, Intf inside, IKE Peer 64.X.X.X  local Proxy Address 192.168.27.0, remote Proxy Address 10.30.18.0,  Crypto map (outside_map)

Is this a device inside the local network requesting access to the 10.30.18.0 network, or is 10.30.18.0 programmed somewhere on the ASA?

[Jennifer Halim]

Should be getting that from the crypto ACL.

Can you pls share the full config from both ends.

[jasonchristophercobb]

Here are the configs.

Thanks,

J

[Jennifer Halim · Accepted solution]

A few things:

1) On the ASA, pls remove "crypto map outside_map 2 set pfs", since PFS is not configured on the router.

2) On the router, your NAT exemption ACL (104) is missing a few deny statements to some subnets and also one of them says UDP while it should say IP.

Should add the following on top of the permit statements:

deny   ip 10.131.16.0 0.0.0.255 192.168.27.0 0.0.0.255

deny   ip 172.21.16.0 0.0.0.255 192.168.27.0 0.0.0.255

deny   ip 172.20.15.0 0.0.0.255 192.168.27.0 0.0.0.255

deny   ip 10.130.15.0 0.0.0.255 192.168.27.0 0.0.0.255

deny   ip 10.30.18.0 0.0.0.255 192.168.27.0 0.0.0.255

3) Should also remove "in-zone" from loopback0 interface since you don't have "out-zone" applied to any interface anyway.

[jasonchristophercobb]

Thank you Jennifer.

I was hoping item #1 would fix it, but same result (error message at beging of thread).

The UDP in item #2 is for voice, and the other missing networks are not needing routing between this ASA.

Like to see any show commands?

Thanks again!

[Jennifer Halim]

Yes pls...

show cry isa sa

show cry ipsec sa

also, pls run:

debug cry isa

debug cry ipsec

on both end and post the debug output. Thanks.

[jasonchristophercobb]

On the router side, debug still doesn't come up with any messages.

The show cry ipsec sa was kind of big, so I inluced that as a text file.

On the ASA, the debug defaulted to level 1, but the log messages are the same, except what's posted above includes info from debug level 255, whats in the file is only level 1.

thanks,

Jason

2811 ROUTER

show cry isa sa
IPv4 Crypto ISAKMP SA
dst             src             state          conn-id slot status
64.0.X.X   108.238.X.X MM_SA_SETUP          0    0 ACTIVE
64.0.X.X   108.238.X.X MM_NO_STATE          0    0 ACTIVE (deleted)
64.0.X.X   108.238.X.X MM_NO_STATE          0    0 ACTIVE (deleted)
64.0.X.X   99.21.Y.Y    QM_IDLE           1059    0 ACTIVE
108.238.X.X 64.0.X.X   MM_NO_STATE          0    0 ACTIVE
108.238.X.X 64.0.X.X   MM_NO_STATE          0    0 ACTIVE (deleted)

IPv6 Crypto ISAKMP SA

ASA5505 #2

show cry isa sa

   Active SA: 1
    Rekey SA: 0 (A tunnel will report 1 Active and 1 Rekey SA during rekey)
Total IKE SA: 1

1   IKE Peer: 64.0.X.X
    Type    : user            Role    : initiator
    Rekey   : no              State   : MM_WAIT_MSG2

show crypto ipsec sa

There are no ipsec sas

[Jennifer Halim]

Base on the debugs and show output, the ASA is initiating the VPN tunnel, however doesn't receive any reply from the router at all, that is why it is stuck at MM_WAIT_MSG2, which means, waiting for the router to reply with MSG2.

If you try to ping from the LAN behind the router towards the ASA, what does the output of "show cry isa sa" says on the router? Does it say waiting for MSG2 as well? If it does, that means UDP/500 might be blocked by the ISP, or a device in front of the ASA. As the router is terminating the VPN to another ASA, it is unlikely it is something on the router side, but more likely the ASA side.

[jasonchristophercobb]

I actually had a ping going to devices at both ASA VPN's from another computer inside the router during the show commands...

But a funny thing happend last Thursday morning... it just started working.  The last changes were what you suggested, but that was made a few days prior to when it started working. Both devices we're reloaded at the time of the changes too, so I don't know how to explain it coming up on it's own a couple days later.

Thank you so much for helping me with this!!!!!!