Problems with VPN disconnections after upgrading ASA to address Cisco security advisory

mitchen · ‎04-23-2014

So, as some background, we have been happily running ASA 8.4(3) for some time. But we actually first experienced this problem a while ago when trying to upgrade ASA software version but downgraded back to 8.4(3) again, the problem went away, and it got sidetracked for various reasons (primarily that we intended upgrading the ASA hardware anyway though, typically, that hasn't actually been carried out yet!)

Anyway, following the latest Cisco security advisory:

http://tools.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20140409-asa

We decided to try upgrading to 8.4(7.15)

BUT - the same problems we experienced last time we tried upgrading (on that occasion it was to 8.4.6) were experienced again. Basically, after upgrading, we experience VPN connectivity problems with (at least) one of our sites - the VPN connectivity with them constantly establishes then drops, then re-establishes again. If we return to 8.4(3) everything is fine.

After upgrading, on the Standby ASA, I notice the following messages constantly appearing:

Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed
Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed
Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed
Apr 22 2014 17:27:16: %ASA-3-210005: LU allocate connection failed
Apr 22 2014 17:27:17: %ASA-3-210005: LU allocate connection failed
Apr 22 2014 17:27:18: %ASA-3-210005: LU allocate connection failed
Apr 22 2014 17:27:19: %ASA-3-210005: LU allocate connection failed

When the Standby ASA is made Active (as the "Old Active" is being updated) we then get the following messages for one particular site (we also have several other sites which didn't appear to be affected):

Apr 22 2014 17:27:31: %ASA-3-713061: Group = 2.2.2.2, IP = 2.2.2.2, Rejecting IPSec tunnel: no matching crypto map entry for remote proxy 192.168.1.0/255.255.255.0/0/0 local proxy 3.3.3.3/255.255.255.255/0/0 on interface OUTSIDE
Apr 22 2014 17:27:31: %ASA-3-713902: Group = 2.2.2.2, IP = 2.2.2.2, QM FSM error (P2 struct &0x72f2a728, mess id 0xf2924361)!
Apr 22 2014 17:27:31: %ASA-3-713902: Group = 2.2.2.2, IP = 2.2.2.2, Removing peer from correlator table failed, no match!
Apr 22 2014 17:27:31: %ASA-3-752015: Tunnel Manager has failed to establish an L2L SA. All configured IKE versions failed to establish the tunnel. Map Tag= outside_map. Map Sequence Number = 3002.

Now, this relates to something of a peculiarity we have in our set-up. At the remote site, we split-tunnel internet traffic EXCEPT for traffic to one particular "extranet" site which is routed back through the datacentre ASA and out to the internet through the DC internet connection. (This all works fine on 8.4(3) though)

This makes me think its due to some sort of mismatch between both ends of the connection but, if so, why does it work ok on version 8.4(3)? And the other strange thing is that we have another remote site almost identically configured which doesn't appear to suffer any such problems. The remote site has a Cisco 2911 running 15.1(4)M.

At the time of the issue, the only messages appearing on the Cisco 2911 were:

Apr 22 17:27:31.238: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=2.2.2.2, prot=50, spi=0x95A15F88(2510380936), srcaddr=1.1.1.1, input interface=GigabitEthernet0/0
Apr 22 17:28:31.250: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=2.2.2.2, prot=50, spi=0x4008F420(1074328608), srcaddr=1.1.1.1, input interface=GigabitEthernet0/0
Apr 22 17:32:59.554: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=2.2.2.2, prot=50, spi=0xA198CD52(2711145810), srcaddr=1.1.1.1, input interface=GigabitEthernet0/0

SET-UP

Datacentre - ASA5520 running version 8.4(3), all works fine. If upgraded to e.g. 8.4(6) or 8.4(7.15) then problems occur but only seems to affect one particular site

Remote Site (Office101) - Cisco 2911 running 15.1(4)M

datacentre firewall peer IP - 1.1.1.1
datacentre subnets - 10.10.0.0/16
172.16.10.0/24
172.16.11.48/28
172.16.12.0/25

remote site (office101) internal subnet - 192.168.1.0
remote site (office101) peer ip - 2.2.2.2

obj-extranet = 3.3.3.3 - extranet server (hosted externally)
obj-extranet-trans = 1.1.1.5 - translation for traffic to extranet server

ASA Config:

!
object network obj-10.10.0.0
subnet 10.10.0.0 255.255.0.0
!
object network obj-192.168.1.0-mask24
subnet 192.168.1.0 255.255.255.0
!
object network obj-172.16.10.0-mask24
subnet 172.16.10.0 255.255.255.0
!
object network obj-172.16.11.48-mask28
subnet 172.16.11.48 255.255.255.240
!
object network obj-172.16.12.0-mask25
subnet 172.16.12.0 255.255.255.0
!
object network obj-extranet
host 3.3.3.3
object network obj-extranet-trans
host 1.1.1.5
!

!
crypto map outside_map 3002 match address office101
crypto map outside_map 3002 set peer 2.2.2.2
crypto map outside_map 3002 set ikev1 transform-set ESP-3DES-MD5
!
access-list office101 line 1 extended permit ip 10.10.0.0 255.255.0.0 192.168.1.0 255.255.255.0
access-list office101 line 4 extended permit ip 172.16.10.0 255.255.255.0 192.168.1.0 255.255.255.0
access-list office101 line 5 extended permit ip 172.16.11.48 255.255.255.240 192.168.1.0 255.255.255.0
access-list office101 line 7 extended permit ip 172.16.12.0 255.255.255.128 192.168.1.0 255.255.255.0
!
nat (inside,OUTSIDE) source static obj-10.10.0.0 obj-10.10.0.0 destination static obj-192.168.1.0-mask24 obj-192.168.1.0-mask24
nat (inside,OUTSIDE) source static obj-172.16.10.0-mask24 obj-172.16.10.0-mask24 destination static obj-192.168.1.0-mask24 obj-192.168.1.0-mask24
nat (inside,OUTSIDE) source static obj-172.16.11.48-mask28 obj-172.16.11.48-mask28 destination static obj-192.168.1.0-mask24 obj-192.168.1.0-mask24
nat (inside,OUTSIDE) source static obj-172.16.12.0-mask25 obj-172.16.12.0-mask25 destination static obj-192.168.1.0-mask24 obj-192.168.1.0-mask24
nat (OUTSIDE,OUTSIDE) source dynamic obj-192.168.1.0-mask24 obj-extranet-trans destination static obj-extranet obj-extranet
!
tunnel-group 2.2.2.2 type ipsec-l2l
tunnel-group 2.2.2.2 ipsec-attributes
pre-shared-key ****

Remote Site Office101 (Cisco 2911) config

crypto isakmp key **** address 1.1.1.1
!
crypto map cryptomap 10 ipsec-isakmp
set peer 1.1.1.1
set transform-set vpn-set
match address datacentre
!
ip access-list extended datacentre
permit ip 192.168.1.0 0.0.0.255 10.10.0.0 0.0.255.255
permit ip 192.168.1.0 0.0.0.255 172.16.11.48 0.0.0.15
permit ip 192.168.1.0 0.0.0.255 172.16.10.0 0.0.0.255
permit ip 192.168.1.0 0.0.0.255 172.16.12.0 0.0.0.127
permit ip 192.168.1.0 0.0.0.255 host 3.3.3.3
!
!###NO NAT ACL####
access-list 112 deny ip 192.168.1.0 0.0.0.255 10.10.0.0 0.0.255.255
access-list 112 deny ip 192.168.1.0 0.0.0.255 172.16.10.0 0.0.0.255
access-list 112 deny ip 192.168.1.0 0.0.0.255 172.16.11.48 0.0.0.15
access-list 112 deny ip 192.168.1.0 0.0.0.255 172.16.12.0 0.0.0.127
access-list 112 deny ip 192.168.1.0 0.0.0.255 host 3.3.3.3
access-list 112 permit ip 192.168.1.0 0.0.0.255 host [cloud-web-scan-ip]
access-list 112 deny ip any any

Has anyone come across similar and/or can suggest what the problem might be? It seems to suggest a misconfiguration but why does it work ok for 8.4(3)? I am concerned now that we cannot address any security vulnerabilities or suchlike because upgrading the ASA version seems to introduce this issue. Also, as I mentioned, we intend upgrading the ASA hardware (to ASA5545x) and, as such, would likely be running 8.6 software on this appliance but I'm worried that introducing the new hardware might also bring in this problem as a result, so I'm naturally keen to find the cause of the issue!

Thanks for any help that can be offered.

ROBERTO TACCON · ‎04-23-2014

Why on the Cisco 2911 there's the

permit ip 192.168.1.0 0.0.0.255 host 3.3.3.3

?

However have you configured on the Cisco ASA :

sysopt connection preserve-vpn-flows

http://www.cisco.com/c/en/us/td/docs/security/asa/asa81/config/guide/config/intro.html#wp1053814

mitchen · ‎04-23-2014

Hi Roberto,

thanks for the response. The permit ip 192.168.1.0 0.0.0.255 host 3.3.3.3 line on the 2911 is to direct traffic to the extranet site (3.3.3.3) through the tunnel to the datacentre ASA (rather than split-tunneling through the 2911's internet connection) and through the datacentre internet connection. So the ACL containing that line is for the "interesting" traffic that will be sent across the VPN tunnel to the datacentre ASA. This seems to work fine with the ASA version 8.4(3)

However, no, I don't have the "sysopt connection preserve-vpn-flows" line - that is interesting, may actually be of benefit? But the thing that would still puzzle me would be why everything works fine at 8.4(3) but only when I upgrade do I experience issue?

l cole · ‎07-03-2014

Hi, did you ever find a good solution to this?

mitchen · ‎07-04-2014

Hi,

it turned out the problem was due to a mismatch in the ACLs between the ASA and the remote site 2911.

We had no matching line in the ASA config for traffic between the office and the extranet server. On the office router, we had:

permit ip 192.168.1.0 0.0.0.255 host 3.3.3.3

So, on the ASA, we had to add:

access-list office101 line 8 extended permit ip host 3.3.3.3 192.168.1.0 255.255.255.0

I'm not sure why the behaviour was different between the ASA versions i.e. why we did not experience any issues when we had the misconfig at version 8.4(3) but it caused problems with higher versions. Also, cannot explain why it only really seemed to be one site that was affected (we had the same config mismatch at other sites which were seemingly alright)

However, ultimately, it was the config mismatch that was causing the problem and, after rectifying that, we were able to upgrade and our remote site VPN connectivity remained stable.