[solved] 8.4.X IKEv2 L2L IPSec problems

viktorlarionov · ‎07-22-2011

Hey folks,

I've been struggling with this problem for a week now, and now it's friday and I feel I'm not getting anywhere, so I could really use a hand of you guys here. So here's the story, I have a central site running a 5510 ASA, with 8.4.1 image on it. This ASA resides inside our corporate network and servers as a VPN concentrator for a bunch of ASA-5505's.

Well, production ASA-5505's currently run 8.3 image and IKEv1 channels, and all's is cool and working well. In the beginning of this week, I took a test ASA-5505 to give 8.4.2 image a try, and that's where weird things started to happen.

Here's a sample conf in wrote for 8.4.2:

boot system disk0:/asa841-k8.bin

asdm image disk0:/asdm-643.bin

interface Vlan1

description network for communication with branch offices.

nameif BRANCHES

security-level 0

no shutdown

ip address 10.0.0.3 255.255.255.0

!

interface Vlan2

description General internal network segment.

nameif LAN

no shutdown

security-level 10

ip address 192.168.14.62 255.255.255.240

[...]

object network carrier-networks

subnet 10.0.0.0 255.255.0.0

object network branch.office

subnet 192.168.14.48 255.255.255.240

access-list BRANCH-OFFICE-ACL remark Consumer traffic will be checked on upstream transit points.

access-list BRANCH-OFFICE-ACL extended permit ip any any log disable

access-list BRANCH-OFFICE-ACL remark Default action is to deny everything.

access-list BRANCH-OFFICE-ACL extended deny ip any any log warnings

access-list BRANCH-OFFICE-CRYPTOMAP remark Traffic targetting internal carrier networks should bypass tunnels.

access-list BRANCH-OFFICE-CRYPTOMAP extended deny ip object branch.office object carrier-networks log disable

access-list BRANCH-OFFICE-CRYPTOMAP remark Everything else, originating from internal network gets tunnelled.

access-list BRANCH-OFFICE-CRYPTOMAP extended permit ip object branch.office any log disable

[...]

nat (LAN,BRANCHES) after-auto source dynamic any interface destination static carrier-networks carrier-networks description Traffic sent to internal carrier networks from inside should be NAT-ed.

route BRANCHES 10.0.0.0 255.255.0.0 10.0.0.1 1

crypto ca import gw-1.branch.office XXXXXXXXXXXXXXXXXXXXXXX

[,..]

quit

crypto ipsec ikev2 ipsec-proposal IKEV2-AES-256-SHA-1

protocol esp encryption aes-256

protocol esp integrity sha-1

crypto ipsec security-association lifetime seconds 3000000

crypto ipsec security-association lifetime kilobytes 10485760

crypto map BRANCH-OFFICE-CRYPTOMAP 1 match address BRANCH-OFFICE-CRYPTOMAP

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set pfs

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set connection-type originate-only

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set peer 10.0.0.2

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set ikev2 ipsec-proposal IKEV2-AES-256-SHA-1

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set security-association lifetime seconds 3000000

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set security-association lifetime kilobytes 10485760

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set trustpoint gw-1.branch.office

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set reverse-route

crypto map BRANCH-OFFICE-CRYPTOMAP interface BRANCHES

crypto ca certificate map BRANCH-OFFICE 10

subject-name attr cn eq gw-1.main.office

crypto isakmp reload-wait

crypto ikev2 policy 1

encryption aes-256

integrity sha

group 2

prf sha

lifetime seconds 86400

crypto ikev2 enable BRANCHES

crypto ikev2 cookie-challenge always

crypto ikev2 limit max-sa 5

crypto ikev1 am-disable

crypto ikev1 policy 1

authentication rsa-sig

encryption aes-256

hash sha

group 1

lifetime none

group-policy BRANCH-OFFICE internal

group-policy BRANCH-OFFICE attributes

vpn-idle-timeout none

vpn-filter value BRANCH-OFFICE-ACL

ipv6-vpn-filter value GLOBAL-IPV6-ACL

vpn-tunnel-protocol ikev2

tunnel-group BRANCH-OFFICE type ipsec-l2l

tunnel-group BRANCH-OFFICE general-attributes

default-group-policy BRANCH-OFFICE

tunnel-group BRANCH-OFFICE ipsec-attributes

ikev1 trust-point gw-1.branch.office

isakmp keepalive threshold 60 retry 5

ikev2 remote-authentication certificate

ikev2 local-authentication certificate gw-1.branch.office

tunnel-group-map enable rules

no tunnel-group-map enable ou

no tunnel-group-map enable ike-id

no tunnel-group-map enable peer-ip

tunnel-group-map default-group DefaultL2LGroup

tunnel-group-map BRANCH-OFFICE 10 BRANCH-OFFICE

I won't bring the central point conf in here because my main problems are local for 5505 and don't even get to connecting to central point.

Ping me if I'm mistaking.

So anyway, now here's where the spooky stuff comes in.

Running this configuration on 5505, and pinging for example 192.168.2.6 from the LAN interface machine (e.g. 192.168.14.60) leaves me with:

Failed to locate egress interface for ICMP from LAN:192.168.14.60/512 to 192.168.2.6/0

Which is fair enough, as it doesn't seem to find a way to reach 192.168.2.6. Funny thing is that on 8.3 it worked and ASA perfectly could make up a default route out of the cryptomap, but heck with it. I add:

route BRANCHES 0.0.0.0 0.0.0.0 10.0.0.1 255

After that i am starting to get:

Tunnel Manager failed to dispatch a KEY_ACQUIRE message. Probable mis-configuration of the crypto map or tunnel-group. Map Tag = Unknown. Map Sequence Number = 0.

Digging into that i do

debug crypto ipsec 255

and I get:

gw-1(config)# IPSEC(crypto_map_check)-3: Looking for crypto map matching 5-tuple: Prot=1, saddr=192.168.14.60, sport=2, daddr=192.168.2.6, dport=2

IPSEC(crypto_map_check)-5: Checking crypto map BRANCH-OFFICE-CRYPTOMAP 1: skipping dormant map.

IPSEC(crypto_map_check)-5: Checking crypto map BRANCH-OFFICE-CRYPTOMAP 1: skipping because 5-tuple does not match ACL OO_temp_BRANCH-OFFICE-CRYPTOMAP1.

IPSEC(crypto_map_check)-1: Error: No crypto map matched.

and in an answer to that I do:

gw-1(config)# show access-list OO_temp_BRANCH-OFFICE-CRYPTOMAP1

access-list OO_temp_BRANCH-OFFICE-CRYPTOMAP1; 1 elements; name hash: 0xb7493f76 (dynamic)

access-list OO_temp_BRANCH-OFFICE-CRYPTOMAP1 line 1 extended permit ip host 10.0.0.3 host 10.0.0.2 (hitcnt=2) 0x3a45b1c8

Hmmm...looks strange to me actually I expected to see something else

Ok, never mind, well where actually the weird stuff comes in, is when in the same configuration I do:

crypto map BRANCH-OFFICE-CRYPTOMAP 1 set connection-type bidirectional

8.4.2 does successfully connect to the central ASA, even let's 3-4 pings and answers to those through and approximately in 5-10 seconds crashes with memory allocation error flushing a long list of adresses and going to reboot eventually.

While struggling with this one, I tried to downgrade to 8.4.1 with the same conf.

The symptoms and behaviour is actually the same, except for the last example with making a crypto-map bidirectional, 8.4.1 in this case doesn't crash - it connects, let's 3-4 pings through and closes the connection, in a second or two opens it once again and the scenario continues indefinetly. The reason for connection closing varies every time from being: Phase 2 Error, to User requested.

Sample output of 8.4.1 in this case below:

4|Jul 22 2011|10:54:10|113019|||||Group = BRANCH-OFFICE, Username = BRANCH-OFFICE, IP = 10.0.0.2, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:02s, Bytes xmt: 0, Bytes rcv: 0, Reason: Phase 2 Error

5|Jul 22 2011|10:54:10|750007|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA DOWN. Reason: no more IPSec SAs

6|Jul 22 2011|10:54:10|602304|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0x6276F541) between 10.0.0.2 and 10.0.0.3 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:54:10|602304|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0xAC62F506) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:54:09|602303|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0x6276F541) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

6|Jul 22 2011|10:54:09|602303|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0xAC62F506) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

5|Jul 22 2011|10:54:09|752016|||||IKEv2 was successful at setting up a tunnel. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

6|Jul 22 2011|10:54:09|113009|||||AAA retrieved default group policy (BRANCH-OFFICE) for user = BRANCH-OFFICE

5|Jul 22 2011|10:54:09|750006|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA UP. Reason: New Connection Established

6|Jul 22 2011|10:54:09|717028|||||Certificate chain was successfully validated with warning, revocation status was not checked.

6|Jul 22 2011|10:54:09|717022|||||Certificate was successfully validated. serial number: 75, subject name: ea=noc.office.ee,cn=gw-1.main.office,ou=network operations team,o=Office,l=Tallinn,st=Harjumaa,c=EE.

5|Jul 22 2011|10:54:08|750001|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:Unknown Received request to establish an IPsec tunnel; local traffic selector = Address Range: 192.168.14.60-192.168.14.60 Protocol: 0 Port Range: 0-65535; remote traffic selector = Address Range: 192.168.2.6-192.168.2.6 Protocol: 0 Port Range: 0-65535

4|Jul 22 2011|10:54:08|752011|||||IKEv1 Doesn't have a transform set specified

5|Jul 22 2011|10:54:08|752003|||||Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv2. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

4|Jul 22 2011|10:54:06|113019|||||Group = BRANCH-OFFICE, Username = BRANCH-OFFICE, IP = 10.0.0.2, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:03s, Bytes xmt: 0, Bytes rcv: 0, Reason: User Requested

5|Jul 22 2011|10:54:06|750007|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA DOWN. Reason: peer request

6|Jul 22 2011|10:54:06|602304|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0xB99F7DA2) between 10.0.0.2 and 10.0.0.3 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:54:06|602304|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0x4C650D47) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:54:03|602303|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0xB99F7DA2) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

6|Jul 22 2011|10:54:03|602303|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0x4C650D47) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

5|Jul 22 2011|10:54:03|752016|||||IKEv2 was successful at setting up a tunnel. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

6|Jul 22 2011|10:54:03|113009|||||AAA retrieved default group policy (BRANCH-OFFICE) for user = BRANCH-OFFICE

5|Jul 22 2011|10:54:03|750006|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA UP. Reason: New Connection Established

6|Jul 22 2011|10:54:03|717028|||||Certificate chain was successfully validated with warning, revocation status was not checked.

6|Jul 22 2011|10:54:03|717022|||||Certificate was successfully validated. serial number: 75, subject name: ea=noc.office.ee,cn=gw-1.main.office,ou=network operations team,o=Office,l=Tallinn,st=Harjumaa,c=EE.

5|Jul 22 2011|10:54:03|750001|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:Unknown Received request to establish an IPsec tunnel; local traffic selector = Address Range: 192.168.14.60-192.168.14.60 Protocol: 0 Port Range: 0-65535; remote traffic selector = Address Range: 192.168.2.6-192.168.2.6 Protocol: 0 Port Range: 0-65535

4|Jul 22 2011|10:54:03|752011|||||IKEv1 Doesn't have a transform set specified

5|Jul 22 2011|10:54:03|752003|||||Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv2. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

6|Jul 22 2011|10:53:58|302021|192.168.2.6|0|192.168.14.60|512|Teardown ICMP connection for faddr 192.168.2.6/0 gaddr 192.168.14.60/512 laddr 192.168.14.60/512

6|Jul 22 2011|10:53:57|302021|192.168.2.6|0|192.168.14.60|512|Teardown ICMP connection for faddr 192.168.2.6/0 gaddr 192.168.14.60/512 laddr 192.168.14.60/512

4|Jul 22 2011|10:53:56|113019|||||Group = BRANCH-OFFICE, Username = BRANCH-OFFICE, IP = 10.0.0.2, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:06s, Bytes xmt: 120, Bytes rcv: 120, Reason: User Requested

5|Jul 22 2011|10:53:56|750007|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA DOWN. Reason: peer request

6|Jul 22 2011|10:53:56|602304|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0xD137A9C1) between 10.0.0.2 and 10.0.0.3 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:53:56|602304|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0x5523DB62) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:53:55|302020|192.168.2.6|0|192.168.14.60|512|Built inbound ICMP connection for faddr 192.168.2.6/0 gaddr 192.168.14.60/512 laddr 192.168.14.60/512

6|Jul 22 2011|10:53:55|302020|192.168.14.60|512|192.168.2.6|0|Built outbound ICMP connection for faddr 192.168.2.6/0 gaddr 192.168.14.60/512 laddr 192.168.14.60/512

6|Jul 22 2011|10:53:50|602303|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0xD137A9C1) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

6|Jul 22 2011|10:53:50|602303|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0x5523DB62) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

5|Jul 22 2011|10:53:50|752016|||||IKEv2 was successful at setting up a tunnel. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

6|Jul 22 2011|10:53:50|113009|||||AAA retrieved default group policy (BRANCH-OFFICE) for user = BRANCH-OFFICE

5|Jul 22 2011|10:53:50|750006|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA UP. Reason: New Connection Established

6|Jul 22 2011|10:53:50|717028|||||Certificate chain was successfully validated with warning, revocation status was not checked.

6|Jul 22 2011|10:53:50|717022|||||Certificate was successfully validated. serial number: 75, subject name: ea=noc.office.ee,cn=gw-1.main.office,ou=network operations team,o=Office,l=Tallinn,st=Harjumaa,c=EE.

5|Jul 22 2011|10:53:50|750001|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:Unknown Received request to establish an IPsec tunnel; local traffic selector = Address Range: 192.168.14.60-192.168.14.60 Protocol: 0 Port Range: 0-65535; remote traffic selector = Address Range: 192.168.2.6-192.168.2.6 Protocol: 0 Port Range: 0-65535

4|Jul 22 2011|10:53:50|752011|||||IKEv1 Doesn't have a transform set specified

5|Jul 22 2011|10:53:50|752003|||||Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv2. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

4|Jul 22 2011|10:53:46|113019|||||Group = BRANCH-OFFICE, Username = BRANCH-OFFICE, IP = 10.0.0.2, Session disconnected. Session Type: LAN-to-LAN, Duration: 0h:00m:02s, Bytes xmt: 0, Bytes rcv: 0, Reason: User Requested

5|Jul 22 2011|10:53:46|750007|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA DOWN. Reason: peer request

6|Jul 22 2011|10:53:46|602304|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0x69F87708) between 10.0.0.2 and 10.0.0.3 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:53:46|602304|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0xAB2B9EDF) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been deleted.

6|Jul 22 2011|10:53:45|602303|||||IPSEC: An inbound LAN-to-LAN SA (SPI= 0x69F87708) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

6|Jul 22 2011|10:53:45|602303|||||IPSEC: An outbound LAN-to-LAN SA (SPI= 0xAB2B9EDF) between 10.0.0.3 and 10.0.0.2 (user= BRANCH-OFFICE) has been created.

5|Jul 22 2011|10:53:45|752016|||||IKEv2 was successful at setting up a tunnel. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

6|Jul 22 2011|10:53:45|113009|||||AAA retrieved default group policy (BRANCH-OFFICE) for user = BRANCH-OFFICE

5|Jul 22 2011|10:53:45|750006|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:BRANCH-OFFICE SA UP. Reason: New Connection Established

6|Jul 22 2011|10:53:45|717028|||||Certificate chain was successfully validated with warning, revocation status was not checked.

6|Jul 22 2011|10:53:45|717022|||||Certificate was successfully validated. serial number: 75, subject name: ea=noc.office.ee,cn=gw-1.main.office,ou=network operations team,o=Office,l=Tallinn,st=Harjumaa,c=EE.

6|Jul 22 2011|10:53:44|302015|10.0.0.3|500|10.0.0.2|500|Built outbound UDP connection 1156 for BRANCHES:10.0.0.2/500 (10.0.0.2/500) to identity:10.0.0.3/500 (10.0.0.3/500)

5|Jul 22 2011|10:53:44|750001|||||Local:10.0.0.3:500 Remote:10.0.0.2:500 Username:Unknown Received request to establish an IPsec tunnel; local traffic selector = Address Range: 192.168.14.60-192.168.14.60 Protocol: 0 Port Range: 0-65535; remote traffic selector = Address Range: 192.168.2.6-192.168.2.6 Protocol: 0 Port Range: 0-65535

4|Jul 22 2011|10:53:44|752011|||||IKEv1 Doesn't have a transform set specified

5|Jul 22 2011|10:53:44|752003|||||Tunnel Manager dispatching a KEY_ACQUIRE message to IKEv2. Map Tag = BRANCH-OFFICE-CRYPTOMAP. Map Sequence Number = 1.

8.4.2 output in such scenario is identical to 8.4.1's except that it crashes after 2 or 3 such connection cycles.

Needless to say that user requested reason is a lie and noone forced the channel to disconnect, in fact ping on the machine behind LAN interface (pinging 192.168.2.6) was put with indefinete amount of retries, something like 999999999, so the traffic was also sent to the channel.

I'm actually stuck with this one folks. I am used to that we have originate-only on the branch office side and dynamic crypto map on the main office side and the stuff flies. Now with this one it doesn't work either the originate-only, nor the bidirectional way.

Any ideas would be really appreciated.

Cheers!

Viktor

viktorlarionov · ‎07-25-2011

upd: With IKE v1 everything is OK. Connections get established and functioning normally on both 8.4.1 and 8.4.2.

(Same configuration, similar parameters for both phases AES-256 / SHA)

Guess, I would have to stick to IKEv1 until v2 gets stable.

Hope this helps anyone else with the same story

Cheers,

vik

viktorlarionov · ‎07-25-2011

Hey folks,

I have managed to solve this one I guess.

Just reporting back in case anyone else bumps into this.

It's a version incompatibility issue. The problem with crashes was specifically that our central host (with dynamic crypto map) was running 8.4.1 and IKEv2 tunnels from 8.4.2 (well actually from 8.4.1 too) would no establish.

As mentioned above in case of 8.4.1 the tunnel kept restarting back and forth and in case of 8.4.2 the client peer (asa5505) just crashed with memory allocation.

After upgrading the central host up to 8.4.2 IKEv2 tunnels establish and work with this configuration.

Though there still are conditions where you can crash the peer (asa5505) router with memory allocation, theese are quite rare conditions.

So in conclusion, in my configuration, IKEv2 L2L tunnels:

8.4.1 <-> 8.4.1 tunnel keeps restarting all the time, no stability

8.4.1 <-> 8.4.2 the client peer crashes with memory allocation error

8.4.2 <-> 8.4.2 works, not as stable as IKEv1, at least with my configuration, but works.

One more important thing: at least in my conf IKEv2 tunnel works only with "bidirectional" on client side. Standart "originate-only" (standart for me) does not.

------------------------------------------------------------

Just as a remark, if someone is interested, as one could see we are using L2L with tunnel exclusion conditions in the cryptomap, in order to access routers management interfaces remotely and provide some other internal services between carrier routers. (L2L peers are geographically far away).

When using 8.4.2 on both sides client peer memory allocation crash conditions come up, when doing for example a long running ping (>= 20 packets) from the central colocation side. Crash looks like this: the central co location in some time closes the channel providing: Internal Error as a reason and ignores further ESP packets from the client.

On the peer side the tunnels looks like as being still up, the client peer still tries to send information to the channel (as it seems the information about channel closing hasn't reached him). And in about a couple of minutes comes the memory allocation crash, stacktrace, and device restart. (I guess when the peer realises that the tunnel is not actually up).

I guess it would be great if Cisco's tech folks would have a look into that, and maybe provide a patch in next firmware version. But anyway 8.4.2 on both sides is much better than the previous scenarios.

Cheers!

Vik