VPN connectivity lost after rekeying (i think)

[jesper_petersen]

Hello

I have a L2L IPSEC tunnel between a set of failover pair of two ASA5510's and a single ASA5505. Over time they will loose connectivity through the tunnel. The tunnel itself stays up, but cannot pass any traffic.

When looking at the tunnel I always see this on the set of 5510's (marked in bold @ IPSEC ID 3):

advdns# sh vpn-sessiondb detail l2l filter ipaddress 93.160.2xx.1xx

Session Type: LAN-to-LAN Detailed

Connection   : 93.160.2xx.1xx
Index        : 14                     IP Addr      : K015-Peer
Protocol     : IPSecLAN2LAN           Encryption   : 3DES
Hashing      : SHA1
Bytes Tx     : 430820527              Bytes Rx     : 9869311
Login Time   : 01:16:13 CEDT Mon Mar 28 2011
Duration     : 7h:46m:47s
Filter Name  : K015-L2L-filter

IKE Sessions: 1
IPSec Sessions: 2

IKE:
  Session ID   : 1
  UDP Src Port : 500                    UDP Dst Port : 500
  IKE Neg Mode : Main                   Auth Mode    : preSharedKeys
  Encryption   : 3DES                   Hashing      : SHA1
  Rekey Int (T): 86400 Seconds          Rekey Left(T): 58390 Seconds
  D/H Group    : 2

IPSec:
  Session ID   : 2
  Local Addr   : HOST_RDC001/255.255.255.255/0/0
  Remote Addr  : 192.168.15.0/255.255.255.0/0/0
  Encryption   : 3DES                   Hashing      : SHA1
  Encapsulation: Tunnel
  Rekey Int (T): 28800 Seconds          Rekey Left(T): 25270 Seconds
  Rekey Int (D): 413696 K-Bytes         Rekey Left(D): 413688 K-Bytes
  Bytes Tx     : 24387                  Bytes Rx     : 12754
  Pkts Tx      : 195                    Pkts Rx      : 195

IPSec:
  Session ID   : 3
  Local Addr   : 10.30.15.0/255.255.255.0/0/0
  Remote Addr  : 192.168.15.0/255.255.255.0/0/0
  Encryption   : 3DES                   Hashing      : SHA1
  Encapsulation: Tunnel
  Rekey Int (T): 28800 Seconds          Rekey Left(T): 25715 Seconds
  Rekey Int (D): 413696 K-Bytes         Rekey Left(D): 1 K-Bytes
  Bytes Tx     : 430796140              Bytes Rx     : 9856557
  Pkts Tx      : 385454                 Pkts Rx      : 207904

This is the result of the same command at the ASA5505 end of the tunnel:

Pff# sh vpn-sessiondb detail l2l

Session Type: LAN-to-LAN Detailed

Connection   : 83.136.xx.xxx
Index        : 1                      IP Addr      : 83.136.xx.xxx
Protocol     : IPSecLAN2LAN           Encryption   : 3DES
Hashing      : SHA1
Bytes Tx     : 9869359                Bytes Rx     : 430815282
Login Time   : 14:00:28 UTC Sun Mar 27 2011
Duration     : 7h:47m:00s
Filter Name  :

IKE Sessions: 1
IPSec Sessions: 2

IKE:
  Session ID   : 1
  UDP Src Port : 500                    UDP Dst Port : 500
  IKE Neg Mode : Main                   Auth Mode    : preSharedKeys
  Encryption   : 3DES                   Hashing      : SHA1
  Rekey Int (T): 86400 Seconds          Rekey Left(T): 58381 Seconds
  D/H Group    : 2

IPSec:
  Session ID   : 2
  Local Addr   : 192.168.15.0/255.255.255.0/0/0
  Remote Addr  : 10.1.11.1/255.255.255.255/0/0
  Encryption   : 3DES                   Hashing      : SHA1
  Encapsulation: Tunnel
  Rekey Int (T): 28800 Seconds          Rekey Left(T): 25256 Seconds
  Rekey Int (D): 4275000 K-Bytes        Rekey Left(D): 4274992 K-Bytes
  Idle Time Out: 30 Minutes             Idle TO Left : 29 Minutes
  Bytes Tx     : 12754                  Bytes Rx     : 24387
  Pkts Tx      : 195                    Pkts Rx      : 195

IPSec:
  Session ID   : 3
  Local Addr   : 192.168.15.0/255.255.255.0/0/0
  Remote Addr  : 10.30.15.0/255.255.255.0/0/0
  Encryption   : 3DES                   Hashing      : SHA1
  Encapsulation: Tunnel
  Rekey Int (T): 28800 Seconds          Rekey Left(T): 25701 Seconds
  Rekey Int (D): 4275000 K-Bytes        Rekey Left(D): 3861311 K-Bytes
  Idle Time Out: 30 Minutes             Idle TO Left : 30 Minutes
  Bytes Tx     : 9856605                Bytes Rx     : 430790895
  Pkts Tx      : 207905                 Pkts Rx      : 385265

On the ASA5505 I can see the following in the log:

Mar 27 2011 21:21:17: %ASA-4-402120: IPSEC: Received an ESP packet (SPI= 0xBB2A21CF, sequence number= 0x1BB08) from 83.136.xx.xxx (user= 83.136.xx.xxx) to 93.160.2xx.1xx that failed authentication.
Mar 27 2011 21:26:12: %ASA-4-402120: IPSEC: Received an ESP packet (SPI= 0xBB2A21CF, sequence number= 0x2EF6E) from 83.136.xx.xxx (user= 83.136.xx.xxx) to 93.160.2xx.1xx that failed authentication.

It has done this 4-5 times now, so i dont think it is a temporary problem. The ASA5505 has been rebooted several times.. Rebooting the failover 5510 is not an option. The 5510 currently holds over 50 IPSEC tunnels and this is the only one behaving like this.

If I do a clear cry ips sa peer "IP of the 5505", then the tunnel goes functional again.

The SW version is:

5510:     7.2.(4)9

5505:     7.2.(4)

This is the configuration that I use for the tunnel:

5510:

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac

crypto map outside_map 15 match address K015-L2L-list
crypto map outside_map 15 set peer K015-Peer
crypto map outside_map 15 set transform-set ESP-3DES-SHA
crypto map outside_map 15 set security-association lifetime seconds 28800
crypto map outside_map 15 set security-association lifetime kilobytes 4608000
crypto map outside_map interface outside
crypto isakmp identity address
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

5505:

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto map VPNMAP 10 match address Hosting_List
crypto map VPNMAP 10 set peer 83.136.xx.xxx
crypto map VPNMAP 10 set transform-set ESP-3DES-SHA
crypto map VPNMAP interface outside
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

Any of you got any good ideas?

Best regards,

Jesper Ross

[Jennifer Halim]

Are you able to try to remove the following 2 lines from ASA5510:

crypto map outside_map 15 set security-association lifetime seconds 28800
crypto map outside_map 15 set security-association lifetime kilobytes 4608000

OR, add the same on ASA 5505:

crypto map VPNMAP 10 set security-association lifetime seconds 28800

crypto map VPNMAP 10 set security-association lifetime kilobytes 4608000

[jesper_petersen]

Hello Jennifer

Thank you for your answer.

I've tried to enter the part for the 5505. That did not help - the tunnel crashed after its KB ran out.

Afterwards I tried to enter the part for the 5510, that did not help either and the tunnel crashed.

As far as I can see both units are using the same numbers for rekeying; 28800 secs and 4608000 kb.

[jesper_petersen]

Here the output of sh run all cry of the ASA 5505

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto ipsec security-association replay window-size 64
crypto ipsec fragmentation before-encryption inside
crypto ipsec fragmentation before-encryption outside
crypto ipsec df-bit copy-df inside
crypto ipsec df-bit copy-df outside
crypto map VPNMAP 10 match address Hosting_List
crypto map VPNMAP 10 set connection-type bi-directional
crypto map VPNMAP 10 set peer 83.136.xx.xxx
crypto map VPNMAP 10 set transform-set ESP-3DES-SHA
crypto map VPNMAP 10 set security-association lifetime seconds 28800
crypto map VPNMAP 10 set security-association lifetime kilobytes 4608000
crypto map VPNMAP 10 set inheritance rule
crypto map VPNMAP 10 set phase1-mode main
crypto map VPNMAP interface outside
crypto isakmp identity auto
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400

------

Here is the same output for the 5510 (other crypto tunnels omitted):

crypto ipsec transform-set ESP-3DES-SHA esp-3des esp-sha-hmac
crypto ipsec transform-set ESP-3DES-MD5 esp-3des esp-md5-hmac
crypto ipsec transform-set ESP-DES-SHA esp-des esp-sha-hmac
crypto ipsec transform-set ESP-AES256-SHA esp-aes-256 esp-sha-hmac
crypto ipsec security-association lifetime seconds 28800
crypto ipsec security-association lifetime kilobytes 4608000
crypto ipsec security-association replay window-size 64
crypto ipsec fragmentation before-encryption outside
crypto ipsec fragmentation before-encryption inside
...

crypto ipsec df-bit copy-df outside
crypto ipsec df-bit copy-df inside
...

crypto map outside_map 15 match address K015-L2L-list
crypto map outside_map 15 set connection-type bi-directional
crypto map outside_map 15 set peer K015-Peer
crypto map outside_map 15 set transform-set ESP-3DES-SHA
crypto map outside_map 15 set security-association lifetime seconds 28800
crypto map outside_map 15 set security-association lifetime kilobytes 4608000
crypto map outside_map 15 set inheritance rule
crypto map outside_map 15 set phase1-mode main
...

crypto map outside_map interface outside
crypto isakmp identity address
crypto isakmp enable outside
crypto isakmp policy 10
authentication pre-share
encryption 3des
hash sha
group 2
lifetime 86400
crypto isakmp policy 20
authentication pre-share
encryption 3des
hash md5
group 2
lifetime 3600
crypto isakmp policy 30
authentication pre-share
encryption aes-256
hash sha
group 5
lifetime 86400
crypto isakmp nat-traversal  20

[Jennifer Halim · Accepted solution]

I just checked and there are a number of bugs in regards to rekey, in ASA version 7.2.4, please kindly upgrade both ASA to at least version 7.2.5.

Here are the bugs for your reference:

CSCtc47782 Malformed IKE traffic causes rekey to fail:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtc47782

CSCso87442  ASA displays smaller traffic-volume lifetime than negotiated:

http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCso87442

CSCsq67954 ASA rekeys at less traffic volume than expected value:
http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCsq67954

Prior to upgrade, you can just remove the following and see if it makes any difference:
crypto map VPNMAP 10 set security-association lifetime kilobytes 4608000
crypto map outside_map 15 set security-association lifetime kilobytes 4608000

Clear tunnels on both end, and monitor to see if you are seeing the same issue.

[jesper_petersen]

Hello Jennifer,

Thank you very much for your help. That gives me more confidence in the two units configuration knowing that there is a bug that could pose the problem that I'm facing.

I've tried to remove the two command by placing a "no" in front of them, but it cannot seem to be done as the values are the default ones.

I will try to update the 5505 and see if that solves the problem. Updating the 5510 is more complicated, so I'll do that one last if possible.

[sachinga.hcl]

There is a workaround and a fix for this issue;

1. Workaround: The lifetime values for the particular VPN tunnel in question needs to be adjusted where the re-key for the VPN should happen with the seconds lifetime and not the data lifetime.

2. Fix: Upgrade to ASA 8.4(3) or the other versions in which the bug is fixed.

The Workaround:

Change the lifetime seconds to a lower value so that the outbound IPsec SA rekey happens when the seconds threshold is reached. And change the lifetime kilobytes to the highest value so that the outbound IPSec SA never rekeys with the kilobytes lifetime because this is where the bug kicks in.

crypto map outside_map 10 set lifetime seconds 3600

crypto map outside_map 10 set lifetime kilobytes 2147483647

Although you can set these lifetime values globally, I wouldn’t recommend it because you do not want the ASA to rekey for all the VPN tunnels you have in every one hour! So, just monitor the VPNs and see which ones would need the workaround to get over this issue.

The Fix:

Upgrading the firewall in a production environment to solve the issue might not be the most appropriate choice specially when you have a workaround that works pretty well. So you can implement the workaround until you get a maintenance window for the upgrade. The bug fixed versions can be found here > http://tools.cisco.com/Support/BugToolKit/search/getBugDetails.do?method=fetchBugDetails&bugId=CSCtq57752CSCtq57752 Bug Details

Hope you find this useful and get your VPN tunnels up and running as quickly as possible!

https://supportforums.cisco.com/thread/1004134