Cisco ASA IKEv2 IPSec VTI site to site tunnel stopped working

[faruk.zaimovic]

Hello ,

I have 2 cisco ASA devices. I made site to site IKEv2/IPSec VTI tunnel between two ASA device. Sometimes that IPSec tunnel stopped working and I have to make shut and no shut tunnel interface to solve that tunnel work again. 

Does anybody have same problem or similar experiance? 

Conf for both ASA device is:

crypto ikev2 policy 5
 encryption aes-256
 integrity sha512 sha384
 group 19 14
 prf sha512 sha384
 lifetime seconds 86400

crypto ikev2 enable OUTSIDE

crypto ipsec ikev2 ipsec-proposal TSET
 protocol esp encryption aes-256 aes-192
 protocol esp integrity sha-512 sha-384 sha-256

crypto ipsec profile IPSEC_PROFILE
set ikev2 ipsec-proposal TSET

group-policy 2.2.2.1 internal
group-policy 2.2.2.1 attributes
 vpn-tunnel-protocol ikev2

tunnel-group x.x.x.x type ipsec-l2l
tunnel-group x.x.x.x general-attributes
 default-group-policy x.x.x.x
tunnel-group x.x.x.x ipsec-attributes
ikev2 remote-authentication pre-shared-key xxxxxxx
ikev2 local-authentication pre-shared-key xxxxxxx

interface Tunnel0
 nameif BRANCH1_VTI
 ip address 172.16.2.1 255.255.255.0
 tunnel source interface OUTSIDE
 tunnel destination 2.2.2.1
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile IPSEC_PROFILE

route BRANCH1_VTI 0.0.0.0 0.0.0.0 172.16.2.3

[Sheraz.Salim]

could you show what configuration you setup for this command.

show run all group policy x.x.x.x

above x.x.x.x means the remote peer ip address. I think what happening between two ASA firewall are they have either vpn-ideal-timeout or session-timeout setup so when they do not receive traffic they tear down the tunnels. having said that, VTI tunnel are classified as route based tunnel where as traditional vpn tunnel are known as policy based vpn.

[faruk.zaimovic]

Hello,

Thank you for your answer

ASA-FW-5516# show running-config all group-policy x.x.x.x
group-policy x.x.x.x internal
group-policy x.x.x.x attributes
vpn-tunnel-protocol ikev2
no vpn-simultaneous-login-delete-no-delay

configuration for group-policy is same for both side.

Other side have IP phones which make communication with IP central all time through tunnel. I think that there is always some traffic through tunnel.

one more thanks 

[Sheraz.Salim · Accepted solution]

If IP phone/s keeping sending the traffic in that case the tunnel should not go down at all. could you change your setting and test them and let us know how you getting on.

group-policy x.x.x.x internal
group-policy x.x.x.x attributes
  vpn-tunnel-protocol ikev2
  no vpn-simultaneous-login-delete-no-delay  
  vpn-idle-timeout none
!
crypto ipsec profile IPSEC_PROFILE
 set ikev2 ipsec-proposal TSET
 set pfs group14
 set security-association lifetime seconds 86400

in case if your ASA firewalls tunnel doing the rekeys when no interesting traffic is tranist/passing, the tunnel will not rebuild until interesting traffic is seen. This is the default behaviour of the ASA firewall is. 

[faruk.zaimovic]

Hello, 

I added your configuration, and I will follow state. I will notify  you.

Thank you very much for your help.

[Rob Ingram]

@faruk.zaimovic this is a route based VPN using VTI's, so you don't need interesting traffic for the tunnel to establish or to rekey, unlike a policy based VPN.

You say the issue is "Sometimes that IPSec tunnel stopped working and I have to make shut and no shut tunnel interface to solve that tunnel work again" - how often is sometimes? Every 12, 24 hours or once a week?

Prior to shut/no shutdown the interface, have you run a debug of ikev2/ipsec?

When there is no communication over the VPN, what is the state of "show crypto ikev2 sa" and "show crypto ipsec sa" of both ASA?

What ASA software version are you using, have you checked for a bug?

[faruk.zaimovic]

Hello,

Thank you for your help.

It is happened at least one time in 24 hours.  show crypto isakmp sa show that tunnel is active. 

ASA sofwear version is 9-16-3-23 , i made upgrade on that version. Previous version was 9-16-2-14. I had same problem on both version. 

I added conf from previous post.

group-policy x.x.x.x internal
group-policy x.x.x.x attributes
  vpn-tunnel-protocol ikev2
  no vpn-simultaneous-login-delete-no-delay  
  vpn-idle-timeout none
!
crypto ipsec profile IPSEC_PROFILE
 set ikev2 ipsec-proposal TSET
 set pfs group14
 set security-association lifetime seconds 86400

and I will follow state. 

Thank you for your help.

[Sheraz.Salim]

@faruk.zaimovic when traffic is not passing but you see the tunnel is up "show crypto isakmp sa".

could you setup the capture on your firewall and also get the debug if the given work around does not work. as it stand this is strange issue the tunnel never goes down but it also do not passing the traffic.

as mentioned by @Rob Ingram you can setup the debug to check whats going on. but having said that you already said the tunnel stay always up in that case you wont be see the debug (the one we wanted to see why this strange behavior happening).

did this tunnel ever work stable prior to going 9.16?

[Sheraz.Salim]

just adding on this the tunnel goes down every 24 hours. does the same configuration are applied on both boxes (asa firewalls). I am thinking it could the phase2 and rekey tear the tunnel down. but again I have not see the route based tunnel goes down ever. but yes policy based does where there is no traffic passing by.

[faruk.zaimovic]

. Hello,

Thanks for help and answer. Both cisco ASA device hasn't had same version, but I made upgrade and now both devices have same version 9.16. I expect that tomorrow happen same situation, and I will make debug and try to see what happen. I will share here my outputs. 

[Rob Ingram]

@faruk.zaimovic does it always happen every 24 hours? this would indicate a rekey issue, but if it happens infrequently (a couple of times a week), then it might be another issue.

If/when it happens again and before you take any action to resolve - make sure you check "show crypto ikev2 sa" and "show crypto ipsec sa" on both ASA, and provide the output of both ASA so we can see. Run ikev2 and ipsec debugs, provide the output.

[faruk.zaimovic]

Hello Rob, 

When I added conf which provide me  Sheraz.Salim , tunnel is up more than 24 hours. I think that solve my problem. I will continue follow state. If there some change I will write here. 

Thanks all for help.

[faruk.zaimovic]

Hello ,

We had same problem today. The tunnel have been working more 5 days without any problem. My customer didn't make debug ikev2. He only made show crypto isakmp sa , and show crypto isakmp sa, and I can see that tunnel is ACTIVE and that traffic encrypt and decrypt.  Traffice didn't pass through tunnel and they had make shut and no shut tunnel interface.

[Rob Ingram]

@faruk.zaimovic that's why I was curious to know whether the tunnels stops at regular intervals or randomly. If after 5 days the issue appears to be random

Unfortunately without seeing the output and the debugs it's going to be hard to determine the issue. I recommend logging a TAC call.

[faruk.zaimovic]

Hello,

I contacted Cisco TAC, help me to solve problem and I want to share with you if someone else hit on same problem. it was Cisco bug CSCwc95290 . Below cisco TAC answer.

kindly note that after checking internally analyzing the outputs, and trying to match your behavior with other cases we found that we are hitting on the following software defect for ASA5516 which may indicate that something freezing on the VPN context:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCwc95290

we suggest upgrading your device to 9.16(4)14 which consider one of the fixed releases and monitoring the behavior.

Please note that while upgrading your device, you may hit the following software defect. The workaround is to unplug the cable and replug it:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvu12608