too many IKE SAs

[Ab26]

what are the possible causes for having TOO many IKE SAs? they all are in negotiations and one is connected. The VPN tunnel seems to be working fine. 

[ccieexpert]

DPD should detect failure and bring down IKE/IPSEC... that is what it was designed to do.. lowering lifetime is not a ideal solution for failover. Can you draw up a topology and explain what you are trying to do so we can workout a better design.. i agree that is not what is causing the problem.. The debugs would be nice... Also what is different between this tunnel and others that dont have the issue ? are the remote peers the same device/IOS version ?

[Ab26]

DPD makes the same router to re-initiate IKE negotiation. In our setup we have 2 routers that have exactly the same config. The routers' interfaces toward the internet share FHRP-IP address. 

The remote peers have different devices based on what the customer uses. From my side all config are the same, but either policy-based on route-based (we have many of each).
I've just uploaded a debug for 2 hours. 

[ccieexpert]

i am not sure if this design really works FHRP to kick in you have to track something ? either connectivity through the tunnel etc ? how do you failover ? if one tunnel goes down, do you failover and make other unit FHRP active unit ? I have never seen this work... maybe we can have a chat about your design offline.

DPD will tear down and reinitiate yes.. but also when IKE/IPSEC lifetime exists, if there is traffic coming from the inside, it will re-initate the tunnel.. for a FHRP failover using this design is really hard to accomplish...

I havent had time to look a lot of details from debugs.

BTW no volume / KB based rekey seen in debugs, so i dont beleive that is the issue.

But I have seen all kind of issues (collisions or timing issues) when you lower the lifetime or keep ike/ipsec the same...

So far based on limited look i see this:

Line 1492: Jan 25 22:32:01.974 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 2830: Jan 25 22:46:32.372 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 4207: Jan 25 23:01:07.948 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 5590: Jan 25 23:15:43.471 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 7019: Jan 25 23:30:13.878 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 8408: Jan 25 23:44:49.406 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 9771: Jan 25 23:59:19.763 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8
Line 11183: Jan 26 00:13:55.299 CET: IPSEC:(SESSION ID = 278924) (ERROR) ident_get_ike_peer_index_from_peer could not find ikmp handle for peer 0x80007FAE97171DB8

it could be a timing condition, but basically these are happening with ipsec delete ....

you didnt get out of "show crypto ikev2 sa detailed" "show crypto session peer <peer>detailed" before and after debugs so not sure if the IKE sessions stayed or not ?

i would suggest setting the ike and ipsec lifetimes to default 86400 and 1 hour and see if it helps..

clear crypto session remote <peer ip>

the clear crypto session would clear out the existing session and take new values into effect, and also you can check of the stale ones are going away. I didnt see anything with ike deletes not working... but the fact is every 15 minutes there is a ike rekey, that may be excessive.. unless you how me the output before and after debugs it is hard to say.

What version are you running ?

Also, if you have TAC support, it maybe best to open a TAC case as there may be a defect in play here...

[Ab26]

Thanks @ccieexpert !

I'll look at the suggestions you advised. 

The output of the 2 show commands were the same before and after the debug. "show crypto ikev2 sa detailed" "show crypto session peer <peer>detailed"

---------------------------------------------------------------------------------------------

Here's how the config looks like for our redundancy setup: 

Primary VPN router:

interface GigabitEthernet0/1
description to Internet
vrf forwarding frontdoor
ip address 10.10.10.4 255.255.255.240
standby 100 ip 10.10.10.3
standby 100 priority 110
standby 100 name REDUNDANCY-VPN
ip tcp adjust-mss 1360
crypto map MAP-VPN redundancy REDUNDANCY-VPN
!
!
crypto ikev2 profile IKEv2-Profile
match fvrf frontdoor
match address local 10.10.10.3
match identity remote address 22.22.22.22 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local IKEv2-Keyring
lifetime 900
ivrf Cust_X-VRF
!
crypto map MAP-VPN 1 ipsec-isakmp
set peer 22.22.22.22
set transform-set IPsec-propsal
set ikev2-profile IKEv2-Profile
match address Cust_X-VPN_ACL
reverse-route remote-peer 10.10.10.1 static
!
ip route vrf Cust_X-VRF 10.10.10.1 255.255.255.255 GigabitEthernet0/1 10.10.10.1
---------------------------------------------------------------------------------------------
---------------------------------------------------------------------------------------------
In the secondary router the only difference is the GigabitEthernet0/1 config. The rest is the same

interface GigabitEthernet0/1
description to Internet
vrf forwarding frontdoor
ip address 10.10.10.5 255.255.255.240
standby 100 ip 10.10.10.3
ip tcp adjust-mss 1360
standby 100 name REDUNDANCY-VPN
ip tcp adjust-mss 1360
crypto map MAP-VPN redundancy REDUNDANCY-VPN
!

[ccieexpert]

ok. first thing is this type of redundancy is very basic... it relies of HSRP failure.. for the most part HSRP will failure in case of lan interface failure unless you track something more.. but you are not going to track 50 tunnels

the only use of "redundancy" is that it will allow you to use a HSRP VIP address for crypto..nothing much other than that.

whether you use 900 sec or default, it wont make any difference.. DPD would be better to track and tear down a tunnel or re-establish the tunnel.. my 2 cents.

Now the reason i asked ike output was to see if any new IKE sa that was created or deleted while running the debugs, if they were still lingering around as stale entries... the debugs didnt show that.. For the entries that are there before debugs were run, i have no way to tell how they got into that state (only with debugs running before they were created would have helped). If you had showed me the ike output, i could have made something out of it whether they were lingering, and the debugs may have highlighted the reason.

Hope that help.. Feel free to run them again with show command if you want me to look at it.. Also feel free to open a TAC case if you have support as it is a complex issues, and perhaps a defect..

[MHM Cisco World]

After you disable KBytes do you see anymore nego ikev2 session?

MHM

[Ab26]

No unfortunately. It didn't make any difference 

[MHM Cisco World]

show crypto map <<- share this please

MHM 

[Ab26]

Crypto Map IPv4 "VPN" 100 ipsec-isakmp
Peer = 22.22.22.22
IKEv2 Profile: IKev2-Profile
Extended IP access list VPN-ACL-Cust_X
access-list VPN-ACL-Cust_X permit ip 10.10.1.0 0.0.0.255 192.168.1.0 0.0.0.255
access-list VPN-ACL-Cust_X permit ip 10.10.2.0 0.0.0.255 192.168.1.0 0.0.0.255
access-list VPN-ACL-Cust_X permit ip host 10.10.10.1 192.168.1.0 0.0.0.255
Current peer: 22.22.22.22
Security association lifetime: 4608000 kilobytes/900 seconds
Responder-Only (Y/N): N
PFS (Y/N): N
Mixed-mode : Disabled
Transform sets={
AES256-SHA256-IPsec-prop: { esp-256-aes esp-sha256-hmac } ,
}
Reverse Route Injection Enabled

[MHM Cisco World]

Get it

There are two life time 

KBytes and time 900

Your Asr is more active so it bytes is end faster than other side 

Try disable KBytes under ipsec map

And I hope your VPN will be stable after that 

MHM

[Ab26]

Thanks @MHM Cisco World ! I did what you suggested but unfortunately it didn't make any difference. (i had the command on for one hour). 

How did you know that my router is the one that initiate the IKE negotiation? 

[MHM Cisco World]

Wait 24 hr (phaseI aging) then new lifetime will take effect' 

Check it tomorrow.

MHM

[liviu.gheorghe]

@Ab26 what IOS version are you running on the box? I have a similar config to yours, but not experiencing your symptoms.