11-30-2016 08:33 AM - edited 03-12-2019 07:22 AM
I am running an AWS Transit VPC setup that automatically instantiates a CSR1000v and configures it. It comes up and works, but once or twice a day the IPSEC tunnels fail with the following messages:
*Nov 30 15:56:22.958: %CRYPTO-6-ISAKMP_MANUAL_DELETE: IKE SA manually deleted. Do 'clear crypto sa peer 52.32.1.245' to manually clear IPSec SA's covered by this IKE SA.
*Nov 30 15:57:19.388: %CRYPTO-6-ISAKMP_MANUAL_DELETE: IKE SA manually deleted. Do 'clear crypto sa peer 52.35.26.206' to manually clear IPSec SA's covered by this IKE SA.
Running 'clear crypto sa' recovers them, but this is an annoying manual process. The tunnels occasionally self-recover as well.
The auto-configuration scripts didn't run during that time so I'm very confused on what 'manually deleted' means.
The configuration is provided by AWS/CloudFormation so I am confident I haven't directly misconfigured something. Here is the config w/credentials stripped in case you can spot the issue:
Current configuration : 7694 bytes
!
! Last configuration change at 16:26:24 UTC Tue Nov 29 2016 by ec2-user
!
version 16.3
service timestamps debug datetime msec
service timestamps log datetime msec
service password-encryption
no platform punt-keepalive disable-kernel-core
platform console virtual
!
hostname ip-192-168-0-11
!
boot-start-marker
boot-end-marker
!
!
logging buffered 32000
logging persistent size 1000000 filesize 8192
!
no aaa new-model
!
ip vrf vpn-15736274
rd 64512:1
route-target export 64512:0
route-target import 64512:0
!
ip vrf vpn-b41009a6
rd 64512:3
route-target export 64512:0
route-target import 64512:0
!
ip vrf vpn0
rd 64512:0
!
subscriber templating
!
multilink bundle-name authenticated
!
crypto pki trustpoint TP-self-signed-2690167130
enrollment selfsigned
subject-name cn=IOS-Self-Signed-Certificate-2690167130
revocation-check none
rsakeypair TP-self-signed-2690167130
!
!
crypto pki certificate chain TP-self-signed-2690167130
certificate self-signed 01
x
quit
!
license udi pid CSR1000V sn x
diagnostic bootup level minimal
!
spanning-tree extend system-id
!
username x
username y
!
redundancy
!
crypto keyring keyring-vpn-b41009a6-4
local-address GigabitEthernet1
pre-shared-key address 52.35.26.206 key x
crypto keyring keyring-vpn-b41009a6-3
local-address GigabitEthernet1
pre-shared-key address 52.32.1.245 key x
crypto keyring keyring-vpn-15736274-2
local-address GigabitEthernet1
pre-shared-key address 52.20.207.139 key x_b
crypto keyring keyring-vpn-15736274-1
local-address GigabitEthernet1
pre-shared-key address 23.22.23.82 key x
!
!
crypto isakmp policy 200
encr aes
authentication pre-share
group 2
lifetime 28800
crypto isakmp keepalive 10 10
crypto isakmp profile isakmp-vpn-15736274-1
keyring keyring-vpn-15736274-1
match identity address 23.22.23.82 255.255.255.255
local-address GigabitEthernet1
crypto isakmp profile isakmp-vpn-15736274-2
keyring keyring-vpn-15736274-2
match identity address 52.20.207.139 255.255.255.255
local-address GigabitEthernet1
crypto isakmp profile isakmp-vpn-b41009a6-3
keyring keyring-vpn-b41009a6-3
match identity address 52.32.1.245 255.255.255.255
local-address GigabitEthernet1
crypto isakmp profile isakmp-vpn-b41009a6-4
keyring keyring-vpn-b41009a6-4
match identity address 52.35.26.206 255.255.255.255
local-address GigabitEthernet1
!
crypto ipsec security-association replay window-size 1024
!
crypto ipsec transform-set ipsec-prop-vpn-aws esp-aes esp-sha-hmac
mode tunnel
crypto ipsec df-bit clear
!
!
crypto ipsec profile ipsec-vpn-aws
set transform-set ipsec-prop-vpn-aws
set pfs group2
!
interface Tunnel1
ip vrf forwarding vpn-15736274
ip address 169.254.44.154 255.255.255.252
ip tcp adjust-mss 1387
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 23.22.23.82
tunnel protection ipsec profile ipsec-vpn-aws
ip virtual-reassembly
!
interface Tunnel2
ip vrf forwarding vpn-15736274
ip address 169.254.46.46 255.255.255.252
ip tcp adjust-mss 1387
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 52.20.207.139
tunnel protection ipsec profile ipsec-vpn-aws
ip virtual-reassembly
!
interface Tunnel3
ip vrf forwarding vpn-b41009a6
ip address 169.254.14.202 255.255.255.252
ip tcp adjust-mss 1387
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 52.32.1.245
tunnel protection ipsec profile ipsec-vpn-aws
ip virtual-reassembly
!
interface Tunnel4
ip vrf forwarding vpn-b41009a6
ip address 169.254.12.146 255.255.255.252
ip tcp adjust-mss 1387
tunnel source GigabitEthernet1
tunnel mode ipsec ipv4
tunnel destination 52.35.26.206
tunnel protection ipsec profile ipsec-vpn-aws
ip virtual-reassembly
!
interface GigabitEthernet1
ip address dhcp
negotiation auto
!
router bgp 64512
bgp log-neighbor-changes
!
address-family ipv4 vrf vpn-15736274
neighbor 169.254.44.153 remote-as 7224
neighbor 169.254.44.153 timers 10 30 30
neighbor 169.254.44.153 activate
neighbor 169.254.44.153 as-override
neighbor 169.254.44.153 soft-reconfiguration inbound
neighbor 169.254.46.45 remote-as 7224
neighbor 169.254.46.45 timers 10 30 30
neighbor 169.254.46.45 activate
neighbor 169.254.46.45 as-override
neighbor 169.254.46.45 soft-reconfiguration inbound
exit-address-family
!
address-family ipv4 vrf vpn-b41009a6
neighbor 169.254.12.145 remote-as 7224
neighbor 169.254.12.145 timers 10 30 30
neighbor 169.254.12.145 activate
neighbor 169.254.12.145 as-override
neighbor 169.254.12.145 soft-reconfiguration inbound
neighbor 169.254.14.201 remote-as 7224
neighbor 169.254.14.201 timers 10 30 30
neighbor 169.254.14.201 activate
neighbor 169.254.14.201 as-override
neighbor 169.254.14.201 soft-reconfiguration inbound
exit-address-family
!
virtual-service csr_mgmt
ip shared host-interface GigabitEthernet1
activate
!
ip forward-protocol nd
ip http server
ip http authentication local
ip http secure-server
!
ip ssh rsa keypair-name ssh-key
ip ssh version 2
ip ssh pubkey-chain
username ec2-user
key-hash ssh-rsa x ec2-user
username automate
key-hash ssh-rsa x
ip ssh server algorithm authentication publickey
!
control-plane
!
line con 0
stopbits 1
line vty 0 4
login local
transport input ssh
!
end
Any ideas what is occurring here?
Solved! Go to Solution.
12-05-2016 01:19 PM
Huge help! I have set up the security group to allow UDP 500 in for all the IPs. We will see if this works better and I have submitted a ticket to AWS to adjust the Cloud Formation\Lamda scripts.
11-30-2016 12:45 PM
The current word from AWS Support is :
It appears that Cisco isn't rekeying phase 1. Here are the snip of the logs:
vpn-15736274
CGW: 34.193.127.69
Tunnel 23.22.23.82
Nov 30 08:01:33 ISAKMP SA established
Nov 30 16:01:33 ISAKMP SA expired
Nov 30 16:01:54 IPsec SA expired since there is no active ISAKMP SA
Nov 30 16:01:56 ISAKMP SA established[snip]
The above process repeat for evey re-key attempt. I have seen this issue before but Cisco has not addressed it. We send rekey proposal before Phase 1 is about to expire but Cisco fails to see/process this proposal. If Cisco processes this request, phase 1 should never go down. However, the core issue here is Cisco is not rekeying Phase 1. Could you please contact Cisco and escalate this issue?
12-05-2016 12:27 PM
Hi Jason,
We are having the EXACT same issue with the AWS transit VPC. Did you head anything back from Cisco? We have the same problem on the Tunnel to our on-prem Palo Altos as well. Any info is appreciated.
Thanks!
Mike
12-05-2016 12:30 PM
Cisco support said 'you don't have support under AWS LicenseIncluded, buy a license'
I opened up incoming traffic from the VGW IP's and that seems to have mitigated the issue. I guess somehow the security group isn't recognizing the phase-1 rekey as being part of an existing connection and was dropping it.
12-05-2016 01:19 PM
Huge help! I have set up the security group to allow UDP 500 in for all the IPs. We will see if this works better and I have submitted a ticket to AWS to adjust the Cloud Formation\Lamda scripts.
06-05-2018 12:49 AM
I had the same issue. I just have one question. Is there any threat opening 500 & 4500 to all the IPs?
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide