I've simplified things a bit:
HUB:
====
no crypto ikev2 client flexvpn Tunnel1
no crypto ikev2 client flexvpn Tunnel2
no crypto ikev2 client flexvpn Tunnel3
no interface tunnel1
no interface tunnel2
interface tunnel2
tunnel destination HUB1
REMOTE:
========
interface Virtual-Template1 type tunnel
no tunnel source
shutdown
interface Virtual-Template2 type tunnel
no tunnel source
shutdown
So there is only tunnel3 at the remote end and only virtual-template 3 listening on the loopback address on the HUB end. In addition to this, the remote is statically configured with the destination of the hub (no flexvpn client hub redundancy). And this still doens't work. There is something in the configuration above with Tunnels 2 and 3 which the routers at both ends do not like and it's not jumping out at me because other than changing the VRF name on the end of the identities, it looks the same.
-------
Here is the debug output from begining to end (with source and destination IP's removed):
004037: .Mar 5 15:32:00.633: IKEv2:% Getting preshared key from profile keyring LAN-to-LAN
004038: .Mar 5 15:32:00.633: IKEv2:% Matched peer block 'LAN-to-LAN'
004039: .Mar 5 15:32:00.633: IKEv2:Searching Policy with fvrf 0, local address SPOKE1
004040: .Mar 5 15:32:00.633: IKEv2:Using the Default Policy for Proposal
004041: .Mar 5 15:32:00.633: IKEv2:Found Policy 'default'
004042: .Mar 5 15:32:00.633: IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH public key, DH Group 19
004043
SPOKE1(config)#: .Mar 5 15:32:00.633: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
004044: .Mar 5 15:32:00.633: IKEv2:(SA ID = 1):Request queued for computation of DH key
004045: .Mar 5 15:32:00.633: IKEv2:IKEv2 initiator - no config data to send in IKE_SA_INIT exch
004046: .Mar 5 15:32:00.633: IKEv2:(SA ID = 1):Generating IKE_SA_INIT message
004047: .Mar 5 15:32:00.633: IKEv2:(SA ID = 1):IKE Proposal: 1, SPI size: 0 (initial negotiation),
Num. transforms: 7
AES-CBC SHA256 SHA1
SPOKE1(config)# SHA256 SHA96 DH_GROUP_256_ECP/Group 19 DH_GROUP_2048_MODP/Group 14
004048: .Mar 5 15:32:00.633: IKEv2:(SA ID = 1):Sending Packet [To HUB1:500/From SPOKE1:500/VRF i0:f0]
Initiator SPI : EECCD9476A6301DE - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
004049: .Mar 5 15:32:00.637: IKEv2:(SA ID = 1):Insert SA
SPOKE1(config)#
004050: .Mar 5 15:32:02.585: IKEv2:(SA ID = 1):Retransmitting packet
004051: .Mar 5 15:32:02.585: IKEv2:(SA ID = 1):Sending Packet [To HUB1:500/From SPOKE1:500/VRF i0:f0]
Initiator SPI : EECCD9476A6301DE - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
004052: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):Received Packet [From HUB1
SPOKE1(config)#54:500/To SPOKE1:500/VRF i0:f0]
Initiator SPI : EECCD9476A6301DE - Responder SPI : E0D14C006A000813 Message id: 0
IKEv2 IKE_SA_INIT Exchange RESPONSE
Payload contents:
SA KE N VID VID NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP)
004053: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):Processing IKE_SA_INIT message
004054: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):Verify SA init message
004055: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):Processing IKE_SA_INIT message
004056: .
SPOKE1(config)#Mar 5 15:32:03.173: IKEv2:(SA ID = 1):Checking NAT discovery
004057: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):NAT INSIDE found
004058: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):NAT detected float to init port 4500, resp port 4500
004059: .Mar 5 15:32:03.173: IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Computing DH secret key, DH Group 19
004060: .Mar 5 15:32:03.197: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] DH key Computation PASSED
004061: .Mar 5 15:32:03.197: IKEv2:(SA ID = 1):Request queued for co
SPOKE1(config)#mputation of DH secret
004062: .Mar 5 15:32:03.197: IKEv2:(SA ID = 1):[IKEv2 -> Crypto Engine] Calculate SKEYSEED and create rekeyed IKEv2 SA
004063: .Mar 5 15:32:03.197: IKEv2:(SA ID = 1):[Crypto Engine -> IKEv2] SKEYSEED calculation and creation of rekeyed IKEv2 SA PASSED
004064: .Mar 5 15:32:03.197: IKEv2:(SA ID = 1):Completed SA init exchange
004065: .Mar 5 15:32:03.197: IKEv2:Config data to send:
004066: .Mar 5 15:32:03.197: Config-type: Config-request
004067: .Mar 5 15:32:03.197: Attrib t
SPOKE1(config)#ype: ipv4-dns, length: 0
004068: .Mar 5 15:32:03.197: Attrib type: ipv4-dns, length: 0
004069: .Mar 5 15:32:03.197: Attrib type: ipv4-nbns, length: 0
004070: .Mar 5 15:32:03.197: Attrib type: ipv4-nbns, length: 0
004071: .Mar 5 15:32:03.197: Attrib type: ipv4-subnet, length: 0
004072: .Mar 5 15:32:03.197: Attrib type: app-version, length: 244, data: Cisco IOS Software, C2900 Software (C2900-UNIVERSALK9-M), Version 15.2(4)M5, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupp
SPOKE1(config)#ort
Copyright (c) 1986-2013 by Cisco Systems, Inc.
Compiled Fri 13-Sep-13 14:59 by prod_rel_team
004073: .Mar 5 15:32:03.197: Attrib type: split-dns, length: 0
004074: .Mar 5 15:32:03.197: Attrib type: banner, length: 0
004075: .Mar 5 15:32:03.197: Attrib type: config-url, length: 0
004076: .Mar 5 15:32:03.197: Attrib type: backup-gateway, length: 0
004077: .Mar 5 15:32:03.201: Attrib type: def-domain, length: 0
004078: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Have config mode data to send
004
SPOKE1(config)#079: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Check for EAP exchange
004080: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Generate my authentication data
004081: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Use preshared key for id SPOKE1_CUSTOMER3, key len 6
004082: .Mar 5 15:32:03.201: IKEv2:[IKEv2 -> Crypto Engine] Generate IKEv2 authentication data
004083: .Mar 5 15:32:03.201: IKEv2:[Crypto Engine -> IKEv2] IKEv2 authentication data generation PASSED
004084: .Mar 5 15:32:03.201:
SPOKE1(config)#IKEv2:(SA ID = 1):Get my authentication method
004085: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):My authentication method is 'PSK'
004086: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Check for EAP exchange
004087: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Generating IKE_AUTH message
004088: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Constructing IDi payload: 'SPOKE1_CUSTOMER3' of type 'FQDN'
004089: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):ESP Proposal: 1, SPI size: 4 (IPSec negotiation)
SPOKE1(config)#,
Num. transforms: 3
AES-CBC SHA96 Don't use ESN
004090: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Building packet for encryption.
Payload contents:
VID IDi AUTH CFG SA TSi TSr NOTIFY(INITIAL_CONTACT) NOTIFY(SET_WINDOW_SIZE) NOTIFY(ESP_TFC_NO_SUPPORT) NOTIFY(NON_FIRST_FRAGS)
004091: .Mar 5 15:32:03.201: IKEv2:(SA ID = 1):Sending Packet [To HUB1:4500/From SPOKE1:4500/VRF i0:f0]
Initiator SPI : EECCD9476A6301DE - Responder SPI : E0D14C006A000813 Message id: 1
IKEv2 IKE_A
SPOKE1(config)#UTH Exchange REQUEST
Payload contents:
ENCR
004092: .Mar 5 15:32:03.617: IKEv2:(SA ID = 1):Packet is a retransmission
004093: .Mar 5 15:32:03.617: IKEv2:Packet is a retransmission
004094: .Mar 5 15:32:03.617: IKEv2:
004095: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Received Packet [From HUB1:4500/To SPOKE1:4500/VRF i0:f0]
Initiator SPI : EECCD9476A6301DE - Responder SPI : E0D14C006A000813 Message id: 1
IKEv2 IKE_AUTH Exchange RESPONSE
Payload contents:
NOTIFY(AUTHEN
SPOKE1(config)#TICATION_FAILED)
004096: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Process auth response notify
004097: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):
004098: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Auth exchange failed
004099: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Auth exchange failed
004100: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Auth exchange failed
004101: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Abort exchange
004102: .Mar 5 15:32:04.337: IKEv2:(SA ID = 1):Deleting SA
