Problem with a S2S (IOS to Azure) over VRF

[Drake22x]

I have this problem trying to set up a S2S VPN between an ASR and Azure environment. My topology is as follows:

server --- (vrf A) switch (vrf A) --- (vrf A) router (vrf Internet) --- azure

I managed to get the session up and I see a few bytes passing through it, but I cannot ping from the server to the host on Azure subnet or vice versa. I suspect I got some of the ivrf/fvrf settings wrong but can't seem to pinpoint the problem and hope somebody can help.

[Drake22x]

I have the following configuration:

Router:

crypto ikev2 proposal XX-PROP-AZURE
encryption aes-cbc-256 aes-cbc-128 3des
integrity sha1
group 2
!
crypto ikev2 policy XX-POLICY-AZURE
proposal XX-PROP-AZURE
!
crypto ikev2 keyring KEY-AZURE
peer 13.80.x.x
address 13.80.x.x
pre-shared-key 6 xxxx
!
!
!
crypto ikev2 profile PROF-PH1-AZURE
match address local interface GigabitEthernet0/0/0.422
match identity remote address 13.80.x.x 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local KEYR-AZURE

crypto ipsec transform-set XFORM-AZURE esp-aes 256 esp-sha-hmac
mode tunnel

crypto ipsec profile PROF-PH2-AZURE
set transform-set XFORM-AZURE
set ikev2-profile PROF-PH1-AZURE

interface Tunnel11
ip vrf forwarding CUST
ip address 169.254.0.1 255.255.255.0
ip tcp adjust-mss 1350
tunnel source GigabitEthernet0/0/0.422
tunnel mode ipsec ipv4
tunnel destination 13.80.x.x
tunnel protection ipsec profile PROF-PH2-AZURE

interface GigabitEthernet0/0/0.422
encapsulation dot1Q 422
ip vrf forwarding INTERNET
ip address 85.x.x.x 255.255.255.240

crypto map CM (carries other customer maps)

ip route vrf CUST 192.168.x.0 255.255.255.0 Tunnel11

ASR#sh crypto ikev2 sa
IPv4 Crypto IKEv2 SA

Tunnel-id Local Remote fvrf/ivrf Status
1 85.x.x.x/500 13.80.x.x/500 none/CUST READY
Encr: AES-CBC, keysize: 256, PRF: SHA1, Hash: SHA96, DH Grp:2, Auth sign: PSK, Auth verify: PSK
Life/Active Time: 86400/767 sec

IPv6 Crypto IKEv2 SA

ASR#sh crypto session interface tunnel11
Crypto session current status

Interface: Tunnel11
Profile: PROF-PH1-AZURE
Session status: UP-ACTIVE
Peer: 13.80.x.x port 500
Session ID: 15868
IKEv2 SA: local 85.x.x.x/500 remote 13.80.x.x/500 Active
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 2, origin: crypto map

ASR#debug crypto ikev2
IKEv2 default debugging is on

*May 15 02:29:11.952: IKEv2:(SESSION ID = 15868,SA ID = 1):Received Packet [From 13.80.x.x:500/To 85.x.x.x:500/VRF i0:f0]
Initiator SPI : AB9FF02644D349E1 - Responder SPI : 4911694719AF0278 Message id: 443
IKEv2 INFORMATIONAL Exchange REQUEST
Payload contents:

*May 15 02:29:11.952: IKEv2:(SESSION ID = 15868,SA ID = 1):Received DPD/liveness query
*May 15 02:29:11.952: IKEv2:(SESSION ID = 15868,SA ID = 1):Building packet for encryption.
*May 15 02:29:11.953: IKEv2:(SESSION ID = 15868,SA ID = 1):Sending ACK to informational exchange

*May 15 02:29:11.953: IKEv2:(SESSION ID = 15868,SA ID = 1):
CSQDC1-R1#Sending Packet [To 13.80.x.x:500/From 85.x.x.x:500/VRF i0:f0]
Initiator SPI : AB9FF02644D349E1 - Responder SPI : 4911694719AF0278 Message id: 443
IKEv2 INFORMATIONAL Exchange RESPONSE
Payload contents:
ENCR

If I put Tu11 in tunnel vrf INTERNET, which I thought I should do, the session doesn't come up. Not sure if using APIPA address for the tunnel is correct also. Totally lost.

[Hitesh Vinzoda]

Please add following statement

interface Tunnel11

tunnel vrf INTERNET

HTH

Hitesh

[Drake22x]

Thanks Hitesh, I had tried that before, but then the session goes completely down:

Interface: Tunnel11
Session status: DOWN
Peer: 13.80.x.x port 500
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map

Tunnel11 169.254.0.1 YES manual up down

[Hitesh Vinzoda]

Your tunnel should be sourced from vrf INTERNET.

Can you paste some debug of isakmp and ipsec

Thanks

Hitesh

[Drake22x]

Sure:

*May 16 02:40:20.849: IKEv2:Received Packet [From 13.80.x.x:500/To 85.13.x.x:500/VRF i0:f8]
Initiator SPI : DC907120570444DB - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP) VID VID VID VID

*May 16 02:40:20.849: IKEv2:(SESSION ID = 16851,SA ID = 1):Verify SA init message
*May 16 02:40:20.849: IKEv2:(SESSION ID = 16851,SA ID = 1):Insert SA
*May 16 02:40:20.849: IKEv2:Searching Policy with fvrf 8, local address 85.13.x.x
*May 16 02:40:20.850: IKEv2:No Matching policy with fvrf 8, local addr 85.13.x.x
CSQDC1-R1#
*May 16 02:40:20.851: IKEv2:(SESSION ID = 16851,SA ID = 1):: Failed to locate an item in the database
*May 16 02:40:20.851: IKEv2:(SESSION ID = 16851,SA ID = 1):Failed SA init exchange
*May 16 02:40:20.851: IKEv2:(SESSION ID = 16851,SA ID = 1):Initial exchange failed: Initial exchange failed
*May 16 02:40:20.851: IKEv2:(SESSION ID = 16851,SA ID = 1):Abort exchange
*May 16 02:40:20.851: IKEv2:(SESSION ID = 16851,SA ID = 1):Deleting SA

[Mohammed al Baqari]

You are missing this. 

crypto ikev2 profile PROF-PH1-AZURE

ivrf CUST

!

interface Tunnel11

tunnel vrf INTERNET

[Drake22x]

Thanks Mohammed. I had tried that as well to no success :(

crypto ikev2 profile PROF-PH1-AZURE
match address local interface GigabitEthernet0/0/0.422
match identity remote address 13.80.x.x 255.255.255.255
authentication remote pre-share
authentication local pre-share
keyring local KEY-AZURE
ivrf CUST

Interface: Tunnel11
Session status: DOWN
Peer: 13.80.x.x port 500
IPSEC FLOW: permit ip 0.0.0.0/0.0.0.0 0.0.0.0/0.0.0.0
Active SAs: 0, origin: crypto map

Is my route correct?

ip route vrf CUST 192.168.x.0 255.255.255.0 Tunnel11

Also, is there a way to specify vrf in the crypto ikev2 keyring command, like you have in a crypto keyring "xx" vrf "yy"?

[Mohammed al Baqari]

Please share the debugs after applying the commands

[Drake22x]

The previous debug was after applying both amendments:

*May 16 03:53:13.709: IKEv2:% Getting preshared key from profile keyring KEY-AZURE
*May 16 03:53:13.709: IKEv2:% Matched peer block '13.80.x.x'
*May 16 03:53:13.709: IKEv2:Searching Policy with fvrf 8, local address 85.13.x.x
*May 16 03:53:13.709: IKEv2:No Matching policy with fvrf 8, local addr 85.13.x.x
*May 16 03:53:13.710: IKEv2:Failed to initiate sa

*May 16 03:53:21.066: IKEv2:Received Packet [From 13.80.x.x:500/To 85.13.x.x:500/VRF i0:f8]
Initiator SPI : 0EBE8E28574B5C20 - Responder SPI : 0000000000000000 Message id: 0
IKEv2 IKE_SA_INIT Exchange REQUEST
Payload contents:
SA KE N NOTIFY(NAT_DETECTION_SOURCE_IP) NOTIFY(NAT_DETECTION_DESTINATION_IP) VID VID VID VID

*May 16 03:53:21.066: IKEv2:(SESSION ID = 17120,SA ID = 1):Verify SA init message
*May 16 03:53:21.067: IKEv2:(SESSION ID = 17120,SA ID = 1):Insert SA
*May 16 03:53:21.067: IKEv2:Searching Policy with fvrf 8, local address 85.13.x.x
*May 16 03:53:21.067: IKEv2:No Matching policy with fvrf 8, local addr 85.13.x.x
CSQDC1-R1#
*May 16 03:53:21.068: IKEv2:(SESSION ID = 17120,SA ID = 1):: Failed to locate an item in the database
*May 16 03:53:21.068: IKEv2:(SESSION ID = 17120,SA ID = 1):Failed SA init exchange
*May 16 03:53:21.068: IKEv2:(SESSION ID = 17120,SA ID = 1):Initial exchange failed: Initial exchange failed
*May 16 03:53:21.068: IKEv2:(SESSION ID = 17120,SA ID = 1):Abort exchange
*May 16 03:53:21.068: IKEv2:(SESSION ID = 17120,SA ID = 1):Deleting SA

Thank you for taking the time to help me 

[Mohammed al Baqari]

Add this as well and share debugs if not working.

crypto ikev2 policy XX-POLICY-AZURE

fvrf INTERNET

[Drake22x]

Added:

crypto ikev2 policy XX-POLICY-AZURE
match fvrf INTERNET
proposal XX-PROP-AZURE

*May 16 04:08:20.310: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=85.13.x.x, prot=50, spi=0xC15A5946(3243923782), srcaddr=13.80.x.x, input interface=GigabitEthernet0/0/0.422

*May 16 04:09:34.567: IKEv2:(SESSION ID = 0,SA ID = 0):Received Packet [From 13.80.x.x:500/To 85.13.x.x:500/VRF i0:f8]
Initiator SPI : 069C7181362762EC - Responder SPI : 782AEAEEA22ACC40 Message id: 0
IKEv2 INFORMATIONAL Exchange REQUEST
*May 16 04:09:34.567: IKEv2:: A supplied parameter is incorrect

Tu11 up/down still

[Mohammed al Baqari]

Can you make sure that proposals are matching. This is mismatch error. 

[Drake22x]

Azure side is very limited in the configuration, so I had used acceptable settings according to Azure from here: https://github.com/Azure/Azure-vpn-config-samples/blob/master/Cisco/Current/ASR/Site-to-Site_VPN_using_Cisco_ASR.md

Is it a matter of trying different parameters until one works?

[Drake22x]

Additionally, this error comes up every time:

*May 16 06:36:23.721: %CRYPTO-4-RECVD_PKT_INV_SPI: decaps: rec'd IPSEC packet has invalid spi for destaddr=85.13.x.x, prot=50, spi=0x36BF9268(918524520), srcaddr=13.80.x.x, input interface=GigabitEthernet0/0/0.422

Article here - http://www.cisco.com/c/en/us/support/docs/security-vpn/ipsec-negotiation-ike-protocols/115801-technote-iosvpn-00.html - suggests to enable crypto isakmp invalid-spi-recovery, but it doesn't seem to apply to ikev2.

Also:

This is an indication that traffic is black-holed and might not recover until the SAs expire on the sending device or until the Dead Peer Detection (DPD) is activated

One of the guides shows the dpd command here, but I cannot seem to have it available on my IOS :(

crypto ikev2 policy azure-policy
  proposal azure-proposal
  dpd 10 30
  exit