CSR VRF-aware VPN problem

[Tobias Moritz]

Hello community,

I'm currently evaluating CSR at AWS (60-day trial) and already got around the usual problems and architectural speciality of AWS network design.

I cannot open a TAC case, because we did not buy licence yet. We will, after this last issue is fixed.

Current setup:

• Two CSRs in one VPC in two AZs
• Transit between both CSRs via GRE tunnel
• running VRF-aware BGP
• using front-door VRF
• both CSRs are connected to multiple AWS VPCs (customers) via AWS VPN feature - fully meshed route based VPNs - one VRF per customer - all running with BGP
• Link to onPremise is done the same way: fully meshed route based VPNs - using front-door VRF - all running with BGP
• VRF import/export rules

This is working fine - no issues here. All HA tests run like expected. So far, everything is fine.

We now needed to create a VPN connection to a special onPremise location of our company. We needed to create a policy based VPN to that location (no support for route based VPN there). This is a two-to-one VPN. Two CSRs connecting to one onPrem gateway. Both tunnels run the same encryption domain. Routing onPrem is based on tunnel state. We put this tunnel in the front-door VRF. Routes are injected to front-door VRF routing table by VPN process (reverse-route static in crypto map). To get these routes exported to customer VRFs, there is a matching network statement in BGP process of front-door VRF.

Well, this is also working fine if we do this only with CSR A. Reachablity is end-to-end. CSR B is routing to CSR because of working VRF-aware VPN. However, if we establish the second tunnel from CSR B, there is something strange happening.

Tunnel is established fine. Traffic received through this tunnel at CSR B is accepted and routed to destination. Traffic created at front-door VRF on CSR B is routed into its own VPN fine. However, traffic originating from a customer VRF which is reaching CSR B (traceroute proved this) is not routed into VPN tunnel, despite routing table of customer VPN is saying so. CSR A is running the same config, there is no problem. Only on CSR B.

I do not understand this. If we remove tunnel configuration from CSR A and only create tunnel on CSR B, it is still not working. I don't understand why, because I did a config compare and found no difference.

Has anyone any idea, whats going on?

How can I debug this problem?

CSR-A:

B        172.29.13.176/28 [20/0] via 1.1.119.182 (vrf-default), 3w4d

CSR-B:

with route (not working for this customer VRF)
B        172.29.13.176/28 [20/0] via 1.1.119.182 (vrf-default), 00:00:02

without route (working, because routed over transit to CSR-A)
B        172.29.13.176/28 [20/0] via 192.168.254.53 (vrf-default), 00:38:23

This problem is hard to describe, I would really appriciate discussing this with a TAC engineer in a WebEx. Is this possible?

Thank you.

[tbanuelo · Accepted solution]

Hello Tobias,

The issue you are describing is going to be outside our CSR platform expertise. It sounds like CSR is running fine and HA is also running fine, and now you are trying find a solution to a networking/VPN issue you are facing.

Our team will need to find a resource internally to address your issue, please allow us a day or so to get back to you with an answer

Regards

Tony

[Tobias Moritz]

If anyone is else is reading this:
The problem was solved by TAC.

Root cause:

The old crypto map feature of IOS (ported to IOS-XE) is not able to handle such VRF infrastructure. Cisco is working on the retirement of crypto maps and provide policy based VPN support at VTI (just put an crypto-ACL on a VTI). No roadmap with release dates yet.

Workaround:

Use VASI. Beside the bad name (vrf aware software instructure), it adds logical interface of type vasileft and vasiright which just do the same, you could do with cables in the old days: just bridge vrfs and create a transit network as overlay. Put the policy based vpn in its own vrf, create vasi links between this vrf and all others, add transit routes over these vasi links, and everything should work. One last thing I which took me another TAC session to figure out: crypto map MUST be applied to physical egress interface (should be fvrf), regardless of ivrf. So far, so normal. But you have to create a route in ivrf which routes the policy based vpn peer gw address to fvrf egress interface - otherwise ivrf does no even try to put the traffic into encryption domain.

Config stub:

vrf definition vrf-amcs

 address-family ipv4

 exit-address-family

crypto isakmp profile isakmp-vpn-amcs

   vrf vrf-amcs

   keyring keyring-vpn-vrf-default

   match identity address 1.1.1.182 255.255.255.255 vrf-default

   local-address GigabitEthernet1

crypto ipsec transform-set ipsec-prop-vpn-amcs esp-aes 256 esp-sha-hmac

 mode tunnel

crypto map cmap-vpn-amcs 10 ipsec-isakmp

 set peer 1.1.1.182

 set security-association lifetime seconds 21600

 set transform-set ipsec-prop-vpn-amcs

 set pfs group2

 set reverse-route distance 10

 set isakmp-profile isakmp-vpn-amcs

 match address vpn-to-amcs

 reverse-route static

ip access-list extended vpn-to-amcs

 permit ip 172.23.0.0 0.0.255.255 172.29.13.176 0.0.0.15

interface GigabitEthernet1

 vrf forwarding vrf-default

 ip address dhcp

 negotiation auto

 crypto map cmap-vpn-amcs

interface vasileft10

 vrf forwarding vrf-10

 ip address 192.168.255.10 255.255.255.254

interface vasiright10

 vrf forwarding vrf-amcs

 ip address 192.168.255.11 255.255.255.254

 no keepalive

ip route vrf vrf-10 172.29.13.176 255.255.255.240 192.168.255.11

ip route vrf vrf-default 172.29.13.176 255.255.255.240 192.168.254.53 20

ip route vrf vrf-amcs 172.23.2.0 255.255.255.0 192.168.255.10

[jkapadnis]

I am running into similar issue. You mentioned that "Use VASI. Beside the bad name (vrf aware software infrastructure)" Is this option enabled by default or do you have to enable it on Cisco 1000v?

Here are the relevant error logs. Cisco CSR 1000v with Crypto Map in VRF isn't matching the Crypto ACL to complete IPsec Phase 1. 

*Apr 10 18:25:42.935: ISAKMP-ERROR: (1021):Expected csr-to-asa profile doesn't match, aborting exchange
*Apr 10 18:25:42.936: ISAKMP-ERROR: (1021):(1021): FSM action returned error: 2
*Apr 10 18:25:42.936: ISAKMP: (1021):Input = IKE_MESG_FROM_PEER, IKE_MM_EXCH
*Apr 10 18:25:42.936: ISAKMP: (1021):Old State = IKE_I_MM5 New State = IKE_I_MM6

I have the exact configuration that you have applied. I also referred VRF cheat sheet still no luck:

https://supportforums.cisco.com/document/51931/vrf-aware-ipsec-cheat-sheet

I have attached the running-config as well. I would really appreciate, if you could share more details from Cisco TAC on how to fix this issue. 

[Tobias Moritz]

There is no need to activate this VASI feature on IOS-XE (like on Nexus platforms with the feature command). Just make sure you have a recent version to avoid hitting bug CSCva27661 (or use ax-licence).

I've checked your running-config, there is not a single vasi interface yet. Please give it a try and create a pair (see my sample config above and don't forget the transit routes).

[jkapadnis]

Hello Tobias,

Thanks for checking the configuration. I saw your vasi interface configuration and added the same to my configuration. Unfortunately, even after creating vasi interface and adding static routes, I am unable bring UP the VPN connection. 

Remote 10.10.0.0/16
Local (CiscoCSRv) 10.8.0.0/16  --> I am configuring vasi interface here. 

I am trying to reach remote network 10.10.0.0/16

Could you please check the config below and let me know if I missed something?

ip vrf default

description Internet VRF

rd 64512:100

!

ip vrf temp

rd 64512:101

!

ip vrf vasi-vrf

description vasi vrf

!

ip vrf vpn-ea76668b

rd 64512:1

route-target export 64512:0

route-target import 64512:0

!

ip vrf vpn-ec76668d

rd 64512:3

route-target export 64512:0

route-target import 64512:0

!

ip vrf vpn0

rd 64512:0

!

!

!

!crypto keyring internet-keyring

  local-address GigabitEthernet1

  pre-shared-key address <IP>

!

!

!

!

!

crypto isakmp policy 200

encr aes

authentication pre-share

group 2

lifetime 28800

crypto isakmp profile csr-to-asa

   vrf vpn-ea76668b

   keyring internet-keyring

   match identity address  <IP>

   local-address GigabitEthernet1

!

crypto ipsec security-association replay window-size 1024

!

crypto ipsec transform-set ipsec-prop-vpn-aws esp-aes esp-sha-hmac

mode tunnel

crypto ipsec transform-set test esp-aes esp-sha-hmac

mode tunnel

crypto ipsec df-bit clear

!

!

crypto ipsec profile ipsec-vpn-aws

set transform-set ipsec-prop-vpn-aws

set pfs group2

!

!

crypto map csr-to-asa 10 ipsec-isakmp

set peer  <IP>

set transform-set test

set pfs group2

set isakmp-profile csr-to-asa

match address csr-to-asa

reverse-route static

!

interface Loopback1

ip vrf forwarding vpn-ea76668b

ip address 10.8.0.1 255.255.255.255

!

ip route 10.8.0.0 255.255.0.0 192.168.255.11

ip route vrf vpn-ea76668b 10.10.0.0 255.255.0.0 192.168.255.10

ip access-list extended csr-to-asa

permit ip 10.8.0.0 0.0.255.255 10.10.0.0 0.0.255.255 log

permit ip 10.7.0.0 0.0.255.255 10.10.0.0 0.0.255.255 log

!

!

!

interface GigabitEthernet1

ip address dhcp

negotiation auto

crypto map csr-to-asa

!

interface GigabitEthernet2

ip vrf forwarding temp

ip address dhcp

negotiation auto

!

interface vasileft1

ip address 192.168.255.10 255.255.255.254

no keepalive

!

interface vasiright1

ip vrf forwarding vpn-ea76668b

ip address 192.168.255.11 255.255.255.254

no keepalive

!