Re: vrf-aware ipsec not working

jni · ‎10-23-2017

Hi all

Would any one be so kind to look at my config and tell me what I am missing. I am to configure several site-to-site VPN. Since the local subnets can be overlapping, I want to use vrf-aware IPsec towards different customers. If I however try to initiate some traffic from local subnet, I don't see any initiation of IPsec tunnel at all. I've tried from two different local subnets (and hence two different customers) with same result.

Some of the config added below

vrf definition vrf-1113
 rd 1.1.1.2:1113
 !
 address-family ipv4
 exit-address-family
!
vrf definition internet-vrf
 rd 1.1.1.2:218
 !
 address-family ipv4
 exit-address-family
!
!
track 1 interface GigabitEthernet0/0/0 line-protocol
track 2 interface GigabitEthernet0/0/1 line-protocol
!
!
crypto keyring cust-1113 vrf internet-vrf
  pre-shared-key address 2.2.2.2 key PSK
!
!
crypto isakmp policy 10
 encr aes 256
 authentication pre-share
 group 5
!
crypto isakmp profile cust-1113
   vrf vrf-1113
   keyring cust-1113
   match identity address 2.2.2.2 255.255.255.255 internet-vrf
!
!
crypto ipsec transform-set aes256-sha esp-aes 256 esp-sha-hmac
 mode tunnel
!
!
crypto map vpn 13 ipsec-isakmp
 set peer 2.2.2.2
 set transform-set aes256-sha
 set isakmp-profile cust-1113
 match address cust-1113
!
!
interface GigabitEthernet0/0/0.218
 encapsulation dot1Q 218
 vrf forwarding internet-vrf
 ip address 1.1.1.2 255.255.255.248
 no ip redirects
 standby version 2
 standby 1 ip 1.1.1.1
 standby 1 priority 105
 standby 1 preempt
 standby 1 authentication somepass
 standby 1 name vpn-peer
 standby 1 track 2 decrement 20
 crypto map vpn redundancy
!
interface GigabitEthernet0/0/1.1113
 encapsulation dot1Q 1113
 vrf forwarding vrf-1113
 ip address 10.4.100.3 255.255.255.240
 no ip redirects
 standby version 2
 standby 113 ip 10.4.100.1
 standby 113 follow HSRP-master-IPv4 (HSRP-master is on another interface)
 standby 113 priority 105 (not nescessary because of the follow)
 standby 113 preempt
 standby 113 authentication somepass
!
!
router bgp 65000
 address-family ipv4 vrf vrf-1113
  redistribute connected
 exit-address-family
 !
ip access-list extended cust-1113
 permit ip 10.4.100.0 0.0.0.15 any
!
ip route vrf vrf-1113 0.0.0.0 0.0.0.0 GigabitEthernet0/0/0.218 1.1.1.6 (not quite sure about next hop - some documentation specifies this as the router next-hop (in this case know through BGP) other the peer address)

I would really appreciate some help or feedback.

Note I have been looking at some documentation, but I do not know what I am doing wrong

https://supportforums.cisco.com/t5/security-documents/vrf-aware-ipsec-cheat-sheet/ta-p/3109449
https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/sec_conn_ikevpn/configuration/xe-3s/sec-ike-for-ipsec-vpns-xe-3s-book/sec-vrf-aware-ipsec.pdf
http://www.networking-forum.com/viewtopic.php?f=33&t=47453

Thank you :o)

jni · ‎10-23-2017

I might add, that there is connection between inside net and the router (arp, ping) and traffic to/from other subnets through IPv6. But If I try to do any connection through IPv4 which should trigger interesting traffic, nothing happens at all.

I have debug crypto isakmp/ipsec running, but no results. A tcpdump on a server on the local subnet looks fine (except no response :)

13:40:42.502583 IP 10.4.100.5 > 8.8.8.8: ICMP echo request, id 23274, seq 1, length 64
13:41:11.751154 IP 10.4.100.5.57416 > 2.2.3.3.domain: [|domain]
13:41:12.751280 IP 10.4.100.5.57416 > 2.2.3.3.domain: [|domain]

jni · ‎10-23-2017

No one? :o)

I would really appreciate it.

Rahul Govindan · ‎10-23-2017

Can you paste the "show ip route vrf vrf-1113" and "show ip route internet-vrf"? I did not see a static route/bgp configuration on the internet-vrf pointing to the outside, but I am assuming that is already there.

Also, if this is a test setup with no other traffic, try running the "debug ip packet detail" command. You may want to add an permit ACL with a "log" keyword on the LAN to process switch traffic to see it in the debug.

jni · ‎10-23-2017

Thank you very much for your answer.

I've tried to add an access-list on the inbound interface with a log statement. I do see the traffic, but still the crypto isn't triggered

Oct 24 08:45:01: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.6(48482) -> x.x.x.x(514), 1 packet
Oct 24 08:45:01: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(59767) -> x.x.x.x(514), 1 packet
Oct 24 08:45:42: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(47942) -> 10.1.0.103(53), 2 packets

The ip routes:

sh ip route vrf vrf-1113

Routing Table: vrf-1113
Gateway of last resort is 1.1.1.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.1.6, GigabitEthernet0/0/0.218
      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.4.100.0/28 is directly connected, GigabitEthernet0/0/1.1113
L        10.4.100.3/32 is directly connected, GigabitEthernet0/0/1.1113

*****

sh ip route vrf internet-vrf

Routing Table: internet-vrf
Gateway of last resort is 1.1.1.6 to network 0.0.0.0

B*    0.0.0.0/0 [200/0] via 1.1.1.6, 17:53:52
      1.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
B        z.z.z.z/28 [200/0] via 1.1.1.6, 17:53:52		(some "upstream" public IP)
B        y.y.y.y/29 [200/0] via 1.1.1.6, 17:53:52		(some "upstream" public IP)
B        x.x.x.x/29 [200/0] via 1.1.1.6, 17:53:52		(some "upstream" public IP)
C        1.1.1.0/29 is directly connected, GigabitEthernet0/0/0.218
L        1.1.1.2/32 is directly connected, GigabitEthernet0/0/0.218
      172.21.0.0/26 is subnetted, 1 subnets
B        172.21.8.0 [200/0] via 1.1.1.6, 17:53:52

A debug ip packet (and a lot of debug crypto like isak/ipsec/engine/packet etc), doesn't really give any useful (to me) output

Oct 24 08:49:55  58504: Oct 24 06:49:51.889: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:49:55  58505: Oct 24 06:49:51.889: IP: s=10.4.100.2 (local), d=224.0.0.102
Oct 24 08:50:06  62081: Oct 24 06:50:00.674: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:50:06  62082: Oct 24 06:50:00.674: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, sending broad/multicast
Oct 24 08:50:06  63766: Oct 24 08:50:04: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(36391) -> 10.1.0.103(53), 1 packet
Oct 24 08:50:14  65616: Oct 24 06:50:09.577: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:50:14  65617: Oct 24 06:50:09.577: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, sending broad/multicast
Oct 24 08:50:22  66743: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.6(48482) -> x.x.x.x(514), 3 packets
Oct 24 08:50:22  66745: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.4(47684) -> x.x.x.x(514), 4 packets
Oct 24 08:50:22  66741: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(59767) -> x.x.x.x(514), 3 packets
Oct 24 08:50:22  69053: Oct 24 06:50:18.229: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:50:22  69054: Oct 24 06:50:18.229: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, sending broad/multicast

So no matter what I do it doesn't seem to trigger the crypto (sh cry isak sa, sh cry ipsec sa peer x.x.x etc).

jni · ‎10-24-2017

Thank you for your reply :o)

The ip routes for the two vrf

sh ip route vrf vrf-1113

Routing Table: vrf-1113
Gateway of last resort is 1.1.1.6 to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via 1.1.1.6, GigabitEthernet0/0/0.218
      10.0.0.0/8 is variably subnetted, 2 subnets, 2 masks
C        10.4.100.0/28 is directly connected, GigabitEthernet0/0/1.1113
L        10.4.100.3/32 is directly connected, GigabitEthernet0/0/1.1113

*****

sh ip route vrf internet-vrf

Routing Table: internet-vrf
Gateway of last resort is 1.1.1.6 to network 0.0.0.0

B*    0.0.0.0/0 [200/0] via 1.1.1.6, 17:53:52
      1.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
B        z.z.z.z/28 [200/0] via 1.1.1.6, 17:53:52		(some "upstream" public IP)
B        y.y.y.y/29 [200/0] via 1.1.1.6, 17:53:52		(some "upstream" public IP)
B        x.x.x.x/29 [200/0] via 1.1.1.6, 17:53:52		(some "upstream" public IP)
C        1.1.1.0/29 is directly connected, GigabitEthernet0/0/0.218
L        1.1.1.2/32 is directly connected, GigabitEthernet0/0/0.218
      172.21.0.0/26 is subnetted, 1 subnets
B        172.21.8.0 [200/0] via 1.1.1.6, 17:53:52

If I added a test-in access-list to the inside interface with "permit ip any any log". I do see the traffic permitted/entering the interface

Oct 24 08:50:06  63766: Oct 24 08:50:04: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(36391) -> 10.1.0.103(53), 1 packet
Oct 24 08:50:22  66743: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.6(48482) -> x.x.x.x(514), 3 packets
Oct 24 08:50:22  66745: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.4(47684) -> x.x.x.x(514), 4 packets
Oct 24 08:50:22  66741: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(59767) -> x.x.x.x(514), 3 packets

I've also tried to do a debug ip packet (and other debugs as e.g. debug crypto isak/ipsec/engine/packets etc), but not much information there.

Oct 24 08:49:55  58504: Oct 24 06:49:51.889: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:49:55  58505: Oct 24 06:49:51.889: IP: s=10.4.100.2 (local), d=224.0.0.102
Oct 24 08:50:06  62081: Oct 24 06:50:00.674: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:50:06  62082: Oct 24 06:50:00.674: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, sending broad/multicast
Oct 24 08:50:06  63766: Oct 24 08:50:04: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(36391) -> 10.1.0.103(53), 1 packet
Oct 24 08:50:14  65616: Oct 24 06:50:09.577: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:50:14  65617: Oct 24 06:50:09.577: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, sending broad/multicast
Oct 24 08:50:22  66743: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.6(48482) -> x.x.x.x(514), 3 packets
Oct 24 08:50:22  66745: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.4(47684) -> x.x.x.x(514), 4 packets
Oct 24 08:50:22  66741: Oct 24 08:50:12: %FMANFP-6-IPACCESSLOGP:fman_fp_image:  list test-in permitted udp 10.4.100.5(59767) -> x.x.x.x(514), 3 packets
Oct 24 08:50:22  69053: Oct 24 06:50:18.229: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, local feature, feature skipped, Auth Proxy(16), rtype 0, forus FALSE, sendself FALSE, mtu 0, fwdchk FALSE
Oct 24 08:50:22  69054: Oct 24 06:50:18.229: IP: s=10.4.100.2 (local), d=224.0.0.102 (GigabitEthernet0/0/1.1113), len 80, sending broad/multicast

To me it looks as the packets are arriving at the interface but for some reason the packets aren't being encrypted and forwarded through the tunnel. No matter what I do I don't see any attempt to create a tunnel (not even unsuccessful). As said before, even with e.g. debug crypto isakmp, I don't see any tunnel establishment attempts.

I don't think it is of relevance, but router is a ISR4431 and running 15.5(3)S5 / 03.16.05.S

Rahul Govindan · ‎10-24-2017

Your routes look correct when I glanced through them. What is strange is that there are no crypto initiation debugs on the box. Can you remove the "redundancy" keyword from the crypto map and test? Usually redundancy is coupled with the HSRP standby group name.

jni · ‎10-24-2017

Yes the redundancy is connected with the standby 1 name vpn-peer (and it didn't help to remove it).

I have a similar problem like this:

https://supportforums.cisco.com/t5/vpn/troubles-using-vrf-aware-ipsec-w-crypto-maps/td-p/2286724

by adding the "reverse-route remote-peer x.x.x.x gateway static" for each customer, I do see some routes when doing a "show crypto route"

crypto map vpn 11 ipsec-isakmp
 set peer x.x.x.x
 set transform-set aes256-sha
 set isakmp-profile cust1
 match address cust1
 reverse-route remote-peer x.x.x.x gateway static
crypto map vpn 12 ipsec-isakmp
 set peer y.y.y.y
 set transform-set aes256-sha
 set pfs group5
 set isakmp-profile cust2
 match address cust2
 reverse-route remote-peer y.y.y.y gateway static


sh crypto route
Routes created in table internet-vrf
x.x.x.x/255.255.255.255 [1/0] via x.x.x.x tag 0 count 1 rtid 23
 on RRI S
Routes created in table internet-vrf
y.y.y.y/255.255.255.255 [1/0] via y.y.y.y tag 0 count 1 rtid 27
 on RRI S

But I don't see any default routes when doing a "sh ip route vrf vrf-1113"

If I add a "ip route vrf vrf-1113 0.0.0.0 0.0.0.0 GigabitEthernet0/0/0.218 x.x.x.x" I can see a default route in the vrf, but still no crypto triggered

sh ip route vrf vrf-1113

Gateway of last resort is x.x.x.x to network 0.0.0.0

S*    0.0.0.0/0 [1/0] via x.x.x.x, GigabitEthernet0/0/0.218
      10.0.0.0/8 is variably subnetted, 3 subnets, 3 masks
S        10.1.0.0/16 [1/0] via x.x.x.x, GigabitEthernet0/0/0.218
C        10.4.100.0/28 is directly connected, GigabitEthernet0/0/1.1113
L        10.4.100.2/32 is directly connected, GigabitEthernet0/0/1.1113

I wonder if it has anything to do with my interesting-traffic / encryption domain ACLs have the any keyword in them (each environment must send all their traffic through the tunnel so it can exit through customer network/IPs)

ip access-list extended cust-1113
 permit ip 10.1.0.16 0.0.0.15 any
ip access-list extended cust-1114
 permit ip 10.133.5.0 0.0.0.15 any

Rahul Govindan · ‎10-26-2017

Your ACL should be matched, even though it has an "any" in the destination. Only have "any any" might create a problem, not your ACL. Another aspect I want to check is NAT. Any chance that you have NAT in your config that could be changing the source of the traffic going to the outside?

jni · ‎11-01-2017

Hi again

Just for your information, it is actually working now. I can't pinpoint what exactly was what made it work, but I think the debugging was made difficult because it wasn't possible to initiate the tunnel from my end right after a "clear cry isak sa" or "clear cry ipsec sa pee x.x.x.x". I think it is due to the HSRP setup and some timeouts. So whenever I cleared the tunnel and tried to initiate again, it would not work. I then configured some DPD which did help, so now I can initiate traffic through the tunnel.

Thank you for your effort :o)