IPSEC tunnel with VRF

[sukesh tandon]

Hi Folks,

I see that there are 4 tunnels build with different sources (loopbacks) but to the same destinations.

Second thing is that tunnel interfaces are part of vrf but loopack interfaces are not.

Third, tunnel protection ipsec profile is used under tunnel interface.

Below is configuration of one of the tunnels, The tunnel IP addresses are different and the sources are different.

STPA222#sh run int tu 880
Building configuration...

Current configuration : 336 bytes
!
interface Tunnel880
bandwidth 500000
ip vrf forwarding NWE-ID
ip address 172.17.89.102 255.255.255.254
logging event link-status
tunnel source loopback10
tunnel mode ipsec ipv4
tunnel destination 135.13.15.18
tunnel protection ipsec profile NWETEST
end

Now my questions is 

1) How does traffic decide, which tunnel to choose when the all the sources are loopbacks and to the same destination  ?

2) What could be the reason behind having 4 tunnels ? Instead one could have also served the same purpose.

3) There is vrf routing to all the tunnel interfaces. Below is an example : 

ip route vrf NWE-ID 0.0.0.0 0.0.0.0 Tunnel880

4) How can i check what is the exit interface for all the tunnels. I am bit confused.

[Seb Rupik]

Hi there,

From the information you have provided, we can see there must be 5 VRFs involved (including the default routing table). Your loopbacks all reside within the default routing table, and so to must the interface which is connected to the internet.

To answer your questions:

1) All traffic destined to 0.0.0.0/0 (ie not locally routed) in any of the 4 named VRFs will leave via one of these tunnels. The output from sh ip route vrf NWE-ID will show you all connected subnets. Any host in one of these subnets will be routed via Tunnel880 if it needs to reach 0.0.0.0/0

2) You have four tunnels because you have four VRFs each requiring a default route. Since none of the named VRFs have an interface with a link to the internet, the tunnels are bound to a loopback in the default VRF. The default VRF does have a route to the internet so the encapsulated packets can reach the remote destination.

3) That routing statements simply says send all traffic destined to 0.0.0.0/0 via Tunnel880

4) The output from sh ip route will give you the default route table. Sh ip route 135.13.15.18 will show you the next hop.

I wrote a blog post about this setup which may help explain the purpose :

https://configif.wordpress.com/2017/12/21/tunnel-vrf/

cheers,

Seb.

[sukesh tandon]

STPA222#sh ip vrf NWE-ID

Name                                 Default RD                          Interfaces
NWE-ID <not set>
                                                                                   Tunnel850
                                                                                   Tunnel860
                                                                                   Tunnel870
                                                                                   Tunnel880
                                                                                    Po20.210                                                                              

STPA222# sh run | in ip route

ip route vrf NWE-ID 0.0.0.0 0.0.0.0 Tunnel850 track 310
ip route vrf NWE-ID 0.0.0.0 0.0.0.0 Tunnel860 track 320
ip route vrf NWE-ID 0.0.0.0 0.0.0.0 Tunnel870 track 330
ip route vrf NWE-ID 0.0.0.0 0.0.0.0 TTunnel880 track 340

STPA222#sh ip route vrf NWE-ID

Routing Table: NWE-ID
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
+ - replicated route, % - next hop override

Gateway of last resort is 0.0.0.0 to network 0.0.0.0

S* 0.0.0.0/0 is directly connected, Tunnel850
                    is directly connected, Tunnel860
                    is directly connected, Tunnel870
                    is directly connected, Tunnel880

Tunnel interface

STPA222#sh ip int brief | in up
Tunnel850 172.17.89.108 YES manual up up
Tunnel860 172.17.89.104 YES manual up up
Tunnel870 172.17.89.106 YES manual up up
Tunnel880 172.17.89.102 YES manual up up

I think i did not mention it correctly. There are total 4 tunnels. All under one vrf.

There is only default route that i see.

All the four tunnels have same destination but different sources.(loopbacks)

So what is the need to establish 4 tunnels ? 

Let me know if you need any other config.

[Seb Rupik]

The plot thickens!

So all fours tunnels are up but the route statement used depends on the track object.

What is the output of:

sh track

sh ip sla

...any chance of seeing the entire config?

cheers,

Seb.

[sukesh tandon]

IPSLAs Latest Operation Summary
Codes: * active, ^ inactive, ~ pending

ID Type Destination Stats Return Last
(ms) Code Run
-----------------------------------------------------------------------
*110 http 148.228.140.105 RTT=0 DNS query e 24 seconds ag
rror o



*120 http 148.228.148.86 RTT=0 DNS query e 47 seconds ag
rror o



*130 http 165.225.44.41 RTT=0 DNS query e 45 seconds ag
rror o



*140 http 216.58.206.68 RTT=0 DNS query e 43 seconds ag
rror o



*150 http 212.58.244.26 RTT=0 DNS query e 41 seconds ag
rror o



*210 http 148.228.140.105 RTT=0 DNS query e 40 seconds ag
rror o



*220 http 148.228.148.86 RTT=0 DNS query e 38 seconds ag
rror o



*230 http 165.225.44.41 RTT=0 DNS query e 36 seconds ag
rror o



*240 http 38.13.90.36 RTT=0 DNS query e 34 seconds ag
rror o



*250 http 212.58.246.94 RTT=0 DNS query e 32 seconds ag
rror o



*311 http 148.228.140.105 RTT=0 DNS query e 1 minute, 3 s
rror econds ago



*312 http 148.228.148.86 RTT=0 DNS query e 42 seconds ag
rror o



*313 http 158.108.68.67 RTT=0 DNS query e 41 seconds ag
rror o



*314 http 212.58.249.208 RTT=0 DNS query e 38 seconds ag
rror o



*315 http 38.13.90.36 RTT=0 DNS query e 36 seconds ag
rror o



*321 http 148.228.140.105 RTT=0 DNS query e 34 seconds ag
rror o



*322 http 148.228.148.86 RTT=0 DNS query e 31 seconds ag
rror o



*323 http 158.108.68.67 RTT=0 DNS query e 29 seconds ag
rror o



*324 http 212.58.249.215 RTT=0 DNS query e 27 seconds ag
rror o



*325 http 38.13.90.36 RTT=0 DNS query e 24 seconds ag
rror o



*331 http 148.228.140.105 RTT=0 DNS query e 23 seconds ag
rror o



*332 http 148.228.148.86 RTT=0 DNS query e 21 seconds ag
rror o



*333 http 158.108.68.67 RTT=0 DNS query e 19 seconds ag
rror o



*334 http 212.58.249.215 RTT=0 DNS query e 17 seconds ag
rror o



*335 http 38.13.90.36 RTT=0 DNS query e 16 seconds ag
rror o



*341 http 148.228.140.105 RTT=0 DNS query e 14 seconds ag
rror o



*342 http 148.228.148.86 RTT=0 DNS query e 12 seconds ag
rror o



*343 http 158.108.68.67 RTT=0 DNS query e 11 seconds ag
rror o



*344 http 212.58.244.66 RTT=0 DNS query e 9 seconds ago
rror



*345 http 157.240.8.35 RTT=0 DNS query e 1 minute, 7 s
rror econds ago



*411 http 148.228.140.105 RTT=66 OK 5 seconds ago




*412 http 148.228.148.86 RTT=67 OK 3 seconds ago




*413 http 158.108.37.67 RTT=32 OK 1 second ago




*414 http 212.58.249.208 RTT=192 OK 59 seconds ag
o



*415 http 38.13.64.35 RTT=64 OK 57 seconds ag
o



*421 http 148.228.140.105 RTT=85 OK 56 seconds ag
o



*422 http 148.228.148.86 RTT=81 OK 54 seconds ag
o



*423 http 158.108.37.67 RTT=34 OK 29 seconds ag
o



*424 http 212.58.249.211 RTT=181 OK 27 seconds ag
o



*425 http 38.13.64.35 RTT=51 OK 25 seconds ag
o



*431 http 148.228.140.105 RTT=84 OK 23 seconds ag
o



*432 http 148.228.148.86 RTT=144 OK 21 seconds ag
o



*433 http 158.108.37.67 RTT=35 OK 20 seconds ag
o



*434 http 212.58.249.213 RTT=193 OK 18 seconds ag
o



*435 http 38.13.64.35 RTT=64 OK 16 seconds ag
o



*441 http 148.228.140.105 RTT=66 OK 14 seconds ag
o



*442 http 148.228.148.86 RTT=60 OK 12 seconds ag
o



*443 http 158.108.37.67 RTT=32 OK 10 seconds ag
o



*444 http 212.58.249.213 RTT=182 OK 8 seconds ago




*445 http 38.13.64.35 RTT=63 OK 7 seconds ago

Track 310
List threshold percentage
Threshold Percentage is Down (0%)
487 changes, last change 3w5d
object 311 Down (0%)
object 312 Down (0%)
object 313 Down (0%)
object 314 Down (0%)
object 315 Down (0%)
Threshold percentage down 59% up 60%
Tracked by:
HSRP Port-channel20.210 1
HSRP Port-channel20.210 2
STATIC-IP-ROUTING 0

Track 320
List threshold percentage
Threshold Percentage is Down (0%)
571 changes, last change 3w5d
object 321 Down (0%)
object 322 Down (0%)
object 323 Down (0%)
object 324 Down (0%)
object 325 Down (0%)
Threshold percentage down 59% up 60%
Tracked by:
HSRP Port-channel20.210 1
HSRP Port-channel20.210 2
STATIC-IP-ROUTING 0

Track 330
List threshold percentage
Threshold Percentage is Down (0%)
533 changes, last change 3w5d
object 331 Down (0%)
object 332 Down (0%)
object 333 Down (0%)
object 334 Down (0%)
object 335 Down (0%)
Threshold percentage down 59% up 60%
Tracked by:
HSRP Port-channel20.210 1
HSRP Port-channel20.210 2
STATIC-IP-ROUTING 0

Track 340
List threshold percentage
Threshold Percentage is Down (0%)
459 changes, last change 3w5d
object 341 Down (0%)
object 342 Down (0%)
object 343 Down (0%)
object 344 Down (0%)
object 345 Down (0%)
Threshold percentage down 59% up 60%
Tracked by:
HSRP Port-channel20.210 1
HSRP Port-channel20.210 2
STATIC-IP-ROUTING 0

[Seb Rupik]

OK then...

So you have four different default routes directed down each of your four tunnels. Each route has a tracking object applied to it which determines if the route is added to the routing table...but although each tracking object is monitoring five different IP SLA jobs these jobs are all similar objects. Certainly Track 320 and 330 are monitoring the same objects, whereas Track 310 and 340 are monitoring 60% of the same objects. This doesn't give you much resilience and you may find all the routes being removed very quickly if you get a set of similar failures.

As you have four possible next-hops for the default route, the router will be performing per-destination load balancing. This is great if the routes are taking different paths, but I get the impression that all of the tunnels are leaving via the same interface so you are not getting any benefit. We can check this...for each of the tunnel interfaces, what is the output of:

sh ip route <tunnel_x_destination_ip>

Cheers,

Seb.

[sukesh tandon]

I get the result as with both global routing table and vrf also .

% Network not in table

So how do i check which is the physical interface that my tunnel has chosen to push the packet from ?

Also could there be any reason for having 4 tunnels

[Seb Rupik · Accepted solution]

Ok, how about the output of:

sh ip cef 135.13.15.18

...or just:

sh ip route

As you say, all of the tunnel loopbacks reside in the default routing table. So the route to say 135.13.15.18 must be in there somewhere! :)

cheers,

Seb.

[sukesh tandon]

Yes , the ip cef command shows that it is going via the physical interface.

STPA222#sh ip cef 135.13.15.18

nexthop 90.173.203.233 TenGigabitEthernet0/1/0

So does CEF here tell me which physical interface the packet will go from ?

Thanks!

[Seb Rupik]

Yes.

It would be interesting to know if all four tunnels are routed out via the same interface.

[sukesh tandon]

yes they will because the destination IP address is same for all of them.

Now to summarize we have default BGP route in the routing table (Global).

1)All the 4 tunnels have different loopback addresses as source and are not part of any VRF.

2) The destination for all the tunnels is same.

3) CEF proves that it is going via which physical interface.

4) All the tunnel interfaces are part of a single VRF.

5 ) Are you aware on how the routing decesion could be made by the device itself since the source does not belong to any VRF and the tunnel does.

6) I am not able track the work flow. There is a BGP neighborship on that exit interface which is tied to a route map.

route-map GOUT permit 110
match ip address TUN880
set ip vrf NWE-ID next-hop 172.17.89.109

route-map GOUT permit 120
match ip address TUN870
set ip vrf NWE-ID next-hop 172.17.89.105

route-map GOUT permit 130
match ip address TUN860
set ip vrf NWE-ID next-hop 172.17.89.103

route-map GOUT permit 140
match ip address TUN850
set ip vrf NWE-ID next-hop 172.17.89.107

Do you think it has got anything to do with the VPN and VRF ?

[Seb Rupik]

Any chance you could share the entire config. Obfuscate any sensitive details.

cheers,

Seb.