GRE egress interface selection

[mattp0002]

Long story short, I have a single router with two Ethernet interfaces connected towards two different ISPs.  Each interface has a different (isp-provided & advertised) public IP address used to peer with the ISPs across a /30 p2p subnet.  eBGP routing is enabled across both connections and I'm accepting a default route from each and advertising my one /24 prefix to both.  Local pref is used to prefer one outbound route over the other while both ISP circuits are online, and each would take all the traffic load in the case of a single circuit failure.  Pretty basic config.

I'd like to turn up two GRE tunnels from this router across the internet.  Each would be sourced from the different ISP-provided public /30 IP addresses.  Both GRE tunnel destinations would be the same IP address of another router across the internet somewhere.

I'd like the traffic from each GRE to stay on it's own ISP symmetrically.  I believe since this router has one route table, whichever default route is currently installed from the RIB would take all the outbound traffic - regardless of the GRE source IP address.  (I understand the return traffic for each GRE should ingress via the proper ISP automatically, I'm talking here about egress from my router towards the internet)

I don't think I can use PBR to set the next-hop for each tunnel, because PBR can't be applied on an egress interface - and these GRE tunnels are sourced at this router rather than passing in and through the router.

Is this type of architecture possible?  How can GRE egress interface be enforced?

Thanks!!

[Peter Paluch]

Hi,

I believe this can be accomplished relatively easily. Let's assume you have Tun1 and Tun2. Tun1 uses tunnel destination X.X.X.X while Tun2 uses tunnel destination Y.Y.Y.Y. Furthermore, let's assume you have two physical interfaces, Fa0/0 and Fa0/1, serving as your uplinks, and finally, let's assume that your uplink next hops are A.A.A.A on Fa0/0 and B.B.B.B on Fa0/1.

Then tying the X.X.X.X to Fa0/0 and Y.Y.Y.Y to Fa0/1 would look as follows;

ip route X.X.X.X 255.255.255.255 Fa0/0 A.A.A.A
ip route X.X.X.X 255.255.255.255 Null0 2
ip route Y.Y.Y.Y 255.255.255.255 Fa0/1 B.B.B.B
ip route Y.Y.Y.Y 255.255.255.255 Null0 2

This configuration tells the router to reach X.X.X.X only via A.A.A.A reachable through Fa0/0, and to reach Y.Y.Y.Y only via B.B.B.B reachable through Fa0/1. Should Fa0/0 or Fa0/1 go down, make sure the X.X.X.X or Y.Y.Y.Y is considered unreachable rather than routed over any other interface.

There is also an option of doing local PBR using a route-map and a local PBR policy but that would be, I guess, just complicating this simple approach.

Would you mind testing this?

Best regards,
Peter

[mattp0002]

Peter,

Thank you for your suggestions.  However, in this case both GRE tunnels will terminate at the same remote IP address, which makes the situation problematic.

In other words, there is no X.X.X.X and Y.Y.Y.Y - both tunnels terminate at X.X.X.X which is reachable concurrently via both ISPs.

Thanks

[Peter Paluch]

Hi Matt,

My bad, I am sorry - I read your original post too quickly.

So what you are saying is that the tunnels are terminated at the same destination but they differ in their source IP address. Am I correct here?

If so then obviously, the PBR is the solution. It would go along these lines, now assuming that Tun1 uses tunnel source M.M.M.M while Tun2 uses tunnel source N.N.N.N and both tunnels use tunnel destination X.X.X.X:

ip access-list extended GRE_ISP1
 permit gre host M.M.M.M host X.X.X.X
!
ip access-list extended GRE_ISP2
 permit gre host N.N.N.N host X.X.X.X
!
route-map PBR_LOCAL permit 10
 match ip address GRE_ISP1
 set ip next-hop A.A.A.A
 set interface Null0
!
route-map PBR_LOCAL permit 20
 match ip address GRE_ISP2
 set ip next-hop B.B.B.B
 set interface Null0
!
ip route A.A.A.A 255.255.255.255 Fa0/0
ip route A.A.A.A 255.255.255.255 Null0 2
ip route B.B.B.B 255.255.255.255 Fa0/1
ip route B.B.B.B 255.255.255.255 Null0 2
!
ip local policy route-map PBR_LOCAL

This configuration first defines two ACLs to differentiate between traffic sourced by different tunnels. Then, a route-map for locally-originated packets is created, stating for each ACL (and thus each tunnel traffic) that the preferred next hop is either A.A.A.A (B.B.B.B) if reachable, and Null0 otherwise. The set ip next-hop has precedence over set interface so the Null0 will only be used if A.A.A.A or B.B.B.B are entirely unreachable.

The static routes make sure that the only accepted path toward the next hops A.A.A.A and B.B.B.B is through their respective interfaces. I will make a test to find out if these are necessary at all but it certainly doesn't harm to have them in place. Finally, the route-map is added as a local PBR policy.

Would this meet your needs?

Best regards,
Peter

[mattp0002]

Peter,

Your configuration makes perfect sense.  However after googling some more I found these:

https://supportforums.cisco.com/discussion/9866081/policy-route-router-gre-generated-traffic

https://supportforums.cisco.com/discussion/9516576/policy-routing-gre-packets

And they've got me wondering if there's a special issue with GRE...  I suppose I can lab it up tomorrow.

[Peter Paluch]

Hi Matt,

Any luck with this? Should I set up a lab and test it? The links you have posted do raise some rather grave issues.

Best regards,
Peter

[mattp0002]

Peter,

Any further thoughts on this?  Sorry for my late reply I was out of town then crushed with other projects.

I need to work this out, I've had a TAC ticket open for 3 weeks on this but so far Cisco hasn't really been helpful at all.

I'm about to grab 5 routers and lab it up here, I'll update you on what I find in a few hours.

UPDATE:

Built it out in the lab, the local IP policy did not work - and did not register any matches whatsoever.

Googled some more, I'm seeing the exact same thing as described here:

http://www.networking-forum.com/viewtopic.php?t=26760

[Peter Paluch · Accepted solution]

Hi Matt,

I've got a solution - it is an ugly hack but it should work. Basically, it is based on the notion that the ip local policy does not catch the self-originated GRE packets, but we can still force them to go over a loopback interface with a PBR applied, and that will finally force the packets to be subject to a PBR.

Assumptions:

• M.M.M.M is used in tunnel source on one Tunnel interface
• N.N.N.N is used in tunnel source on the other Tunnel interface
• X.X.X.X is used in tunnel destination on both Tunnel interfaces
• A.A.A.A is one of the possible next hops
• B.B.B.B is another of the possible next hops

The configuration would then would be as follows:

access-list 1 permit M.M.M.M
access-list 2 permit N.N.N.N
!
route-map TunPBR permit 10
 match ip address 1
 set ip next-hop A.A.A.A
!
route-map TunPBR permit 20
 match ip address 2
 set ip next-hop B.B.B.B
!
ip route X.X.X.X 255.255.255.255 Loopback1000
!
interface Loopback1000
 ip address 192.0.2.1 255.255.255.255
 ip policy route-map TunPBR

This config tries to match the source loopbacks M.M.M.M and N.N.N.N by ACLs 1 and 2. Then a normal route-map is matchign these sources addresses, and setting the next hops as expected. Afterwards, the route toward X.X.X.X (the tunnel tailend) is forced through Loopback1000, and that interface has the route-map applied - and that one finally catches the packets and causes them to be PBRed.

Would you mind testing this out?

Best regards,
Peter

[Jose Jara · Accepted solution]

Hi Matt, 

I see a solution for this case, it will require to configure BGP multipath in order to install the default route from both ISP´s. So, the config in BGP would be like this:

maximum-paths 2

bgp bestpath as-path multipath-relax ---- because you are receiving the default from different ASN´s

Also, I assume the rest of BGP attributes of the default route injected from both ISPs will be the same.

Now, in the GRE Tunnel you have to specify the interface where you want to route the traffic:

interface Tunnel 1

Tunnel Route-Via <interface>

This feature is available from 12.4(11)T and it allows to configure it in two forms, 

- mandatory: if there is no route available through the interface, it will drop the traffic

- preferred: if there is no route available, it will use any route. 

Also, as far as I know there is a caveat and it is that this feature does not work with IPSec (tunnel protection).

Other option would be to configure two VRFs, one per interface.

Best Regards,

Jose.

[Peter Paluch]

Hi Jose,

Just as in the thread regarding suppressing BGP advertisements, you have pinpointed us to an existing solution that may be readily available and which I have missed. Rated as deserved, and thank you very much for doing that! You have become something of a conscience of mine :)

One comment, though: You have suggested to configure BGP to allow installing both default routes into the routing table, so that the tunnel route-via can point to any one of them. I do not think that doing such an extensive change to BGP's configuration is necessary. All we need the router to have is a route toward the tunnel endpoint out both ISPs, and this route can be comfortably added statically:

ip route X.X.X.X 255.255.255.255 A.A.A.A
ip route X.X.X.X 255.255.255.255 B.B.B.B

with A.A.A.A being the ISP1 next hop and B.B.B.B being the ISP2 next hop.

What do you think?

Best regards,
Peter

[mattp0002]

The plot thickens!

Jose - Very interesting as that seems to be exactly what I need.  I'm trying to test out these commands right now.  I have some old 1841 routers I was using to lab this up.  Advanced IP Services 12.4(25f).  I've also checked on a new ISR4451-X running the Universal 15.4(3)S2 ios.

Neither of these routers shows that command "tunnel route-via" as available under the GRE tunnel interface section.  When I type "?" it doesn't show up, and additionally when I just type it in, I get the ^ underneath the 'o' in "route-via."  So my ios doesn't seem to know about it.

Now I just found this link: http://blog.ipspace.net/2010/06/tunnel-route-selection-and-dmvpn-tunnel.html which basically states that ? hmm maybe it's not really supported?

This blog claims it is: https://mellowd.co.uk/ccie/?tag=route-via

I'm going to play around with my lab ios and maybe change it to a T ios hopefully that will work.

Peter - I like your hack.  It makes sense.  I am also going to lab that one out, as it does not seem to be reliant on any special ios featureset.  My only concern with it is if you had any legitimate non-gre traffic on this router destined for the remote-end, that might fail in the event of a circuit failure.  But, in reality in my case that wouldn't be an issue as all legit traffic should transverse inside my tunnel mesh.

Thanks both of you I will continue testing this and let you know the results!

[Peter Paluch]

Matt,

There certainly is a more, let's say, clean way of accomplishing what you need by creating two so-called Front-door VRFs (FVRF), have each tunnel's endpoints exist in the separate FVRF, and point each FVRF to a different upstream ISP. This would not even require doing PBR.

Does your IOS support VRFs and the tunnel vrf command?

Best regards,
Peter

[mattp0002]

Peter:

Yes that command is available.  Just trying to imagine what you are suggesting - are you saying to create two FVRFs, each with let's say a /32 host route via the 2 ISPs to the tunnel endpoint(s) but via the 2 different next-hop addresses.

Then, those /32 statics would be used to build my GRE tunnels, and these tunnel interfaces would be bound into the global route table so their routes will be available to all my normal traffic?

That is way cleaner and easier than the use of the loopback PBR.

Could you show a few lines of config you'd suggest?  I'm thinking "tunnel vrf ISP_A" along with the tunnel dest host route "ip route vrf ISP_A x.x.x.x 255.255.255.255 z.z.z.z"  Something like that?

Jose:

Swapped my ios over to advipservices 124-20t and the command is there.  Still need to lab it out I will update you all.  Looks very promising.

[Peter Paluch]

Hi Matt,

The solution using VRFs would be similar to this:

ip vrf ISP1
 rd 1:1
!
ip vrf ISP2
 rd 1:2
!
interface Loopback1
 ip vrf forwarding ISP1
 ip address 192.0.2.1 255.255.255.255
!
interface Loopback2
 ip vrf forwarding ISP2
 ip address 192.0.2.2 255.255.255.255
!
ip route 192.0.2.1 255.255.255.255 Loopback1
ip route 192.0.2.2 255.255.255.255 Loopback2
!
ip route vrf ISP1 0.0.0.0 0.0.0.0 1.2.3.4 global
ip route vrf ISP2 0.0.0.0 0.0.0.0 5.6.7.8 global
!
interface Tunnel1
 tunnel vrf ISP1
 tunnel source Loopback1
 tunnel destination 198.51.100.1
 ...
!
interface Tunnel2
 tunnel vrf ISP2
 tunnel source Loopback2
 tunnel destination 198.51.100.1
 ...
!

The basic idea of this config is to first create two VRFs and then put the loopbacks used as tunnel source interfaces into these VRFs. However, because they will not be reachable anymore from traffic arriving via interfaces in the global routing table, two static routes pointing to their IPs and respective interfaces, causing the traffic arriving to these IP addresses to "bleed" from the global routing table into the respective VRFs thanks to the egress interface that belongs to it. Afterwards, both VRFs are populated by a default route that points to a different next hop in the global routing table for each VRF, making each tunnel reach its endpoint over a different next hop. And finally, the tunnel interfaces are configured with respective FVRFs.

Best regards,
Peter

[Jose Jara]

Hi Peter,

thank you for the rating!

Regarding the BGP configuration in my example, I totally agree with you and configuring static routes to the destination would be much easier and better.

In the solution you propose I do not see one thing and this is how would be reachable these loopbacks through Internet, as the tunnel source must be the addressing of the providers, which is configured in the /30 WAN interface. I think it would be easier to just configure the WAN interfaces in different VRFs like this:

interface Gi0/0

description ISP-1

ip vrf forwarding ISP1

interface Gi0/1

description ISP-2

ip vrf forwarding ISP2

Then, configure two default routes, one per VRF:

ip route vrf ISP1 0.0.0.0 0.0.0.0 1.2.3.4

ip route vrf ISP2 0.0.0.0 0.0.0.0 5.6.7.8

Finally, configure the GRE tunnels to route the traffic through the VRF´s:

interface Tunnel1

tunnel vrf ISP1

interface Tunnel2

tunnel vrf ISP2

Matt,

in the link of ipspace, by the way the source where I found that tunnel protection does not work with this feature and one of the best technical blogs I know, it states that this feature does not work with tunnel protection and apart from that I do not know any other caveat and it should work in your case. I tested before proposed you this solution and it worked fine. Anyway, before any configuration of a new feature you should verify that this is supported in your IOS/platform.

For this, check always the feature navigator of Cisco: www.cisco.com/go/fn

Also, I recommend you to test any unknown feature before configure it in production. 

This feature is known as Tunnel Route Selection and you can find more information here: http://www.cisco.com/c/en/us/td/docs/ios/12_4t/12_4t11/ht_trsel.html

I´ve seen that in 4451 is not supported but in the IOS of the 1841 you have installed it should work fine.

Best Regards,

Jose