DMVPN and OSPF route preferences

[Isaiah]

I have several routers in a DMVPN with each other running OSPF across it.  I want some spokes to prefer routes from other specific spokes.  For example, suppose I have four offices: two in the US and two in Asia.  One office in the US has an Internet connection, as does one in Asia, while the other two do not.  They all have (and need) mesh connectivity with each other and full OSPF routing through a DMVPN across a private MPLS network.  I want the US office that does not have its own Internet connection to prefer the route across the DMVPN to the other US office, while the Asia office without its own Internet to prefer the other Asia office.

The challenge is that OSPF costs are normally based on the interface through which the route is learned.  In this case, all sites appear to be equally distant from all other sites since all are learned through the same tunnel interface.  With OSPF point-to-multipoint or NBMA network types you can specify OSPF costs per neighbor, however those network types are not compatible with the intent of DMVPN.  Is there a technique to accomplish these goals with OSPF?

[Rolf Fischer]

Hi,

With OSPF point-to-multipoint or NBMA network types you can specify OSPF costs per neighbor, however those network types are not compatible with the intent of DMVPN.

Specifiing costs on a per-neighbor basis is possible with the network types

• point-to-multipoint
• point-to-multipoint non-broadcast.

The latter (rather an option than a own network type) is not applicable because in a DMVPN we want automatic neighbor discovery. However, point-to-multipoint could be a possible solution.

Advantages: It is robust, discovers neighbors automatically and you can assign individual costs to neighbors manually where you need them (be aware that this affects all routes learned from this neighbor).

Disadvantages: It advertises only hostroutes for every p-to-mp interface, not the subnet itself. So an non-OSPF IP host in that subnet would not be reachable from other routers. And many p-to-mp interfaces result in many hostroutes, but there are ways to suppress them. The default-timers are "slow" but they are easy to adjust.

I haven't used p-to-mp over a DMVPN yet, but several times in other environments (for instance over static Layer-2 multipoint-tunnels) and had no bad experiences so far.

HTH
Rolf

[Isaiah]

Thank you Rolf.  However, I also see the following behavior with point-to-multipoint: with DMVPN you do not have neighbor relationships with every spoke, but only with the configured hub(s).  So you do not learn any routes from the other spokes directly, but you learn everything from the hub.  With a network type "broadcast" this is not a problem because the hub is acting as the DR and gives you everybody else's routes with their address set as the next hop.

But with p-to-mp, it is assumed that you do not have connectivity between spokes (as possibly in a frame relay network), meaning that the hub router sets the next hop address for every route as itself.  This causes all spoke to spoke communication to flow through the hub.  Also, because you are learning the spoke routes from the hub rather than from the spoke directly, the cost assigned to the hub neighbor applies to spoke routes as well.

In order to allow the formation of neighbor relationships directly between spoke routers, you would have to put in "ip nhrp map multicast {public IP}" statements on every spoke for every other spoke.  This would also not be scalable at all, as the number of mesh neighbors relationships would grow exponentially in a large DMVPN.

So I don't think p-to-mp is the solution.  Or am I misunderstanding something?

[Rolf Fischer]

Unfortunately I have only little practical experience with DMVPN so far, but my understanding is that Phase 3's "Shortcut Switching Enhancement" solves the next-hop problem:

Migrating from Dynamic Multipoint VPN Phase 2 to Phase 3

"This lookup process is how data packets are redirected over the spoke-to-spoke direct tunnel, rather than being forwarded to the hub as the routing table states.
(...)
Because the spokes use routes with the IP next-hop set to the hub router (not the remote spoke router as before), you can configure OSPF to use point-to-multipoint network mode rather than broadcast network mode."

[Isaiah]

I appreciate your insights, Rolf.  Yes, you are right, that is true that DMVPN Phase 3 does override the FIB next hop with the dynamically learned entries in the NHRP mapping table.  However, that NHRP redirect/shortcut is for the next hop address, which is dependent on which route in the FIB was already selected.  But it is that selection, before the redirect/shortcut even comes into play, that is the one that I want to influence.

Referring to the diagram on the original post, if I am in Osaka, and I am only learning of routes through the California hub router with that router's IP set as the next hop, it is still going to screw up my route selection abilities because I cannot weight routes originally advertised from Tokyo differently than routes advertised from Oregon or California, since they are all both coming in on the same tunnel interface and also all being learned from (and the next hop set to) the California hub.  Yes, the DMVPN phase 3 redirect/shortcut would still cause me to build a direct tunnel to Tokyo if I have traffic that could be routed directly there.  But in this case that doesn't help because the routing table is not going to prefer the Tokyo default route in the first place so as to even get to the point of setting up the direct tunnel.

I think I am becoming fairly convinced that OSPF really does not have a way to manipulate individual routes like this.  We will probably have to look at BGP for manipulation abilities at this level.

[Rolf Fischer]

You're correct - I really had to verify some things with a simplified gns3 lab. As I mentioned, we don't use DMVPN in our production environmtent so I made some wrong conclusions, sorry for the confusion.

I don't think there is a way to prefer an external route originated from a spoke without having an OSPF neighborship to this spoke.

In theory you could combine the point-to-multipoint (broadcast) network-type on the hub with the point-to-multipoint non-broadcast network-type on those spokes where you need spoke-to-spoke adjacencies. Both types are compatible and the Cisco OSPF implementation doesn't care whether the received Hellos are uni- or multicasts, as long as the connection is bidirectional. The hub would still auto-discover this spokes (even without neighbor-statements for the hub configured on the spokes) and you could form spoke-to-spoke adjacencies where you really need them. The cost option would then provide some flexibility in terms of path-selection.

But unfortunately I don't think this is recommendable for a production environment. At least I have a new interesting lab challenge for the next rainy weekend ;)

[Isaiah]

I think I found my answer.  The following is from the session presentation for Cisco Live BRKSEC-4054 - Slide 28:

Multiple DMVPNs versus Single DMVPN
•Multiple DMVPNs
  •Easier to manipulate RP metrics between DMVPNs for Load-sharing
    •EIGRP –Route tags, Delay; iBGP –Communities, MED; OSPF –Cost
•Single DMVPN
  •Slightly more difficult to manipulate RP metrics within DMVPN for Load-sharing
    •EIGRP –Route tags, delay; iBGP –Communities, MED; OSPF –Can’t do

It does not yield extremely granular control, but you can run smaller "regional" DMVPN communities with only a subset of all your nodes in each, and then prefer them by giving their tunnel interfaces a better OSPF cost than your global DMVPN tunnel interface.  Duplicate route prefixes (like the default route) being advertised from your regional DMVPN will have better metrics than your global one. But if they go away, the global one is still there.