Hierarchical DMVPN Design Questions

[Javier Zamora]

Is there any example of a hierarchical DMVPN setup out there?

I have a couple of questions that might be answered by that based on the diagram:

1) What are the advantages of regional hubs vs a consolidated hub and leaving the rest to dynamic tunnels?

2) From the Regional Hub's perspective, is the regional cloud and the East/West clouds on the same tunnel interface? 

3) How will a tunnel from Spoke 4 to Spoke 1 will flow? is it a real spoke to spoke? or will it flow through the regional hub?

I have been doing phase two for several years on other networks, but I am considering doing a hierarchical phase three to replace a global MPLS network 

[Philip D'Ath]

The question is a bit to do with scale.

In the worst case, how many spokes world wide do you think there might be?

Also, where is the majority of the traffic going?  US East?  Somewhere else?  All over the place?

[Javier Zamora]

Worldwide I am thinking in the neighborhood of 150 spokes. Most of Asia spoke traffic is to the Asia regional hub (about 50%) to the US East about 30% and the rest to the spokes.

East coast is the main DC on Americas and Europe. West coast is the DR DC. Americas and Europe spoke will mainly talk to East coast DC (70%) the rest to the spokes.

[Philip D'Ath]

I would go with your design as is.

[Javier Zamora]

Thanks for the response, any examples to clarify how the regional hubs are configured?

[vikasgupta2k]

Hi Javier,

I'm also trying to accomplish the similar design what you showed in the diagram. Did you encounter any weird issues with this design, would it be possible to share any sample configs for hubs and spokes for reference.

Thanks,

Vikas

[Javier Zamora]

I had to lab it myself, as there are a lot of unanswered questions. The link given by vikasgupta2k is useful:

https://www.cisco.com/c/en/us/support/docs/security/dynamic-multipoint-vpn-dmvpn/211292-Configure-Phase-3-Hierarchical-DMVPN-wit.html

I did notice that some of the examples out there are partially working, but they are not scalable.  Here is my biggest takeaway:

"When using a second tunnel interface, you must be sure that the NHRP network-ID is the same as that of the spoke-facing tunnel interface, thus associating both interfaces to the same NHRP process.

The NHRP network-ID is important when building a spoke-to-spoke tunnel across regions. In this case, the NHRP network-ID must be the same on each GRE interface, that is, the GRE tunnel interfaces for the hubto-spoke tunnel, and for the hub-to-hub tunnel. The regional hub knows to send the NHRP redirect only if the packet is forwarded out a GRE tunnel having the same network-ID as the spoke originating the packet. As stated previously, the regional hub is treated as a spoke of the core hub. Therefore, note that when packets are sent between the regional hubs, a hub to hub tunnel can be formed if the hubs are in the same NHRP network-ID on the core hub, and the redirect and shortcut commands are configured. Also, note that you cannot use a tunnel key when using multiple mGRE tunnel interfaces with the same NHRP network-ID. Therefore, each mGRE interface must have a unique tunnel source IP address."

What this is roughly saying, is that you cannot use tunnel keys when doing hierarchical and using the same interface, which becomes an issue when you scale it to use Dual Cloud Dual DMVPN, making redundancy next to impossible. It also states that you shouldn't use a single tunnel for both functions "It is recommended to use a separate mGRE tunnel interface for the hub-to-hub connections, rather than using the same tunnel interface that the spokes connect to."

As the document states that it required a unique source IP address, I tried it with secondary IP address, and it works, but I've worked enough with IOS to know that this might cause unexpected bugs. So I moved it to a loopback interface and it worked, please bear in mind that I have not been able to do it in production yet. 

I am attaching some basic templates sanitized with secondary IPs, feel free to test them and tear them apart. 

[Mohammad Alhyari]

Does this help

https://www.cisco.com/c/en/us/support/docs/security/dynamic-multipoint-vpn-dmvpn/211292-Configure-Phase-3-Hierarchical-DMVPN-wit.html