Technical background
One of the major concerns and challenges that pertain to site to site VPN implementations using hub and spoke topology with a large number of sites is scalability. In spit of the fact that implementing many GRE tunnels over IPSec with a dynamic routing protocol can scale well. However the number of Access lists and point to point tunnels will be hard to manage when there is a large number of remote sites using fully of partially meshed topology. In addition to the scalability issues, the implementation of a large number of site to site VPNs using hub and spoke topology with a big number of spoke to spoke communications, this will lead to a high overhead on the CPU and memory of the hub router because all of the spoke to spoke traffic must transit the hub.
One of the most scaleable solutions to the above VPN issues using hub and spoke topology with a large number of spoke to spoke communications required is Cisco preparatory Dynamic Multipoint VPNs (DMVPNs). One of the main requirement of DMVPN implementations is the use of hub and spoke topology, but that dos not mean spoke to spoke traffic must traverse the hub. The hub required for spokes’ address registration using NHRP. NBMA Next Hop Resolution Protocol (NHRP) Defined in RFC 2332 used for spokes address registration in DMVPN implementations. With DMVPN any traffic flowing between routers is sent via a GRE tunnel, but the interesting feature that distinguishes DMVP among other VPN implementations is that this GRE tunnel is a multipoint GRE tunnel. In other words the hub and the spokes will require one tunnel each to achieve a fully meshed DMVPN connectivity.
From the given information above it is obvious that DMVPN can provide the following benefits:
· Simplify the configuration portion of the hub router by eliminating the need of configuring crypto maps, tunnel interfaces, ACLs for each spoke
· The spoke routers can obtain their IP addresses dynamically, for example an internet edge router connected to an ADSL link can obtain its IP automatically from the ISP and then the tunnel will register with Hub using NHRP
Design Example:
In this document we will apply some of the DMVP concepts and features to achieve a redundant path to an existing WAN link, as it shown in the bellow diagram the company has a primary WAN link to a service provider using MPLS VPN, HUB1 and HUB2 routers connected to the head office and all other sites use HUB1 as the primary path to reach the head office. Also each site has its own internet link such as ADSL with dynamic IP address assigned to each remote site by the ISP, except the head office routers Hub1 and HUB2 they have to have static IP addresses from the ISP in order to let the remote Sites register with the hub routers using NHRP to form direct spoke to spoke DMVPN communications.
As illustrated in the above diagram, the routing protocol between the remote sites and the MPLS SP is RIPv2 while the routing between the MPLS SP and the Hub routers is EBGP. With DMVPN, the dynamic routing protocol that will be used to form a redundant dynamic path over mGRE tunnels is OSPF. Bellow is the configuration portion of Hub1 DMVPN :
Note:
The network 172.16.1.0/24 is used in this example to represent the internet IP addressing ( in a real networks implementations this will be a dynamic or static public IPs in the spokes and a public static IP in the Hub side).
Hub1 internet IP address: 172.16.1.1
Hub2 internet IP address: 172.16.1.10
Hub1:
crypto isakmp policy 100
encr aes
hash md5
authentication pre-share
crypto isakmp key key address 0.0.0.0 0.0.0.0 no-xauth
!
!
crypto ipsec transform-set set1 esp-aes esp-md5-hmac
mode transport
!
crypto ipsec profile profile1 -- this will be used to protect mGRE tunnel with IPSec
set transform-set set1
!
!
interface Tunnel0
bandwidth 1000
ip address 150.1.1.1 255.255.255.0 --- DMVPN shared subnet
no ip redirects
ip mtu 1400
ip nhrp authentication key1
ip nhrp map multicast dynamic
ip nhrp map 150.1.1.10 172.16.1.10 --- Static NHRP to Hub2
ip nhrp map multicast 172.16.1.10
ip nhrp network-id 150
ip nhrp holdtime 360 -- this tells the spoke how long to keep an NHRP reply
ip tcp adjust-mss 1360
ip ospf network broadcast -- required for direct spoke to spoke communications with ospf
ip ospf priority 200 -- to make this router as the OSPF DR
tunnel source FastEthernet1/1
tunnel mode gre multipoint
tunnel key 1500
tunnel protection ipsec profile profile1 --- IPSec encryption
!
Now let’s see the configuration portion of one of the spokes (site 2):
!
interface Tunnel0
bandwidth 1000
ip address 150.1.1.2 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication key1
ip nhrp map 150.1.1.1 172.16.1.1 ---Static NHRP mapping to Hub1 and Hub2
ip nhrp map multicast 172.16.1.1
ip nhrp map 150.1.1.10 172.16.1.10
ip nhrp map multicast 172.16.1.10
ip nhrp network-id 150
ip nhrp nhs 150.1.1.1
ip nhrp nhs 150.1.1.10
ip tcp adjust-mss 1360
ip ospf network broadcast
ip ospf priority 0 --- 0 to make sure this spoke will never become an OSPF DR or BDR
tunnel source FastEthernet1/1
tunnel mode gre multipoint
tunnel key 1500
tunnel protection ipsec profile profile1
!
Site2 verifications:
SITE2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
192.168.1.1 200 FULL/DR 00:00:38 150.1.1.1 Tunnel0
192.168.1.10 20 FULL/BDR 00:00:38 150.1.1.10 Tunnel0
SITE2#show ip route
R 10.10.1.0 [120/1] via 10.2.1.10, 00:00:17, FastEthernet1/0
R 10.3.1.0 [120/1] via 10.2.1.10, 00:00:17, FastEthernet1/0
O IA 192.168.1.0/24 [110/101] via 150.1.1.10, 00:00:07, Tunnel0
[110/101] via 150.1.1.1, 00:00:07, Tunnel0
150.1.0.0/24 is subnetted, 1 subnets
30.0.0.0/24 is subnetted, 1 subnets
O 30.1.1.0 [110/101] via 150.1.1.3, 00:00:07, Tunnel0
SITE2#
From Site 2 routing table we can see that the MPLS link is up, however Site2 router prefers the path through DMVPN tunnel over the MPLS SP link. This is normal behavior as OSPF administrative distance AD is less than RIP, the simplest way to solve this issue using the above topology is to make RIP AD lees than OSPF AD ( this solution may not be the best for every topology ) .
router rip
distance 109
SITE2#show ip route
R 10.10.1.0 [109/1] via 10.2.1.10, 00:00:11, FastEthernet1/0
R 10.3.1.0 [109/1] via 10.2.1.10, 00:00:11, FastEthernet1/0
C 10.2.1.0 is directly connected, FastEthernet1/0
R 192.168.1.0/24 [109/1] via 10.2.1.10, 00:00:11, FastEthernet1/0
150.1.0.0/24 is subnetted, 1 subnets
C 150.1.1.0 is directly connected, Tunnel0
30.0.0.0/24 is subnetted, 1 subnets
R 30.1.1.0 [109/2] via 10.2.1.10, 00:00:11, FastEthernet1/0
Now all the routes is through the MPLS link using RIPv2.
After the routing issue has been resolved lets do some verification and see if Hub1 link to the MPLS SP goes down what will happen to the spoke routing table:
HUB1#show ip route
20.0.0.0/24 is subnetted, 1 subnets
O 20.1.1.0 [110/101] via 150.1.1.2, 00:04:45, Tunnel0
30.0.0.0/24 is subnetted, 1 subnets
O 30.1.1.0 [110/101] via 150.1.1.3, 00:04:45, Tunnel0
As we can see Hub1 now using DMVPN to reach the remote sites because the primary link to the MPLS SP is down.
SITE2#show ip route
R 10.10.1.0 [109/1] via 10.2.1.10, 00:00:17, FastEthernet1/0
R 10.3.1.0 [109/1] via 10.2.1.10, 00:00:17, FastEthernet1/0
R 192.168.1.0/24 [109/1] via 10.2.1.10, 00:00:17, FastEthernet1/0
30.0.0.0/24 is subnetted, 1 subnets
R 30.1.1.0 [109/2] via 10.2.1.10, 00:00:17, FastEthernet1/0
Although Hub1 link to the MPLS SP is down the spokes will continue use the MPLS SP as the primary path, this because Hub2 is advertising the networks, which is the optimal path for this topology. This behavior happened as result to the adjustment we did to the RIP administrative distance above.
Now if Hub1 and Hub 2 links to the MPLS SP go down the only path to reach the HQ network ( 192.168.1.0) by the spokes will be over the internet using DMVPN:
SITE2#show ip route
R 10.10.1.0 [109/1] via 10.2.1.10, 00:00:06, FastEthernet1/0
R 10.3.1.0 [109/1] via 10.2.1.10, 00:00:06, FastEthernet1/0
C 10.2.1.0 is directly connected, FastEthernet1/0
O IA 192.168.1.0/24 [110/101] via 150.1.1.10, 00:00:33, Tunnel0
[110/101] via 150.1.1.1, 00:00:33, Tunnel0
150.1.0.0/24 is subnetted, 1 subnets
C 150.1.1.0 is directly connected, Tunnel0
30.0.0.0/24 is subnetted, 1 subnets
R 30.1.1.0 [109/2] via 10.2.1.10, 00:00:06, FastEthernet1/0 -- site 3 LAN is reachable via the MPLS SP link
SITE2#
From the above routing table of site 2 we can see that site 2 has both hub1 and hub 2 as next hops to reach the HQ network while the design requirements require hub1 to be the primary, this issue can be fixed by using several ways, the simplest way is by tuning the dynamic routing protocol used in the topology. This document will demonstrate the ability of achieving the same goal by using OSPF and EIGRP as the routing protocol over DMVPN tunnels. With OSPF in this example ospf cost will be used to control the path selection, by increasing the ospf cost on the LAN interface of Hub2 to make the path through Hub1 more preferred.
SITE2#show ip route
O IA 192.168.1.0/24 [111/101] via 150.1.1.1, 00:00:50, Tunnel0 -- Hub1 is the primary next hop to reach the HQ network over mGRE tunnel ( less ospf cost).
In the case of Site 2 link to the MPLS SP goes down the remote networks in the routing table will be shown as bellow:
SITE2#show ip route
O IA 192.168.1.0/24 [111/101] via 150.1.1.1, 00:00:50, Tunnel0
O 30.1.1.0 [111/101] via 150.1.1.3, 00:00:30, Tunnel0
SITE2#traceroute 30.1.1.1
Type escape sequence to abort.
Tracing the route to 30.1.1.1
1 150.1.1.3 284 msec 272 msec * -- direct spoke to spoke
EIGRP as the Routing Protocol over DMVPN Tunnels:
If EIGR used as the routing protocol over DMVPN mGRE Tunnels using the above topology, the first issue will be faced by Site 2 is that HQ network will be seen with two equal cost paths then Site2 will loadbalnce the trafficto that network. While the required behavior is the primary path to be through Hub1. Earlier in this document we have seen how this issue fixed with OSPF by changing the OSPF Cost of an interface, with EIGRP we will use administrative distance to prefer one eigrp neighbor over others:
Router eigrp 10 --- this configurations required in every spoke
Distance 80 150.1.1.1 0.0.0.0
EGRP internal route AD is 90, the above configuration changed the AD of all routes coming from hub1 150.1.1.1 to be 80 which will make it the preferred neighbor ( please note that if there is any external EIGRP route you may need to consider another approach such as delay, as with EIGRP you can not change administrative distance of external eigrp routes from a specific peer only you can change it globally from all peers).
SITE2#show ip route 192.168.1.0
Routing entry for 192.168.1.0/24
Known via "eigrp 10", distance 80, metric 15362560, type internal
Redistributing via eigrp 10
Last update from 150.1.1.1 on Tunnel0, 00:01:36 ago
Routing Descriptor Blocks:
* 150.1.1.1, from 150.1.1.1, 00:01:36 ago, via Tunnel0
Route metric is 15362560, traffic share count is 1
Total delay is 500100 microseconds, minimum bandwidth is 1000 Kbit
Reliability 255/255, minimum MTU 1400 bytes
Loading 1/255, Hops 1
First issue fixed, let’s see what other facts need to be consider when we use EIGRP.
With EIGRP and RIP implementations over DMVPN tunnels we have to disable spilt horizon in the hub tunnel interface (same concept used with NBMA networks such as hub and spoke frame-relay topology)
SITE2#traceroute 30.1.1.1
Type escape sequence to abort.
Tracing the route to 30.1.1.1
1 150.1.1.1 136 msec 304 msec 296 msec
2 150.1.1.3 316 msec 268 msec *
also from the above traceroute from Site 2 to Site 3 LAN network, the traffic is traversing the hub, which means the communications between the spokes not direct !!
with ospf we used a network type of broadcast over the tunnels which resulted in direct communications between spokes after each spoke register with the hub using NHRP.
With eigrp we need to add an additional command under the hub tunnel interface to enable the direct spoke to spoke communications ( this behavior might be useful for some implementations that requires the spoke to spoke traffic to go through the hub router first for example to do traffic inspection )
Hub1:
interface Tunnel0
no ip next-hop-self eigrp 10
no ip split-horizon eigrp 10
SITE2#show ip route
D 30.1.1.0 [80/15490560] via 150.1.1.3, 00:00:16, Tunnel0
Now Site 2 see Site 3 LAN network with Site 3 tunnel interface IP
SITE2#traceroute 30.1.1.1
Type escape sequence to abort.
Tracing the route to 30.1.1.1
1 150.1.1.1 140 msec
150.1.1.3 276 msec 204 msec
SITE2#traceroute 30.1.1.1
Type escape sequence to abort.
Tracing the route to 30.1.1.1
1 150.1.1.3 140 msec 192 msec *
SITE2#
The first initial traffic will go through the Hub then all the subsequent traffic will be direct spoke to spoke.
To Sum up, DMVPN is a very flexible, dynamic and secure feature that can suit various type of implementations, also we can control our traffic and path selection by using simple dynamic routing protocol adjustments
.
Thank you
Marwan Alshawi
