Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to get an update on Dynamic Multipoint VPN with Mike Sullenberger. Mike has been working with TCP/IP networking for 19 years and has been with Cisco for 14+ years where he is a Distiguished Support Engineer (DSE) in Customer Advocacy. His technical expertise is in the areas of TCP/IP, IPsec VPNs, Routing Protocols, NAT and HSRP. He is the principle architect of the Dynamic Multipoint VPN (DMVPN) solution, where he works on DMVPN network designs, troubleshooting and the design of new DMVPN features. Mike has a Bachelors of Science degree in Mathematics and he is a CCIE in Routing & Switching since 1997.
Remember to use the rating system to let Mike know if you have received an adequate response.
Mike might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through October 1, 2010. Visit this forum often to view responses to your questions and the questions of other community member
#3. Can we have multihoming spoke router for dmvpn setup?
#7. Can I have a spoke connecting back to 2 different hubs using 2 different ISP's an
seperate the traffic over two tunnels, one tunnel going over each ISP?
Yes this is supported, there are basically two ways that this is done.
There are a couple of issues to take into account for this type of network setup with DMVPN.
#4. If I would like to sell dmvpn service to multiple customers bydedicating hub router at my WAN
cloud, do we need to run VRF or mpls on the spoke and hub router?
I assume that in this scenario that any particular spoke is only supporting a single customer. In this case
you would use the DMVPN VRF-lite type of configuration, where you have a different DMVPN network for
each customer. On the hub that means that you would have an mGRE tunnel per customer and each
mGRE tunnel would be in a different VRF in order to keep the customer traffic and routing separate on the
hub router. Since a spoke only supports a single customer it would have only one mGRE tunnel, the one
that matches with the correct mGRE tunnel for that customer on the hub. Since there is only one mGRE
tunnel on the spoke you do not need to use a VRF on the spoke.
If on the other hand a spoke router supports more than one customer (2 for example), then that spoke would
have two mGRE tunnels one for each customer and you would configure those mGRE tunnels into separate
VRFs. Again you would want to configure the two mGRE tunnels and VRFs on the spoke to match with the
corresponding mGRE tunnel on the hub.
#8. What is the best and easiest way to route the Internet traffic from a spoke through the
tunnel back to the hub and then routed to our corporate Internet gateway which is in a
different location than the hub ?
I assume that you want Internet destined traffic coming in on the tunnel interface from the
spoke to be routed out the LAN interface on Hub router to go to the corporate gateway rather
than to go directly out thre WAN interface on the Hub router.
You can do this two ways:
#9. I have a dual cloud design terminated on the same hub router connected to different ISPs. I use PBR to forward router's traffic to the proper ISP with no luck. It has been a bug reported that PBR doesn't handle GRE traffic for route manipulation, so I stick with static routes to the spokes. Is there any workaround of it?
This is correct PBR and local PBR is not designed to be able to redirect tunnel and IPsec encrypted packets.
For what you want there are two ways to "lock" a tunnel to a particular outbound interface, a 'vrf-lite' or 'tunnel route-via' type configuration.
For 'tunnel route-via' you configure 'tunnel route-via
If you have two WAN interfaces Serial0 and Serial1 and you want Tunnel0 to use Serial0 and Tunnel1 to use Serial1 then you could configure something like:
ip route 0.0.0.0 0.0.0.0 serial0
ip route 0.0.0.0 0.0.0.0 serial1
tunnel route-via serial0 mandatory
tunnel route-via serial1 mandatory
Then tunnel 0 tunnel packets will be routed out serial0, as long as there is matching route for the tunnel destination that points out serial0 and the same is true for tunnel1 tunnel packets. The mandatory means that if there is not a matching route pointing out the configured interfaces then drop the packet. If you don't use mandatory then it will fallback to use any other matching route presumably going out a different interface.
For 'tunnel vrf ...', you configure VRFs, put the WAN interface into the VRF and the matching tunnel to use that VRF.
ip vrf ISP1
ip vrf ISP2
ip vrf forwarding ISP1
ip vrf forwarding ISP2
tunnel vrf ISP1
tunnel vrf ISP2
ip route vrf ISP1 0.0.0.0 0.0.0.0 Serial0
ip route vrf ISP2 0.0.0.0 0.0.0.0 Serial1
#10. If we have a lot HUB link connection will the SPOKE to SPOKE connect continue to work ? (NHRP issue)
I think you meant to say "If we have a loss of the Hub link (spoke-hub tunnel).
For simplicity lets first assume that there is only a single DMVPN Hub. When the hub goes down the spoke will remove the routes that it learned from the Hub from its routing table, because it lost the Hub as its routing neighbor. BUT the spoke will not delete any spoke-to-spoke tunnels (NHRP mappings or ISAKMP/IPsec SAs). Even though the spoke-to-spoke tunnel is still there the spoke will not be able to use these tunnels because the routing table no longer has a route to the destination network.
Furthermore, when the routing entries are removed there isn't any trigger into NHRP, for NHRP to remove NHRP mapping entries. Eventually NHRP will time out the current dynamic NHRP mapping entries that it had when the Hub went down since they are not being used. Only at that time does NHRP remove the mapping entry and send triggers to tear down ISAKMP and IPsec.
If there still happens to be a route in the routing table (could be a static route) with the correct IP next hop then the spoke should still use the spoke-spoke tunnel even when the hub is down. NHRP won't be able to refresh the mapping entry since the NHRP resolution request/response would need to go to the hub.
In DMVPN Phase 3 this gets a bit cleaner. In this case you would only need a route that points out the tunnel interface, it wouldn't have to have the correct IP next-hop (NHRP ignores the IP next-hop in Phase 3).Also NHRP will be able to refresh the NHRP mapping entry, because the NHRP resolution request/response will go over the direct spoke-spoke tunnel.
Note: Recently I have noticed that there may be a bug in sending the "refresh" NHRP resolution request via the spoke-hub-spoke path rather than via direct spoke-spoke path.
If you have two (or more) DMVPN hubs within a single DMVPN network (single mGRE interface), then when the first (primary) hub goes down, the spoke router will still remove the routes from the routing table that it learned from this Hub, but it will also be learning the same routes (higher metric) from the second (backup) hub, so it will immediately install these routes. So the spoke-to-spoke traffic would continue going over the spoke-to-spoke tunnel, and be unaffected by the primary hub outage.
#12. If you're running DMVPN over the Internet . . . and there is a problem with a path between two spokes - does DMVPN deal with this issue by sending traffic?
Depending on when the problem occurs DMVPN does slightly different things.
When DMVPN is buildng a new spoke-spoke tunnel it will continue to send spoke to spoke
data traffic via the spoke-hub-spoke tunnel path. The last step in bringing up the spoke-spoke
tunnel is that the NHRP resolution reply is sent directly over the spoke-spoke tunnel. If the
NHRP resolution reply doesn't make it to through then the remote spoke will not forward data
traffic over the spoke-spoke tunnel and this data traffic will continue to be forwarded via the
spoke-hub-spoke path. So in this case when the spoke-spoke tunnel doesn't work spoke to
spoke data traffic still makes it through via the spoke-hub-spoke path.
The other case is where the spoke-spoke tunnel is already up and functioning, data traffic is
being forwarded over it and then the spoke-spoke tunnel breaks. In this case the data forwarding
code will doesn't realize that anything is wrong and will continue to forward data traffic over the
direct spoke-spoke tunnel, effectively black-holing traffic. In the background we have ISAKMP
keepalives running between the spokes. Presumably the ISAKMP keepalives will be triggered
and fail and therefore ISAKMP would tear-down the IPsec and ISAKMP SAs and trigger NHRP
to remove the NHRP mappings. The data traffic would then revert to the spoke-hub-spoke path
and trigger to try to bring up the direct spoke-spoke tunnel again. Note, ISAKMP keepalives
(DPD) usually takes 30-60 secondsto detect a dead peer and during tat time is when we would
be black-holing data traffic.
#13. I want to use DMVPN as a backup with dual hub and MPLS as theprimary connection, if a spoke does not have either MPLS connectivityand DMVPN to the primary, and have MPLS or DMVPN connectivity to thebackup hub and the backup hub still has connectivity to the primary hubcan we reroute the traffic to the hub?
Yes you can do this.
DMVPN sets up a dynamic tunnel network, over which you run a routing protocol that
tells the router how to forward traffic over DMVPN and other parts of the network.
In your case you setup the tunnels to the primary and backup hubs and set the
routing protocol metric to prefer the path through the primary hub. When the tunnel
to the primary hub goes down then the spoke will remove the routes learned from the
primary hub and then install the routes learned from the backup hub (higher metric).
The spoke will then route traffic to the backup hub, once the traffic gets to the backup
hub there will be route there that would forward this to the primary hub and on to the
Note, normally you configure a tunnel (via the same mGRE interface) between the
hubs so that they can "talk" to each other. In DMVPN Phase 1 this is optional,
in Phase 2&3 it is required.
#14. Can we use all the kind of encryption that we use in VPN
DMVPN basically uses ISAKMP/IPsec like a black box. You can use any encryption
and/or hashing algorithm that is supported on all the boxes in the DMVPN network.
You can even setup the hub and spokes to accept multiple different encryption/hashes
so that different peer pairs can use different encryption/hashing.
I recommend not using AH, since ESPAuth basically does the same thing and AH
will block any DMVPN nodes that are behind NAT from working. Also you want to use
IPsec transport mode for DMVPN.
#15. Is there any limitation for spoke router? How can we use spoke in single dmvpn layout?
I am not sure what you are asking.
For DMVPN the spoke router must be an Cisco IOS router (ISR, 7200, 7300, ASR) also
with some caveats you can use a 6500 and 7600. Normally you size the spoke router so
that will have sufficient IPsec encryption throughput for the traffic that will be going in/out
of that spoke site. Small spoke site could have a small router, and a big spoke site a big
The spokes are pre-configured to connect to two (for redundancy) DMVPN hubs and
when they come up they connect to the hub, NHRP register, become RP neighbors,
pass routes and then they are in the DMVPN network.
#16. I would like to see the trouble shooting parameter for DMVPN, If my NHRP us not stable then what should I do ?
Here is a methodology for troubleshooting DMVPN.
To check out layer 4 you would look at the routing protocol neighbor table and possibly run routing protocol debugs.
#17. As per Cisco doucmentation for DMVPN we can should use EIGRP . Why is that? Is it that we can tweak the cost parameter used by EIGRP ie. K1 K2 K3 -- K5?
With DMVPN you can use any Routing Protocol that is based on IP, (EIGRP, OSPF, RIP, BGP) and even ODR, but not ISIS.
It turns out that with the Non-Broadcast-Multi-Access (NBMA) style network that DMVPN sets up a distance-vector type protocol usually works best. RIP and EIGRP are distance-vector protocols, and of these two EIGRP is the best for convergence time, etc, but RIP and BGP will scale to a larger number of neighbors. In general for scaling to the largest number of neighbors from a single DMVPN hub the order from lowest to highest is (OSPF, EIGRP, ODR, RIP(passive mode), BGP).
I usually say, that if you can, use the same routing protocol over DMVPN that you use in the rest of your network, since switching (redistributing in/out) of a different routing protocol to go over DMVPN can be complicated. This is usually not a problem in smaller DMVPN networks (<1000) nodes, but when you get larger than this you will probably have to pick a routing protocol that scales to the most number of neighbors over DMVPN. We are doing work with both EIGRP and BGP so that they will scale higher, and therefore you may be able to match up with the rest of your network.
With OSPF being a link-state protocol and its area concept it is much harder to scale this to larger number of neighbors for the NBMA type of network that DMVPN builds, but in most cases you should be able to get to 500-600 neighbors, perhaps more, for the larger routers (7200, 3945e, ASR).
Note, I never play with the K