Re: ASK THE EXPERT - DYNAMIC MULTIPOINT VPN - Page 2

ciscomoderator · ‎09-21-2010

Welcome to the Cisco Networking Professionals Ask the Expert conversation. This is an opportunity to get an update on Dynamic Multipoint VPN with Mike Sullenberger. Mike has been working with TCP/IP networking for 19 years and has been with Cisco for 14+ years where he is a Distiguished Support Engineer (DSE) in Customer Advocacy. His technical expertise is in the areas of TCP/IP, IPsec VPNs, Routing Protocols, NAT and HSRP. He is the principle architect of the Dynamic Multipoint VPN (DMVPN) solution, where he works on DMVPN network designs, troubleshooting and the design of new DMVPN features. Mike has a Bachelors of Science degree in Mathematics and he is a CCIE in Routing & Switching since 1997.

Remember to use the rating system to let Mike know if you have received an adequate response.

Mike might not be able to answer each question due to the volume expected during this event. Our moderators will post many of the unanswered questions in other discussion forums shortly after the event. This event lasts through October 1, 2010. Visit this forum often to view responses to your questions and the questions of other community member

Michael Sullenberger · ‎09-28-2010

#3. Can we have multihoming spoke router for dmvpn setup?

#7. Can I have a spoke connecting back to 2 different hubs using 2 different ISP's an

seperate the traffic over two tunnels, one tunnel going over each ISP?

Yes this is supported, there are basically two ways that this is done.

Use a single DMVPN network with a tunnel source that is routable over both physical paths (ISPs).
Use two DMVPN networks where each one is "locked" to a physical outbound interface (ISP).

There are a couple of issues to take into account for this type of network setup with DMVPN.

The addresses that are used for the 'tunnel source ...' on the DMVPN routers are routable over both ISPs (MPLS networks).

In this case you can use a single DMVPN cloud, single mGRE tunnel on each node and just route the IPsec/GRE tunnel packets over either one of the ISPs (MPLS networks).

This works well when you want to have a primary ISP and a backup ISP. In this case tunnel packets will be routed over the Primary ISP if it is up and over the backup ISP only if the Primary ISP path is down.

If you want to load-balance your tunnel traffic over both ISPs if they are both up than this case doesn't work well, you don't normally get very good load-balancing because you are statistically load-balancing a few number of tunnels, rather than statistically load-balancing many host flows. In this case you want to look at option 2.
The addresses that are used for the 'tunnel source ...' on the DMVPN routers are NOT routable over both ISPs (MPLS networks). In this case you need to use a different tunnel source if the packets are to be routed over the ISP1 versus ISP2.

In this case you will need to use two DMVPN networks (clouds) one for each ISP. This is a Dual DMVPN Dual Hub type scenario. You use either a 'vrf-lite' or 'tunnel route-via' type configuration to "lock"a tunnel to a particular outbound interface for a particular ISP on the hubs and those spokes that are attached to two ISPs. On the spokes that are attached to only one ISP you can either have it be a member of only one DMVPN could or both, where both tunnels would use the same outbound physical interface over the single ISP. In either case where a spoke only has a single tunnel interface or two tunnel interfaces the routing protocol running over the tunnels is used to decide which tunnel to use for forwarding packets.

One thing to note is that you can only build spoke-spoke tunnels within a DMVPN network (cloud), not between DMVPN networks (clouds). If two spokes end up routing packets for each other out different DMVPN networks (clouds) than they will not be able to build a spoke-spoke tunnel to each other. These nodes will still be able to reach each other using a spoke-hub-spoke path. This situation could arise because either the routing happens to be set up that way or because of network failures (access to one ISP or the other) has caused the spokes to be on different DMVPN networks (clouds).

If you want to load-balance your use of the two ISPs than this is the only way to get reasonable statistical load-balancing of the host traffic, since the host traffic will have two routes one for each tunnel interface, where each tunnel goes over a different ISP. In this way we are able to statistically balance many host flows over the two ISPs. This can play havoc with trying to bring up and use spoke-spoke tunnels. Depending on how the two directions of the flows are forwarded out a tunnel, you may or may not get a spoke-spoke tunnel and you may or may not use the spoke-spoke tunnel in one direction and this can be different for each host flow. Load-balancing over the two DMVPN networks (ISPs) is fairly easy to setup, but as noted spoke-spoke tunnel usage may be fairly erratic. Often for this type of network it is best to have a primary/backup ISP setup rather than a try for a load-balanced ISP setup.
There is now available a third option. In this case you setup a primary DMVPN network (mGRE tunnel interface) and a backup DMVPN network (mGRE tunnel interface), using the backup interface command to hold the backup mGRE interface in the down state while the primary mGRE interface is in the up state. The code will then monitor the configured NHRP NHSs on the primary mGRE tunnel and if ALL of them are unreachable (not returning NHRP registration replies) than the primary mGRE tunnel interface will be marked as up/down and the backup mGRE tunnel interface will be triggered to come up. The NHS on the primary mGRE tunnel interface will continue to "probed" with NHRP registration request and as soon as the spoke receives an NHRP registration reply from on of the primary NHSs it will mark the primary mGRE tunnel interface as up/up and the backup mGRE tunnel interface will go down.

Note, while a spoke is using the backup DMVPN network it will not be able to build a spoke-spoke tunnel with any spokes using
the primary DMVPN network, but these spokes will still be able to communicate using a spoke-hub-spoke path.

Mike.

Michael Sullenberger · ‎09-28-2010

#4. If I would like to sell dmvpn service to multiple customers bydedicating hub router at my WAN

cloud, do we need to run VRF or mpls on the spoke and hub router?

I assume that in this scenario that any particular spoke is only supporting a single customer. In this case

you would use the DMVPN VRF-lite type of configuration, where you have a different DMVPN network for

each customer. On the hub that means that you would have an mGRE tunnel per customer and each

mGRE tunnel would be in a different VRF in order to keep the customer traffic and routing separate on the

hub router. Since a spoke only supports a single customer it would have only one mGRE tunnel, the one

that matches with the correct mGRE tunnel for that customer on the hub. Since there is only one mGRE

tunnel on the spoke you do not need to use a VRF on the spoke.

If on the other hand a spoke router supports more than one customer (2 for example), then that spoke would

have two mGRE tunnels one for each customer and you would configure those mGRE tunnels into separate

VRFs. Again you would want to configure the two mGRE tunnels and VRFs on the spoke to match with the

corresponding mGRE tunnel on the hub.

Mike.

Michael Sullenberger · ‎09-28-2010

#8. What is the best and easiest way to route the Internet traffic from a spoke through the

tunnel back to the hub and then routed to our corporate Internet gateway which is in a

different location than the hub ?

I assume that you want Internet destined traffic coming in on the tunnel interface from the

spoke to be routed out the LAN interface on Hub router to go to the corporate gateway rather

than to go directly out thre WAN interface on the Hub router.

You can do this two ways:

Use Policy Based Routing (PBR) on the tunnel interface to force the traffic incoming on
the tunnel interface to go out the LAN interface, rather than follow the routing table
(default route) and go out the WAN interface.
Use VRFs on the hub router. It is easiest to put the WAN interface (ip vrf forwarding ...)
and the tunnel itself (tunnel vrf ...) into a VRF. The inside of the tunnel (data traffic) and
the LAN interface stay in global. In this way you have two routing tables (VRF and global),
which gives you two default routes, one for tunnel packets (in VRF pointing out the WAN)
interface and one for data packets (in global pointing out the LAN interface).

Mike.

Michael Sullenberger · ‎09-29-2010

#9. I have a dual cloud design terminated on the same hub router connected to different ISPs. I use PBR to forward router's traffic to the proper ISP with no luck. It has been a bug reported that PBR doesn't handle GRE traffic for route manipulation, so I stick with static routes to the spokes. Is there any workaround of it?

This is correct PBR and local PBR is not designed to be able to redirect tunnel and IPsec encrypted packets.

For what you want there are two ways to "lock" a tunnel to a particular outbound interface, a 'vrf-lite' or 'tunnel route-via' type configuration.

For 'tunnel route-via' you configure 'tunnel route-via mandatory' on the tunnel interface and you must have multiple equal cost routes in the routing table for the tunnel destinations. Usually this means two default routes 0.0.0.0/0 via the two different physical interfaces.

For Example:

If you have two WAN interfaces Serial0 and Serial1 and you want Tunnel0 to use Serial0 and Tunnel1 to use Serial1 then you could configure something like:

ip route 0.0.0.0 0.0.0.0 serial0

ip route 0.0.0.0 0.0.0.0 serial1

interface tunne0

...

tunnel route-via serial0 mandatory

interface tunnel1

...

tunnel route-via serial1 mandatory

Then tunnel 0 tunnel packets will be routed out serial0, as long as there is matching route for the tunnel destination that points out serial0 and the same is true for tunnel1 tunnel packets. The mandatory means that if there is not a matching route pointing out the configured interfaces then drop the packet. If you don't use mandatory then it will fallback to use any other matching route presumably going out a different interface.

For 'tunnel vrf ...', you configure VRFs, put the WAN interface into the VRF and the matching tunnel to use that VRF.

For Example:

ip vrf ISP1
   rd 1:1
ip vrf ISP2
   rd 2:2
interface Serial0
   ip vrf forwarding ISP1
   ...
interface Serial1
   ip vrf forwarding ISP2
   ...

interface Tunnel0
   ...
   tunnel vrf ISP1
interface Tunnel1
   ...
   tunnel vrf ISP2
...
ip route vrf ISP1 0.0.0.0 0.0.0.0 Serial0
ip route vrf ISP2 0.0.0.0 0.0.0.0 Serial1

Mike.

Michael Sullenberger · ‎10-01-2010

#10. If we have a lot HUB link connection will the SPOKE to SPOKE connect continue to work ? (NHRP issue)

I think you meant to say "If we have a loss of the Hub link (spoke-hub tunnel).

For simplicity lets first assume that there is only a single DMVPN Hub. When the hub goes down the spoke will remove the routes that it learned from the Hub from its routing table, because it lost the Hub as its routing neighbor. BUT the spoke will not delete any spoke-to-spoke tunnels (NHRP mappings or ISAKMP/IPsec SAs). Even though the spoke-to-spoke tunnel is still there the spoke will not be able to use these tunnels because the routing table no longer has a route to the destination network.

Furthermore, when the routing entries are removed there isn't any trigger into NHRP, for NHRP to remove NHRP mapping entries. Eventually NHRP will time out the current dynamic NHRP mapping entries that it had when the Hub went down since they are not being used. Only at that time does NHRP remove the mapping entry and send triggers to tear down ISAKMP and IPsec.

If there still happens to be a route in the routing table (could be a static route) with the correct IP next hop then the spoke should still use the spoke-spoke tunnel even when the hub is down. NHRP won't be able to refresh the mapping entry since the NHRP resolution request/response would need to go to the hub.

Note:With DMVPN Phase 2, the NHRP resolution request/response (refresh) should go directly to the other spoke. Thus allowing the NHRP mapping entry timer to be refreshed and the spoke-to-spoke tunnel to be "refreshed" and used, even though the DMVPN Hub is down.

In DMVPN Phase 3 this gets a bit cleaner. In this case you would only need a route that points out the tunnel interface, it wouldn't have to have the correct IP next-hop (NHRP ignores the IP next-hop in Phase 3).Also NHRP will be able to refresh the NHRP mapping entry, because the NHRP resolution request/response will go over the direct spoke-spoke tunnel.

Note: Recently I have noticed that there may be a bug in sending the "refresh" NHRP resolution request via the spoke-hub-spoke path rather than via direct spoke-spoke path.

If you have two (or more) DMVPN hubs within a single DMVPN network (single mGRE interface), then when the first (primary) hub goes down, the spoke router will still remove the routes from the routing table that it learned from this Hub, but it will also be learning the same routes (higher metric) from the second (backup) hub, so it will immediately install these routes. So the spoke-to-spoke traffic would continue going over the spoke-to-spoke tunnel, and be unaffected by the primary hub outage.

Mike.

Michael Sullenberger · ‎10-01-2010

#12. If you're running DMVPN over the Internet . . . and there is a problem with a path between two spokes - does DMVPN deal with this issue by sending traffic?

Depending on when the problem occurs DMVPN does slightly different things.

When DMVPN is buildng a new spoke-spoke tunnel it will continue to send spoke to spoke

data traffic via the spoke-hub-spoke tunnel path. The last step in bringing up the spoke-spoke

tunnel is that the NHRP resolution reply is sent directly over the spoke-spoke tunnel. If the

NHRP resolution reply doesn't make it to through then the remote spoke will not forward data

traffic over the spoke-spoke tunnel and this data traffic will continue to be forwarded via the

spoke-hub-spoke path. So in this case when the spoke-spoke tunnel doesn't work spoke to

spoke data traffic still makes it through via the spoke-hub-spoke path.

The other case is where the spoke-spoke tunnel is already up and functioning, data traffic is

being forwarded over it and then the spoke-spoke tunnel breaks. In this case the data forwarding

code will doesn't realize that anything is wrong and will continue to forward data traffic over the

direct spoke-spoke tunnel, effectively black-holing traffic. In the background we have ISAKMP

keepalives running between the spokes. Presumably the ISAKMP keepalives will be triggered

and fail and therefore ISAKMP would tear-down the IPsec and ISAKMP SAs and trigger NHRP

to remove the NHRP mappings. The data traffic would then revert to the spoke-hub-spoke path

and trigger to try to bring up the direct spoke-spoke tunnel again. Note, ISAKMP keepalives

(DPD) usually takes 30-60 secondsto detect a dead peer and during tat time is when we would

be black-holing data traffic.

Mike.

Michael Sullenberger · ‎10-01-2010

#13. I want to use DMVPN as a backup with dual hub and MPLS as theprimary connection, if a spoke does not have either MPLS connectivityand DMVPN to the primary, and have MPLS or DMVPN connectivity to thebackup hub and the backup hub still has connectivity to the primary hubcan we reroute the traffic to the hub?

Yes you can do this.

DMVPN sets up a dynamic tunnel network, over which you run a routing protocol that

tells the router how to forward traffic over DMVPN and other parts of the network.

In your case you setup the tunnels to the primary and backup hubs and set the

routing protocol metric to prefer the path through the primary hub. When the tunnel

to the primary hub goes down then the spoke will remove the routes learned from the

primary hub and then install the routes learned from the backup hub (higher metric).

The spoke will then route traffic to the backup hub, once the traffic gets to the backup

hub there will be route there that would forward this to the primary hub and on to the

final destination.

Note, normally you configure a tunnel (via the same mGRE interface) between the

hubs so that they can "talk" to each other. In DMVPN Phase 1 this is optional,

in Phase 2&3 it is required.

Mike.

Michael Sullenberger · ‎10-01-2010

#14. Can we use all the kind of encryption that we use in VPN

DMVPN basically uses ISAKMP/IPsec like a black box. You can use any encryption

and/or hashing algorithm that is supported on all the boxes in the DMVPN network.

You can even setup the hub and spokes to accept multiple different encryption/hashes

so that different peer pairs can use different encryption/hashing.

I recommend not using AH, since ESPAuth basically does the same thing and AH

will block any DMVPN nodes that are behind NAT from working. Also you want to use

IPsec transport mode for DMVPN.

Mike.

Michael Sullenberger · ‎10-01-2010

#15. Is there any limitation for spoke router? How can we use spoke in single dmvpn layout?

I am not sure what you are asking.

For DMVPN the spoke router must be an Cisco IOS router (ISR, 7200, 7300, ASR) also

with some caveats you can use a 6500 and 7600. Normally you size the spoke router so

that will have sufficient IPsec encryption throughput for the traffic that will be going in/out

of that spoke site. Small spoke site could have a small router, and a big spoke site a big

router.

The spokes are pre-configured to connect to two (for redundancy) DMVPN hubs and

when they come up they connect to the hub, NHRP register, become RP neighbors,

pass routes and then they are in the DMVPN network.

Mike.

Michael Sullenberger · ‎10-01-2010

#16. I would like to see the trouble shooting parameter for DMVPN, If my NHRP us not stable then what should I do ?

Here is a methodology for troubleshooting DMVPN.

The main issue with troubleshooting DMVPN is that are a number of different layers that you need to worry about.

Physical (NBMA or tunnel endpoint) Routing layer -- this is getting the encrypted tunnel packets between the tunnel endpoints (DMVPN hub and spoke or between spoke and spoke routers).
The IPsec Encryption layer -- this is encrypting the GRE tunnel packet going out and decrypting the IPsec packet coming in to reveal the GRE encapsulated packet.
The GRE Encapsulation layer -- this is GRE encapsulating the data IP packet going out and GRE decapsulating the GRE packet (after IPsec encryption) coming in to get the data IP packet.
The VPN Routing layer -- this is routing packets in/out of the p-pGRE and/or mGRE interfaces on the tunnel endpoint routers. This is done by running a dynamic routing protocol over the DMVPN tunnels.

To troubleshoot a problem you need to look at each layer and see if that layer is working correctly. I usually start with the lowest layer (1) and work my way to the highest layer (4). Also when troubleshooting it is best to pick out a single spoke that is having a problem and concentrate on that spoke, rather then try to troubleshoot at the same time all of the spokes that are having a problem. It is also useful if possible to have a spoke that is working that is similar to the spoke that is having problems as a comparison.

In general when troubeshooting it is best to do the following:

Sync up the timestamps between the hub and spoke
Enable msec debug and log timestamps
Enable "terminal exec prompt timestamp" for the debugging sessions.

This way you can easily correlate the debug output with the show command output.

So at this point you pick a spoke that is having a problem. You then need to get the tunnel IP address of that spoke and the current physical IP address of that spoke. It is also best if you can log into that spoke using TELNET or SSH directly (not through the DMVPN tunnel) to that spokes physical address. If you can't do this then log in through the DMVPN tunnel assuming it is up. To find a spoke's current physical address (assuming you know the spokes tunnel IP address) you can use 'show ip nhrp' on the hub. Look for the spokes tunnel IP address (they are in sorted order) and once you find the matching entry you want the NBMA address.

To check out layer 1 you can ping from the hub to the spoke's NBMA address and from the spoke to the hub's NBMA address (from the output of 'show ip nhrp' on the spoke). These pings should go directly out the physical interface, not through the DMVPN tunnel. Hopefully there isn't a Firewall that blocks ping packets. You can also use traceroute to check the path that the encrypted tunnel packets are taking. If this doesn't work you need to check the routing and any firewalls between the hub and spoke routers.

To check out layer 2 you need to look for ISAKMP SAs and IPsec SAs between the NBMA addresses of the hub and spoke.

show crypto isakmp sa detail
show crypto ipsec sa peer

will show you these things. Take a look at the SA lifetime values. If they are close to the configured lifetimes (default -- 24 hrs for ISAKMP and 1 hour for IPsec) then that means these SAs have been recently negotiated. If you look a little while later and they have been re-negotiated again, then the ISAKMP and/or IPsec may be bouncing up and down. To check this out you can run the debugs

debug crypto isakmp
debug crypto ipsec
debug crypto engine

You want to run these on both the spoke and the hub at the same time, also need to run them long enough so that we can see the ISAKMP and IPsec SAs re-negotiated two to three times if possible. Also on the hub router you can use:

debug crypto condition ...

to restrict the crypto debugs to only show you debugs for the particular spoke in question.
It is also good to look at the communication between NHRP and IPsec by showing the crypto map and socket tables.

show crypto map
show crypto socket

To check out layer 3 you need to look at NHRP. The spoke should be sending an NHRP registration packet on a regular basis, every 1/3 NHRP holdtime (on spoke) or 'ip nhrp registration timeout ' value. You can check on this on the spoke with

show ip nhrp nhs detail

This will tell you whether the spoke is sending NHRP registration requests and getting replies from the hub. It will also tell you if the spokes is currently retransmitting NHRP registration requests (hub is not responding).
On the hub you can use

show ip nhrp

to see if the hub has a mapping entry for that spoke. Also on the hub check the 'created' and 'expire' timer. The 'created' timer tells you how long this NHRP mapping entry has continuously been in the NHRP mapping table. The 'expire' timer tells you how long before this NHRP mapping entry would be deleted, if the hub were not to receive another NHRP registration from the spoke. If the 'created' timer is low and gets reset a lot then that means tat the NHRP mapping entry is getting reset, this could be a bug in NHRP or be triggered from IPsec (when the IPsec SAs are cleared NHRP is triggered to clear the mapping). You can also try pinging from the hub to the spoke's tunnel ip address and the reverse. You should also check that data packets will go through, i.e. ping from the inside LAN interface IP address to the inside LAN interface address on the other end. These packets should be going through the tunnel.

If there looks like there is a problem in this area then you can use the following debugs on the hub router.

debug nhrp
debug nhrp packet   (this shows you NHRP packets in/out, you can look for packets from/to the spoke in question)
debug nhrp cache    (this one can be verbose so you want to use it with caution)
debug tunnel protection
debug crypto socket      (these last two show communication between NHRP and IPsec)

To check out layer 4 you would look at the routing protocol neighbor table and possibly run routing protocol debugs.

Mike.

Michael Sullenberger · ‎10-01-2010

#17. As per Cisco doucmentation for DMVPN we can should use EIGRP . Why is that? Is it that we can tweak the cost parameter used by EIGRP ie. K1 K2 K3 -- K5?

With DMVPN you can use any Routing Protocol that is based on IP, (EIGRP, OSPF, RIP, BGP) and even ODR, but not ISIS.

It turns out that with the Non-Broadcast-Multi-Access (NBMA) style network that DMVPN sets up a distance-vector type protocol usually works best. RIP and EIGRP are distance-vector protocols, and of these two EIGRP is the best for convergence time, etc, but RIP and BGP will scale to a larger number of neighbors. In general for scaling to the largest number of neighbors from a single DMVPN hub the order from lowest to highest is (OSPF, EIGRP, ODR, RIP(passive mode), BGP).

I usually say, that if you can, use the same routing protocol over DMVPN that you use in the rest of your network, since switching (redistributing in/out) of a different routing protocol to go over DMVPN can be complicated. This is usually not a problem in smaller DMVPN networks (<1000) nodes, but when you get larger than this you will probably have to pick a routing protocol that scales to the most number of neighbors over DMVPN. We are doing work with both EIGRP and BGP so that they will scale higher, and therefore you may be able to match up with the rest of your network.

With OSPF being a link-state protocol and its area concept it is much harder to scale this to larger number of neighbors for the NBMA type of network that DMVPN builds, but in most cases you should be able to get to 500-600 neighbors, perhaps more, for the larger routers (7200, 3945e, ASR).

Note, I never play with the K values of EIGRP, it is too dangerous.

Mike.