DMVPN with Policy based routing

[Cory King]

Hi all, 

I'm having an issue that i hope somebody can help me with.  We are running a DMVPN environment and I have a couple of spoke offices that run voip over their tunnel to each other.  The issue is that we swapped the ISP in spoke A but now the latency has doubled from spoke A to spoke B so the phones will cut out here and there.  To try and fix the issue, we had a lower bandwidth connection installed in spoke A which is the same ISP as spoke B so latency would be reduced and thus the phones should be better.  So what I tried was creating two tunnels in each location and on the hub sites so that they would connect.  Spoke B only has the one isp so they are using the shared tag on the tunnel protection profile.  But when I try and add both ISP's on spoke A, I have two default routes with the same metric and the primary tunnel loses connection to some other spokes.  If I change the distance on the 2nd defualt route, the 2nd tunnel doesn't come up.  I've tested this in GNS3 and it works completely fine which is the frusterating part as it doesn't work in production.  I've searched around and there doesn't seem to be the same issue as I have.  Any insight would be great.

[Georg Pauwen]

Cory,

do you have any Qos for voice traffic configured at all ? I am just thinking that you could try that first before configuring a separate tunnel just for VoIP traffic.

Either way, can you post the configs of both the hub and a spoke ?

[Cory King]

Thank you for the reply.  Attached are the relevent parts of the configs of a hub (there are two), and the two spokes in question.  I have interfaces shutdown as they don't function and don't have a policy map in place, but the major factor is the tunnels.  Again this config works perfect in GNS3.

[Rich Uline]

Cory,

Latency shouldn't really be a problem for VoIP. If the calls are dropping out then it's more likely you have a problem with jitter. Depending on where the jitter is introduced, obtaining that second connection may not solve your issue.

As I understand it, you have two tunnels on Spoke A - one on each ISP-facing interface. You also have two tunnels on SpokeB, but they both use the same interface. I didn't see you mention how you handled the routing configuration needed on SpokeA in order to establish the second tunnel. You could do it with PBR. Let's say that the source interface (the one with the same ISP as SpokeB) of your second tunnel on SpokeA is using 192.168.0.2 and the default router for that network is 192.168.0.1. (Edited for your posted configs)Your PBR configuration might look something like this:

access-list 10 permit 3.3.3.3
access-list 11 permit 4.4.4.4
!
route-map TUNNELS permit 10
  match ip address 10
  set ip default next-hop 3.3.3.1
route-map TUNNELS permit 11
  match ip address 11
  set ip default next-hop 4.4.4.1
!
ip local policy route-map TUNNELS

[Georg Pauwen]

Hello,

I agree with Rich, if latency is the problem, you might want to try a VoIP QoS first. Below is an example of your modified HUB, apply that to the spokes as well:

crypto keyring DMVPN
pre-shared-key address 0.0.0.0 0.0.0.0 key xxxxxxxxx
!
crypto isakmp policy 1
encr aes 256
authentication pre-share
group 5
crypto isakmp keepalive 10 periodic
crypto isakmp profile DMVPN
keyring DMVPN
match identity address 0.0.0.0
!
crypto ipsec transform-set TS esp-aes 256 esp-sha-hmac
mode transport
!
crypto ipsec profile Internet
set transform-set TS
set isakmp-profile DMVPN
!
interface Loopback1
ip address 10.1.255.1 255.255.255.255
!
interface Port-channel1
ip address 10.1.1.3 255.255.240.0
ip flow monitor nbar-mon input
no negotiation auto
!
interface Tunnel1
bandwidth 100000
ip address 172.16.200.1 255.255.254.0
no ip redirects
ip mtu 1400
ip nhrp authentication vpnmesh
ip nhrp map multicast dynamic
ip nhrp network-id 1
ip nhrp holdtime 600
ip tcp adjust-mss 1350
ip ospf network broadcast
ip ospf dead-interval 10
ip ospf hello-interval 5
ip ospf priority 200
ip ospf cost 20
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 123456
tunnel protection ipsec profile Internet shared
service-policy output PRIORITY_PARENT
!
interface Tunnel2
bandwidth 100000
ip address 172.16.202.1 255.255.254.0
no ip redirects
ip mtu 1400
ip nhrp authentication vpnmesh
ip nhrp map multicast dynamic
ip nhrp network-id 2
ip nhrp holdtime 600
ip tcp adjust-mss 1350
ip ospf network broadcast
ip ospf dead-interval 10
ip ospf hello-interval 5
ip ospf priority 200
ip ospf cost 21
qos pre-classify
tunnel source GigabitEthernet0/0/0
tunnel mode gre multipoint
tunnel key 7891011
tunnel protection ipsec profile Internet shared
service-policy output PRIORITY_PARENT
!
interface TenGigabitEthernet0/0/0
no ip address
shutdown
!
interface TenGigabitEthernet0/0/1
no ip address
shutdown
!
interface GigabitEthernet0/0/0
description outside
bandwidth 100000
ip address 1.1.1.1 255.255.255.248
negotiation auto
service-policy output PRIORITY_PARENT
!
interface GigabitEthernet0/0/1
description inside
no ip address
negotiation auto
channel-group 1 mode active
!
interface GigabitEthernet0/0/2
no ip address
negotiation auto
!
interface GigabitEthernet0/0/2.3001
description MPLS
bandwidth 100000
encapsulation dot1Q 3001
ip address 172.16.90.10 255.255.255.248
ip flow monitor nbar-mon input
service-policy output MPLS-QOS-Parent
!
interface GigabitEthernet0/0/3
no ip address
negotiation auto
channel-group 1 mode active
!
interface GigabitEthernet0/0/4
no ip address
shutdown
negotiation auto
!
interface GigabitEthernet0/0/5
no ip address
shutdown
negotiation auto
!
interface TenGigabitEthernet0/1/0
description VPLS
bandwidth 2000000
ip address 172.16.1.1 255.255.255.0
ip flow monitor nbar-mon input
ip policy route-map REDIRECT-INTERNET-RM
ip ospf network broadcast
ip ospf dead-interval 10
ip ospf hello-interval 5
ip ospf priority 200
ip ospf cost 10
service-policy output VPLS-QOS-Parent
!
interface GigabitEthernet0
vrf forwarding Mgmt-intf
ip address 10.1.100.3 255.255.255.0
negotiation auto
!
router ospf 1
area 1 range 10.1.0.0 255.255.0.0
redistribute static metric-type 1 subnets route-map STATIC-TO-OSPF
passive-interface default
no passive-interface TenGigabitEthernet0/1/0
no passive-interface Tunnel1
no passive-interface Tunnel2
network 10.0.0.0 0.255.255.255 area 1
network 172.16.0.0 0.0.255.255 area 0
default-information originate metric 10
!
router bgp 65535
bgp log-neighbor-changes
neighbor 172.16.90.9 remote-as 65012
!
address-family ipv4
network 10.1.0.0 mask 255.255.0.0
network 10.253.1.0 mask 255.255.255.0
neighbor 172.16.90.9 activate
neighbor 172.16.90.9 send-community both
neighbor 172.16.90.9 soft-reconfiguration inbound
exit-address-family
!
ip forward-protocol nd
!
ip ftp source-interface Loopback1
no ip http server
no ip http secure-server
ip tftp source-interface Loopback1
ip route 0.0.0.0 0.0.0.0 98.158.83.97
ip route 10.1.0.0 255.255.0.0 10.1.1.50
ip route 10.253.1.0 255.255.255.0 10.1.1.2
ip route vrf Mgmt-intf 0.0.0.0 0.0.0.0 10.1.100.1
ip tacacs source-interface Loopback1
ip ssh version 2
ip scp server enable
!
ip access-list extended REDIRECT-ACL
deny ip 10.0.0.0 0.255.255.255 10.0.0.0 0.255.255.255
permit ip 10.0.0.0 0.255.255.255 any
!
access-list 10 permit 10.253.1.0 0.0.0.255
!
access-list 100 permit udp any any range 2048 65535
access-list 100 permit ip any any dscp ef
!
class-map match-any VoIP
match access-group 100
match ip rtp 16383 16383
!
policy-map PRIORITY_CHILD
class VoIP
priority percent 50
policy-map PRIORITY_PARENT
class class-default
shape average 512000
service-policy PRIORITY_CHILD

!

route-map STATIC-TO-OSPF permit 10
match ip address 10
!
route-map REDIRECT-INTERNET-RM permit 10
match ip address REDIRECT-ACL
set ip next-hop 10.1.1.2

[Cory King]

Thank you for everyones reply and input.  I do have qos on the circuit, which I see I don't have in the configs, and it does help but the other issue that I forgot to mention is the drops between the two carriers.  I agree also that the latency shouldn't be a factor it's more the drops that occur which is why I was going for the same carrier as the drops would be less.  What I really want to know is how to get PBR working when you have two gateways as my setup does not.  I do have other sites also that I would like to get PBR working for the purpose of voice and data.   Does that part of the configuration look correct or do I need to create multiple ipsec profiles?  From reading all the documentation and forum posts it doesn't appear so but I'm not totally sure.

[Rich Uline · Accepted solution]

Cory,

Let's take a look at SpokeA. From your posted configs we have:

interface Tunnel1
 tunnel source GigabitEthernet0/0
...
interface GigabitEthernet0/0
 ip address 3.3.3.3 255.255.255.248
...
interface Tunnel2
 tunnel source GigabitEthernet0/2
...
interface GigabitEthernet0/2
 ip address 4.4.4.4 255.255.255.128

So, these are the source IP addresses and interfaces we are using to establish the two tunnels. Now let's look at Hub1.

interface Tunnel1
 tunnel source GigabitEthernet0/0/0
...
interface Tunnel2
 tunnel source GigabitEthernet0/0/0
...
interface GigabitEthernet0/0/0
 ip address 1.1.1.1 255.255.255.248

And now we have our destination IP address that SpokeA must reach. Finally, there is one last bit of information I want to look at. On SpokeA we have our default routes set up like so:

ip route 0.0.0.0 0.0.0.0 3.3.3.1
ip route 0.0.0.0 0.0.0.0 4.4.4.1

This looks to me like there are equal cost default routes. This would normally be fine except that there is no guarantee for which interface is used - the router will dynamically decide. This is in conflict with what we need - one specific source IP to use one specific exit interface, while another specific source uses a separate specific exit interface. A more specific route will not help us in this case because the destination is the same. If we create a static route to our destination specifying one exit interface, then both tunnels attempt to establish on that interface. We need PBR. Please try implementing something like this:

access-list 10 permit 3.3.3.3
access-list 11 permit 4.4.4.4
!
route-map TUNNELS permit 10
  match ip address 10
  set ip next-hop 3.3.3.1
route-map TUNNELS permit 11
  match ip address 11
  set ip next-hop 4.4.4.1
!
ip local policy route-map TUNNELS

[Cory King]

Thanks Rich that did it!

So I've labbed this up before deploying and here is what I have for anyone that could use this info as well. 

Along with the local PBR for the tunnels, I also wanted to keep traffic outbound going out the main connection as it's faster, and I wanted the 2nd connection dedicated for the spoke a - spoke b traffic only.  Here is my labbed config:

interface Tunnel1
tunnel source GigabitEthernet0/0
...
interface GigabitEthernet0/0
ip address 3.3.3.3 255.255.255.0
...
interface Tunnel2
tunnel source GigabitEthernet0/2
...
interface GigabitEthernet0/2
ip address 4.4.4.4 255.255.255.0

!

interface GigabitEthernet0/1

 ip address 10.252.1.3 255.255.255.0

 ip policy route-map REDIRECT

!

ip route 0.0.0.0 0.0.0.0 3.3.3.254

ip route 0.0.0.0 0.0.0.0 4.4.4.254

!

ip local policy route-map TUNNELS

!

access-list 10 permit 3.3.3.3
access-list 11 permit 4.4.4.4

!

ip access-list extended REDIRECT-TUNNEL2

permit ip 10.252.0.0 0.0.255.255 10.119.0.0 0.0.255.255

!

ip access-list extended REDIRECT-OUTSIDE
deny ip 10.252.0.0 0.0.255.255 10.0.0.0 0.255.255.255
permit ip 10.252.0.0 0.0.255.255 any

!

ip sla 1
icmp-echo 3.3.3.254 source-interface GigabitEthernet0/0
frequency 10
ip sla schedule 1 life forever start-time now
ip sla 2
icmp-echo 4.4.4.254 source-interface GigabitEthernet0/2
frequency 10
ip sla schedule 2 life forever start-time now

!

track 1 ip sla 1 reachability
track 2 ip sla 2 reachability
!

route-map REDIRECT permit 10

match ip address REDIRECT-TUNNEL2

set ip next-hop 172.16.202.119 

!
route-map REDIRECT permit 11
match ip address REDIRECT-OUTSIDE
set ip next-hop verify-availability 3.3.3.254 1 track 1
!
route-map REDIRECT permit 12
match ip address REDIRECT-OUTSIDE
set ip next-hop verify-availability 4.4.4.254 1 track 2
!

route-map TUNNELS permit 10
match ip address 10
set ip next-hop 3.3.3.254

!
route-map TUNNELS permit 11
match ip address 11
set ip next-hop 4.4.4.254