cancel
Showing results for 
Search instead for 
Did you mean: 
cancel
2855
Views
10
Helpful
9
Replies

Dual-Cloud DMVPN Source Address Problem

jerecassidy
Level 1
Level 1

Greetings,

I am running a single-hub dual-cloud DMVPN in phase2 operation (spoke-to-spoke enabled).  I am very suprised that the following issue does not come up more often.

Here is my setup:

Each headend has its own ISP.

Each remote site has a single router connected to 2 ISPs (interface1 and interface2)


Each headend public-IP is statically routed (/32) across an individual interface.

The default-route is floating based on an IP SLA tracking mechanism.

Note the following picture (showing default and static host routes)

dmvpn_source.jpg

With both default routes set to the interface doing DMVPN-X, spoke-to-spoke on DMVPN-X works well.  But what about spoke-to-spoke on DMVPN-Y?  It gets broken in the following manner:

In Site A, my TunnelY Interface is sourced from 10.2.0.2.  After it gets Site B;s public IP (10.4.0.2) via NHRP, it tries to form a spoke to spoke tunnel.  But how does it get to 10.4.0.2?  It uses its default-route out the 10.1.0.2 interface with a source address of 10.2.0.2.    A few things might happen:

1) ISP blocks bad sources completely either explicitly or via uRPF.

2) Spoke-to-Spoke Tunnel comes up, but assymetic routing occurs (this is rare)

3) The ISP Nat's all sources to itself (Comcast SMC gateways do this)  In the above example, you'd see 10.1.0.1 crypto packets arriving at 10.4.0.2!  Imagine the confusion this caused

In most cases, isakmp is hosed.  Even if the tunnel comes up, I do not want assymetic routering with all of the bandwidth on one interface - i'd like to actively use both ISP connections.

So... how to handle this?  I anticipated it, but I believed the NHRP/DMVPN mechanism would handle this situation.  i.e. if i learn a spoke via TunnelY and TunnelY is sourced on Interface2, that it would naturally send packets out interface2.  Alas, this is not the case.

Here are a couple of ways i thought of solving:

1) Since my endpoints aren't dyamic, I could statically host route all possible Y spokes out all the interface2s, all X out the interface1s.  (with 30 sites, this is so ugly I hesitate to even include it)

2) Route map each external interface and match against the source addresses.  If interface1 detects an interface2 source, set-next-hop to interface2.  Do the same on interface2 - if it hears a source matching interface1's IP address, set next-hop to interface1.  This is repeatable, but seems somewhat ugly as well.

3) post to the Cisco forums and see what the consensus is

Thank you very much in advance.  Here are my configs of the spoke sites if you need them:

Example using site A above:

(using PKI for isakmp)

interface TunnelX
bandwidth 10000
ip address 192.168.X.13 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication [redact]
ip nhrp map multicast 1.1.1.1
ip nhrp map 192.168.X.1 1.1.1.1
ip nhrp network-id X
ip nhrp holdtime 240
ip nhrp nhs 192.168.X.1
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/1
tunnel mode gre multipoint
tunnel key X
tunnel protection ipsec profile DMVPN_IPSEC
!

interface TunnelY
bandwidth 10000
ip address 192.168.Y.13 255.255.255.0
no ip redirects
ip mtu 1400
ip nhrp authentication [redact]
ip nhrp map multicast 2.2.2.2
ip nhrp map 192.168.Y.1 2.2.2.2
ip nhrp network-id Y
ip nhrp holdtime 240
ip nhrp nhs 192.168.Y.1
ip tcp adjust-mss 1360
tunnel source GigabitEthernet0/2
tunnel mode gre multipoint
tunnel key Y
tunnel protection ipsec profile DMVPN_IPSEC
!

ip route 1.1.1.1 255.255.255.255 10.1.0.1

ip route 2.2.2.2 255.255.255.255 10.2.0.1

ip route 0.0.0.0 0.0.0.0 10.1.0.1 track 1

ip route 0.0.0.0 0.0.0.0 10.2.0.1 250   (for failover if track 1 goes down)

1 Accepted Solution

Accepted Solutions

Marcin Latosiewicz
Cisco Employee
Cisco Employee

This is typically solved by separating ISPs into front VRFs (keeping the inside VRF global if you chose to) , allowing two defeault routes.

It's late (almost 1AM) but I think tunnel route-via could potentially work too.

View solution in original post

9 Replies 9

Marcin Latosiewicz
Cisco Employee
Cisco Employee

This is typically solved by separating ISPs into front VRFs (keeping the inside VRF global if you chose to) , allowing two defeault routes.

It's late (almost 1AM) but I think tunnel route-via could potentially work too.

Cliff Campbell
Level 1
Level 1

Since you're looking for a consensus, I'll chime in with I agree with Marcin!    FVRF is the way to go.  Here's a white paper that I found useful in switching my sites to use FVRF: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6586/ps6660/prod_white_paper0900aecd8034be03_ps6658_Products_White_Paper.html

Thank you very much Marcin and Cliff!

After looking at my needs for interaction between the outside (front) and inside VRFs (I also use these interfaces for internet traffic / inbound NAT / etc), I decided to forego the FVRF for the moment.

The "tunnel route-via mandatory"   seems like the perfect command for me though!

EXCEPT

When putting an IPSEC protection policy on a DMVPN tunnel, the route-via command appears to be ignored.  As pointed out by this CCIE on his blog:

http://blog.ipspace.net/2010/06/tunnel-route-selection-and-dmvpn-tunnel.html

Can you (or someone from cisco) verify this behavior - it seems like it should work as advertised regardless of the tunnel protection policy.

Thanks again!

Hey guys, sorry I was out of office.

My suggestion - open a TAC case - there were a few bugs open since the article Ivan wrote I was involved in one case at least (in which we migrated customer to VRF-lite),  there are too many moving parts (config,version etc) to confirm one way or another.

IMHO VRF-lite is a nicer solution.

Thanks again Marcin.

I did test with the latest version,

Cisco IOS Software, C2951 Software (C2951-UNIVERSALK9-M), Version 15.3(3)M1, RELEASE SOFTWARE (fc1)

with similar results.

The "route-via mandatory"  command does not work when a IPSEC protection policy is applied to a GRE tunnel.

Curiously enough, I thought i found a solution would would work.  In the example above, applied the following LOCAL policy statement.  It works for traceroutes, pings, telnet, etc.  It even appears to work for IKE !!  All my sessions went to QM_IDLE.   I was temporarily elated.

For SITE A:

ip local policy route-map INTERNET_OUT permit 10

match ip access-list X_SOURCE

set next hop 10.1.0.1

ip local policy route-map INTERNET_OUT permit 20

match ip access-list Y_SOURCE

set next hop 10.2.0.1

ip access-list standard X_SOURCE

  permit 10.1.0.2

ip access-list standard Y_SOURCE

  permit 10.2.0.2

In this way, it does not matter which source address the router uses to form the packets - they get sent out the correct interface.

BUT

IPSEC is not working.  I get encaps only on one end - im not sure what is happening to them - but i believe my applying the local policy, i have somehow hosed the crypto ipsec function.

The first 5 pings succeed before the DMVPN brings up the spoke-to-spoke IKE session.  As soon as the IKEsession is up and the IPSEC SA is formed, the traffic dies.  Now instead of traffic bouncing off the head-end, it does not reach.

f i add a static route for the remote spoke's public ip, it picks up and starts working immediately.  crazy thing is that this shouldnt matter - it doesnt change the path the packets are taking, right?

Also - the FVRF method is starting to sound reasonable, but I really dont understand how i would do failover and interaction with the inner VRF.   If I was running a default in the inner-VRF and bring back all traffic to the headend, I would do this in a heartbeat, but i have extensive interaction with the internet locally at this site.  So, i would do the following:

interface gi0/1  (internet A)

   ip vrf forwarding vpnX

interface gi0/2 (internet B)

  ip vrf forwarding vpnY

int tunnel X

  tunnel vrf vpnX

int tunnel T

  tunnel vrf vpnY

ip route vrf vpnX 0.0.0.0 0.0.0.0 [x-gateway]

ip route vrf vpnY 0.0.0.0 0.0.0.0 [y-gateway]

I am using certificate based DMVPN, but i would guess that I could get this to work for establishing spoke-to-spoke between sites, but what about my traffic that has to go from the global-vrf (inner) to the internet on these 2?

I have implemented route-maps for NAT, static inbound NATs, tracking slas, etc.  Can you point me in the direction of doing inter-VRF interaction for NAT, etc?  I was under the impression that this would not be possible without using extra interfaces on my routers and switches to create a convoluted mess

Thanks!

Thanks all!  I decided to power through with the VRF solution and solve my NAT / border issues with brute force trial and error.

Luckily I also found this thread which was helpful:

https://supportforums.cisco.com/thread/2199101

I created 2 INTERNET vrfs and a LOCAL VRF.  I then use the vrf import facility to leak the default routes between them.  Initially, i was trying to avoid the "extra" local VRF and use the vrf export facility to leak the defaults to global.  I could not get a working NVI setup with this.

Use of the "tunnel vrf " command on the DMVPN tunnels works the same as the expected behavior of "route-via"

The key was understanding the NVI for the nat piece.  I had been trying to brute force it with domain-based NAT.

if anyone is interested in my full config, please let me know.  Thanks again for the help.

Hi Jerecassidy,

If you would like to share the results, I'm interested to see how the actual solution looks like.

Thanks,

Wesley

benterry3
Level 1
Level 1

Hello jerecassidy, if you wouldn't mind I'd really like to take a look at the configs you settled upon.

 

Thanks in advance.

 

-Ben

Getting Started

Find answers to your questions by entering keywords or phrases in the Search bar above. New here? Use these resources to familiarize yourself with the community: