DMVPN NRHP Claimed NBMA IP conflicting with Routing table | DMVPN expert only

lap · ‎09-07-2011

Hi all,

Today I have experimented an issue at our customer running DMVPN. I have solved it but I would to understand more in details.

the setup is the following:

HUB DMVPN router-------INTERNET------ISP-NAT-DEVICE------SPOKE DMVPN router

So basically the SPOKE DMVPN router was building the DMVPN to the HUB but it the tunnel was immediately teared down.

The spoke router IP was 192.168.1.30 which was seen as claimed NBMA IP on the HUB by doing sh ip nhrp.

But I could see that there was a route on the HUB router to 192.168.1.0 already.

So what I did it change the IP on the spoke router to 192.168.20.30 and everything was working.

So I don´t understand why having an NBMA IP on the spoke which was present on the HUB routing table could prevent the DMVPN from forming?

I miss something somewhere.

Any ideas?

Regards,

Laurent

Marcin Latosiewicz · ‎09-07-2011

Laurent,

In the absence of DMVPN experts you can get me :-)

I'm curious to give this a try in the lab, it might be as simple as trying to understand what comes first - routing lookup or NHRP and if the packet is going to be routed out the wrong interface.

Care to share with me the software versions involved?

Marcin

lap · ‎09-07-2011

Hi Marcin,

IOS is on spoke platform c1812: c181x-advipservicesk9-mz.124-15.T15.bin.

Thanks for helping.

What I did to make it work is change the IP of F0 (outside interface connected to ISP-NAT-DEVICE)

on the spoke to 192.168.20.34 instead of 192.168.1.34 and it was working.

I can see that on the spoke there was 3 entry for 192.168.1.0/24, one directly connected, 192.168.1.0/25 learned via HUB through OSPF and the last 192.168.1.128/30 learned via HUB through OSPF.

I would like to understand the process here. For example GRE is first encapsulating OSPF multicast packets with source IP tunnel 0 X and destination 224.0.0.5 then what is happening? Was the GRE packet sent through the tunnel instead of F0 on the spoke as the router will route based on the more specific prefix match?

That is maybe why tunnel was going up and down every second as first there was only one entry in the rib for

192.168.1.0/24 and then when the DMVPN was up the spoke was receiving the routing prefixes from the HUB but then the router was routing IPsec (containing) GRE packets through DMVPN instead of F0

(outside interface connected to ISP-NAT-DEVICE)? What do you think?

Best regards,

Laurent

Marcin Latosiewicz · ‎09-08-2011

Laurent,

(please note that this is before I started anything)

On NHRP level there should be no problem over the tunnel.

What I think might have happened is that we learned the IP address of default route over DMVPN. I.e. you have a default route via 192.168.1.1 (possibly?) if you recive a more specific route for 192.168.1.1 than your directly connected 192.168.1.0/24 (and we recived 192.168.1.0/25) we assume that 192.168.1.1 is available over whoever presented us with 192.168.1.0/25 (in this case the tunnel interface).

I'll check it in the lab thoroughly, but obviously will need some time.

Marcin

Marcin Latosiewicz · ‎09-08-2011

Laurent,

This is after introducing a more specific prefix.

Spoke#

*Sep 8 09:08:43.143: %OSPF-5-ADJCHG: Process 1, Nbr 21.24.21.34 on Tunnel0 from FULL to DOWN, Neighbor Down: Dead timer expired

Spoke#

*Sep 8 09:08:52.123: %OSPF-5-ADJCHG: Process 1, Nbr 21.24.21.34 on Tunnel0 from LOADING to FULL, Loading Done

Spoke#

*Sep 8 09:09:37.803: %OSPF-5-ADJCHG: Process 1, Nbr 21.24.21.34 on Tunnel0 from FULL to DOWN, Neighbor Down: Dead timer expired

Spoke#

*Sep 8 09:09:45.231: %OSPF-5-ADJCHG: Process 1, Nbr 21.24.21.34 on Tunnel0 from LOADING to FULL, Loading Done

Spoke#

If you check how the CEF adj should look like:

Spoke#sh ip cef 11.1.1.1 in

0.0.0.0/0, version 14, epoch 0, cached adjacency 192.168.1.1

0 packets, 0 bytes

via 192.168.1.1, 0 dependencies, recursive

next hop 192.168.1.1, Ethernet0/0 via 192.168.1.1/32

valid cached adjacency

edit:

debugs showing this behavior (debugs taken on 12.4(15)T14 )

*Sep 8 10:28:06.939: %OSPF-5-ADJCHG: Process 1, Nbr 212.244.251.34 on Tunnel0 from FULL to DOWN, Neighbor Down: Dead timer expired

Spoke#

*Sep 8 10:28:12.439: RT: del 192.168.1.0/25 via 172.16.1.1, ospf metric [110/11121]

*Sep 8 10:28:12.439: RT: delete subnet route to 192.168.1.0/25

*Sep 8 10:28:12.439: RT: NET-RED 192.168.1.0/25

Spoke#

*Sep 8 10:28:15.139: %OSPF-5-ADJCHG: Process 1, Nbr 212.244.251.34 on Tunnel0 from LOADING to FULL, Loading Done

Spoke#

*Sep 8 10:28:22.439: RT: network 192.168.1.0 is now variably masked

*Sep 8 10:28:22.439: RT: add 192.168.1.0/25 via 172.16.1.1, ospf metric [110/11121]

*Sep 8 10:28:22.439: RT: NET-RED 192.168.1.0/25

Spoke#

*Sep 8 10:28:56.727: IP-Static: 0.0.0.0 0.0.0.0 192.168.1.1 Path = 2 3 5 7, route table no change, recursive flag clear

*Sep 8 10:28:56.727: RT: NET-RED 0.0.0.0/0

Additional logging on 15.1.4M1

*Sep 8 10:42:31.819: %OSPF-5-ADJCHG: Process 1, Nbr 212.244.251.34 on Tunnel0 from LOADING to FULL, Loading Done

Spoke>

*Sep 8 10:42:44.439: %ADJ-5-PARENT: Midchain parent maintenance for IP midchain out of Tunnel0, addr 172.16.1.1 - looped chain attempting to stack

Marcin

lap · ‎09-09-2011

Excellent Marcin,

I did also a test in a lab:

Issue description

**************

At first look the DMVPN tunnel was going up and down every 5-10 seconds. The routing information appeared and disappeared from the routing table in the SPOKE router.

Cisco Crawley WAN IP was 192.168.1.X /24.

OSPF has a dead timer of 40 seconds (4 times hello). So in the log of the router we could see the following every 40 seconds:

%OSPF-5-ADJCHG: Process 100, Nbr 10.128.253.1 on Tunnel0 from FULL to DOWN, Neighbor Down: Dead timer expired

Issue explanation

**************

The issue appeared when the DMVPN came up and the SPOKE router router was receiving the full routing table from HUB router. In this routing table there was 1 OSPF prefix that was causing the issue and it was: 192.168.1.0/25. As the default route at spoke location was 192.168.1.1. the SPOKE router was then confused how to route packets received from HUB router.

Before receiving the routing table everything works fine on the SPOKE router (DMVPN is UP and stable):

s= 80.10.10.1 [HUB-Public IP] (FastEthernet0), d=192.168.1.X (FastEthernet0), routed via RIB

Routing table:

C 192.168.1.0/24 is directly connected, FastEthernet0/0

S* 0.0.0.0/0 [254/0] via 192.168.1.1

After receiving the routing table form HUB Denmark there was a routing loop because the SPOKE router routed the IPsec packets received form the DMVPN HUB towards Tunnel 0 instead of FastEthernet0 because 192.168.1.0/25 is more specific than 192.168.1.0/24 so the SPOKE never received OSPF packets (routing info) from DMVPN HUB router and after 40 seconds the routing information disappear from the routing table:

After receiving the routing table :

s=80.10.10.1 (FastEthernet0/0), d=192.168.1.2 (Tunnel0), routed via RIB

Routing table:

192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks

O 192.168.1.0/25 [110/101] via 10.128.253.1, 00:00:13, Tunnel0 [more specific route]

C 192.168.1.0/24 is directly connected, FastEthernet0/0

S* 0.0.0.0/0 [254/0] via 192.168.1.1

Solution:

********

To solve this issue I changed the net in SPOKE location y from 192.168.1.0/24 to 192.168.20.0/24 and the problem was gone.

********************************************************************************************************************************’

But Marcin, the thing I don´t understand is why packets send by SPOKE are not routed out Tunnel0 ? see a debug IP packets on SPOKE when OSPF info are in SPOKE routing table:

Routing table:

192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks

O 192.168.1.0/25 [110/101] via 10.128.253.1, 00:00:13, Tunnel0 [more specific route]

C 192.168.1.0/24 is directly connected, FastEthernet0/0

S* 0.0.0.0/0 [254/0] via 192.168.1.1

SPOKE1#

*Mar 1 00:09:31.691: IP: tableid=0, s=80.10.10.1 (FastEthernet0/0), d=192.168.1.2 (Tunnel0), routed via RIB [only received packets are routed wrongly to Tunnel0 instead for F0/0]

*Mar 1 00:09:31.695: IP: s=80.10.10.1 (FastEthernet0/0), d=192.168.1.2 (Tunnel0), g=10.128.253.1, len 108, forward

*Mar 1 00:09:31.703: IP: tableid=0, s=192.168.1.2 (local), d=80.10.10.1 (FastEthernet0/0), routed via FIB [But send packets are routed right, not to Tunnel0. Should they not be routed to Tunnel0 like received packets?]

*Mar 1 00:09:31.703: IP: s=192.168.1.2 (local), d=80.10.10.1 (FastEthernet0/0), len 136, sending

Best regards!

Marcin Latosiewicz · ‎09-12-2011

Laurent,

Sorry missed this thread in recent floof of emails.

The problem is that once you learn 192.168.1.0/25 the default route is changing from:

everything without a specific match via 192.168.1.1 via directly known interface fa0/0 to:

everything ... via 192.168.1.1 via 192.168.1.0/25 which was learned via tunnel0.

This of course causes problems since for all traffic that wants to get to the internet (note that traffic excused from CEF might be working differently) we are trying to send out through tunnel0, BUT we don't know how to route traffic once it's encapsulated GRE/IPsec (since it's potiing out to through the tunnel itself).

This is causing the mid-chain messages on newer IOS software.

M.

lap · ‎09-13-2011

Hi Marcin,

What I meant in my previous post is that although there is a more specific route in the routing table in the spoke, Packet from SPOKE to HUB are still going through. Please see attached a Wireshark dump when pinging from SPOKE to HUB tunnel IP when more specific prefix /25 is in the routing table.

As you can see the packets are going through and reach the HUB which is stange right? As the packets from SPOKE should be encapsulated into GRE than a routing lookup occur on WAN HUB IP which is going through SPOKE tunnel and so on. But that is not the case here. Packets are reaching the HUB still but packet send back to SPOKE are not processed correctly and are loop. See my previous post.

I am a bit comfused;-)

Marcin Latosiewicz · ‎09-14-2011

Laurent,

Indeed that's why I mentioned that situation is different for packet to and from the box (i.e. not CEF switched) and CEF switched.

For me in the lab I never see that more specific entry installed (as seen by "show ip route") although the debugs for ip routing are showing it.

I would need to have a look further in my lab, but lack the time.

(edit: if time allows me I will try before Friday but I cannot commit)

Marcin

lap · ‎10-21-2011

Hi Marcin,

Thanks for your help.

Best regards,

Laurent