DMVPN Dynamic Spoke Issue

[SajeshB]

We are having a DMVPN connectivity(Attached NW Diagram). Dynamic spoke-to-spoke communication between 2 and 3 is having an issue but the underlay NBMA IP is having recahability and also other connectivity is working fine and as expected between spoke 1 and 3.
As a workaround after creating ipsla between spoke 2 and 3 the tunnel is Up with attribute Static.
I need help to identify the issue or, if someone can help guide me with troubleshooting or debugging steps so that I can investigate this further.

During Issue:
192.168.2.115 10.134.131.117 UP 00:02:53 I2 10.134.131.117/32

After creating IPSLA:
1 192.168.2.117 10.134.131.117 UP 10:58:11 S

HUB1: NBMA IP 192.168.2.115
HUB2: 192.168.2.119
SPOKE 1: 192.168.2.109
SPOKE 2: 192.168.2.117
SPOKE 3: 192.168.2.119

While sharing this output, i noticed that the issue logs pasted above is showing 192.168.2.115 IP which is HUB1, so just wanted to know whether spoke 2 advertsiement should learn via HUB2 192.168.2.119

[balaji.bandi]

Is the HUB to HUB have connectivity ?

Spoke 1 and Spoke 2 working?

basic design all spokes need to register with HUB, and HUB to HUB have a connection.

In some case Spoke have a dual home connection like Spoke connect o HUB1 and HUB 2

as per the diagram, you looking at spoke2 is a transit point...(personally that is wrong design)

Can you share the config ?

[MHM Cisco World]

I need to see the Spoke config 

[SajeshB]

I have attached in reply

[SajeshB]

HI @balaji.bandi ,

Yes there is HUB to HUB connectivity.

Spoke 1 and 2 is working.

chcrt8001ccci#pi vr CCCI 10.134.131.109
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.134.131.109, timeout is 2 seconds:
.!!!!

1 192.168.2.109 10.134.131.109 UP 00:00:05 D

Spoke 1/2/3 is not a transit connectivity for any HUB to HUB or for any spoke to spoke device we have different underlay backbone connectivity for that.

I have attached the config, it will be really helpful if you look into that and see if there is any incorrect config or issue in that.

[MHM Cisco World]

in Spoke 1 can you share 
show ip ospf neighbor ??

[SajeshB]

Spoke 1 Ouptut:

lonrt8001ccci#sh ip osp ne

Neighbor ID Pri State Dead Time Address Interface
10.134.131.110 0 FULL/ - 00:00:39 10.51.145.70 GigabitEthernet0/0/3.2152
10.134.130.115 100 FULL/DR 918 msec 10.134.131.115 Tunnel801
10.134.131.119 45 FULL/BDR 926 msec 10.134.131.119 Tunnel801
10.134.131.116 100 FULL/DR 936 msec 10.134.130.116 Tunnel802
10.134.131.120 45 FULL/BDR 848 msec 10.134.130.120 Tunnel802
10.116.5.208 0 FULL/ - 00:00:31 10.51.145.74 GigabitEthernet0/0/4.459
lonrt8001ccci#

Just to update the issue is between Spoke 2 and 3 and also currently as a workaround IPSLA is configured in spoke 3  and addition to that nhs is configured in Spoke 3 for Spoke2 NBMA IP

[MHM Cisco World]

sorry I want show ip ospf neighbor in Spoke 3 not Spoke 1

[SajeshB]

Spoke 3 Output:

chcrt7301gdn#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.51.234.228 100 FULL/DR 978 msec 10.51.232.99 Tunnel1044
10.51.234.227 100 FULL/DR 870 msec 10.51.232.67 Tunnel1034
10.49.23.4 100 FULL/DR 920 msec 10.51.232.34 Tunnel1024
10.49.23.2 100 FULL/DR 838 msec 10.51.232.1 Tunnel1014
10.62.5.95 0 2WAY/DROTHER 968 msec 10.134.131.117 Tunnel801
10.134.130.115 100 FULL/DR 936 msec 10.134.131.115 Tunnel801
10.134.131.119 45 FULL/BDR 848 msec 10.134.131.119 Tunnel801

[MHM Cisco World]

10.62.5.95 0 2WAY/DROTHER 968 msec 10.134.131.117 Tunnel801 <<- that what we looking for, the OSPF not full establish between Spoke3 and Spoke2.

I think that Spoke3 use different Hub than Spoke2 and hence the ospf and reach issue.  

check the tracroute from both side, see fist hop which one ?

[SajeshB]

Thankyou, So Could this be an issue ? and will you please let me know how can i make this neighbor full establish.

Also traceroute to Tunnel IP or the multicast IP ? I checked trace for tunnel IP it is not going on 1st hop for any of the spoke but i can do E2E ping between spoke1/2/3

Also looking at Spoke 1 and 3 which is working fine i dont see any neigbor establishment between them.

Spoke 1 output:
10.134.131.110 0 FULL/ - 00:00:37 10.51.145.70 GigabitEthernet0/0/3.2152
10.134.130.115 100 FULL/DR 987 msec 10.134.131.115 Tunnel801
10.134.131.119 45 FULL/BDR 925 msec 10.134.131.119 Tunnel801
10.134.131.116 100 FULL/DR 855 msec 10.134.130.116 Tunnel802
10.134.131.120 45 FULL/BDR 817 msec 10.134.130.120 Tunnel802
10.116.5.208 0 FULL/ - 00:00:39 10.51.145.74 GigabitEthernet0/0/4.459

[MHM Cisco World]

sorry for late reply I will run lab and share result with you soon 

[Thomas Schmitt]

I don’t know any reason for your dmvpn design with just one tunnel interface, and wirhout any priorities for nhrp; the behavior isn’t deterministic in this case.

Anyway, your problem is the third NHS on Spoke3, that is unknown on other Spokes and unfortunately for you, spoke3 use that NHS

so just remove it or add to another spoke router