STRANGE PROBLEM WITH OSPF

[qualxarnu]

Dear Community,

I would like to to ask you for a little help or hint regarding the strange issue which I have with the OSPF protocol.
Below I will try to explain how the network topology looks like.

I have few locations which in most cases have two routers. Few of them have only one router.
In main location we have two routers which are DMVPN hubs.
Each router in HQ is a hub for own DMVPN cloud and in branches each router is connected only to one DMVPN hub.
Exception are location with one router which have DMVPN connection to both hubs.

Then each location have also a multilayer switch which has SVI interfaces for specific location.

Now the OSPF is configured in a way that all routers using DMVPN are in area 0 and interface towards the multilayer switch is a normal area. Each location has own area number.

Now the problem is that some multilayer switches doesn't install routes to the routing tables from other areas and I'm not sure what can be the problem.

From things which I checked are:
- all DMVPN routers are in OSPF FULL state with particular DMVPN hub (branch router 1 is in FULL state with hub1, router 2 with hub 2)
- in specific branch location both routers are in FULL state with each other and with multilayer switch
- network type for tunnel interfaces is set to point-multipoint
- all routers in area 0 has information about all subnets in the environment
- databases on multilayer switches however looks kind of suspicious for me.

I aslo tried in two locations to switch from EIGRP to OSPF which should replace the first one, and first branch which has one DMVPN router and multilayer switch it worked like a charm.
But other two locations with two DMVPN routers and multilayer switches didn't go so well.
The problem was that not all routes were installed to the routing table by OSPF o multilayer switches.

What is interesting,both locations have the same type of a switch, the same software version and the same license level (for the first time it was ipbase, then increasedto ipservices - same result).
But for some reason one switch dosent install all routes.
Configurations looks the same I think, but maybe I missed something and maybe there is some reason why it's happening like that.

Can someone give a hint what will be the good/fastest way to check/debug why multilayer switch doesn't install all routes?
I will be really grateful.

[MHM Cisco World]

- network type for tunnel interfaces is set to point-multipoint <<- why you not make it broadcast, if you decide to make network type broadcast then make sure that the Hub is the DR of DMVPN. 

if you can please share the config of hub and one spoke 

[qualxarnu]

I've changed the network type, but it didn't change anything.
I'll share config and topology in a separate reply.

[MHM Cisco World]

Did you check that hub is now DR for this domain?

[qualxarnu]

Yes, hub1 became DR and hub 2 became BDR, but it's still doesn't solve the issue.

[MHM Cisco World]

Yes I see, 
I already run lab for dual Hub's and OSPF I failed 
but I think the solution is using Dual Cloud not Single Cloud
DMVPN Dual Hub using OSPF (juantron.wixsite.com)

[Joseph W. Doherty]

If hub 1 (10.100.1.0/24) and hub 2 (10.100.2.0/24) don't share a network, they shouldn't be DR and BDR (for each other).  Each hub should (ideally) be DR for its DMVPN network.

[Joseph W. Doherty]

Digging into this a little deeper (NB: my DMVPN/OSPF hands-on is out-of-date; I had last used it when Phase II was "real soon to brand new" i.e. I've never use Phase II or III), it appears there's slightly different OSPF recommended setups depending on whether your doing Phase I, II or III.

One recommendation (which I'm unsure only applies to a specific Phase), often is, setting OSPF priority to zero on all spoke tunnel interfaces.  That doesn't appears to have been done in your posted configuration data.

Also, there's a bit of a difference between dual hubs, same DMVPN cloud, and dual DMVPN clouds, each with a single hub.  (Yours I believe, as you've described [including in the OP], is the latter.)

I mention the above, to help insure "we're all on the same page".

I suspect the provided posted configuration information has been "edited".  If so, we may need to see the actual (relevant) configuration statements of a hub, spoke single VPN router, spoke dual VPN routers and L3 switch config for the prior two cases.

Also, what are the actual devices, IOS version, and feature licenses, being used?

What DMVPN Phase approach are you using (or hope to)?

[Joseph W. Doherty]

Oh, just noticed a OSPF difference, that could possibly influence what routes you see on a spoke L3 switch.

On the spoke single VPN router, cost to your spoke is same, as both DMVPN share the same port to the L3 switch.

On the spoke with dual VPN routers, cost to your spoke is different, as your secondary VPN router has a 10K interface cost to L3 switch.

[MHM Cisco World]

I run lab and full success, 
please check the attachment photo, 
if you need more detail or you want something specific in lab please ask. 
thanks 
MHM 

[qualxarnu]

Thanks for this checking, however I'm affraid that this lab topology is too simple.
The thing is that in my production network this OSPF works somehow partially, which means that some subnets are visible on the multilayer switches, but some not.
I had to turno off the OSPF on production right now, but I recall that DMVPN routers were fine in case of learnig routes, but the problem was rather in having these routes on the multilayer switches.

Maybe I will have to build more compliated lab environment by myself, but I can only do it in the next weekend

[MHM Cisco World]

OK let me guess, the Spoke with area 6 which you face issue with it ??

[MHM Cisco World]

the idea with dual hub and why we need dual cloud is simple 
if you use one cloud then Spoke must connect to both hub in same time, and hub must interconnect to each other. 
this need if we run ospf network broadcast 

it need to make sure that DR can talk to BDR and both can talk to OSPF other neighbor. 
so to solve the issue that some spoke connect to one hub not to both and to solve issue of interconnect hubs we need dual cloud 
one for each hub and by make hub is DR then we sure that Spoke prefix is learn between hub and spokes and Spoke-to-Spoke

[Joseph W. Doherty]

Some questions - just to insure nothing overlooked . . .

DMVPN hub routers DO have (same) area 0 connectivity to each other?

When you upgraded switch(es) from IPBase to IPServices, was reboot done?

You're not doing any address summarization on ABRs?

Each branch has a unique area number?

Each branch area is an "ordinary" OSPF area, i.e. not a stub, etc.?

You describe when you have two VPN routers at a site, they have their own area 0 connection.  This is done how?

Is you intention to do direct spoke-to-spoke traffic, or spoke-to-hub-to-spoke?

(about) How many branches?

When there is but one VPN router, hosting both DMVPNs, all good, but when there are two VPN routers, each hosting one of the DMVPNs, that's when L3 switches sometimes missing routes for other areas, correct?

[qualxarnu]

Answers:
- DMVPN hub routers do have the same area 0 on the tunnel interfaces. Interfaces facing multilayer switch are asisgned to area 1 (normal area). However in some tests I also tried to assign these interfaces also to area 0 and SVI of the switch to area 0 to extend it a little, but it didn't change anything.

- Yes, when I did upgrade from IPbase to IPservices, there was an upgrade.

- No, I'm not doing any address summarization on ABRs

- Yes, each branch has unique area number and I have around 15 normal areas (not stub, or any other types,. Just normal).

- If we have 2 branch routers, then tunnel interface of the first router points to the primary DMVPN hub, while secondary router points to the secondary hub. Primary router doesn't have connection with secodnary hub via DMVPN as wel as seondary spoke doesn't have connection via DMVPN with primary hub. However let's say the DMVPN space is area 0 for all routers.

- The intention is to use opportunity which DMVPN gives, so to have dynamic VPN tunnels between spoke routers, so to have direct connection to spoke without going all the time via hub and then to spoke.

- I have 15 branches and for now I've tested 3 locations where 2 have 2 spoke routers while 1 have only one spoke router.
So in case of 2 spoke routers if I start to use OSPF, then not all routes land in the routing table and we can say that the connection doesn't work.
However in case of router which have connection with both hubs, then everything works fine.

Simplified topology looks like this: