topic Re: STRANGE PROBLEM WITH OSPF in Routing and SD-WAN

STRANGE PROBLEM WITH OSPF

qualxarnu — Sun, 12 Feb 2023 19:01:34 GMT

Dear Community,

I would like to to ask you for a little help or hint regarding the strange issue which I have with the OSPF protocol.
Below I will try to explain how the network topology looks like.

I have few locations which in most cases have two routers. Few of them have only one router.
In main location we have two routers which are DMVPN hubs.
Each router in HQ is a hub for own DMVPN cloud and in branches each router is connected only to one DMVPN hub.
Exception are location with one router which have DMVPN connection to both hubs.

Then each location have also a multilayer switch which has SVI interfaces for specific location.

Now the OSPF is configured in a way that all routers using DMVPN are in area 0 and interface towards the multilayer switch is a normal area. Each location has own area number.

Now the problem is that some multilayer switches doesn't install routes to the routing tables from other areas and I'm not sure what can be the problem.

From things which I checked are:
- all DMVPN routers are in OSPF FULL state with particular DMVPN hub (branch router 1 is in FULL state with hub1, router 2 with hub 2)
- in specific branch location both routers are in FULL state with each other and with multilayer switch
- network type for tunnel interfaces is set to point-multipoint
- all routers in area 0 has information about all subnets in the environment
- databases on multilayer switches however looks kind of suspicious for me.

I aslo tried in two locations to switch from EIGRP to OSPF which should replace the first one, and first branch which has one DMVPN router and multilayer switch it worked like a charm.
But other two locations with two DMVPN routers and multilayer switches didn't go so well.
The problem was that not all routes were installed to the routing table by OSPF o multilayer switches.

What is interesting,both locations have the same type of a switch, the same software version and the same license level (for the first time it was ipbase, then increasedto ipservices - same result).
But for some reason one switch dosent install all routes.
Configurations looks the same I think, but maybe I missed something and maybe there is some reason why it's happening like that.

Can someone give a hint what will be the good/fastest way to check/debug why multilayer switch doesn't install all routes?
I will be really grateful.

Re: STRANGE PROBLEM WITH OSPF

MHM Cisco World — Sun, 12 Feb 2023 19:27:04 GMT

- network type for tunnel interfaces is set to point-multipoint <<- why you not make it broadcast, if you decide to make network type broadcast then make sure that the Hub is the DR of DMVPN.

if you can please share the config of hub and one spoke

Re: STRANGE PROBLEM WITH OSPF

Joseph W. Doherty — Sun, 12 Feb 2023 21:02:08 GMT

Some questions - just to insure nothing overlooked . . .

DMVPN hub routers DO have (same) area 0 connectivity to each other?

When you upgraded switch(es) from IPBase to IPServices, was reboot done?

You're not doing any address summarization on ABRs?

Each branch has a unique area number?

Each branch area is an "ordinary" OSPF area, i.e. not a stub, etc.?

You describe when you have two VPN routers at a site, they have their own area 0 connection. This is done how?

Is you intention to do direct spoke-to-spoke traffic, or spoke-to-hub-to-spoke?

(about) How many branches?

When there is but one VPN router, hosting both DMVPNs, all good, but when there are two VPN routers, each hosting one of the DMVPNs, that's when L3 switches sometimes missing routes for other areas, correct?

Re: STRANGE PROBLEM WITH OSPF

David Ruess — Fri, 19 Jan 2024 11:11:49 GMT

Hello,

Can you share your configurations and diagram with us. Doesn't have to be the whole thing just a sample of 1 working one and non-working site since you said only some of the L3 switches don't have all the routes.

Some show commands could also be helpful:

show ip ospf int brief

show ip ospf nieghbor

Things to check also.

Make sure the same RID is not being used on more than 1 device. The router/switch will ignore LSUs from its own RUD so if they are conflicting it wont learn the routes

Another thing to check is to make sure your areas are configured correctly. Discontiguous areas can be a problem especially in a DMVPN network

And finally are there any other routing protocols in use that would exclude OSPF form being installed in the routing table such as EIGRP or external BGP?

-David

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Sun, 12 Feb 2023 22:26:33 GMT

I've changed the network type, but it didn't change anything.
I'll share config and topology in a separate reply.

Re: STRANGE PROBLEM WITH OSPF

MHM Cisco World — Sun, 12 Feb 2023 22:33:30 GMT

Did you check that hub is now DR for this domain?

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Sun, 12 Feb 2023 22:50:26 GMT

Answers:
- DMVPN hub routers do have the same area 0 on the tunnel interfaces. Interfaces facing multilayer switch are asisgned to area 1 (normal area). However in some tests I also tried to assign these interfaces also to area 0 and SVI of the switch to area 0 to extend it a little, but it didn't change anything.

- Yes, when I did upgrade from IPbase to IPservices, there was an upgrade.

- No, I'm not doing any address summarization on ABRs

- Yes, each branch has unique area number and I have around 15 normal areas (not stub, or any other types,. Just normal).

- If we have 2 branch routers, then tunnel interface of the first router points to the primary DMVPN hub, while secondary router points to the secondary hub. Primary router doesn't have connection with secodnary hub via DMVPN as wel as seondary spoke doesn't have connection via DMVPN with primary hub. However let's say the DMVPN space is area 0 for all routers.

- The intention is to use opportunity which DMVPN gives, so to have dynamic VPN tunnels between spoke routers, so to have direct connection to spoke without going all the time via hub and then to spoke.

- I have 15 branches and for now I've tested 3 locations where 2 have 2 spoke routers while 1 have only one spoke router.
So in case of 2 spoke routers if I start to use OSPF, then not all routes land in the routing table and we can say that the connection doesn't work.
However in case of router which have connection with both hubs, then everything works fine.

Simplified topology looks like this:

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Fri, 19 Jan 2024 11:22:41 GMT

Simplified topology:

So the areas like area 2 doesn't work, areas like this with number 6 seems to be working. The last varation like area 15 is not checked yet.

Config for HUB1 router:
router ospf 10
router-id 10.100.0.1
priority 100
auto-cost reference-bandwidth 100000
nsf
area 0 authentication
area 1 authentication message-digest
redistribute static metric-type 1 subnets
redistribute bgp 65210 metric-type 1 subnets route-map BGP2OSPF
passive-interface default
no passive-interface GigabitEthernet0/0/0
no passive-interface Tunnel101
network 10.100.0.1 0.0.0.0 area 0
network 10.100.1.0 0.0.0.255 area 0
network 10.101.100.0 0.0.0.255 area 1
default-information originate metric 180 metric-type 1

interface Tunnel101
ip ospf authentication key-chain WAN-OSPF
ip ospf network broadcast
ip ospf priority 100
ip ospf bfd
ip ospf 10 area 0

interface GigabitEthernet0/0/0
ip ospf message-digest-key 1 md5 xxxxxxxxxxxxxx
ip ospf dead-interval 15
ip ospf hello-interval 5
ip ospf priority 100
ip ospf bfd
ip ospf 10 area 1

----------------------
WORKING SCENARIO
----------------------

SPOKE RTR:
router-id 10.100.0.16
auto-cost reference-bandwidth 100000
nsf
area 0 authentication
area 7 authentication
redistribute static metric-type 1 subnets
redistribute bgp 65228 metric-type 1 subnets route-map BGP2OSPF
passive-interface default
no passive-interface GigabitEthernet0/0/0
no passive-interface Tunnel101
no passive-interface Tunnel201
network 10.100.0.16 0.0.0.0 area 0
network 10.100.1.0 0.0.0.255 area 0
network 10.100.2.0 0.0.0.255 area 0
network 10.107.100.0 0.0.0.255 area 7

interface Tunnel101
ip ospf authentication key-chain WAN-OSPF
ip ospf network broadcast
ip ospf bfd
ip ospf 10 area 0

interface Tunnel201
ip ospf authentication key-chain WAN2-OSPF
ip ospf network broadcast
ip ospf bfd
ip ospf 10 area 0
ip ospf cost 10000

interface GigabitEthernet0/0/0
ip ospf authentication key-chain LAN-OSPF
ip ospf dead-interval 15
ip ospf hello-interval 5
ip ospf priority 50
ip ospf bfd
ip ospf 10 area 7


show ip ospf nieghbor:
Neighbor ID Pri State Dead Time Address Interface
10.100.0.2 90 FULL/DR 00:00:35 10.100.2.1 Tunnel201
10.100.0.1 100 FULL/DR 00:00:39 10.100.1.1 Tunnel101
10.107.100.4 1 FULL/BDR 00:00:13 10.107.100.4 GigabitEthernet0/0/0

show ip ospf int brief:
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Lo1 10 0 10.100.0.16/32 1 LOOP 0/0
Tu201 10 0 10.100.2.8/24 10000 BDR 1/1
Tu101 10 0 10.100.1.8/24 3333 BDR 1/1
Gi0/0/0 10 7 10.107.100.2/24 100 DR 1/1


MULTILAYER SWITCH:
router ospf 10
router-id 10.107.100.4
auto-cost reference-bandwidth 100000
nsf
area 7 authentication
passive-interface default
no passive-interface Vlan100
network 10.107.0.0 0.0.255.255 area 7
network 192.168.208.0 0.0.0.255 area 7
network 192.168.228.0 0.0.0.255 area 7

interface Vlan100
ip ospf authentication key-chain LAN-OSPF
ip ospf dead-interval 15
ip ospf hello-interval 5
ip ospf bfd
ip ospf 10 area 7


show ip ospf nieghbor:
Neighbor ID Pri State Dead Time Address Interface
10.100.0.16 50 FULL/DR 00:00:11 10.107.100.2 Vlan100

show ip ospf int brief:
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Vl228 10 7 192.168.228.5/24 100 DR 0/0
Vl208 10 7 192.168.208.1/24 100 DR 0/0
Vl100 10 7 10.107.100.4/24 100 BDR 1/1


----------------------
NOT WORKING SCENARIO
----------------------

SPOKE RTR1:
router ospf 10
router-id 10.100.0.22
auto-cost reference-bandwidth 100000
nsf
area 0 authentication
area 10 authentication
redistribute bgp 65224 metric-type 1 subnets route-map BGP2OSPF
passive-interface default
no passive-interface GigabitEthernet0/0/0
no passive-interface Tunnel101
network 10.100.0.22 0.0.0.0 area 0
network 10.100.1.0 0.0.0.255 area 0
network 10.110.100.0 0.0.0.255 area 10

interface Tunnel101
ip ospf authentication key-chain WAN-OSPF
ip ospf network broadcast
ip ospf bfd
ip ospf 10 area 0

interface GigabitEthernet0/0/0
ip ospf authentication key-chain LAN-OSPF
ip ospf dead-interval 15
ip ospf hello-interval 5
ip ospf priority 100
ip ospf bfd
ip ospf 10 area 10


show ip ospf nieghbor:
Neighbor ID Pri State Dead Time Address Interface
10.100.0.1 100 FULL/DR 00:00:32 10.100.1.1 Tunnel101
10.100.0.23 50 FULL/BDR 00:00:12 10.110.100.3 GigabitEthernet0/0/0
10.110.100.4 1 FULL/DROTHER 00:00:12 10.110.100.4 GigabitEthernet0/0/0


show ip ospf int brief:
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Lo1 10 0 10.100.0.22/32 1 LOOP 0/0
Tu101 10 0 10.100.1.11/24 10000 BDR 1/1
Gi0/0/0 10 10 10.110.100.2/24 100 DR 2/2

SPOKE RTR2:
router ospf 10
router-id 10.100.0.23
auto-cost reference-bandwidth 100000
nsf
area 0 authentication
area 10 authentication
redistribute bgp 65224 metric-type 1 subnets route-map BGP2OSPF
passive-interface default
no passive-interface GigabitEthernet0/0/0
no passive-interface Tunnel201
network 10.100.0.23 0.0.0.0 area 0
network 10.100.1.0 0.0.0.255 area 0
network 10.110.100.0 0.0.0.255 area 10

interface Tunnel201
ip ospf authentication key-chain WAN-OSPF
ip ospf network broadcast
ip ospf bfd
ip ospf 10 area 0
ip ospf cost 10000

interface GigabitEthernet0/0/0
ip ospf authentication key-chain LAN-OSPF
ip ospf dead-interval 15
ip ospf hello-interval 5
ip ospf priority 50
ip ospf bfd
ip ospf 10 area 10
ip ospf cost 10000


show ip ospf nieghbor:
Neighbor ID Pri State Dead Time Address Interface
10.100.0.2 90 FULL/DR 00:00:38 10.100.2.1 Tunnel201
10.100.0.22 100 FULL/DR 00:00:10 10.110.100.2 GigabitEthernet0/0/0
10.110.100.4 1 FULL/DROTHER 00:00:11 10.110.100.4 GigabitEthernet0/0/0


show ip ospf int brief:
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Lo1 10 0 10.100.0.23/32 1 LOOP 0/0
Tu201 10 0 10.100.2.11/24 10000 BDR 1/1
Gi0/0/0 10 10 10.110.100.3/24 10000 BDR 2/2


MULTILAYER SWITCH:
router ospf 10
router-id 10.110.100.4
auto-cost reference-bandwidth 100000
nsf
area 10 authentication
passive-interface default
no passive-interface Vlan100
network 10.110.0.0 0.0.255.255 area 10
network 192.168.224.0 0.0.0.255 area 10

interface Vlan100
ip ospf authentication key-chain LAN-OSPF
ip ospf dead-interval 15
ip ospf hello-interval 5
ip ospf bfd
ip ospf 10 area 10


show ip ospf nieghbor:
Neighbor ID Pri State Dead Time Address Interface
10.100.0.22 100 FULL/DR 00:00:11 10.110.100.2 Vlan100
10.100.0.23 50 FULL/BDR 00:00:11 10.110.100.3 Vlan100


show ip ospf int brief:
Interface PID Area IP Address/Mask Cost State Nbrs F/C
Vl224 10 10 192.168.224.1/24 100 DR 0/0
Vl100 10 10 10.110.100.4/24 100 DROTH 2/2

Regarding few more questions:

- RID should be unique for each router and multilayer switch using OSPF.
- I hope that areas are configured correctly. Each hub and spoke router has tunnel interface in area 0 while LAN interface in specific and unique area. Multilayer switches are only in normal areas (without area 0), but they are connected to these routers which have connection with area 0.
- in case of other protocols, yes there is BGP, but it's used only for private IP address scope which is redistributed to the OSPF. What is more, there is a EIGRP which I would like to replace with OSPF because later I would like to also use dynamic routing on on Cisco devices which are not talking via EIGRP. But in situation where I turn off EIGRP in one scenario OSPF works ok, but in other scenario communication is lost as not all routes are installed in routing table.

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Sun, 12 Feb 2023 23:25:59 GMT

To be honest I'm starting to wonder, if in this case if branch has 2 spoke routers, then maybe connection both spokes with both hubs will not solve the issue. However in case of EIGRP the current scenario works fine, so I'm wondering why......

Re: STRANGE PROBLEM WITH OSPF

MHM Cisco World — Sun, 12 Feb 2023 23:50:04 GMT

Freind you config ospf priority 100 in hub

That good only clear ip ospf process in not work sites and check again.

Re: STRANGE PROBLEM WITH OSPF

Joseph W. Doherty — Mon, 13 Feb 2023 00:44:30 GMT

Have to study your answers further, and the info your provided David, but in your "simplified" diagram, only the area 6 ABR precludes area 0 from being partitioned.

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Mon, 13 Feb 2023 07:23:57 GMT

Yes, hub1 became DR and hub 2 became BDR, but it's still doesn't solve the issue. 😞

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Mon, 13 Feb 2023 07:26:37 GMT

Regarding area 0 continuity, I did small test where I turned off OSPF on the HUB2 router.
So in this case only HUB1 was working, so something which can also happen due to the router failure.
But with only 1 DMVPN hub working it also didn't solve the issue.

Re: STRANGE PROBLEM WITH OSPF

MHM Cisco World — Mon, 13 Feb 2023 14:24:44 GMT

Yes I see,
I already run lab for dual Hub's and OSPF I failed
but I think the solution is using Dual Cloud not Single Cloud
DMVPN Dual Hub using OSPF (juantron.wixsite.com)

Re: STRANGE PROBLEM WITH OSPF

Joseph W. Doherty — Mon, 13 Feb 2023 14:52:58 GMT

If hub 1 (10.100.1.0/24) and hub 2 (10.100.2.0/24) don't share a network, they shouldn't be DR and BDR (for each other). Each hub should (ideally) be DR for its DMVPN network.

Re: STRANGE PROBLEM WITH OSPF

Joseph W. Doherty — Mon, 13 Feb 2023 17:35:32 GMT

Deactivation of a hub, wasn't something I suggested, as it should NOT partition area 0 as long as you have any other router like your area 6 example.

Even if you did partition your area 0, the way you're configured, I'm unsure you would create a problem.

Just for laughs, in a problem area, like one represented by area 2 in your diagram (i.e. two VPN routers), you might try creating secondary area 0 addresses on your VPN routers, on the same interfaces working with the branch area L3 switch (latter not to be given a similar secondary). This would be somewhat like your area 6 example, but with two branch routers rather that one.

I've still not yet (haven't had the time) to fully analyze the additional information you've provided. So far, haven't seen a obvious reason why you're having this problem. (To really analyze this, might need to dig into the LSA database.)

One of our great(est) contributors, @Peter Paluch, I believe, is very knowledgeable about many routing protocols, including OSPF. If he has the time, perhaps he'll join us.

Re: STRANGE PROBLEM WITH OSPF

MHM Cisco World — Mon, 13 Feb 2023 15:23:46 GMT

I run lab and full success,
please check the attachment photo,
if you need more detail or you want something specific in lab please ask.
thanks
MHM

Re: STRANGE PROBLEM WITH OSPF

qualxarnu — Mon, 13 Feb 2023 17:05:45 GMT

Thanks for this checking, however I'm affraid that this lab topology is too simple.
The thing is that in my production network this OSPF works somehow partially, which means that some subnets are visible on the multilayer switches, but some not.
I had to turno off the OSPF on production right now, but I recall that DMVPN routers were fine in case of learnig routes, but the problem was rather in having these routes on the multilayer switches.

Maybe I will have to build more compliated lab environment by myself, but I can only do it in the next weekend 😞

Re: STRANGE PROBLEM WITH OSPF

MHM Cisco World — Mon, 13 Feb 2023 17:37:01 GMT

OK let me guess, the Spoke with area 6 which you face issue with it ??

Re: STRANGE PROBLEM WITH OSPF

Joseph W. Doherty — Mon, 13 Feb 2023 18:41:11 GMT

Digging into this a little deeper (NB: my DMVPN/OSPF hands-on is out-of-date; I had last used it when Phase II was "real soon to brand new" i.e. I've never use Phase II or III), it appears there's slightly different OSPF recommended setups depending on whether your doing Phase I, II or III.

One recommendation (which I'm unsure only applies to a specific Phase), often is, setting OSPF priority to zero on all spoke tunnel interfaces. That doesn't appears to have been done in your posted configuration data.

Also, there's a bit of a difference between dual hubs, same DMVPN cloud, and dual DMVPN clouds, each with a single hub. (Yours I believe, as you've described [including in the OP], is the latter.)

I mention the above, to help insure "we're all on the same page".

I suspect the provided posted configuration information has been "edited". If so, we may need to see the actual (relevant) configuration statements of a hub, spoke single VPN router, spoke dual VPN routers and L3 switch config for the prior two cases.

Also, what are the actual devices, IOS version, and feature licenses, being used?

What DMVPN Phase approach are you using (or hope to)?