Everything that cofee pointed out is correct, but not the exact reason that the neighboring router is not reachable from a SPF point of view.
The neighboring routers, if one is configured for broadcast and one for point-to-point, will actually form a complete adjacency with each other and completely synchonize their LSDBs. The reason for this being the network type is not encoded into OSPF packets and does not have to match. Even though a DROTHER router on a broadcast network normally sends LSUs to 224.0.0.6 (the DR), the point-to-point end will send them to 224.0.0.5 (all OSPF routers) and the DR will still process these. The DR would send LSUs to the 224.0.0.5 group anyway so these are processed by the point-to-point side. Also, because the router configured as point-to-point will not set the DR address in its hellos it will never compete for the DR position and the broadcast router will always declare itself DR.
The reason the neighboring router and networks attached to that router actually end up being unreachable and not installed in the RIB is due to the discrepancy in the LSDB and SPF computation. The following occurs when the routers exchange LSAs and become adjacent. The broadcast router will create a router LSA that contains entries for all its directly attached networks including the mismatched link. Because the link on its side is set to broadcast mode it will create an entry in the LSA for it as a transit network with its own interface address as the DR and router interface address. This DR address in the LSA is a reference to the network LSA. Here is an example:
OSPF Router with ID (2.2.2.2) (Process ID 1)
Router Link States (Area 0)
LS age: 1091
Options: (No TOS-capability, DC)
LS Type: Router Links
Link State ID: 2.2.2.2
Advertising Router: 2.2.2.2
LS Seq Number: 80000003
Checksum: 0xC519
Length: 48
Number of Links: 2
Link connected to: a Stub Network
(Link ID) Network/subnet number: 2.2.2.2
(Link Data) Network Mask: 255.255.255.255
Number of TOS metrics: 0
TOS 0 Metrics: 1
Link connected to: a Transit Network
(Link ID) Designated Router address: 10.0.0.2
(Link Data) Router Interface address: 10.0.0.2
Number of TOS metrics: 0
TOS 0 Metrics: 10
It will also create a network LSA as it is the DR on the link. This network LSA will contain a list of the two attached routers RIDs along with a reference to the DR router's interface address in the LSID. Below is an example:
OSPF Router with ID (2.2.2.2) (Process ID 1)
Net Link States (Area 0)
Routing Bit Set on this LSA
LS age: 58
Options: (No TOS-capability, DC)
LS Type: Network Links
Link State ID: 10.0.0.2 (address of Designated Router)
Advertising Router: 2.2.2.2
LS Seq Number: 80000001
Checksum: 0x43D6
Length: 32
Network Mask: /24
Attached Router: 2.2.2.2
Attached Router: 1.1.1.1
All good up to this point. The problem occurs when the router configured for point-to-point creates its router LSA. This LSA will contain entries for all its directly attached networks including this link. The problem is how this link is listed in the LSA. Instead of being listed as a transit network, it will be listed as connected to another router (point-to-point) with the neighboring routers RID and interface address., and another entry as a stub network including the link prefix. Below is an example:
OSPF Router with ID (1.1.1.1) (Process ID 1)
Router Link States (Area 0)
LS age: 490
Options: (No TOS-capability, DC)
LS Type: Router Links
Link State ID: 1.1.1.1
Advertising Router: 1.1.1.1
LS Seq Number: 80000002
Checksum: 0xC027
Length: 60
Number of Links: 3
Link connected to: a Stub Network
(Link ID) Network/subnet number: 1.1.1.1
(Link Data) Network Mask: 255.255.255.255
Number of TOS metrics: 0
TOS 0 Metrics: 1
Link connected to: another Router (point-to-point)
(Link ID) Neighboring Router ID: 2.2.2.2
(Link Data) Router Interface address: 10.0.0.1
Number of TOS metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.0.0.0
(Link Data) Network Mask: 255.255.255.0
Number of TOS metrics: 0
TOS 0 Metrics: 1
Now even though the two routers have full link state information, when they go to compute SPF and rebuild the network from the LSDB the network will be partitioned on the link between the point-to-point router's router LSA and the pseudonode (network LSA). Here is a diagram representation:
Psuedonode (Transit Network)
To RID 1.1.1.1 LSA Type 2 To RID 2.2.2.2
R1 (point to point) IF 10.0.0.1 DR 10.0.0.2 IF 10.0.0.2 R2 (broadcast)
LSA Type 1 --------->X<--------- Subnet 10.0.0.0/24 ---------><--------- LSA Type 1
RID 1.1.1.1 f0 f0 ID 2.2.2.2
Stub 10.0.0.1/24 IF 10.0.0.2
To RID 2.2.2.2 Transit w/ DR 10.0.0.2
Cost 10 Cost 10
The link that I bolded is the issue. This is the link between the point-to-point node and the psuedonode (network LSA) in the SPF tree. It has mismatched information, one side points to a RID and interface address, the other to a stub network and RID. Realistically even my repesentation is flawed as from an SPF point of view the links wouldn't even look matched up between the psuedonode and point-to-point router, it would look like both have a link pointing to nowhere.
Also as a note, not all network type mismatches will result in no connectivity. The only mismatches that result in no connectivity are ones between those that create a network LSA (elect a DR) and those that do not. As an example configuring one link for point-to-multipoint and one for point-to-point works fine. Same thing for between broadcast and nonbroadcast.
The only other mismatch that will cause issues possibly with connectivity are between point-to-point and point-to-mulitpoint or point-to-multipoint non broadcast with more than two neighbors. This will cause repeated flapping between neighbors and control plane churning, as when the point-to-point end creates a new adjacency it will flush the old and bring the neighbor down because it is only allowed to have one neighbor on the link.
Hope that explains everything clearly.
