iBGP route preferred over OSPF, different results with the same setup

[rw13]

Our leaf set up is fairly straight forward. We have two routers at each leaf site for redundancy that receive a default route from our hub router via eBGP. Router1 is the "primary",  we use local preference of 200 on it. Router2 is the "secondary",  we use local preference of 100 on that. Both routers have OSPF for the igp for iBGP reachability, they are directly connected and peer with their loopbacks. OSPF is configured on both routers to advertise a default route via default-information-originate with metric-type 1 for the downstream L3 distribution switches. Here's where things get strange and we have this deployed at many sites....

-At some sites Router2 will prefer the default advertised from Router1 via iBGP.
-At some sites Router2 will prefer the default advertised from Router1 via OSPF.

In the configs nothing has been changed to tweak a routing protocols admin distance. There is not PBR or any otherwise config that would cause this. They are identical configurations aside from IP addresses quite literally.

I would expect that Router2 prefers the default route from Router1's OSPF advertisement since local preference on Router1 is higher, this iBGP route is the preferred BGP path on Router2. The iBGP (200 AD) route would not be installed in the routing table since there is OSPF route available (110 AD). With that said I can't imagine any scenario that would cause the iBGP route to be preferred over OSPF at some sites, especially when I look at the tables they look nearly identical aside from the ones preferring OSPF show a rib failure in BGP which is to be expected. What's going on here or have I forgotten something about OSPF and BGP's relationship?

Example, SiteA (iBGP preferred)

router1# show ip route
Gateway of last resort is (eBGP neighbor) to network 0.0.0.0
B* 0.0.0.0/0 [20/0] via (eBGP neighbor), 8d16h

router1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 (eBGP neighbor) 200 0 65001 65002 i

router2#show ip route
Gateway of last resort is (iBGP Neighbor) to network 0.0.0.0
B* 0.0.0.0/0 [200/0] via (iBGP Neighbor), 8d16h

router2#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*>i 0.0.0.0 (iBGP Neighbor) 0 200 0 65001 65002 i
*                (eBGP neighbor) 100 0 65000 65002 i

Example Site B, (OSPF preferred)

router1#show ip route
Gateway of last resort is (eBGP neighbor) to network 0.0.0.0
B* 0.0.0.0/0 [20/0] via (eBGP neighbor), 6w4d

router1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 (eBGP neighbor) 200 0 65001 65002 i

router2#show ip route
Gateway of last resort is (router1) to network 0.0.0.0
O*N1 0.0.0.0/0 [110/12] via (router1), 6w4d, TenGigabitEthernet0/1/0

router2#show ip bgp
Network Next Hop Metric LocPrf Weight Path
r>i (iBGP Neighbor) 0 200 0 65001 65002 i
r 0.0.0.0 (eBGP neighbor) 100 0 65000 65002 i

[rw13]

I tried this as well, this didn't give any more info than just show ip bgp for 0.0.0.0/0

router2#show ip bgp 0.0.0.0/0 bestpath
BGP routing table entry for 0.0.0.0/0, version 6052
Paths: (2 available, best #1, table default)
Multipath: eBGP
Not advertised to any peer
Refresh Epoch 1
65001 65002, (received & used)
(R1 iBGP peer) (metric 11) from R1 iBGP peer (R1 iBGP peer)
Origin IGP, metric 0, localpref 200, valid, internal, best
rx pathid: 0, tx pathid: 0x0

[Kanan Huseynli]

Hi,

could you share OSPF external database when scenario is not working one (ibgp is preferred), please?

[rw13]

router2#show ip ospf database nssa-external

OSPF Router with ID (R2 Router ID) (Process ID 1)

Type-7 AS External Link States (Area 111)

LS age: 1211
Options: (No TOS-capability, Type 7/5 translation, DC, Upward)
LS Type: AS External Link
Link State ID: 0.0.0.0 (External Network Number )
Advertising Router: R1 router ID
LS Seq Number: 8000011B
Checksum: 0xC613
Length: 36
Network Mask: /0
Metric Type: 1 (Comparable directly to link state metric)
MTID: 0
Metric: 1
Forward Address: R1 address
External Route Tag: 0

[Kanan Huseynli]

---

@rw13 wrote:

router2#show ip ospf database nssa-external

OSPF Router with ID (R2 Router ID) (Process ID 1)

Type-7 AS External Link States (Area 111)

LS age: 1211
Options: (No TOS-capability, Type 7/5 translation, DC, Upward)
LS Type: AS External Link
Link State ID: 0.0.0.0 (External Network Number )
Advertising Router: R1 router ID
LS Seq Number: 8000011B
Checksum: 0xC613
Length: 36
Network Mask: /0
Metric Type: 1 (Comparable directly to link state metric)
MTID: 0
Metric: 1
Forward Address: R1 address
External Route Tag: 0

---

Could you explain what is R1 address in forwarding address? Which interface/ IP and on R2 do sh ip route to this address and share here.

[rw13]

That is showing up as the R1 loopback address. Show ip route for that address is via the directly connected link between R2 and R1. 

[Kanan Huseynli]

It is important type of route on R2. Is it OSPF route or BGP?

Do "show ip route [forwarding address]" route should be OSPF learned (intra or inter-area route, not external).

[rw13]

Yes route it is OSPF intra area route, thanks Kanan.

[Kanan Huseynli]

Strange..actually there is no reason to not install this route based on all information that you now and previously provided.

Could you share "sh ip os rib 0.0.0.0 0.0.0.0" and "sh ip os database nssa-external internal" as well on R2?

It would be better to share "show ip ospf rib [forwarding_address]" as well

[rw13]

Yes exactly! Sorry I cannot share exact router IP's/ID's since they are public.

router 2#show ip osp rib 0.0.0.0 0.0.0.0
%OSPF: Route not in OSPF Base Topology (MTID 0) local RIB

router2#show ip ospf rib (forwarding address)

OSPF Router with ID (R2 router ID) (Process ID 1)
Base Topology (MTID 0)
OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator
* (R1's loopback)/32, Intra, cost 11, area 111
SPF Instance 3349, age 6d17h
Flags: HiPrio
via (R1's directly connected int IP), TenGigabitEthernet0/1/0, label 1048578, strict label 1048578
Flags: none
LSA: 1/(R1 Router ID)/(R1 Router ID)

router2#sh ip os database nssa-external internal
OSPF Router with ID (R2 Router ID) (Process ID 1)
Type-7 AS External Link States (Area 111)
LSA prefix priority: Low
Distance: 16777215
Now in min table
Table index: 45 min 49 sec
LS age: 1222
Options: (No TOS-capability, Type 7/5 translation, DC, Upward)
LS Type: AS External Link
Link State ID: 0.0.0.0 (External Network Number )
Advertising Router: (R1 Router ID)
LS Seq Number: 80000121
Checksum: 0xBA19
Length: 36
Network Mask: /0
Metric Type: 1 (Comparable directly to link state metric)
MTID: 0
Metric: 1
Forward Address: (R1's loopback ID)
External Route Tag: 0

[Kanan Huseynli]

You don't have route in OSPF RIB, because Type7 LSA has infinity metric (Distance: 16777215), so can't be installed to RIB.

The reason is OSPF can not see R1's loopback interface (i.e forwarding address) as best and installed into RIB. See below:

router2#show ip ospf rib (forwarding address)

OSPF Router with ID (R2 router ID) (Process ID 1)
Base Topology (MTID 0)
OSPF local RIB
Codes: * - Best, > - Installed in global RIB
LSA: type/LSID/originator
* (R1's loopback)/32, Intra, cost 11, area 111  --> there is no installed in global RIB code (>)
SPF Instance 3349, age 6d17h
Flags: HiPrio --> there is no "RIB" flag
via (R1's directly connected int IP), TenGigabitEthernet0/1/0, label 1048578, strict label 1048578
Flags: none --> there is no "RIB" flag
LSA: 1/(R1 Router ID)/(R1 Router ID)

Could you recheck routing table for forwarding address? Yes, from LSA output even it is understandable that it is intra are route with 11 metric over area 111, but there is no RIB flag (installed to routing table).

let's look at router LSA then, share output of router LSA of R1 and is there type2 LSA in network?

share "show ip bgp [forward address]" as well.

[MHM Cisco World]

Hi, 
you are totally right, 
LO learn from intra area and must used for forward address. 
NOW, the only case that this LO not inject to RIB, if Router receive it from eBGP, i.e. eBGP is be as transit between two edge routers.
so @rw13 can you confirm that eBGP advertise this LO to router??

[MHM Cisco World]

last think @rw13  are the work site use router  IOS XR and non work site use router IOS ??

[rw13]

Yes that is correct XR for hub, IOS for leaf. 

[MHM Cisco World]

The IOS-XR and IOS have different behave for forwarding address use in ospf lsa5.

[rw13]

I do have it in the routing table. There are no type2 LSA, all are p2p links. I see what you mean but it does not make sense since it is there. 

router1#show ip ospf database

OSPF Router with ID (R1 Rrouter ID) (Process ID 111)

Router Link States (Area 111)

Link ID ADV Router Age Seq# Checksum Link count
(link to l3 switch) (l3 switch router ID) 1163 0x80009C9F 0x00EA16 19
(R1 loopback) (R1 loopback router ID) 1186 0x80009D70 0x00C1FB 9
(R2 loopback) (R2 loopback router ID) 947 0x8000A84A 0x003D98 9

Type-7 AS External Link States (Area 111)

Link ID ADV Router Age Seq# Checksum Tag
0.0.0.0 (R1 loopback ID) 1186 0x80000173 0x00166B 0