iBGP route preferred over OSPF, different results with the same setup

[rw13]

Our leaf set up is fairly straight forward. We have two routers at each leaf site for redundancy that receive a default route from our hub router via eBGP. Router1 is the "primary",  we use local preference of 200 on it. Router2 is the "secondary",  we use local preference of 100 on that. Both routers have OSPF for the igp for iBGP reachability, they are directly connected and peer with their loopbacks. OSPF is configured on both routers to advertise a default route via default-information-originate with metric-type 1 for the downstream L3 distribution switches. Here's where things get strange and we have this deployed at many sites....

-At some sites Router2 will prefer the default advertised from Router1 via iBGP.
-At some sites Router2 will prefer the default advertised from Router1 via OSPF.

In the configs nothing has been changed to tweak a routing protocols admin distance. There is not PBR or any otherwise config that would cause this. They are identical configurations aside from IP addresses quite literally.

I would expect that Router2 prefers the default route from Router1's OSPF advertisement since local preference on Router1 is higher, this iBGP route is the preferred BGP path on Router2. The iBGP (200 AD) route would not be installed in the routing table since there is OSPF route available (110 AD). With that said I can't imagine any scenario that would cause the iBGP route to be preferred over OSPF at some sites, especially when I look at the tables they look nearly identical aside from the ones preferring OSPF show a rib failure in BGP which is to be expected. What's going on here or have I forgotten something about OSPF and BGP's relationship?

Example, SiteA (iBGP preferred)

router1# show ip route
Gateway of last resort is (eBGP neighbor) to network 0.0.0.0
B* 0.0.0.0/0 [20/0] via (eBGP neighbor), 8d16h

router1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 (eBGP neighbor) 200 0 65001 65002 i

router2#show ip route
Gateway of last resort is (iBGP Neighbor) to network 0.0.0.0
B* 0.0.0.0/0 [200/0] via (iBGP Neighbor), 8d16h

router2#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*>i 0.0.0.0 (iBGP Neighbor) 0 200 0 65001 65002 i
*                (eBGP neighbor) 100 0 65000 65002 i

Example Site B, (OSPF preferred)

router1#show ip route
Gateway of last resort is (eBGP neighbor) to network 0.0.0.0
B* 0.0.0.0/0 [20/0] via (eBGP neighbor), 6w4d

router1#show ip bgp
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 (eBGP neighbor) 200 0 65001 65002 i

router2#show ip route
Gateway of last resort is (router1) to network 0.0.0.0
O*N1 0.0.0.0/0 [110/12] via (router1), 6w4d, TenGigabitEthernet0/1/0

router2#show ip bgp
Network Next Hop Metric LocPrf Weight Path
r>i (iBGP Neighbor) 0 200 0 65001 65002 i
r 0.0.0.0 (eBGP neighbor) 100 0 65000 65002 i

[David Ruess]

Hello,

Can you provide the routing section output of R1 and R2 at a "working" site and a :non-working" site. Specifically looking for the BGP and OSPF config section to see the redistribution and and network advertisements/peerings.

-David

[rw13]

Router1 and Router2 from working and not working are the same, really! I have to ommit IP's for privacy but you get the point. OSPF networks that are configured are point to points locally for an adjacency between router1 and 2 and downstream L3 distribution switches. Some OSPF networks are passive just for advertising within the area and redistribution into BGP, like the loopback the routers peer with. OSPF areas are NSSA. Our routing used to be OSPF back to the hub router. They are just a local area within each site now and have no real NSSA function or connectivity to any other area.

Router 1 -

router ospf 1
auto-cost reference-bandwidth 100000
area 111 authentication message-digest
area 111 nssa default-information-originate metric-type 1
passive-interface default
no passive-interface TenGigabitEthernet0/1/0 (router to router)
no passive-interface Port-channel1 (to downstream L3 switch)
network X.X.X.X 0.0.0.0 area 111
network X.X.X.X 0.0.0.0 area 111
network X.X.X.X 0.0.0.0 area 111
network X.X.X.X 0.0.0.0 area 111
!
router bgp 65500
bgp log-neighbor-changes
redistribute ospf 1 match internal nssa-external 1 nssa-external 2 route-map drop_default
neighbor ibgp peer-group
neighbor ibgp update-source Loopback0
neighbor ibgp next-hop-self
neighbor ibgp send-community
neighbor ibgp soft-reconfiguration inbound
neighbor 1.1.1.1 remote-as 65001
neighbor 1.1.1.1 description "eBGP to Hub 1"
neighbor 1.1.1.1 ebgp-multihop 2
neighbor 1.1.1.1 send-community both
neighbor 1.1.1.1 advertisement-interval 3
neighbor 1.1.1.1 soft-reconfiguration inbound
neighbor 1.1.1.1 route-map local_pref_200 in
neighbor 1.1.1.1 route-map local_as out
neighbor 10.10.10.2 remote-as 65500
neighbor 10.10.10.2 peer-group ibgp
neighbor 10.10.10.2 description "ibgp peer with router2"
maximum-paths 2
!

Router 2

router ospf 1
auto-cost reference-bandwidth 100000
area 111 authentication message-digest
area 111 nssa default-information-originate metric-type 1
passive-interface default
no passive-interface TenGigabitEthernet0/1/0 (router to router)
no passive-interface Port-channel1 (to downstream L3 switch)
network X.X.X.X 0.0.0.0 area 111
network X.X.X.X 0.0.0.0 area 111
network X.X.X.X 0.0.0.0 area 111
network X.X.X.X 0.0.0.0 area 111
!
router bgp 65500
bgp log-neighbor-changes
redistribute ospf 1 match internal nssa-external 1 nssa-external 2 route-map drop_default
neighbor ibgp peer-group
neighbor ibgp update-source Loopback0
neighbor ibgp next-hop-self
neighbor ibgp send-community
neighbor ibgp soft-reconfiguration inbound
neighbor 2.2.2.2 remote-as 65000
neighbor 2.2.2.2 description "eBGP to Hub 2"
neighbor 2.2.2.2 ebgp-multihop 2
neighbor 2.2.2.2 send-community both
neighbor 2.2.2.2 advertisement-interval 3
neighbor 2.2.2.2 soft-reconfiguration inbound
neighbor 2.2.2.2 route-map local_as out
neighbor 10.10.10.1 remote-as 65500
neighbor 10.10.10.1 peer-group ibgp
neighbor 10.10.10.1 description "ibgp peer with router1"
maximum-paths 2
!

[David Ruess]

Thanks for the info. First thing that comes to mind is the NSSA area. Is every single site using an NSSA area? I ask because you're redistributing NSSA route types and could have impact (not sure yet) I know you say you're using the eBGP to the sites. Do those connect to each Router (1 and 2) individually? If that's the case each router should be using its eBGP connection default route out since the AD would be 20 and not use OSPF.

I know you say they are the same config but can you check to see the redistribute/NSSA/and default information originate statements are exactly the same.

I presume the one "not-working" getting its default route from iBGP is either not peered with its OSPF neighbors at the local site to get the default route or the default route is not being advertised (or blocked by route-map/policy) by OSPF in some form or fashion. Maybe the default route metric type.

This of course is all assuming the default route for all is 0.0.0.0/0

You can check this by looking at the OSPF database to see if it even has a default route learned from OSPF. This would be on the Router getting its default route from iBGP in your example.

-David

[rw13]

Yes every single site uses NSSA. eBGP to the leaf connects to router 1 and 2 individually. I agree each router would be using its eBGP connection as you mentioned if they both have no manipulation to the default route they receive but R1 is set up with a higher local pref than R2 to create a primary/backup function. As far as checking the OSPF redistribution and NSSA/Default originate statements they really are the same, same exact. OSPF/BGP adjacencies are all up.

router 1#show ip bgp sum
BGP router identifier 10.10.10.1, local AS number 65500

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
1.1.1.1 4 65001 100373 98050 5905 0 0 8d19h 1
10.10.10.2 4 65500 15398958 15398656 5905 0 0 2y26w 5

router 2#show ip bgp sum
BGP router identifier 10.10.10.2, local AS number 65500

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
2.2.2.2 4 65000 100373 98050 5905 0 0 5d19h 1
10.10.10.1 4 65500 15398958 15398656 5905 0 0 2y26w 5

router 1#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
10.10.10.2 0 FULL/ - 00:00:10 10.10.10.255 TenGigabitEthernet0/1/0 (to router 2)
10.10.20.1 0 FULL/ - 00:00:11 10.10.10.101 Port-channel1 (to L3 distribution switch)

router 2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
10.10.10.1 0 FULL/ - 00:00:10 10.10.10.254 TenGigabitEthernet0/1/0 (to router 1)
10.10.20.1 0 FULL/ - 00:00:11 10.10.10.105 Port-channel1 (to L3 distribution switch)

This has me leaning towards a software defect or race condition, but the latter I don't know how. 

[David Ruess]

Thank you for providing more info. As far as the local preference I don't think it would be used yet. If routers learn the default route from eBGP and iBGP and OSPF then it would prefer eBGP first as the AD is 20. So the metrics used in iBGP would be irrelevant. Metrics are only used after longest match and then AD comparison.

A couple commands I would check on the non working site (learning default from iBGP):

On the HUB issue the command below and check to see if it is sending the default route to both peers (R1 and R2)

show ip bgp neighbor x.x.x.x advertised-routes

On each peer (R1 and R2) issue the command below to see if its receiving the default route:

show ip bgp neighbors x.x.x.x received-routes

If the default route is being sent to both peers and being received by both peers according to those commands and you have verified this, try the following:

Check the OSPF link-state database of both remote routes to see if it has a default route in it. 

show ip ospf database

-David

[rw13]

Thanks David, but to your point about local preference not being used that is what it is using. It is the first decision it is making since one of the iBGP paths has local preference of 200. The route is sent by the hub, it is also received by the leaf, see the outputs I provided. The OSPF database does have both routes in it. See my diagram hopefully it clears things up. 

[David Ruess]

So I had some time to lab it up. Minimal config but same concept. If R1 and R2 are learning the default route from an eBGP neighbor then they will use that respective neighbor for the default route since the AD is 20. The fact that R2 is using R1 for its default route at all suggests that its not getting a default route from its eBGP neighbor. I agree with you higher local preference is preferred but this should not apply if everything is working as it should. Local preference applies to iBGP peers. Which means in order for local preference to take effect you would first have to lose your eBGP peering and your OSPF peering as their ADs both beat out the AD of iBGP. I’m still trying to understand your topology and what is iBGP vs eBGP.

I will say this - as long as you havent manipulated anything the only reason that the router would choose an iBGP peering over OSPF is somehow its not seeing the default route being advertise by OSPF. In my lab everything worked as it should. Both routers chose their respective eBGP neighbor for the default route. Once I killed the eBGP link the OSPF default route took over since its AD is lower. 

We are missing something. Can you provide the output to the command I requested earlier of:

sh ip bgp neighbors <eBGP IP> received-routes - on both R1 and R2 of the site thats using iBGP for the default instead of OSPF

Here is my lab:

My R3 is your R2 in this case.

Once I killed the link to R4 and it lost its eBGP neighborship the OSPF default route took over

-David

[rw13]

David, with respect I understand your point but you keep missing R1 and R2 both receive a default, but R1 has added local pref 200 the the default it receives by design. Recalling BGP compares local pref in the path selection process as the 2nd, R2 will choose the path R1 has since it has higher local pref. As far as the topology clarification R1 and R2 are iBGP, the routers with different AS's are eBGP. The default is sent from AS65002 to each hub. 

[David Ruess]

OK I misread your drawing for one of the initial questions I had.

We both understand the local preference and how its used. Where I think the miscommunication is happening is on order of operations when it comes to how a route is installed in the routing table. Local preference comes into play only if iBGP is being used. If both R1 and R2 are getting a default route from their upstream eBGP neighbor then iBGP wont even be considered because its AD is 200, while eBGP is 20.

That being said the only way R2 is preferring its iBGP default route with AD of 200 (which it is based on your output) then its NOT getting its default route from its eBGP neighbor upstream AND its also not getting it from its OSPF neighbor (or is and not putting it in the RIB) of R1 since OSPF AD is 110, still lower than the AD of 200. 

Remember order of operations. If a router receives the exact same route (longest match) from multiple sources it will compare AD next, then metric (like local preference).

My theory is R2 is NOT learning a default route from its eBGP neighbor or OSPF in this case (or if it is its not installing in the RIB). That's what we are trying to figure out. 

[rw13]

Again with respect I believe you have misunderstood the path selection process. BGP does not compare whether it is iBGP or an eBGP route as a first step. If a router sees two paths and compares those two paths, one being a higher preference, it will use that path with the higher preference before comparing others. In this case that is why R2 choose to use R1's route into the routing table. OSPF is then preferred over iBGP since the AD is lower. This is turning into a point of how BGP works and derailing the question.The question is why isn't OSPF's default route being received and/or used by R2. I know the default route is received from it's eBGP and iBGP peer. We have done failover testing to show that it is there and works. 

[David Ruess]

Have you tried removing the iBGP peer in the spoke/remote network. Since you have OSPF I am not sure the reason for iBGP as you can manipulate path selection there too. You only need the eBGP peerings for the network to connect fully. Unless you have specific requirements I would remove it to see if it converges correctly.

[David Ruess]

The only other thing I could think of is the OSPF route doesnt take over if it comes from BGP first. 

You can try to remove the iBGP peering between R1 and R2 - let the default OSPF route install and then add the iBGP neighbor back.

-David

[MHM Cisco World]

Note:- iBGP is 200 and eBGP is 20 OSPF have AD 110 
Note:- r>i (iBGP Neighbor) 0 200 0 65001 65002 i
r meaning that the router learn same prefix from lower IGP routing protocol, and hence the router refuse it. 

there is no issue it is normal behave of BGP 

[rw13]

Thanks MHM, I realize this is the expected behavior. The question is why are other sites not behaving the same way and preferring the iBGP learned default instead.