OSPF Equal-Cost Load Balancing not working as expected

[tlxbx]

• I have a situation where ASR-903 has OSPF adj. with 2 ASR1k. ASR1k are in HSRP pair.
• ASR1k is re-distributing BGP into OSPF so ASR-903 learns the same subnets from Both ASR1k. OSPF & BGP configs are EXACTLY same on both. 
• However the issue is ASR-903 routing table only includes routes learned from ASR1k-A but not B.
• OSPF database on ASR903 shows the subnet is learned from Both. 
• When I issue 

  sh ip ospf database external 

  I noticed one has Forward Address: 0.0.0.0 while other has Forward Address: 172.27.200.106. Not sure why this is but the one with 0.0.0.0 is added in RIB. 

So why it is not doing the ECMP and suppressing one subnet?

One of the actual example below from the Router. 

ASR903-RTR#sh ip ospf neighbor

Neighbor ID Pri State Dead Time Address Interface
10.255.1.9 1 2WAY/DROTHER 00:00:54 172.27.137.2 Port-channel1
10.255.1.10 1 FULL/BDR 00:00:58 172.27.137.3 Port-channel1

***************************************************************************************

ASR903-RTR#sh ip route ospf | i 172.27.144.
O E1 172.27.144.0/23 [110/2] via 172.27.137.2, 2w1d, Port-channel1

***************************************************************************************
ASR903-RTR#sh ip ospf database | i 172.27.144.
172.27.144.0 10.255.1.9 1028 0x80001CDE 0x00854C 64512
172.27.144.0 10.255.1.10 1660 0x80001CDB 0x003A9E 64512
***************************************************************************************

ASR903-RTR#sh ip ospf database external 172.27.144.0

OSPF Router with ID (172.27.137.4) (Process ID 1)

Type-5 AS External Link States

LS age: 320
Options: (No TOS-capability, DC, Upward)
LS Type: AS External Link
Link State ID: 172.27.144.0 (External Network Number )
Advertising Router: 10.255.1.9
LS Seq Number: 80001CFE
Checksum: 0x456C
Length: 36
Network Mask: /23
Metric Type: 1 (Comparable directly to link state metric)
MTID: 0
Metric: 1
Forward Address: 0.0.0.0
External Route Tag: 64512

LS age: 1330
Options: (No TOS-capability, DC, Upward)
LS Type: AS External Link
Link State ID: 172.27.144.0 (External Network Number )
Advertising Router: 10.255.1.10
LS Seq Number: 80001CFB
Checksum: 0xF9BE
Length: 36
Network Mask: /23
Metric Type: 1 (Comparable directly to link state metric)
MTID: 0
Metric: 1
Forward Address: 172.27.200.106
External Route Tag: 64512

[MHM Cisco World]

this is not equal cost it different 
one is forward via forwarding address 
other is forward via advertise address 

now you will return to BGP to check 
both use

next-hop self ?

both have best path toward other neighbor not toward each other ?

this two point effect your config.

[tlxbx]

Can you please explain what is the difference between those 2 addresses?

[MHM Cisco World]

https://www.cisco.com/c/en/us/support/docs/ios-nx-os-software/ios-xr-software/213533-compare-ospf-forwarding-address-behaviou.html

take look this doc.

[tlxbx]

Thank you. I read the document but still have few follow up questions?

1. I am still not sure what is making one of the Forward address as zero & other non-zero. I checked the criteria and both config looks the same. 
2. Document says if the non-zero forward address subnet is learned via OSPF it will be added into the RIB. In my case it is learned via OSPF but it is still not added into the RIB. What could be the reason for that?

Forward Address: 172.27.200.106
External Route Tag: 64512

ASR903-RTR#sh ip route 172.27.200.106
Routing entry for 172.27.200.104/30
Known via "ospf 1", distance 110, metric 2, type intra area

[MHM Cisco World]

can I see 

show ip bgp 

in both ASR HSRP ? from here the story begin

[tlxbx]

ASR1001-A#sh ip bgp 172.27.144.0
BGP routing table entry for 172.27.144.0/23, version 685
BGP Bestpath: med
Paths: (2 available, best #2, table default)
Advertised to update-groups:
4
Refresh Epoch 1
64512, (received & used)
172.27.200.110 from 172.27.200.110 (10.0.100.130)
Origin incomplete, metric 0, localpref 100, valid, external
Extended Community: RT:64512:50000
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
64512, (received & used)
172.27.200.114 from 172.27.200.114 (10.0.100.131)
Origin incomplete, metric 0, localpref 100, valid, external, best
Extended Community: RT:64512:50000
rx pathid: 0, tx pathid: 0x0

******************************************************************************

ASR1001-B#sh ip bgp 172.27.144.0
BGP routing table entry for 172.27.144.0/23, version 566
BGP Bestpath: med
Paths: (2 available, best #2, table default)
Advertised to update-groups:
4
Refresh Epoch 1
64512, (received & used)
172.27.200.102 from 172.27.200.102 (10.0.100.130)
Origin incomplete, metric 0, localpref 100, valid, external
Extended Community: RT:64512:50000
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
64512, (received & used)
172.27.200.106 from 172.27.200.106 (10.0.100.131)
Origin incomplete, metric 0, localpref 100, valid, external, best
Extended Community: RT:64512:50000
rx pathid: 0, tx pathid: 0x0

[MHM Cisco World]

the best path learn from
so ASR1001-A add External OSPF with NON-Zero forward address  

172.27.200.106 <-Forward Address: 172.27.200.106

so ASR1001-B add External OSPF with Zero forward address

172.27.200.114 <-Forward Address: 0.0.0.0 

in ASR1001-B check 172.27.200.114 is learn in RIB or NOT.

[tlxbx]

It;s other way around ASR-A adds zero forward & ASR-B Adds non-zero 172.27.200.106. 

ASR1001-A#sh ip route 172.27.200.114
Routing entry for 172.27.200.112/30
Known via "connected", distance 0, metric 0 (connected, via interface)
Redistributing via ospf 1
Routing Descriptor Blocks:
* directly connected, via TenGigabitEthernet0/0/0.2
Route metric is 0, traffic share count is 1

What is noticed is 

TenGigabitEthernet0/0/0.2 

is configured as OSPF Passive-interface so I assume why ASR-A adds 0.0.0.0 as forward address while ASR-B doesn't?

TenGigabitEthernet0/0/0.2 is up, line protocol is up
Internet Address 172.27.200.113/30, Interface ID 18, Area 0
Attached via Network Statement
Process ID 1, Router ID 10.255.1.9, Network Type BROADCAST, Cost: 1
Topology-MTID Cost Disabled Shutdown Topology Name
0 1 no no Base
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 10.255.1.9, Interface address 172.27.200.113
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
oob-resync timeout 40
No Hellos (Passive interface)
Supports Link-local Signaling (LLS)
Cisco NSF helper support enabled
IETF NSF helper support enabled
Index 1/3/3, flood queue length 0
Next 0x0(0)/0x0(0)/0x0(0)
Last flood scan length is 0, maximum is 0
Last flood scan time is 0 msec, maximum is 0 msec
Neighbor Count is 0, Adjacent neighbor count is 0
Suppress hello for 0 neighbor(s)

[MHM Cisco World]

These conditions set the forwarding address field to a non-zero address:

• OSPF is enabled on the ASBR's next hop interface AND 

• ASBR's next hop interface is non-passive under OSPF AND 

• ASBR's next hop interface is not point-to-point AND 

• ASBR's next hop interface is not point-to-multipoint AND 

• ASBR's next hop interface address falls under the network range specified in the router ospf command.

To be sure that check the same for ASR-B.

[tlxbx]

ASR-B has a non-passive interface.

So if I make

asr-a next-hop 

interface as non-passive I assume it will add a non-zero address?

In that case, ASR-903 will receive 2 subnets with non-zero address so will it added both in it's RIB for ECMP?

[MHM Cisco World · Accepted solution]

Great Job friend, 
if ASR-B forward address is non-passive and add it to LSA-5 then Yes 
ASR-A will do same 
and ASR903 will see two ECMP and use it
if the metric ASR-B to forward address = ASR-A to forward address

[Harold Ritter]

Hi @tlxbx ,

Make sure the interface towards 172.27.200.102 is owned by ospf and that it is not configured as a p2p interface. This should cause the forwarding address to be set on the external routes coming from that BGP peer.

Regards,

[tlxbx]

Another confusion is why both ASRs is choosing BEST path as NX-B?

All Path selection criteria are equal such as AS#, Orion Type, MED, etc.

I am unable to check Age of the Route but looking into Uplink Switch(nexus) uptime, NX-A has 16 min higher uptime than NX-B so I assume the NX-A BGP was established sooner making it oldest route. 

COREA# 172.27.200.114 (10.0.100.131) --> NX-B

COREB#  172.27.200.106 (10.0.100.131--> NX-B

[MHM Cisco World]

I think that this why forwarding address is 0.0.0.0 in one ASR and NON-Zero in other ASR
NON-Zero in ASR direct connect to NX-B ?

and for your Q. the Answer is 
You are right OLDEST or
Router-ID is lowest.

So as I mention the story begin in BGP,
make the ASR prefer the direct connect NX
so to solve this config Weight.