iBGP multi-path, what am I missing?

[Steven Williams]

It seems I have to paths to a network, also physically have two paths, but only one is getting into the route table? 

CORE-9500-01#show ip bgp 10.20.41.0
BGP routing table entry for 10.20.41.0/24, version 13
Paths: (2 available, best #2, table default)
Multipath: iBGP
Not advertised to any peer
Refresh Epoch 1
Local, (received & used)
172.16.63.5 (metric 20) from 172.16.63.4 (172.16.63.4)
Origin IGP, metric 0, localpref 100, valid, internal
Originator: 172.16.63.5, Cluster list: 10.111.111.1
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
Local, (received & used)
172.16.63.5 (metric 20) from 172.16.63.3 (172.16.63.3)
Origin IGP, metric 0, localpref 100, valid, internal, best
Originator: 172.16.63.5, Cluster list: 10.111.111.1
rx pathid: 0, tx pathid: 0x0

!

!
CORE-9500-01#show ip bgp
BGP table version is 15, local router ID is 172.16.63.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
* i 10.20.40.0/24 172.16.63.5 0 100 0 i
*>i 172.16.63.5 0 100 0 i
* i 10.20.41.0/24 172.16.63.5 0 100 0 i
*>i 172.16.63.5 0 100 0 i
*>i 10.20.42.0/24 172.16.63.5 0 100 0 i
* i 172.16.63.5 0 100 0 i
*>i 10.20.50.0/24 172.16.63.5 0 100 0 i
* i 172.16.63.5 0 100 0 i

!

!
CORE-9500-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 12 subnets, 3 masks
B 10.20.40.0/24 [200/0] via 172.16.63.5, 00:08:43
B 10.20.41.0/24 [200/0] via 172.16.63.5, 00:08:37
B 10.20.42.0/24 [200/0] via 172.16.63.5, 00:08:49
B 10.20.50.0/24 [200/0] via 172.16.63.5, 00:08:32
C 10.53.100.0/30 is directly connected, FortyGigabitEthernet1/0/29
L 10.53.100.2/32 is directly connected, FortyGigabitEthernet1/0/29
C 10.53.100.4/30 is directly connected, FortyGigabitEthernet1/0/30
L 10.53.100.6/32 is directly connected, FortyGigabitEthernet1/0/30
O 10.53.100.8/30
[110/11] via 10.53.100.1, 5d02h, FortyGigabitEthernet1/0/29
O 10.53.100.12/30
[110/11] via 10.53.100.5, 5d02h, FortyGigabitEthernet1/0/30
O 10.53.100.16/30
[110/11] via 10.53.100.1, 2d01h, FortyGigabitEthernet1/0/29
O 10.53.100.20/30
[110/11] via 10.53.100.5, 2d01h, FortyGigabitEthernet1/0/30
172.16.0.0/32 is subnetted, 5 subnets
C 172.16.63.1 is directly connected, Loopback0
O E2 172.16.63.2
[110/20] via 10.53.100.5, 5d02h, FortyGigabitEthernet1/0/30
[110/20] via 10.53.100.1, 5d02h, FortyGigabitEthernet1/0/29
O E2 172.16.63.3
[110/1] via 10.53.100.1, 5d02h, FortyGigabitEthernet1/0/29
O E2 172.16.63.4
[110/1] via 10.53.100.5, 5d02h, FortyGigabitEthernet1/0/30
O E2 172.16.63.5
[110/20] via 10.53.100.5, 2d01h, FortyGigabitEthernet1/0/30
[110/20] via 10.53.100.1, 2d01h, FortyGigabitEthernet1/0/29
CORE-9500-01#

I do have some router reflectors in the mix and I know there is limitations with RR I remember but not sure how they restrict multi-path to networks. 

R1 peers iBGP with R3 and R4 via loopback. 

R2 peer iBGP with R3 and R4 via lookback

Layer 3 switches peer iBGP with R3 and R4. 

R3 and R4 are route reflectors. R1 clearly has two paths to any network on Layer 2 switches but only one shows in router table. 

[Steven Williams]

Ya so it is a RR thing. Even though both paths are in the bgp table the RR only advertises the best path to the neighbor. Is there any way around this?

[Martin L]

Feature called bgp additional-paths  (or smthg like that).  it may work for you as it did for me in similar topo

router bgp 65000
add vpnv4
neighbor iBGP send-community extended
neighbor iBGP route-reflector-client
bgp additional-paths select backup
neighbor iBGP advertise diverse-path backup

[Martin L]

also, maximum-path iBGP x command may help . i think restriction is that next-hop value must be different for iBGPs; restriction for max-path ebgp is routes from same ASn.

[Giuseppe Larosa]

Hello Martin, Steven

in address-family VPNv4 is enough to use different RDs on the PE nodes to make the RRS servers  to think the two VPNv4 prefixes  are different and to propagate both of them.

In address family ipv4 unicast this option is not available.

Edit:

the following feature looks like interesting and should apply to Steven's network

https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/iproute_bgp/configuration/15-s/irg-15-s-book/irg_diverse_path.html?dtid=osscdc000283

Yes, it is the feature described by Martin. I see the same commands

The only possible issue is that Steven's route reflectors may be non Cisco devices but Palo Alto firewalls if I correcty remember.

Hope to help

Giuseppe

[Steven Williams]

Correct. Firewalls are the RR. I am thinking I will not have "tricks" to make ECMP work here.

[Steven Williams]

I had multi-path working when I wasn't peering with loopback addresses. When my core 9500 switches (R1, R2) peered with interface IPs on R3 and R4 i had multiple paths to my network held on the switch stack.

B 10.101.0.0/24 [200/0] via 10.51.105.2, 00:07:25
[200/0] via 10.51.104.2, 00:07:25

So why does peering with loopbacks change this?

[Giuseppe Larosa]

Hello Steven,

the use of loopbacks endpoints fo iBGP sessions should not change the behaviour.

However, I remember that you have tried to implement an iBGP only solution without an IGP running and you faced serious issues caused by non deterministic behaviour in BGP related to BGP scanning activity as we discussed in a previous thread.

Could you try to move the RRS on Cisco devices and to use the diverse path feature mentioned in previous post?

Hope to help

Giuseppe

[paul driver]

Hello

Just like to add, I notice in your topology the RR-Clients are connected to each other -Are these also ibgp peers with each other?
RR-Clients should not really have a ibgp peering with each other if they are already peering upstream with Route Reflectors.
The whole idea behind RR is that you reduce the number of peer sessions so not rely on having a fully meshed ibgp topology

if you are to partly meshing the RR Clients you may need to disable client-client reflection on the route reflectors to accommodate that partially meshed RR-Client ibgp peering.

route reflectors
router bgp xx
no bgp client-to-client reflection

[Steven Williams]

Route reflectors are not connected and do not peer ibgp. Yes they are Palo Alto firewalls. 

[paul driver]

Hello

So R3-R4 are RR but they are not peered to each other and R1-R2 are RR-clients but not ibgp peered to each other correct?

[Steven Williams]

Only route reflectors are r3 and r4. 1,2, and switch stack are clients.

[paul driver]

Hello

So are 1-2 ibgp peered with each other if so disable client-to-client reflection on the RR (3-4)

[Steven Williams]

1 and 2 not peered with each other. 1 and 2 only peered with 3 and 4. And 3 and 4 peered with switch stack on two different layer 3 port channels. 

[Steven Williams]

Should all peers just be peered for simplicity?