Re: iBGP multi-path, what am I missing? - Page 3

Steven Williams · ‎06-24-2019

It seems I have to paths to a network, also physically have two paths, but only one is getting into the route table?

CORE-9500-01#show ip bgp 10.20.41.0
BGP routing table entry for 10.20.41.0/24, version 13
Paths: (2 available, best #2, table default)
Multipath: iBGP
Not advertised to any peer
Refresh Epoch 1
Local, (received & used)
172.16.63.5 (metric 20) from 172.16.63.4 (172.16.63.4)
Origin IGP, metric 0, localpref 100, valid, internal
Originator: 172.16.63.5, Cluster list: 10.111.111.1
rx pathid: 0, tx pathid: 0
Refresh Epoch 1
Local, (received & used)
172.16.63.5 (metric 20) from 172.16.63.3 (172.16.63.3)
Origin IGP, metric 0, localpref 100, valid, internal, best
Originator: 172.16.63.5, Cluster list: 10.111.111.1
rx pathid: 0, tx pathid: 0x0

!

!
CORE-9500-01#show ip bgp
BGP table version is 15, local router ID is 172.16.63.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
* i 10.20.40.0/24 172.16.63.5 0 100 0 i
*>i 172.16.63.5 0 100 0 i
* i 10.20.41.0/24 172.16.63.5 0 100 0 i
*>i 172.16.63.5 0 100 0 i
*>i 10.20.42.0/24 172.16.63.5 0 100 0 i
* i 172.16.63.5 0 100 0 i
*>i 10.20.50.0/24 172.16.63.5 0 100 0 i
* i 172.16.63.5 0 100 0 i

!

!
CORE-9500-01#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is not set

10.0.0.0/8 is variably subnetted, 12 subnets, 3 masks
B 10.20.40.0/24 [200/0] via 172.16.63.5, 00:08:43
B 10.20.41.0/24 [200/0] via 172.16.63.5, 00:08:37
B 10.20.42.0/24 [200/0] via 172.16.63.5, 00:08:49
B 10.20.50.0/24 [200/0] via 172.16.63.5, 00:08:32
C 10.53.100.0/30 is directly connected, FortyGigabitEthernet1/0/29
L 10.53.100.2/32 is directly connected, FortyGigabitEthernet1/0/29
C 10.53.100.4/30 is directly connected, FortyGigabitEthernet1/0/30
L 10.53.100.6/32 is directly connected, FortyGigabitEthernet1/0/30
O 10.53.100.8/30
[110/11] via 10.53.100.1, 5d02h, FortyGigabitEthernet1/0/29
O 10.53.100.12/30
[110/11] via 10.53.100.5, 5d02h, FortyGigabitEthernet1/0/30
O 10.53.100.16/30
[110/11] via 10.53.100.1, 2d01h, FortyGigabitEthernet1/0/29
O 10.53.100.20/30
[110/11] via 10.53.100.5, 2d01h, FortyGigabitEthernet1/0/30
172.16.0.0/32 is subnetted, 5 subnets
C 172.16.63.1 is directly connected, Loopback0
O E2 172.16.63.2
[110/20] via 10.53.100.5, 5d02h, FortyGigabitEthernet1/0/30
[110/20] via 10.53.100.1, 5d02h, FortyGigabitEthernet1/0/29
O E2 172.16.63.3
[110/1] via 10.53.100.1, 5d02h, FortyGigabitEthernet1/0/29
O E2 172.16.63.4
[110/1] via 10.53.100.5, 5d02h, FortyGigabitEthernet1/0/30
O E2 172.16.63.5
[110/20] via 10.53.100.5, 2d01h, FortyGigabitEthernet1/0/30
[110/20] via 10.53.100.1, 2d01h, FortyGigabitEthernet1/0/29
CORE-9500-01#

I do have some router reflectors in the mix and I know there is limitations with RR I remember but not sure how they restrict multi-path to networks.

R1 peers iBGP with R3 and R4 via loopback.

R2 peer iBGP with R3 and R4 via lookback

Layer 3 switches peer iBGP with R3 and R4.

R3 and R4 are route reflectors. R1 clearly has two paths to any network on Layer 2 switches but only one shows in router table.

Steven Williams · ‎07-08-2019

Also this is still confusing to me....I see BGP multi-path from one side of the network to the destination, but not the other way around, I feel like they should match?

network 10.100.0.0/24 lives beyond the next hop (firewalls) If 10.100.0.0/24 lives on BGP 172.16.63.10 and OSPF shows two paths to get to 172.16.63.10, why would BGP not be marked as Multi-path?

CORE-9500-02#show ip bgp
BGP table version is 5, local router ID is 172.16.63.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
*>i 0.0.0.0 172.16.63.3 100 0 ?
*mi 172.16.63.4 100 0 ?
*>i 10.100.0.0/24 172.16.63.10 0 100 0 i
*> 192.168.1.0 0.0.0.0 0 32768 i
* i 172.16.63.1 0 100 0 i
*> 192.168.12.0 0.0.0.0 0 32768 i
* i 172.16.63.1 0 100 0 i

!

!
CORE-9500-02#show ip route
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR

Gateway of last resort is 172.16.63.4 to network 0.0.0.0

B* 0.0.0.0/0 [200/0] via 172.16.63.4, 01:12:41
[200/0] via 172.16.63.3, 01:12:41
10.0.0.0/8 is variably subnetted, 9 subnets, 3 masks
O 10.53.100.0/30
[110/11] via 10.53.100.9, 01:14:10, FortyGigabitEthernet1/0/29
O 10.53.100.4/30
[110/11] via 10.53.100.13, 01:14:10, FortyGigabitEthernet1/0/30
C 10.53.100.8/30 is directly connected, FortyGigabitEthernet1/0/29
L 10.53.100.10/32 is directly connected, FortyGigabitEthernet1/0/29
C 10.53.100.12/30 is directly connected, FortyGigabitEthernet1/0/30
L 10.53.100.14/32 is directly connected, FortyGigabitEthernet1/0/30
O 10.53.100.24/30
[110/11] via 10.53.100.9, 01:14:10, FortyGigabitEthernet1/0/29
O 10.53.100.28/30
[110/11] via 10.53.100.13, 01:14:10, FortyGigabitEthernet1/0/30
B 10.100.0.0/24 [200/0] via 172.16.63.10, 00:13:26
172.16.0.0/32 is subnetted, 5 subnets
O E2 172.16.63.1
[110/20] via 10.53.100.13, 01:14:10, FortyGigabitEthernet1/0/30
[110/20] via 10.53.100.9, 01:14:10, FortyGigabitEthernet1/0/29
C 172.16.63.2 is directly connected, Loopback0
O E2 172.16.63.3
[110/1] via 10.53.100.9, 01:14:10, FortyGigabitEthernet1/0/29
O E2 172.16.63.4
[110/1] via 10.53.100.13, 01:14:10, FortyGigabitEthernet1/0/30
O E2 172.16.63.10
[110/20] via 10.53.100.13, 01:14:10, FortyGigabitEthernet1/0/30
[110/20] via 10.53.100.9, 01:14:10, FortyGigabitEthernet1/0/29
192.168.1.0/24 is variably subnetted, 2 subnets, 2 masks
B 192.168.1.0/24 is directly connected, 01:12:41, Vlan11
L 192.168.1.3/32 is directly connected, Vlan11
192.168.12.0/24 is variably subnetted, 2 subnets, 2 masks
B 192.168.12.0/24 is directly connected, 01:12:41, Vlan112
L 192.168.12.3/32 is directly connected, Vlan112
CORE-9500-02#

BUT....

I see this:

CORE-9500-02#show ip cef 10.100.0.0
10.100.0.0/24
nexthop 10.53.100.9 FortyGigabitEthernet1/0/29
nexthop 10.53.100.13 FortyGigabitEthernet1/0/30

paul driver · ‎07-08-2019

Hello

@Steven Williams wrote:

why would BGP not be marked as Multi-path?

could you try -

router bgp xx
bgp dmzlink-bw <- all ibgp routers

maximum-paths ibgp xx
neighbor xxxx send-community
neighbor xxxx dmzlink-bw <ebgp peers

Please rate and mark as an accepted solution if you have found any of the information provided useful.
This then could assist others on these forums to find a valuable answer and broadens the community’s global network.

Kind Regards
Paul

Steven Williams · ‎07-08-2019

I have max paths. what is the rest? These are all iBGP peers fully meshed.

router bgp 65001
bgp router-id 172.16.63.2
bgp log-neighbor-changes
neighbor 172.16.63.1 remote-as 65001
neighbor 172.16.63.1 update-source Loopback0
neighbor 172.16.63.3 remote-as 65001
neighbor 172.16.63.3 update-source Loopback0
neighbor 172.16.63.4 remote-as 65001
neighbor 172.16.63.4 update-source Loopback0
neighbor 172.16.63.10 remote-as 65001
neighbor 172.16.63.10 update-source Loopback0
!
address-family ipv4
neighbor 172.16.63.1 activate
neighbor 172.16.63.1 soft-reconfiguration inbound
neighbor 172.16.63.3 activate
neighbor 172.16.63.3 soft-reconfiguration inbound
neighbor 172.16.63.4 activate
neighbor 172.16.63.4 soft-reconfiguration inbound
neighbor 172.16.63.10 activate
maximum-paths ibgp 2
exit-address-family
!
address-family ipv4 vrf CORP_PROD
network 192.168.1.0
network 192.168.12.0
maximum-paths ibgp 2
exit-address-family

Steven Williams · ‎07-08-2019

If the BGP peer is 172.16.63.10 that is advertising the route 10.100.0.0/24 in BGP why would I have multiple paths to 172.16.63.10, but not 10.100.0.0/24?

CORE-9500-01#show ip route 10.100.0.0
Routing entry for 10.100.0.0/24
Known via "bgp 65001", distance 200, metric 0, type internal
Last update from 172.16.63.10 03:38:27 ago
Routing Descriptor Blocks:
* 172.16.63.10, from 172.16.63.10, 03:38:27 ago
Route metric is 0, traffic share count is 1
AS Hops 0
MPLS label: none
CORE-9500-01#show ip route 172.16.63.10
Routing entry for 172.16.63.10/32
Known via "ospf 10", distance 110, metric 20, type extern 2, forward metric 11
Last update from 10.53.100.5 on FortyGigabitEthernet1/0/30, 04:45:06 ago
Routing Descriptor Blocks:
10.53.100.5, from 172.16.63.10, 04:45:06 ago, via FortyGigabitEthernet1/0/30
Route metric is 20, traffic share count is 1
* 10.53.100.1, from 172.16.63.10, 04:45:06 ago, via FortyGigabitEthernet1/0/29
Route metric is 20, traffic share count is 1
CORE-9500-01#

Giuseppe Larosa · ‎07-08-2019

Hello Steven,

as already explained in previous post in this thread you are performing direct iBGP sessions between loopbacks that are advertised in OSPF.

So what you see is normal, if device using loopback address 172.16.63.10 is the only one to advertise in iBGP network 10.100.0.0/24 you will have a single iBGP route for prefix installed in the IP routing table.

However, BGP uses recursion over the BGP next-hop, so traffic destined to prefix 10.100.0.0 can use both paths to 172.16.63.10 this is what you see in the CEF table.

>> CORE-9500-02#show ip cef 10.100.0.0
10.100.0.0/24
nexthop 10.53.100.9 FortyGigabitEthernet1/0/29
nexthop 10.53.100.13 FortyGigabitEthernet1/0/30

The key point here is to look at the output of

show ip bgp 10.100.0.0

if there is a single BGP path iBGP multipath can install only one path. However, thanks to BGP recursion over next-hop traffic can be sent over the two OSPF paths to BGP next-hop.

If multiple BGP paths are present for prefix 10.100.0.0 look at the IGP metric to next-hop the second advertisement can come from a device with an higher OSPF metric then device with IP 172.16.63.10.

I think mBGP can use paths that have same values of IGP metric to next-hop as seen by OSPF.

Hope to help

Giuseppe

Steven Williams · ‎07-09-2019

Sorry, still trying to fully understand. It really gets confusing when looking at it that way. Also when using the show ip bgp 10.100.0.0 i am looking for two paths since there is two paths. Like my default route is marked for "multi-path" i would expect my 10.100.0.0 route to be also since both peers advertising the default route also each have a path to 10.100.0.0/24. But the default route comes from TWO different peer addresses so I can see how it shows that way and since 10.100.0.0 comes from only ONE peer address it only sees one entry. I bet if I was to peer to the each port-channel interface and not a loopback it would show multipath in the BGP table because it would see the 10.100.0.0/24 from TWO different peer addresses.

CORE-9500-01#show ip bgp
BGP table version is 14, local router ID is 172.16.63.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
*mi 0.0.0.0 172.16.63.4 100 0 ?
*>i 172.16.63.3 100 0 ?
*>i 10.100.0.0/24 172.16.63.10 0 100 0 i
* i 192.168.1.0 172.16.63.2 0 100 0 i
*> 0.0.0.0 0 32768 i
* i 192.168.12.0 172.16.63.2 0 100 0 i
*> 0.0.0.0 0 32768 i
CORE-9500-01#show ip bgp 10.100.0.0
BGP routing table entry for 10.100.0.0/24, version 14
Paths: (1 available, best #1, table default)
Multipath: iBGP
Flag: 0x100
Not advertised to any peer
Refresh Epoch 1
Local
172.16.63.10 (metric 20) from 172.16.63.10 (172.16.63.10)
Origin IGP, metric 0, localpref 100, valid, internal, best
rx pathid: 0, tx pathid: 0x0
CORE-9500-01#

Giuseppe Larosa · ‎07-09-2019

Hello Steven,

the show ip bgp 10.100.0.0 output has a single entry.

You are using iBGP with a full mesh of iBGP sessions because standard iBGP peers do not propagate iBGP routes to another iBGP peer. This can be done by RRS, but you have removed them because they defeat / break the iBGP multipath feature.

Devices with IP 172.16.63.3 and 172.16.63.4 cannot send the BGP advertisement received from 172.16.63.10 to the 172.16.63.1, because they are not RRS anymore.

The missing part is the concept of recursion over BGP next-hop. This is part of BGP from the beginning.

Just to make an example it is the recursion over BGP next-hop that allows a router to use an MPLS Label Switched Path LSP with destination = remote PE loopback = BGP next-hop.

You have load balancing over OSPF for the BGP destination prefix thanks to BGP recursion to BGP next-hop, that means route the packets for BGP prefix like they were for the BGP next-hop, and so use all available paths to reach the BGP next-hop.

Of course, routers in the middle must be aware of the BGP prefix and agree on the BGP next-hop, so that traffic is correctly routed end to end.

In your case having a full mesh of iBGP sessions there is no risk of black holing traffic for net 10.100.0.0/24.

Hope to help

Giuseppe

Steven Williams · ‎07-09-2019

Makes sense. Now I am confused one more thing....When I pull the received routes from each firewall peer address on one of the 9500s I get this output.

CORE-9500-01#show ip bgp neighbors 172.16.63.3 received-routes
BGP table version is 14, local router ID is 172.16.63.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
*>i 0.0.0.0 172.16.63.3 100 0 ?

Total number of prefixes 1
CORE-9500-01#show ip bgp neighbors 172.16.63.4 received-routes
BGP table version is 14, local router ID is 172.16.63.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path, L long-lived-stale,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
*mi 0.0.0.0 172.16.63.4 100 0 ?

Total number of prefixes 1

Why is only one marked with "m"?

Giuseppe Larosa · ‎07-09-2019

Hello Steven,

a single best path is still chosen and it is the path provided via the lowest BGP router-id / BGP next-hop 172.16.63.3, so this path is flagged as * >i *= valid i = IBGP and > = best

The additional path provided by 172.16.63.4 is flagged as *mi *= valid, i = iBGP and m means this path is picked up by iBGP multipath feature, but it is not the single best path to prefix 0.0.0.0/0 as the > flag is present only on the path via 172.16.63.3.

However, the show ip route 0.0.0.0 provides you both entries as B [200/0] because iBGP multipath is occurring for this prefix.

Hope to help

Giuseppe

Steven Williams · ‎07-09-2019

makes sense, and something to consider when creating your router-ids depending on the design. Lots of little things to remember when looking at all this since it is not all "defined clearly" if you do not know what to look for. I appreciate the help.

Steven Williams · ‎08-05-2019

So this is complete, but I seem to be having weird issues with traffic flow or asymmetric traffic. I think ECMP on the palo altos are maybe the issue, not sure though.

I am also wondering how to make traffic from the upstream firewalls return to the active hsrp node. The issue is the network is connected on BOTH hsrp nodes so the upstream firewalls see both as equal cost path which I think causes some issues with arp, maybe not. But if a packet goes out the active hsrp node to the firewall and firewall sees two equal cost paths back to the hsrp nodes it could send it to the standby node for that network, which would cause the traffic to have to cross the layer 2 link between the peers.

Can you do some kind of prepending from the network statements in each ipv4 family within bgp? Like prepend an extra AS on the standby hsrp node?