2 ISPs with BGP sessions | I fail to change the default route

florinmarian · ‎07-18-2023

Hello!
Let me give you a little context.
We have 2 ISPs, one called Digi and another called Orange.
Digi has a guaranteed bandwidth of 150MBps IPv4 and 1Gbps best-effort IPv6 and Orange has 500Mbps guaranteed, without best-effort both IPv4 and IPv6.
What I am trying to do is to combine the 2 ISPs so that I have the best possible speeds for upload and download from the 3 subnets announced through the BGP session (2 IPv4, 1 IPv6).
At first I tried to change the default route coming from Digi with Orange, but I can't do anything, the weight of the neighbor is ignored.
Any suggestion is welcome.
Thank you!

router bgp 12345
bgp router-id 10.192.63.19
no bgp default ipv4-unicast
bgp fast-external-fallover
bgp log-neighbor-changes
bgp graceful-restart restart-time 120
bgp graceful-restart stalepath-time 360
bgp graceful-restart
bgp maxas-limit 50
bgp bestpath as-path multipath-relax
neighbor XX.XXX.51.73 remote-as 8953
neighbor XX.XXX.213.144 remote-as 8953
neighbor 2a02:XXXX:4009:2::1 remote-as 8953
neighbor 2a02:XXXX:8953::1 remote-as 8953
neighbor 10.192.63.17 remote-as 8708
neighbor 10.192.63.18 remote-as 8708
neighbor 2a02:XXXX:fff::1 remote-as 8708
neighbor 2a02:XXXX:fff::2 remote-as 8708
!
address-family ipv4
 neighbor 10.192.63.17 activate
 neighbor 10.192.63.18 activate
 neighbor XX.XXX.51.73 activate
 neighbor XX.XXX.213.144 activate
 neighbor 10.192.63.17 weight 17500
 neighbor 10.192.63.18 weight 17500
 neighbor XX.XXX.51.73 weight 47500
 neighbor XX.XXX.213.144 weight 47500
 neighbor 10.192.63.17 maximum-prefix 2000
 neighbor 10.192.63.18 maximum-prefix 2000
 neighbor XX.XXX.51.73 maximum-prefix 2000
 neighbor XX.XXX.213.144 maximum-prefix 2000
 neighbor XX.XXX.51.73 ebgp-multihop 7
 neighbor XX.XXX.51.73 route-map DENY-OUT in
 neighbor XX.XXX.51.73 route-map PERMIT-IN out
 neighbor XX.XXX.213.144 ebgp-multihop 7
 neighbor XX.XXX.213.144 route-map DENY-IN in
 no neighbor 2a02:XXXX:fff::1 activate
 no neighbor 2a02:XXXX:fff::2 activate
 no neighbor 2a02:XXXX:4009:2::1 activate
 no neighbor 2a02:XXXX:8953::1 activate
 no auto-summary
 no synchronization
 maximum-paths 4
 network XXX.241.240.0 mask 255.255.255.0
 network XXX.241.241.0 mask 255.255.255.0
exit-address-family
!
address-family ipv6
 neighbor 2a02:XXXX:fff::1 activate
 neighbor 2a02:XXXX:fff::2 activate
 neighbor 2a02:XXXX:4009:2::1 activate
 neighbor 2a02:XXXX:8953::1 activate
 neighbor 2a02:XXXX:fff::1 weight 47500
 neighbor 2a02:XXXX:fff::2 weight 47500
 neighbor 2a02:XXXX:4009:2::1 weight 17500
 neighbor 2a02:XXXX:8953::1 weight 17500
 neighbor 2a02:XXXX:fff::1 maximum-prefix 2000
 neighbor 2a02:XXXX:fff::2 maximum-prefix 2000
 neighbor 2a02:XXXX:4009:2::1 maximum-prefix 2000
 neighbor 2a02:XXXX:8953::1 maximum-prefix 2000
 neighbor 2a02:XXXX:4009:2::1 ebgp-multihop 7
 neighbor 2a02:XXXX:4009:2::1 route-map DENY-OUT-V6 in
 neighbor 2a02:XXXX:4009:2::1 route-map PERMIT-IN-V6 out
 neighbor 2a02:XXXX:8953::1 ebgp-multihop 7
 neighbor 2a02:XXXX:8953::1 route-map DENY-IN-V6 in
 network 2a0e:8f02:f04f::/48
 network 2a0e:8f02:f04f::1/128
 no synchronization
 maximum-paths 4
 exit-address-family
!

Switch#show ip bgp
BGP table version is 4, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 10.192.63.17 47500 8708 i
*m 10.192.63.18 47500 8708 i
0.0.0.0 0 i
*> XXX.241.240.0/24 0.0.0.0 0 32768 i
*> XXX.241.241.0/24 0.0.0.0 0 32768 i

MHM Cisco World · ‎07-18-2023

bgp bestpath as-path multipath-relax
maximum-paths 2
bgp dmzlink-bw

as I understand you need two path install in RIB
try above command

florinmarian · ‎07-18-2023

Hey!

I applied those commands and then `clear ip bgp *` but still traceroute shows only 1st ISP's IPs and also the speed isn't improved yet (still capped at 150Mbps)

Switch#show ip bgp
BGP table version is 4, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *m  0.0.0.0          10.192.63.18                       17500 8708 i
 *>                   10.192.63.17                       17500 8708 i
                      0.0.0.0                                0 i
 *>  XXX.241.240.0/24 0.0.0.0                  0         32768 i
 *>  XXX.241.241.0/24 0.0.0.0                  0         32768 i
Switch#

MHM Cisco World · ‎07-18-2023

Can I see show ip route 0.0.0.0

florinmarian · ‎07-18-2023

Switch#show ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "bgp 57403", distance 20, metric 0, candidate default path
  Tag 8708, type external
  Last update from 10.192.63.17 00:03:11 ago
  Routing Descriptor Blocks:
  * 10.192.63.18, from 10.192.63.18, 00:03:11 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 8708
      MPLS label: none
      MPLS Flags: NSF
    10.192.63.17, from 10.192.63.17, 00:03:11 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 8708
      MPLS label: none
      MPLS Flags: NSF
Switch#

MHM Cisco World · ‎07-18-2023

shared count is 1 meaning half/half sharing is done.

florinmarian · ‎07-18-2023

Hey!

I saw that dmzlink-bw has to be applied also to BGP neighbors and it works for neighbors of 1st ISP but not for the second one.

Switch(config-router-af)# neighbor 10.192.63.17 dmzlink-bw
Switch(config-router-af)# neighbor 10.192.63.18 dmzlink-bw
Switch(config-router-af)# neighbor 92.180.51.73 dmzlink-bw
%BGP: Propagation of DMZ-Link-Bandwidth is supported only for single-hop EBGP peers

Joseph W. Doherty · ‎07-19-2023

Yup, it appears you've bumped into one of the feature's restrictions.

BTW, I was unaware of this feature, until reading these posts, but this recent BGP feature also appears to only do static UCMP, unlike PfR's dynamic UCMP.

florinmarian · ‎07-18-2023

@MHM Cisco World it is normal to have default route announced from 2nd ISP pointing to the first one?

Switch#show ip bgp neighbors 10.192.63.17 advertised-routes
BGP table version is 5, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  0.0.0.0          10.192.63.18                       17500 8708 i
 *>  188.241.240.0/24 0.0.0.0                  0         32768 i
 *>  188.241.241.0/24 0.0.0.0                  0         32768 i

Total number of prefixes 3
Switch#show ip bgp neighbors 10.192.63.18 advertised-routes
BGP table version is 5, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  0.0.0.0          10.192.63.18                       17500 8708 i
 *>  188.241.240.0/24 0.0.0.0                  0         32768 i
 *>  188.241.241.0/24 0.0.0.0                  0         32768 i

Total number of prefixes 3
Switch#show ip bgp neighbors 62.217.213.144 advertised-routes

Total number of prefixes 0
Switch#show ip bgp neighbors 92.180.51.73 advertised-routes
BGP table version is 5, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  0.0.0.0          10.192.63.18                       17500 8708 i
 *>  188.241.240.0/24 0.0.0.0                  0         32768 i
 *>  188.241.241.0/24 0.0.0.0                  0         32768 i

Total number of prefixes 3
Switch#

MHM Cisco World · ‎07-18-2023

I dont get your Q?

florinmarian · ‎07-18-2023

Shouldn't Orange advertise default route for 0.0.0.0/0 to it's own router instead of 10.192.63.18 which is default route from another ISP?

Switch#show ip bgp neighbors 10.192.63.17 advertised-routes
BGP table version is 5, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  0.0.0.0          10.192.63.18                       17500 8708 i
 *>  188.241.240.0/24 0.0.0.0                  0         32768 i
 *>  188.241.241.0/24 0.0.0.0                  0         32768 i

Total number of prefixes 3
Switch#show ip bgp neighbors 10.192.63.18 advertised-routes
BGP table version is 5, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  0.0.0.0          10.192.63.18                       17500 8708 i
 *>  188.241.240.0/24 0.0.0.0                  0         32768 i
 *>  188.241.241.0/24 0.0.0.0                  0         32768 i

Total number of prefixes 3
Switch#show ip bgp neighbors 92.180.51.73 advertised-routes
BGP table version is 5, local router ID is 10.192.63.19
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
              x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found

     Network          Next Hop            Metric LocPrf Weight Path
 *>  0.0.0.0          10.192.63.18                       17500 8708 i
 *>  188.241.240.0/24 0.0.0.0                  0         32768 i
 *>  188.241.241.0/24 0.0.0.0                  0         32768 i

Total number of prefixes 3

MHM Cisco World · ‎07-18-2023

correct but did you check neighbor? the next-hop will change when advertise. there is eBGP so next-hop must change (no show by show ip route advertised-route)

Joseph W. Doherty · ‎07-18-2023

"Any suggestion is welcome."

Cisco's PfR technology has (or had) the capability to proportionally low balance, dynamically. Doing though for ingress, may require cooperation from you ISPs. Egress balancing, though, is strictly controlled by PfR.

florinmarian · ‎07-21-2023

I am still looking for a solution to use the 3 routers (two routers of ISP 1 and one router of ISP 2).
Deactivating the neighbor from AS8953 we download through IPV6 from source X with 600Mbps, the same with the neighbors from AS8708 (also 600Mbps) and then activating it, the speed didn't increase either, but we couldn't have 2 600Mbps connections in parallel, but the traffic was shared even if the settings seem more than OK.

router bgp 57403
bgp router-id 10.192.63.19
no bgp default ipv4-unicast
bgp fast-external-fallover
bgp log-neighbor-changes
bgp graceful-restart restart-time 120
bgp graceful-restart stalepath-time 360
bgp graceful-restart
bgp maxas-limit 150
bgp bestpath as-path multipath-relax
neighbor 92.180.51.73 remote-as 8953
neighbor 2a02:a58:4009:2::1 remote-as 8953
neighbor 10.192.63.17 remote-as 8708
neighbor 10.192.63.18 remote-as 8708
neighbor 2a02:2f08:fff::1 remote-as 8708
neighbor 2a02:2f08:fff::2 remote-as 8708
!
address-family ipv4
 bgp dmzlink-bw
 neighbor 10.192.63.17 activate
 neighbor 10.192.63.18 activate
 neighbor 92.180.51.73 activate
 neighbor 10.192.63.17 dmzlink-bw
 neighbor 10.192.63.18 dmzlink-bw
 neighbor 92.180.51.73 dmzlink-bw
 neighbor 10.192.63.17 send-community extended
 neighbor 10.192.63.18 send-community extended
 neighbor 92.180.51.73 send-community extended
 neighbor 10.192.63.17 maximum-prefix 20000
 neighbor 10.192.63.18 maximum-prefix 20000
 neighbor 92.180.51.73 maximum-prefix 20000
 neighbor 10.192.63.17 prefix-list ALLOWED-PREFIXES-IPv4 out
 neighbor 10.192.63.18 prefix-list ALLOWED-PREFIXES-IPv4 out
 neighbor 92.180.51.73 prefix-list ALLOWED-PREFIXES-IPv4 out
 no neighbor 2a02:2f08:fff::1 activate
 no neighbor 2a02:2f08:fff::2 activate
 no neighbor 2a02:a58:4009:2::1 activate
 no auto-summary
 no synchronization
 maximum-paths 2
 network 188.241.240.0 mask 255.255.255.0
 network 188.241.241.0 mask 255.255.255.0
exit-address-family
!
address-family ipv6
 bgp dmzlink-bw
 neighbor 2a02:2f08:fff::1 activate
 neighbor 2a02:2f08:fff::2 activate
 neighbor 2a02:a58:4009:2::1 activate
 neighbor 2a02:2f08:fff::1 dmzlink-bw
 neighbor 2a02:2f08:fff::2 dmzlink-bw
 neighbor 2a02:a58:4009:2::1 dmzlink-bw
 neighbor 2a02:2f08:fff::1 send-community extended
 neighbor 2a02:2f08:fff::2 send-community extended
 neighbor 2a02:a58:4009:2::1 send-community extended
 neighbor 2a02:2f08:fff::1 maximum-prefix 20000
 neighbor 2a02:2f08:fff::2 maximum-prefix 20000
 neighbor 2a02:a58:4009:2::1 maximum-prefix 20000
 neighbor 2a02:2f08:fff::1 prefix-list ALLOWED-PREFIXES-IPv6 out
 neighbor 2a02:2f08:fff::2 prefix-list ALLOWED-PREFIXES-IPv6 out
 neighbor 2a02:a58:4009:2::1 prefix-list ALLOWED-PREFIXES-IPv6 out
 network 2a0e:8f02:f04f::/48
 network 2a0e:8f02:f04f::1/128
 no synchronization
 maximum-paths 2
 exit-address-family
!

Joseph W. Doherty · ‎07-21-2023

Beating the same drum, cannot say what PfRv3 actually now supports, but having worked with the "original" OER and the follow-on next version, i.e. the first version of PfR, both of those dynamically (proportionally) balanced across links regardless of what router they were attached to. This also included dealing with BGP's natural preference to use an equal value route from its eBGP peer rather than via its iBGP peer. I.e. it could/would shift flows between your WAN edge routers.

Besides using OEM/PfR on our dual Internet routers, I turned it lose on all our internal WAN routers using private BGP (international) WAN clouds. The one unexpected issue/complaint was from our WAN fault network monitoring, most "faults" disappeared because OER/PfR, as I had it configured, would detect a problem, and reroute around it before the fault/issue tripped a WAN monitoring alert. Great for our network traffic, not so great to know when to beat up our WAN SPs. (Figuring out how to "know" of WAN issues, was more difficult than getting OER/PfR to do its thing.)

Laugh - but wait there's more - OER/PfR can also do it's own internal SLA monitoring. It there's a choice, it will try to obtain the best actual performance for flows, from each controlled edge router to destinations.

Personally, I considered comparing OER/PfR vs. using a dynamic routing protocol, to comparing using a dynamic routing protocol vs. static routing.

I also found, on my dual Internet routers, we no longer needed anything beyond default routes. In theory, all destinations were equally reachable, and of equal quality, via either ISP, but if they weren't OER/PfR could rapidly adjust.

The forgoing may appear to look like a paid endorsement, but it's not, I really found the technology amazing. It also handled issues like black holes or "brown outs" within our SP WAN clouds, when BGP looks just dandy. Or, as all our sites had at least two WAN cloud connections, if one cloud to site link was congested, it would see the slower throughput to the site, and juggle flows, again, to obtain the best performance. Heck, on occasion, it would even use another of our sites as a transit to another of our sites.