BGP multipath using single path

[Daniel Graham]

I have BGP multi-path setup but only one router in AS65222 is receiving traffic for the prefix 192.0.2.254.

Goal is to have traffic from cloud distributed between AS65222 routers to the anycast ip.

Thoughts on what is going wrong here?

Topology is:

                

192.0.2.1# show run
router bgp 65111 
    router-id 192.0.2.1
    neighbor 192.0.2.2 remote as 65222
    neighbor 192.0.2.3 remote as 65222
    address-family ipv4
        maximum-paths 2

# show ip route
192.0.2.254/32, ubest/mbest: 2/0
*via 192.0.2.2, [20/0], 00:12:48, bgp-65111, external, tag 65222
*via 192.0.2.3, [20/0], 00:12:48, bgp-65111, external, tag 65222

# show ip bgp 192.0.2.254/32
BGP routing table information for VRF default, address family IPv4 Unicast
BGP routing table entry for 192.0.2.254/32, version 411
Paths: (2 available, best #1)
Flags: (0x00001a) on xmit-list, is in urib, is best urib route
Multipath: eBGP

  Advertised path-id 1
  Path type: external, path is valid, received and used, is best path
  AS-Path: 65222 , path sourced external to AS
    192.0.2.2 (metric 0) from 192.0.2.2 (192.0.2.2)
      Origin IGP, MED not set, localpref 100, weight 0

  Path type: external, path is valid, received and used, not best reason: Router Id, multipath
  AS-Path: 65222 , path sourced external to AS
    192.0.2.3 (metric 0) from 192.0.2.3 (192.0.2.3)
      Origin IGP, MED not set, localpref 100, weight 0

  Path-id 1 advertised to peers:
    192.0.2.2

# show ip bgp
BGP routing table information for VRF default, address family IPv4 Unicast
BGP table version is 414, local router ID is 192.0.2.1
Status: s-suppressed, x-deleted, S-stale, d-dampened, h-history, *-valid, >-best
Path type: i-internal, e-external, c-confed, l-local, a-aggregate, r-redist
Origin codes: i - IGP, e - EGP, ? - incomplete, | - multipath

   Network            Next Hop            Metric     LocPrf     Weight Path
*>e192.0.2.254/32     192.0.2.2                                      0 65222 i
*|e                   192.0.2.3                                      0 65222 i

[Giuseppe Larosa]

Hello @Daniel Graham ,

BGP multipath is configured and working

show ip route
192.0.2.254/32, ubest/mbest: 2/0
*via 192.0.2.2, [20/0], 00:12:48, bgp-65111, external, tag 65222
*via 192.0.2.3, [20/0], 00:12:48, bgp-65111, external, tag 65222

But the issue here is with the way CEF load balancing is performed:

Load balancing is performed per flow and with flow is meant a source IP , destination IP pair

Because your destination prefix is actually a single host 192.0.2.254/32 you can try to change the source address.

IF you can do this, you should see used also the second path.

CEF load balancing uses  a binary EXOR of least significant bits in source IP and destination IP and a seed value chosen at router boot to choice the exit link for a specific flow.

Enabling per packet load balancing even if possibile is not recommended

Hope to help

Giuseppe

[Daniel Graham]

Testing was done with multiple source IP’s. In this diagram the router performing multipath is running NX-OS. I tried changing load-sharing options with no luck. Is NX-OS able to perform load sharing for bgp learned routes?

[Giuseppe Larosa]

Hello @Daniel Graham ,

looking at load balancing documents for Nexus I have found the following feature

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/nx-os/itd/configuration/guide/b-Cisco-Nexus-7000-Series-Intelligent-Traffic-Director-Configuration-Guide/m-conf-itd-72.html

I don't know if this can apply to your network scenario but I think it is wise to give a look at it.

Hope to help

Giuseppe

[Daniel Graham]

I'm running Nexus 3k, doesn't look like its supported.

Also found this, no matter what source IP I use it has the same result:

192.0.2.1# show routing hash 10.0.0.2 192.0.2.254
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0xed
No IPv4 protocol specified, defaulting to UDP
Hash for VRF "default"
Hashing to path *192.0.2.2
Out Interface: Vlan10
For route:
192.0.2.254/32, ubest/mbest: 2/0
    *via 192.0.2.2, [20/0], 00:00:57, bgp-65111, external, tag 65222
    *via 192.0.2.3, [20/0], 00:01:02, bgp-65111, external, tag 65222

192.0.2.1# show routing hash 192.168.0.2 192.0.2.254
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0xed
No IPv4 protocol specified, defaulting to UDP
Hash for VRF "default"
Hashing to path *192.0.2.2
Out Interface: Vlan10
For route:
192.0.2.254/32, ubest/mbest: 2/0
    *via 192.0.2.2, [20/0], 00:00:57, bgp-65111, external, tag 65222
    *via 192.0.2.3, [20/0], 00:01:02, bgp-65111, external, tag 65222

[Giuseppe Larosa]

Hello @Daniel Graham ,

for the way the hash is calculated the best way to see if load balancing is happening is to change the least significant byte of the source address

For example

source1:  10.0.0.2

source2:   192.168.0.3

the hash happens on the last meaningful bits of source and destination address so if you use the same value on the last 4th byte the same link is chosen regardless of what the first three bytes of the source address are.

Hope to help

Giuseppe

[Daniel Graham]

Good to know its based on least significant bit. Unfortunately no change -

192.0.2.1# show routing hash 10.0.0.2 192.0.2.254
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0xed
No IPv4 protocol specified, defaulting to UDP
Hash for VRF "default"
Hashing to path *192.0.2.2
Out Interface: Vlan10
For route:
192.0.2.254/32, ubest/mbest: 2/0
    *via 192.0.2.2, [20/0], 00:00:48, bgp-65111, external, tag 65222
    *via 192.0.2.3, [20/0], 00:00:57, bgp-65111, external, tag 65222

192.0.2.1# show routing hash 192.168.0.3 192.0.2.254
Load-share parameters used for software forwarding:
load-share mode: address source-destination port source-destination
Universal-id seed: 0xed
No IPv4 protocol specified, defaulting to UDP
Hash for VRF "default"
Hashing to path *192.0.2.2
Out Interface: Vlan10
For route:
192.0.2.254/32, ubest/mbest: 2/0
    *via 192.0.2.2, [20/0], 00:01:01, bgp-65111, external, tag 65222
    *via 192.0.2.3, [20/0], 00:01:10, bgp-65111, external, tag 65222

I also tested with change to ip load-sharing mode and tested with different protocols and ports, all with no change. Here is the mode currently set.

192.0.2.1# show ip load-sharing
IPv4/IPv6 ECMP load sharing:
Universal-id (Random Seed): 237
Load-share mode : address source-destination

[Giuseppe Larosa · Accepted solution]

Hello @Daniel Graham ,

the last show tells what the device is actually doing

It makes an EXOR of the last byte of source with destination and then an EXOR with the random seed value 237.

192.0.2.1# show ip load-sharing
IPv4/IPv6 ECMP load sharing:
Universal-id (Random Seed): 237
Load-share mode : address source-destination

It is not so immediate to find two flows that take a different path.

Hope to help

Giuseppe

[Daniel Graham]

That makes sense, thanks for pointing these things out.