Traffic being assymetric and CEF

[filipequintela]

Hey Guys,

I have configured BGP multipath to get BGP load balancing between two ASs on my company.

The scenario is in annex. Basically i have two routers interconnected with 2 links running in different ASs.

BGP multipath was configured so that i could achieve load balacing.

The problem is i'm seeing an assymetry on my traffic. output traffic is going through one interface and the replies are going from the other one. Just a ping test right..

As per CEF i'm seeing that it would be using a single path. Why this is not happening? Using a single link?

ROUTER1#sh cef state
CEF Status:
RP instance
common CEF enabled
IPv4 CEF Status:
CEF enabled/running
dCEF disabled/not running
CEF switching enabled/running
universal per-destination load sharing algorithm, id 9B4C5E9D
IPv6 CEF Status:
CEF disabled/not running
dCEF disabled/not running
universal per-destination load sharing algorithm, id 9B4C5E9D

ROUTER3#sh cef state
CEF Status:
RP instance
common CEF enabled
IPv4 CEF Status:
CEF enabled/running
dCEF disabled/not running
CEF switching enabled/running
universal per-destination load sharing algorithm, id F968AD29
IPv6 CEF Status:
CEF disabled/not running
dCEF disabled/not running
universal per-destination load sharing algorithm, id F968AD29

ROUTER1#sh ip cef exact-route 10.10.0.1 10.20.0.1
10.10.0.1 -> 10.20.0.1 =>IP adj out of GigabitEthernet0/3, addr 192.168.1.2

ROUTER3#sh ip cef exact-route 10.20.0.1 10.10.0.1
10.20.0.1 -> 10.10.0.1 =>IP adj out of GigabitEthernet0/3, addr 192.168.1.1

ROUTER1#sh int g0/1
GigabitEthernet0/1 is up, line protocol is up
Hardware is iGbE, address is 5000.0001.0001 (bia 5000.0001.0001)
Internet address is 192.168.0.1/24
MTU 1500 bytes, BW 145000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Unknown, Unknown, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 91000 bits/sec, 100 packets/sec
170 packets input, 27984 bytes, 0 no buffer
Received 5 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
62966 packets output, 7172653 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
ROUTER1#sh int g0/3
GigabitEthernet0/3 is up, line protocol is up
Hardware is iGbE, address is 5000.0001.0003 (bia 5000.0001.0003)
Internet address is 192.168.1.1/30
MTU 1500 bytes, BW 145000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Unknown, Unknown, link type is auto, media type is RJ45
output flow-control is unsupported, input flow-control is unsupported
ARP type: ARPA, ARP Timeout 04:00:00
Last input 00:00:06, output 00:00:04, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
30 second input rate 90000 bits/sec, 99 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
63540 packets input, 7252177 bytes, 0 no buffer
Received 5 broadcasts (0 IP multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 0 multicast, 0 pause input
440 packets output, 44873 bytes, 0 underruns
0 output errors, 0 collisions, 1 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out

BGP for R1:

router bgp 1
bgp router-id 10.10.10.2
bgp log-neighbor-changes
neighbor 192.168.0.2 remote-as 2
neighbor 192.168.1.2 remote-as 2
!
address-family ipv4
redistribute ospf 100
neighbor 192.168.0.2 activate
neighbor 192.168.1.2 activate
maximum-paths 2
exit-address-family

BGP for R2:

router bgp 2
bgp router-id 10.10.10.4
bgp log-neighbor-changes
neighbor 192.168.0.1 remote-as 1
neighbor 192.168.1.1 remote-as 1
!
address-family ipv4
redistribute ospf 100
neighbor 192.168.0.1 activate
neighbor 192.168.1.1 activate
maximum-paths 2
exit-address-family

IP Route for R1:

ROUTER1#sh ip route 10.20.0.1
Routing entry for 10.20.0.0/24
Known via "bgp 1", distance 20, metric 2
Tag 2, type external
Redistributing via ospf 100
Advertised by ospf 100 subnets
Last update from 192.168.0.2 00:45:11 ago
Routing Descriptor Blocks:
* 192.168.1.2, from 192.168.1.2, 00:45:11 ago
Route metric is 2, traffic share count is 1
AS Hops 1
Route tag 2
MPLS label: none
192.168.0.2, from 192.168.0.2, 00:45:11 ago
Route metric is 2, traffic share count is 1
AS Hops 1
Route tag 2
MPLS label: none

IP Route for R2:

ROUTER3#sh ip route 10.10.0.1
Routing entry for 10.10.0.0/24
Known via "bgp 2", distance 20, metric 2
Tag 1, type external
Redistributing via ospf 100
Advertised by ospf 100 subnets
Last update from 192.168.1.1 00:45:35 ago
Routing Descriptor Blocks:
192.168.1.1, from 192.168.1.1, 00:45:35 ago
Route metric is 2, traffic share count is 1
AS Hops 1
Route tag 1
MPLS label: none
* 192.168.0.1, from 192.168.0.1, 00:45:35 ago
Route metric is 2, traffic share count is 1
AS Hops 1
Route tag 1
MPLS label: none

[Giuseppe Larosa · Accepted solution]

Hello @filipequintela ,

what you see is normal for the following reasons:

a) CEF load balancing is flow based that is given a pair of addresses sourceIP destinationIP they make a flow and they use a single exit link ( this is done to avoid the possible out of order issues that per packet load balancing can cause).

b)  the exit interface is chosen using an algorithm that involves the less significant bits of sourceIP , destinationIP and a seed value that is node specific and can change after a router reload.

The operation is a binary EXOR :

10.10.0.1  EXOR 10.20.0.1  EXOR SEED gives a value that makes a choice on first router.

On the second router what changes is its own SEED and this leads to a different choice of the exit interface.

To be sure that mulipath is happening you need to check

show ip bgp 10.10.0.0

show ip bgp 10.20.0.0

you should see one path as best and the other one labelled as "m" that means taken by multipath

To see load balancing in action you need multiple flows with different sourceIP/ destination IP

Edit:

reviewing your network diagram what happens does not even qualify for asymmetric routing it is simple the result of different choices made by the two eBGP peers that have two links between themselves as explained above for the way CEF works.

Hope to help

Giuseppe

[Joseph W. Doherty]

https://ipwithease.com/bgp-multipath-as-path-relax/

[Giuseppe Larosa]

Hello Joseph,

in this specific case looking at the network diagram we see thay only two EBGP routers are involved and only two BGP ASes.

So this scenario should not need the as path relax to work as the two eBGP sessions are with the same peer AS.

Hope to help

Giuseppe

[Joseph W. Doherty]

Giuseppe, thank you for your correction, you're quite correct!

My bad, I didn't look at diagram, just went by:

"I have configured BGP multipath to get BGP load balancing between two ASs on my company.

The scenario is in annex. Basically i have two routers interconnected with 2 links running in different ASs."

and, mistakenly, took that as the two ASs were external to OP's. (A good example of how mistakes can easily happen when you rush.)

That said, CEF load balancing is deterministic per flow (Giuseppe describes the details), and if you have some flows with heavy bandwidth usage (or just one flow), your load balancing can be "skewed", especially short term, but possibly long term too.

Your posted stats, show they've been never cleared, but 60K packets seem "low". How long were OP stats collected?

As Giuseppe describes, what you're seeing might be normal.

Your posted show ip route command results, also confirm both paths are in the route table for the same destination.