Load balancing on sub-interfaces (3 links)

Stanislav Zaikin · ‎06-04-2013

Hello.

I am trying to load balance between the three links of a bundle. Traffic comes and goes with the same bundle interface.

Launched 5 threads TCP\UDP with different SRC DST IP addresses and see the following balances:

IOS-XR Monitor Time: 00:00:30 SysUptime: 106:39:28

Last Clear: 00:00:22

Protocol:General

Interface In(pps) Out(pps) InPkts/Delta OutPkts/Delta

Te0/1/0/0 11381 628 102062/25512 256/64

Te0/1/0/1 33849 55965 303244/75700 505364/126230

Te0/1/0/2 11363 0 100800/25200 0/0

Quit='q', Clear='c', Freeze='f', Thaw='t',

Next set='n', Prev set='p', Bytes='y', Packets='k'

(General='g', IPv4 Uni='4u', IPv4 Multi='4m', IPv6 Uni='6u', IPv6 Multi='6m')

We have 10G switch connected to asr9010 three ports and the following configuration:

interface TenGigE0/1/0/0

bundle id 1 mode active

bundle port-priority 2

interface TenGigE0/1/0/1

bundle id 1 mode active

bundle port-priority 2

interface TenGigE0/1/0/2

bundle id 1 mode active

interface Bundle-Ether1.75

ipv4 address 25.0.0.1 255.255.255.252

encapsulation dot1q 75

interface Bundle-Ether1.76

ipv4 address 26.0.0.1 255.255.255.252

encapsulation dot1q 76

RP/0/RSP0/CPU0: ios # sh bundle load-balancing bundle-e1 detail location 0/1/CPU0

Tue Jun 4 07:03:07.605 UTC

Bundle-Ether1

Type: Ether (L3)

Members <current/max>: 3/3

Total Weighting: 3

Load balance: Default

Locality threshold: 65

Avoid rebalancing? False

Sub-interfaces: 3

Member Information:

Port: LON ULID BW

-----------------------------

Te0/1/0/0 0 0 1

Te0/1/0/1 1 1 1

Te0/1/0/2 2 2 1

Sub-interface Information:

Sub-interface Type Load Balance Locality

Hash Threshold

-------------------------------------------------- ---

Bundle-Ether1.76 L3 Default 65

Bundle-Ether1.75 L3 Default 65

Bundle-Ether1.100 L3 Default 65

Platform Information:

=====================

* Bundle Summary Information *

--------------------------

Interface: Bundle-Ether1 Ifhandle: 0x08000160

Lag ID: 1 Virtual Port: 255

Number of Members: 3 Local to LC: Yes

Hash Modulo Index: 3

Member Information:

LON Interface ifhandle SFP port slot remote / rack_id

-------------------------------------------------- -----

Te0/1/0/0 0x02000140 0 12 0 1 0/0

Te0/1/0/1 0x02000180 1 13 0 1 0/0

Te0/1/0/2 0x020001c0 11 2 0 1 0/0

* Bundle Table Information *

------------------------

[NP 0]:

----------------------------------

Unicast (Global) LAG table

----------------------------------

idx local LON VQI port

----------------------------------

1 0 0 12 0

2 0 1 13 0

3 0 2 11 0

[NP 1]

----------------------------------

Unicast (Global) LAG table

----------------------------------

idx local LON VQI port

----------------------------------

1 0 0 12 0

2 0 1 13 0

3 0 2 11 0

[NP 2]:

----------------------------------

Unicast (Global) LAG table

----------------------------------

idx local LON VQI port

----------------------------------

1 0 0 12 0

2 0 1 13 0

3 0 2 11 0

[NP 3]

-------------------------------------------------- -----------------

Unicast (Global) LAG table | Multicast (Local) LAG table

-------------------------------------------------- -----------------

idx local LON VQI port | idx local LON VQI port

-------------------------------------------------- -----------------

1 0 0 12 0 1 1 2 11 0

2 0 1 13 0 2 0 0 0 0

3 1 2 11 0 3 0 0 0 0

[NP 4]:

-------------------------------------------------- -----------------

Unicast (Global) LAG table | Multicast (Local) LAG table

-------------------------------------------------- -----------------

idx local LON VQI port | idx local LON VQI port

-------------------------------------------------- -----------------

1 1 0 12 0 1 1 0 12 0

2 0 1 13 0 2 0 0 0 0

3 0 2 11 0 3 0 0 0 0

[NP 5]

-------------------------------------------------- -----------------

Unicast (Global) LAG table | Multicast (Local) LAG table

-------------------------------------------------- -----------------

idx local LON VQI port | idx local LON VQI port

-------------------------------------------------- -----------------

1 0 0 12 0 1 1 1 13 0

2 1 1 13 0 2 0 0 0 0

3 0 2 11 0 3 0 0 0 0

[NP 6]

----------------------------------

Unicast (Global) LAG table

----------------------------------

idx local LON VQI port

----------------------------------

1 0 0 12 0

2 0 1 13 0

3 0 2 11 0

[NP 7]

----------------------------------

Unicast (Global) LAG table

----------------------------------

idx local LON VQI port

----------------------------------

1 0 0 12 0

2 0 1 13 0

3 0 2 11 0

================================================== =============================

xthuijs · ‎06-04-2013

Yeah 5 flows is not enough to create enough spread.

You could use the show cef exact route and or the bundle bundle-hash commands to findout which path or member a flow owuld take. Note however that there was a bug in these show commands if you are not running 423 with pack2.

Also have you maybe found the asr9000 loadbalancing architecture?

https://supportforums.cisco.com/docs/DOC-26687

‎it holds a lot of detail on the LB architecture.

xander

Stanislav Zaikin · ‎06-05-2013

20 flows and a bit better result:

IOS-XR Monitor Time: 00:00:08 SysUptime: 133:33:44

Last Clear: 00:00:06

Protocol:General

Interface In(pps) Out(pps) InPkts/Delta OutPkts/Delta

Te0/1/0/0 11794 14977 0/44696 0/44484

Te0/1/0/1 10682 8786 0/37924 0/25456

Te0/1/0/2 18243 16958 0/44596 0/57579

Quit='q', Clear='c', Freeze='f', Thaw='t',

Next set='n', Prev set='p', Bytes='y', Packets='k'

(General='g', IPv4 Uni='4u', IPv4 Multi='4m', IPv6 Uni='6u', IPv6 Multi='6m')

Can the ASR9K more or less normal balance on uneven number of links?

xthuijs · ‎06-05-2013

Yup, that confirms the suspicion that the entropie in your flows was too little.

You'll also notice that instead of using, sequential addresses for your test flows using more randomized L3 and L4 combo's (especially random high source ports to simulate clients), the spread will get better too.

We use a hash result of 256 buckets, these buckets are shared over the members you have.

So the spread, when entropie is good enough, will be a near equal spread. 20 flows is likely not good enough to produce that spread. In non facto 2 number of members with a facto 2 number of buckets, the spread will never be a perfect x/members.

In older releases we only had 8 buckets which inherently means that a non facto 2 number of members will

be problematic in terms of member loading.

Make sure you verify this with either XR 4.2.3 and smu pack2 or XR 4.3.1

regards

xander