ASR 9000 Load balancing issues

[IBEngTeam]

Hi all,

We are running ASR 9000 in our core and edge. all routers are running isis, bgp and mpls. we have multiple redundant links and have configured isis for UCMP. Since some of the redundant links have small difference in delay, we are experiencing out of order packet, this effect some application, so we tried to resolve the issue using the following configuration:

cef load-balancing
fields l3 global
!

The result is very strange! for applications using tcp with fix source and destination ports - all works well. but for tcp application using multiple source ports - we can see packets traversing different links - this is not expected, since we configured the router  to ignore layer 4 data in hash!

Can someone help and explain this? or suggest how can we prevent the out of order packets in redundant paths topology?

All routers are using asr 9000 enhanced ethernet line cards.

Thanks,

Adi.

[balaji.bandi]

with this information, it's very hard to identify what is the issue.

do you have any network diagram of how they connected? where is the source and where is the destination this must be defined to assist further.

[IBEngTeam]

Hi,

Thanks for the reply.

This is the network topology, source is connected to PE2 and destination to PE1, there are redundant paths, and we want to prevent out of order packets - but for some reason packets with same L3 information traverse different paths.

Regarding traffic flow - packet enters PE2 and goes into a VRF, from the it is switched using MPLS.

Network

Thanks,

Adi

[MHM Cisco World]

can you use per-destination instead of per-packet of the there are many traffic between two PE. 

[IBEngTeam]

Hi,

thanks for the reply.

how do we configure the per destination?

adi.

[MHM Cisco World]

ip load-sharing [per-packet] [per-destination]

we usually use above command to change from per-packet to per-destination  

[IBEngTeam]

Hi,

this command does not exists in IOS XR asr9000

[smilstea]

In XR we don't load balance based on per packet, we do per flow which can be based on source/destination mac/ip. We will always hash the same tuple of information in a packet out the same interface. Per packet load balancing will always lead to OOO packets which is one reason why we don't support it. Your understanding of l3 and l4 fields is correct, that command you applied should make it so only src/dst ip and router ID is looked at and not L4 fields. You also have MPLS and Bundles though. Have you adjusted the bundle load balancing?

https://community.cisco.com/t5/service-providers-knowledge-base/asr9000-xr-load-balancing-architecture-and-characteristics/ta-p/3124809

Sam

[balaji.bandi]

in normal condition all works as expected, but any of the link failed in the path you having issue ? is this correct ?

when the failiure take place, some where you have convergence issue, have you capture the outputs when the path failed and you having any issue before you do any other method to fix the issue.

how big the network, you need to get more evidency before make any changes, if this is Service provider network you may see different results. so be cautious making changes and also 1 change at a time and if not work role back.

Do you have any snippet config of ISIS and BGP and routing views, what is 1000, 600, 400 ? is the port-channel or vrf ?

[IBEngTeam]

Hi,

Thanks for the reply.

Let me explain again - the issue is not related to link failure but only to the way asr9000 performs the load balance on redundant paths. we configured all 9k routers with:

cef load-balancing
fields l3 global
!

We did this to prevent out of order packets, by forcing all packets with same source and destination ip to be directed to the same path. but for some reason this does not work, and we see packets with same L3 information but different L4 information travailing different paths.

When checking the cef table for destination 10.10.10.50, it remains the same regardless of L4 information:

show cef vrf wan_a exact-route 3.250.123.161 10.10.10.50 protocol UDP source-port 3915 destination-port 7877 ingress-interface TenGigE0/0/0/0

10.10.10.48/30, version 378897, internal 0x5000001 0x0 (ptr 0x71fd5c04) [1], 0x0 (0x0), 0x208 (0x732ce648)
Updated Dec 5 12:07:59.579
Prefix Len 30, traffic index 0, precedence n/a, priority 3
via Bundle-Ether103.100
via 10.1.1.1/32, 15 dependencies, recursive, bgp-multipath [flags 0x6080]
path-idx 0 NHID 0x0 [0x72538b78 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 10.1.1.1/32 via 24010/0/21
next hop 10.254.252.81/32 BE103.100 labels imposed {24002 33472}

But actually, when changing the L4 information (udp ports for example), some packets traverse a different path.

Maybe this is because the route is recursive? the route to 10.10.10.48/30 is via  10.1.1.1, and 10.1.1.1 is multipath:

10.1.1.1/32, version 1222071, internal 0x1000001 0x0 (ptr 0x71f098d8) [5], 0x0 (0x71ed5180), 0xa28 (0x732e6108)
Updated Dec 5 13:32:19.233
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via 10.254.252.77/32, Bundle-Ether102.100, 4 dependencies, weight 4294967295, class 0, protected, backup (Local-LFA) [flags 0x600]
path-idx 0 bkup-idx 1 NHID 0x0 [0x731573c0 0x0], Parent interface: BE102
next hop 10.254.252.77/32
local label 24010 labels imposed {24005}
via 10.254.252.81/32, Bundle-Ether103.100, 5 dependencies, weight 3323255238, class 0, protected, backup (Local-LFA) [flags 0x600]
path-idx 1 bkup-idx 0 NHID 0x0 [0x73157450 0x0], Parent interface: BE103
next hop 10.254.252.81/32
local label 24010 labels imposed {24002}

Any ideas? our main goal is to prevent out of order packets.

Adi.

[MHM Cisco World]

fiend I do lab to understand and help you to solve issue, 
the lab MPLS R1 connect to R2&R3 which connect to R4 
there is iBGP VPNv4 between R1 and R4 

the R1 and R4 use per-destination include-port (source & destination)

when both R4 and R1 have two equal link to both R2&R3 the per-destination work perfect, 
I see the traffic pass from R1 to R4 through same path, and from R4 to R1 through same path 

then I remove R3 from my network and connect R3 to R1 and R4 with two link 

and still per-destination work perfect because R1 and R2 have two equal path to R2 now

last case I think what you see 
I remove one link between R4 and R1 

here the issue 
R4 when forward traffic to R1 since it have one link the packet with MPLS label receive by R2 
BUT R2 dont have any idea about the CEF algo use by R1!!
so R2 will forward traffic select any path toward R1, 
here R1 can receive out of order packet.

so the router along the path must know how to deal with packet from same traffic. 

I think that can achieve only via MPLS TE

this is my view 
and thanks 

[IBEngTeam]

Hi

Thanks for the detailed response. but this does not answer the issue. and this is related to cisco ios not ios xr.

adi.

[MHM Cisco World]

I will ask you one Q, PE use IP source and destination + L4 port for load sharing 
the P not see IP header it see MPLS label, what algorithm P router use to forward traffic if it have two path ??

[IBEngTeam]

Hi,

Thanks for the reply.

We are running some test and will update.

Adi.