Load balancing over 5xECMP ASR903

[Peyman Zadmehr]

Hi Everybody

I am using an ASR903- RSP3_400. This router has five TenG connections to another router and formed OSPF Adjacency over These five links. Prefixes are advertised from upstream router to ASR900 via these five links. so for each destination, there will be five entries in Routing table...

When the this router wants to send traffic toward destination learnt from upstream router, It only uses four links out of five and the last link is not used.

I am using following IOS version:

show version | include IOS
Cisco IOS XE Software, Version 03.18.01.S - Extended Support Release
Cisco IOS Software, ASR900 Software (PPC_LINUX_IOSD-UNIVERSALK9_NPE-M), Version 15.6(2)S1, RELEASE SOFTWARE (fc1)
Cisco IOS-XE software, Copyright (c) 2005-2016 by cisco Systems, Inc.
All rights reserved. Certain components of Cisco IOS-XE software are

And this is my OSPF configuration:

router ospf 1
router-id 10.234.100.103
ispf
area 0 authentication message-digest
redistribute connected metric-type 1 subnets route-map Connected--->OSPF1
redistribute static metric-type 1 subnets route-map STATIC--->OSPF1
passive-interface default
no passive-interface TenGigabitEthernet0/1/0
no passive-interface TenGigabitEthernet0/1/1
no passive-interface TenGigabitEthernet0/2/0
no passive-interface TenGigabitEthernet0/2/1
no passive-interface TenGigabitEthernet0/3/0
network 10.234.100.103 0.0.0.0 area 0
network 10.234.196.76 0.0.0.3 area 0
network 10.234.196.80 0.0.0.3 area 0
network 10.234.196.84 0.0.0.3 area 0
network 10.234.196.88 0.0.0.3 area 0
network 10.234.196.92 0.0.0.3 area 0
maximum-paths 8
!

I also enable this command on the box and reloaded it:

show running-config | section platform
platform loadbalance max-paths 8

show ip cef details also shows all five links are used in CEF table but interface TenGigabitEthernet0/3/0 is not used at all and only 0/1/0,0/2/0,0/1/1,0/2/1 are utilized.

What is the root cause of this issue?

Would you please help me out?

Thank you

[Georg Pauwen]

Hello,

are all 5 routes installed in the routing table (sh ip route) ?

[Peyman Zadmehr]

Dear Georg

Yes, For example:

show ip cef 10.2.0.8/30 detail
10.2.0.8/30, epoch 2, per-destination sharing
nexthop 10.234.196.77 TenGigabitEthernet0/1/0
nexthop 10.234.196.81 TenGigabitEthernet0/2/0
nexthop 10.234.196.85 TenGigabitEthernet0/1/1
nexthop 10.234.196.89 TenGigabitEthernet0/2/1
nexthop 10.234.196.93 TenGigabitEthernet0/3/0

This entry is not used

show ip route 10.2.0.8
Routing entry for 10.2.0.8/30
Known via "ospf 1", distance 110, metric 100058, type extern 1
Last update from 10.234.196.93 on TenGigabitEthernet0/3/0, 01:34:28 ago
Routing Descriptor Blocks:
10.234.196.93, from 10.199.233.7, 01:34:28 ago, via TenGigabitEthernet0/3/0
Route metric is 100058, traffic share count is 1
10.234.196.89, from 10.199.233.7, 01:34:28 ago, via TenGigabitEthernet0/2/1
Route metric is 100058, traffic share count is 1
10.234.196.85, from 10.199.233.7, 01:34:28 ago, via TenGigabitEthernet0/1/1
Route metric is 100058, traffic share count is 1
10.234.196.81, from 10.199.233.7, 01:34:28 ago, via TenGigabitEthernet0/2/0
Route metric is 100058, traffic share count is 1
* 10.234.196.77, from 10.199.233.7, 01:34:28 ago, via TenGigabitEthernet0/1/0
Route metric is 100058, traffic share count is 1

I wonder is this related to number of the link. I mean Do we need power of two number of the links here?

Thank you

Peyman

[Peyman Zadmehr]

Hi 

Yes I sent the info before(Start of the thread)

One question here:

I am going to get The mentioned version, Will I encounter Smart Licensing Here? Or Can I disable it and making the device continue working....

Thank you

[Georg Pauwen]

Hello,

smart licensing is mandatory starting with 16.9.1, if you upgrade to 16.8.1, you should be fine without.

[Peyman Zadmehr]

Thank you for the information

I am going to test the IOS.. I hope I wont encounter any other bugs. I say this based on experience..

[MHM Cisco World]

show ip cef interface 0/3/0
check if the per-packet it enable ?

[Peyman Zadmehr]

Hi

Show ip cef interface Ten0/3/0 does not contain this information. it just shows the route which are reachable via this interface.

The output is same for all five interfaces.

Thank you

[MHM Cisco World]

show cef interface x

please try this and send me the output.

[Peyman Zadmehr]

Hi

I attached the requested log. The "Not-Utilized" Interface is 0/3/0

Thank you

[Peyman Zadmehr]

Hi 

I also included the show commands which show information about IP addresses and Utilization

Thanks

[Georg Pauwen · Accepted solution]

Hello,

I found the bug below:

Max of ECMP path to 8 for RSP3C
CSCve16000
Description
Symptom:
RSP3 platform supports only for 4 paths for ECMP load-balancing in hardware.

Conditions:
Only happens when more that 4 ECMP paths exist and IGP (OSPF/ISIS) maximum-paths value is configured to be greater than 4.

Workaround:
Configure IGP (OSPF/ISIS) maximum-paths value to be 4 or less.

Further Problem Description:
None

Product:

Cisco ASR 900 Series Aggregation Services Routers
Cisco Network Convergence System 4200 Series

Known Affected Releases: (1)
15.6(2)SP

Known Fixed Releases: (3)
Fuji-16.8.1
16.8.1
16.8(0.23)

[MHM Cisco World]

Thanks a lot, can you check other interface with 
show cef interface "x"?
I see that this is bug but let be sure.

[Peyman Zadmehr]

Hi this is the requested output on "not using" Interface

show cef interface tenGigabitEthernet 0/3/0
TenGigabitEthernet0/3/0 is up (if_number 23)
Corresponding hwidb fast_if_number 23
Corresponding hwidb firstsw->if_number 23
Internet address is 10.234.196.94/30
ICMP redirects are always sent
IP unicast RPF check is disabled
IP policy routing is disabled
BGP based policy accounting on input is disabled
BGP based policy accounting on output is disabled
Hardware idb is TenGigabitEthernet0/3/0
Fast switching type 1, interface type 89
IP CEF switching enabled
IP CEF switching turbo vector
IP Null turbo vector
IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
Input fast flags 0x0, Output fast flags 0x0
ifindex 22(22)
Slot 0 (0) Slot unit 192 VC -1
IP MTU 1500

This is the output of "using" Interface:

show cef interface tenGigabitEthernet 0/2/0
TenGigabitEthernet0/2/0 is up (if_number 15)
Corresponding hwidb fast_if_number 15
Corresponding hwidb firstsw->if_number 15
Internet address is 10.234.196.82/30
ICMP redirects are always sent
IP unicast RPF check is disabled
IP policy routing is disabled
BGP based policy accounting on input is disabled
BGP based policy accounting on output is disabled
Hardware idb is TenGigabitEthernet0/2/0
Fast switching type 1, interface type 89
IP CEF switching enabled
IP CEF switching turbo vector
IP Null turbo vector
IP prefix lookup IPv4 mtrie 8-8-8-8 optimized
Input fast flags 0x0, Output fast flags 0x0
ifindex 14(14)
Slot 0 (0) Slot unit 128 VC -1
IP MTU 1500

Apart IP address and IfID they are seemed identical to me. What output line must show the problem?

Thank you

[Peyman Zadmehr]

This is a sample of RT:

O E1 10.2.0.0/30
[110/100058] via 10.234.196.93, 1d00h, TenGigabitEthernet0/3/0
[110/100058] via 10.234.196.89, 23:55:26, TenGigabitEthernet0/2/1
[110/100058] via 10.234.196.85, 1d00h, TenGigabitEthernet0/1/1
[110/100058] via 10.234.196.81, 1d00h, TenGigabitEthernet0/2/0
[110/100058] via 10.234.196.77, 21:34:05, TenGigabitEthernet0/1/0
O E1 10.2.0.4/30
[110/100058] via 10.234.196.93, 1d00h, TenGigabitEthernet0/3/0
[110/100058] via 10.234.196.89, 23:55:26, TenGigabitEthernet0/2/1
[110/100058] via 10.234.196.85, 1d00h, TenGigabitEthernet0/1/1
[110/100058] via 10.234.196.81, 1d00h, TenGigabitEthernet0/2/0
[110/100058] via 10.234.196.77, 21:34:05, TenGigabitEthernet0/1/0