eBGP convergence on A9K

[smailmilak]

Hi,

we did a stress test this morning on our IGW (ASR 9912; RP2; TH HW).

There are multiple eBGP sessions and we get full RT from them.

If we disable ISP1, it takes around 35 seconds to failover to ISP2. It takes a few seconds to tear down the BGP session and under the BGP table ISP2 is selected for path XX, but it takes around 35 seconds to get this route in the routing table.

When BGP session to neighbor 80.91.255.123 goes down the route is still in the routing table.

show route vrf Internet 31.13.93.36
Wed Jun 7 06:02:46.321

Routing entry for 31.13.93.0/24
Known via "bgp 65101", distance 20, metric 0
Tag 1299, type external
Installed Jun 7 05:33:50.113 for 00:28:56
Routing Descriptor Blocks
80.91.255.123, from 80.91.255.123, BGP external
Route metric is 0

Under show bgp vrf Internet table the backup path is selected. See Path #1

show bgp vrf Internet 31.13.93.36
Wed Jun 7 06:03:13.948
BGP routing table entry for 31.13.93.0/24, Route Distinguisher: 10.100.96.1:2
Versions:
Process bRIB/RIB SendTblVer
Speaker 253200139 253200139
Last Modified: Jun 7 06:03:05.067 for 00:00:08
Paths: (3 available, best #1)
Advertised to PE peers (in unique update groups):
10.100.96.2
Advertised to CE peers (in unique update groups):
xxxxxxxxxxxxxxxxxxxxxxxx
Path #1: Received by speaker 0
Advertised to PE peers (in unique update groups):
10.100.96.2
Advertised to CE peers (in unique update groups):
xxxxxxxxxxxxxxxxxxxxxx
1299 32934, (received & used)
10.100.96.2 (metric 31) from 10.100.96.2 (10.100.96.2)
Received Label 57225
Origin IGP, localpref 101, valid, internal, best, group-best, import-candidate, imported
Received Path ID 0, Local Path ID 0, version 253200139
Community: 1299:30000
Extended community: RT:65101:50 RT:65101:101
Source AFI: VPNv4 Unicast, Source VRF: default, Source Route Distinguisher: 10.100.96.2:2
Path #2: Received by speaker 0
Not advertised to any peer
5391 3320 32934
195.29.254.121 from 195.29.254.121 (195.29.254.121)
Origin IGP, localpref 101, valid, external, group-best
Received Path ID 0, Local Path ID 0, version 0
Community: 10200:0
Extended community: RT:65101:50 RT:65101:101
Path #3: Received by speaker 0
Not advertised to any peer
5391 3320 32934, (received-only)
195.29.254.121 from 195.29.254.121 (195.29.254.121)
Origin IGP, localpref 100, valid, external
Received Path ID 0, Local Path ID 0, version 0
Community: 10200:0

BGP table has selected new path but under the routing table it's still the old path that is down.

Compare timestamp.

show route vrf Internet 31.13.93.36
Wed Jun 7 06:03:15.625

Routing entry for 31.13.93.0/24
Known via "bgp 65101", distance 20, metric 0
Tag 1299, type external
Installed Jun 7 05:33:50.112 for 00:29:25
Routing Descriptor Blocks
80.91.255.123, from 80.91.255.123, BGP external
Route metric is 0

After around 35 seconds new next-hop is installed.

show route vrf Internet 31.13.93.36
Wed Jun 7 06:04:14.556

Routing entry for 31.13.93.0/24
Known via "bgp 65101", distance 200, metric 0
Tag 1299, type internal
Installed Jun 7 06:03:56.989 for 00:00:17
Routing Descriptor Blocks
10.100.96.2, from 10.100.96.2
Nexthop in Vrf: "default", Table: "default", IPv4 Unicast, Table Id: 0xe0000000
Route metric is 0
No advertising protos.

IGW is running MP-BGP. Full RT is in vrf "Internet".

I did some testing on XRv and failover time is only a few seconds, BUT it's only a few prefixes. Not 600K.

What can I do to speed this process up?

[Tim.Yu · Accepted solution]

Try this command:

router bgp xxx
vrf Internet
address-family ipv4 unicast
maximum-paths ibgp 16

If you are using RR, I think you should also do add-path on both RR and IGW

router bgp 65545
address-family vpnv4 unicast
additional-paths receive
additional-paths send
additional-paths selection route-policy multipath

route-policy multipath
set path-selection all advertise
end-policy

I have done this on xrv, it works well. But in my case med is same for both paths. 

RP/0/0/CPU0:Border#show cef vrf internet 10.0.0.10/32
Mon May 22 06:24:30.359 UTC
10.0.0.10/32, version 30, internal 0x5000001 0x0 (ptr 0xa1409af4) [1], 0x0 (0x0), 0x208 (0xa1851050)
Updated May 22 06:21:08.903
Prefix Len 32, traffic index 0, precedence n/a, priority 3
via 10.127.224.1/32, 7 dependencies, recursive, bgp-multipath [flags 0x6080]
path-idx 0 NHID 0x0 [0xa15f3b74 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 10.127.224.1/32 via 17001/0/21
next hop 3.3.3.11/32 Gi0/0/0/2 labels imposed {17001 24000}
via 10.127.224.2/32, 5 dependencies, recursive, bgp-multipath [flags 0x6080]
path-idx 1 NHID 0x0 [0xa15f4b74 0x0]
recursion-via-/32
next hop VRF - 'default', table - 0xe0000000
next hop 10.127.224.2/32 via 17002/0/21
next hop 3.3.3.11/32 Gi0/0/0/2 labels imposed {17002 24004}

If the med is different, you should use PIC-edge feature. Try this command " additional-paths install backup"   After this, you should see there is backup path installed in bgp and cef.

[smailmilak]

Hi,

I did some testing today and adding a backup path into RIB and FIB was simple. 

Your way is also correct (the one with different MED). The syntax is slightly different. I will mark it as correct.

router bgp 65101

vrf Internet

   address-family ipv4 unicast

    additional-paths selection route-policy BACKUP_PATH

 RP/0/0/CPU0:IGW_APOLJE#sh route vrf Internet 8.8.8.8

Routing entry for 8.8.8.8/32

  Known via "bgp 65101", distance 20, metric 0

  Tag 1299

  Number of pic paths 1 , type external

  Installed Jun  8 12:06:35.778 for 00:02:24

  Routing Descriptor Blocks

    50.2.2.1, from 50.2.2.1, BGP external

      Route metric is 0

    50.2.2.9, from 50.2.2.9, BGP external, BGP backup path

      Route metric is 0

show cef vrf Internet 8.8.8.8

Thu Jun  8 12:12:51.282 UTC

8.8.8.8/32, version 211, internal 0x1000001 0x0 (ptr 0xa13f4b74) [1], 0x0 (0x0), 0x200 (0xa175228c)

Updated Jun  8 12:11:38.867

local adjacency 50.2.2.1

Prefix Len 32, traffic index 0, precedence n/a, priority 3

   via 50.2.2.1/32, 5 dependencies, recursive, bgp-ext [flags 0x6020]

    path-idx 0 NHID 0x0 [0xa13f25f4 0x0]

    next hop 50.2.2.1/32 via 50.2.2.1/32

     local label 24011

     next hop 50.2.2.1/32 Gi0/0/0/0    labels imposed {None}

   via 50.2.2.9/32, 6 dependencies, recursive, bgp-ext, backup [flags 0x6120]

    path-idx 1 NHID 0x0 [0xa13f2b74 0x0]

    next hop 50.2.2.9/32 via 50.2.2.9/32

    local label 24011

     next hop 50.2.2.9/32 Gi0/0/0/1    labels imposed {None}

sh run route-policy BACKUP_PATH

route-policy BACKUP_PATH

  set path-selection backup 1 install

end-policy

!

[joshuacmoore]

Do you happen to have any data on how much additional RAM this consumed on your ASR9k?

[smailmilak]

Hi Joshua,

it seems that it did not consume much with PIC Edge feature enabled.

HW is RP2 with 16GB. LC's are all Tomahawk TR.

before PIC Edge

show memory
Thu May 25 10:21:12.709 
Physical Memory: 16384M total (11037M available)
Application Memory : 16042M (11037M available)

With PIC Edge 

show memory
Fri Jun 23 10:17:06.405 
Physical Memory: 16384M total (9857M available)
Application Memory : 16042M (9857M available)

[Tim.Yu]

This is reasonable because BGP tables are saved "next-hop independently", witch means they share a same pointer with next hop information if the next hop of routes are same. Also RIB and FIB should have similar structure. I am not developer from cisco but I think normally they should implement like this..

So only when there are lots of different next hops in your routing table, enabling BGP PIC will consume a lot memory

[smailmilak]

Sound logical. 

Customer has three possible next-hops for most of the routes.