Floating Static routes redistribution into BGP

[rajpanchal1978]

Hello All,

I am trying to setup a dual site multihomed network. Attached is the diagram showing topology and how the routing tables are anticipated to be populated using BGP and static routing.

The site on the left (mas11/12.gw) is primary site and one on right (bla11/12.gw) is secondary site. Each site is eBGP dual homed into Transit as shown.

Primary and Secondary site are of same MPLS core and use MPLS VPN . M19:Interchange VPN is external VPN facing Trasit and M26:OCIX is a internal VPN facing internal networks. Both VPNs are interconnected using a FW at each site, hence any traffic being exchanged between Internal Networks and Transit is subject for Firewall .

Secondary site should not be used in normal working scenario , it will be only used when the primary site is entirely out of service !

For this reason , on the secondary side , I am costing out all static and BGP routes higher making it NOT preferable as far as routes from primary site are available.For staic routes I am trying to use higher AD of 250 , for BGP routes I am trying to set the lower local-preference for routes recieved from eBGP peers

As secondary site is backing up primary site , routing information available at each site will be exactly the same (except the interface address) but with difference in cost/preference facilating primary/secondary site arrangement.

The static routes on Primary site are redistributed into BGP for advertising it to Transit peers(using eBGP) and also to secondary site(using MP(i)BGP.

The floating static routes(with AD 250) on secondary site are also being redistributed into BGP for advertising them to Transit peers(using eBGP when primary site is out of service) and also primary site (using MP(i)BGP)

Both primary and secondary sites are recieving same routing information from Transit but on primary site the local pref is set to 100 while on secondary site the local pref is set to 90 for the routing information recieved from transit

Now comes the problem description !

Everything works fine with this arrangement as far I dont flap the primary site to simulate a Primary site outage resulting in static routes at primary site being stopped getting redistributed into BGP. I achieve this by shutting down the VLAN 61 SVI on mas11/12.gw routers.

Before I flap the primary site , I can see the BGP routes (with AD of 200 ) recieved from primary site routers in M19:INTERCHANGE VPN  are preferred against the floating static routes ( with AD of 250 ) .. sweet as expected . Also the eBGP routes advertised out to Transit peer at secondary site are the one leant via primary site as expected.

Now with the flap induced , no routes (static routes redistrbuted at Primary site ) are recieved at secondary from primary site in M19:INTERCHANGE VPN  and Floating route routes (with AD 250 ) are installed in the routing table of M19:INTERCHANGE VPN, as that is the obvious choice. These also gets advertised to Transit Peers as expected.

However when the primary site is restored back , I can see that though the static routes of primary site , which are recieved on secondary site as iBGP routes ( with AD of 200 ) are not able to replace the floating static routes ( with AD of 200 ) upon restoration .

I have gone through some blogs/posts , and undestand that many have encoutered a similar issue , and the fix is to set the weight = 0 and local-preference = less than primary site routes for the static routes redistributed into BGP at secondary site.

I tried it and seems it working for me !!!!!!!

However I am trying to understand why is it so that we need to tune bgp metrics for static-to bgp redistrbution and the AD  arrangement doesnt work on its own in this case.

  ... As per cisco , Administrative distance is the first criterion that a router uses to determine which routing protocol to use if two protocols provide route information for the same destination. Administrative distance is a measure of the trustworthiness of the source of the routing information. Administrative distance has only local significance, and is not advertised in routing updates.

If that is the case , in the tie breaker to choose the best route to same destination  offered by iBGP routes(with AD 200) and Floating static routes (with AD 250) , the RIB should prefer iBGP route isnt it ?

Where as looks like it doesnt happes that way at first point .. and we go by tuning weight and local preference of the floating static route while it is being redistributed into BGP at secondary site ... What is the need for this ... as redistrbution is being generally done in order to propogate the routing information to remote routers . On the same router RIB should have been looking into AD and route metrics to choose the best path ????

Cheers

Raj

[Vaibhava Varma]

Hi Raj

From my understanding below listed is what's actaully happening in this setup.

Primary-Site (P) and Secondary-Site (S) have iBGP b/w them and also P & S have eBGP peering to Transit.

Now we have a Static Route (SR) to Destination X both at P & S. S is using poor AD value and also S is redistributing them into eBGP . Now on S its receing Route to X from P as an iBGP with AD-200 and also since S is locally redistributing this destination X into eBGP when the BGP best path algo runs it will select its own redistributed X into eBGP as best path owing to the contention on Weight Criterion as the Locally Redistributed X Route has Weight of 32768 and the iBGP Route X from P would have Weight 0 as Weight is local to router. Same applies fro eBGP route from Transit. Thats why we do not see it via iBGP Peer P despite of AD 200. The only way to override this is to manipulate set the Weight to 32768 to avoid contention on Weight Criterion and set the Higher LP on S for destination X and it will select X from P always as the very second criterion for Best Path Selection is LP beofre Locally Originated Routes.

Also this will remove the Floating-Static Route from the BGP Table of S as it was not selected for Best Path in BGP and now has poor AD 250. I think if we notice and take a look on S and do a show ip route for X that it will show to be learnt locally always and  traffic for destination X will always flow locally and not via P in this scenario which we are using even when Site P is in production.If we really want to keep Site as Secondary always we need to make the above changes and then as long as Site P is available Traffic for X will flow via P.

Also on a side note depending upon the BGP convergence and RIB Compuation it might happen that once the route X is preffered from iBGP Peer P even if we restore to default the Weight and LP for P it will still be preffered as best route ( what we originally wanted ) as X would have been discarded from RIB earlier and also from BGP-RIB-Out and hence the iBGP route will be preffered over Floating-Static Route - the normal behaviuor based on AD value.

So the safe option is to manipulate the weight and LP for Peer P.

Hope this clarifies your doubt. Do let me know if still some clarification required.

Regards

Varma

[Marwan ALshawi]

Excellent description Varma. 5+

[Vaibhava Varma]

Thank You Very Much marwanshawi

Regards

Varma