Border Node routing issue

[techno.it]

Hello,

I'm having trouble wrapping my head around a weird issue. I have two borders and one single fusion switch and they only have B1F1 and B2F1 connections. I have imported a route on the Fusion router so that I can leak into the fabric but the border node giving surprising results upon checking the routing table,  FB1 learning routes via LISP instead of BGP, and  CEF entry display the next-hop using the LISP tunnel!!

As an end result, the host inside the fabric cannot reach this subnet located outside the fabric.

I have checked border nodes have no static routes configured for this route.

Attaching some outputs.

Appreciate any suggestions

[jalejand]

Looks like your external route was imported as LISP registration (in case you did internal borders) and know this is known via LISP on Borders.

Important questions:
  did you setup iBGP between your borders in VRF PROD? and if you did, please share the config
  did you add any weight on eBGP neighbors on the Fusion? on Borders should be 65535 but no weight on fusion config

[techno.it]

@jalejand 

Yes, there is iBGP peering for each VRF. Can I share the config in DM?

did you setup iBGP between your borders in VRF PROD? and if you did, please share the config

Yes, it is added on both Borders and Fusion

did you add any weight on eBGP neighbors on the Fusion? on Borders should be 65535 but no weight on fusion config

[techno.it]

@jalejand 

Here you go with BGP config for Border nodes and Fusion  for the prod VRF

FB1

router bgp 65000
bgp router-id interface Loopback0
bgp log-neighbor-changes
bgp graceful-restart

address-family ipv4 vrf PROD
bgp aggregate-timer 0
network 10.1.4.50 mask 55.55.55.55
network 10.1.5.50 mask 55.55.55.55
network 10.1.55.76 mask 55.55.55.252
aggregate-address 10.1.5.0 55.55.55.0 summary-only
aggregate-address 10.1.4.0 55.55.55.0 summary-only
redistribute lisp metric 10
neighbor 10.1.50.73 remote-as 65000
neighbor 10.1.50.73 update-source Port-channel1.20
neighbor 10.1.50.73 fall-over bfd
neighbor 10.1.50.73 activate
neighbor 10.1.55.78 remote-as 65001
neighbor 10.1.55.78 update-source Vlan3020
neighbor 10.1.55.78 fall-over bfd
neighbor 10.1.55.78 activate
neighbor 10.1.55.78 weight 65535
exit-address-family

FB2

router bgp 65000
bgp router-id interface Loopback0
bgp log-neighbor-changes
bgp graceful-restart

address-family ipv4 vrf PROD
bgp aggregate-timer 0
network 10.1.4.50 mask 55.55.55.55
network 10.1.5.50 mask 55.55.55.55
network 10.1.55.12 mask 55.55.55.252
aggregate-address 10.1.5.0 55.55.55.0 summary-only
aggregate-address 10.1.4.0 55.55.55.0 summary-only
redistribute lisp metric 10
neighbor 10.1.50.74 remote-as 65000
neighbor 10.1.50.74 update-source Port-channel1.20
neighbor 10.1.50.74 fall-over bfd
neighbor 10.1.50.74 activate
neighbor 10.1.55.14 remote-as 65001
neighbor 10.1.55.14 update-source Vlan3004
neighbor 10.1.55.14 fall-over bfd
neighbor 10.1.55.14 activate
neighbor 10.1.55.14 weight 65535
exit-address-family

Fusion

router bgp 65001
bgp log-neighbor-changes
bgp graceful-restart

address-family ipv4 vrf PROD
bgp aggregate-timer 0
neighbor 10.1.55.13 remote-as 65000
neighbor 10.1.55.13 update-source Vlan3004
neighbor 10.1.55.13 fall-over bfd
neighbor 10.1.55.13 activate
neighbor 10.1.55.13 weight 65535
neighbor 10.1.55.77 remote-as 65000
neighbor 10.1.55.77 update-source Vlan3020
neighbor 10.1.55.77 fall-over bfd
neighbor 10.1.55.77 activate
exit-address-family

[iores]

How did the loop happened in this scenario? External route was distributed in LISP.

Then, it was redistributed again in BGP. To border neighbor it was redistributed with AD 200 and with no AS path but the neighbor already has eBGP route with AD 20. What have I missed?

[jalejand · Accepted solution]

You might have ended with a LISP-BGP redistribution loop, but also you have a weight on the Fusion facing Borders.

Configure this on your iBGP peers:

Border 1:

route-map tag_local_eids permit 5
set community 65537  

router bgp 65000

  address-family ipv4 vrf PROD

     neighbor 10.1.50.73 route-map tag_local_eids out

end

clear ip bgp 10.1.50.73 vrf PROD out

Border 2

route-map tag_local_eids permit 5
set community 65537  

router bgp 65000

  address-family ipv4 vrf PROD

     neighbor 10.1.50.74 route-map tag_local_eids out

end

clear ip bgp 10.1.50.74 vrf PROD out

Fusion

router bgp 65001

  address-family ipv4 vrf PROD

   no neighbor 10.1.55.13 weight 65535

end

clear ip bgp 10.1.55.13 vrf PROD in

These steps are mentioned on the Fusion Router config guide:

https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/dna-center/213525-sda-steps-to-configure-fusion-router.html

[techno.it]

@jalejand 

those configurations has fixed the issue. thank you for the help

I have removed weight command from fusion

just a small problem, border 2 is still preferred border node for inside and outside fabric hosts.

Any suggestions to fix this ?

[jalejand]

External Borders can't be deterministic, Edge nodes will always send traffic to both of them based on ECMP hashing algorithms, for return traffic it really depends on BGP, you can use BGP multipath on Fusion to do load balance between borders

router bgp 65001

 add ipv4
   maximum-paths 2

add ipv4 vrf PROD

   maximum-paths 2

[techno.it]

@jalejand Just trying to understand, any reasons for removing weight command on Fusion?

[jalejand]

Sure thing, as you are using Anywhere borders and CP, take in consideration that both Borders are also considered "internal", meaning that BGP routes are iimported on LISP DB (because that is what internal borders do)

So imagine the following scenario, where you have a weight of 65536 pointing to Border 2.

1) Fusion announces a route to Border1, this route could be announced by network, redistribution or from BGP. When locally originated (network statement/redistribution) the route is installed on the Fusion table with a weight of 32768, when advertised from a BGP peer, it can contain AS_PATH

2) Route is received on Border1, then Border 1 registers that route into LISP CP, Border 2 which is also a CP will have a LISP registration for that route

3) LISP registrations are exported as LISP routes with AD of 250 on the CPs, thats why you had a route pointing to null0, that is an exported LISP route in one of your borders, in this case B2

4) B2 takes the LISP null0 route and redistributes into BGP (via redistribute lisp metric 10 on BGP),  this route is locally injected, it has no AS_PATH from Fusion anymore.

5) At some point B2 loses its route from Fusion (weight 65k), but it has its route redistributed from LISP (weight 32K), because of timing, it advertises the route to Fusion with no AS_PATH

6) Fusion receives a route with no AS_PATH, and even worse, it is weighted with 65k because of the weight command on the neighbor. It wins over it original source (network, redistribution, from BGP neighbors, etc). A loop is formed.

Thats why you do not weight on Fusions, because of the possibility of Internal Borders advertising a BGP route with no AS_PATH that fusions could take as preferred rather than the original route.

[iores]

Between which nodes is the loop being formed? Is it between fusion-B1-B2-fusion? 

The fusion receives the route with AS path containing 65000. How can this have priority over the locally originated route on fusion?

[techno.it]

@jalejand 
Wow! Super clear.

Thank you for sharing detailed and informative explanation. That was very helpful. I appreciate it.

Last query for this thread, let us assume we have CP + Border on same device and it is only exit point outside the fabric so is it prefer to configure borders as external or anywhere. Please suggest.

Thank you

[jalejand]

I'm glad it was useful!

External only is fine, in the end is the only exit point, the internal capability is to force the traffic to some specific prefixes to specific borders.

[techno.it]

@jalejand 

Do we need to remove weight from Fusion for GRT BGP peering as well?

[Sylvain_Che]

Hi,

Did you get a reply about removing the WEIGHT command for GRT BGP peering?

Thanks,

Sylvain.