topic Re: SXP with multiple border nodes recommended design in Software-Defined Access (SD-Access)

SXP with multiple border nodes recommended design

Jakob Mellberg — Fri, 08 Sep 2023 13:28:44 GMT

Hi!

I was wondering what the recommended approach is for implementing IP-SGT SXP connections to ISE for every VRF from multiple border nodes.
In guides the DNA Center created network loopback address is used with a single border node. In environments with multiple external borders nodes this does not work. And if the network is deleted or changed the SXP connection needs to be remade. I don't want to use the peer-networks because the SXP connection will go down when the peer goes down and takes additional time after the peer-link is up to restore, possibly not enforcing policies correctly for a moment until the SXP connection is back up.

I have managed to get it to work by creating local loopbacks for each vrf and adding it manually in lisp and bgp. But as I understand it is not recommended/supported to manually edit lisp as DNA Center overrides it when configuring new networks.

Re: SXP with multiple border nodes recommended design

Andrii Oliinyk — Fri, 08 Sep 2023 13:47:22 GMT

u need to legalize manual modifications by creating DayN network templates & associating them with devices of interest. After u provision devices w/ that templates from DNAC configuration becomes compliant & always be used as part of device provisioning

Re: SXP with multiple border nodes recommended design

Jakob Mellberg — Fri, 08 Sep 2023 14:08:28 GMT

It seems that "router lisp" is a blacklisted command in dna center templates and cannot be used to override lisp changes.

Re: SXP with multiple border nodes recommended design

Andrii Oliinyk — Fri, 08 Sep 2023 16:47:37 GMT

not sure i get why do u need to use your custom loopbacks under router lisp. u need to have them in the VN & u can directly inject their host-routes into BGP vpnv4 af

Re: SXP with multiple border nodes recommended design

Jakob Mellberg — Fri, 08 Sep 2023 18:56:15 GMT

I have a Pub/Sub fabric with 2 Anywhere Border nodes (Catalyst 9500) that does outbound policy enforcement on the peer vlans.
I am using ISE to push IP-SGT mappings from ISE to all vrfs. The scenario I am trying to mitigate is when one border node looses its upstream bgp I want the SXP connection to still be active through the other border node. Otherwise when the border node recovers its upstream bgp the SXP will not be active and IP-SGT mappings will not be enforced until SXP is back up. I have injected the loopbacks into BGP so the upstream fusion routers can see the route.

The only way I found that the border nodes can reach each others loopbacks is when I manually add the route to lisp using the database-mapping command in the respective vrfs (instance-id).

Without: database-mapping 10.128.4.13/32 locator-set rloc_xxxxxx

ORE-LH-TA05-BN1#sh ip cef vrf INTERNAL 10.128.4.13/32
10.128.4.13/32
nexthop 10.129.0.49 TwentyFiveGigE1/0/45 unusable: no label
nexthop 10.129.0.53 TwentyFiveGigE1/0/46 unusable: no label
ORE-LH-TA05-BN1#ping vrf INTERNAL 10.128.4.13 source lo 3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.128.4.13, timeout is 2 seconds:
Packet sent with a source address of 10.128.3.13
.....
Success rate is 0 percent (0/5)

With manual lisp mapping
ORE-LH-TA05-BN1#sh ip cef vrf INTERNAL 10.128.4.13/32
10.128.4.13/32
nexthop 10.129.0.2 LISP0.4101
ORE-LH-TA05-BN1#ping vrf INTERNAL 10.128.4.13 source lo 3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.128.4.13, timeout is 2 seconds:
Packet sent with a source address of 10.128.3.13
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Is there some other approach I can take?

Re: SXP with multiple border nodes recommended design

jalejand — Fri, 08 Sep 2023 19:01:44 GMT

If you deploy fabric multicast (headend or native, it does not matter), you will now have a unique loopback for the VRF in each Border which you can use as SXP source 🙂

Re: SXP with multiple border nodes recommended design

Andrii Oliinyk — Fri, 08 Sep 2023 19:10:43 GMT

Hi Jay
it make a sense but it still dosnt prevent admin from creation custom interfaces within VNs & advertising them in BGP. assuming this non-DNAC-native configuration will be legalized via network templates/profiles to cut off DNAC incompliance woos :0)
lucky IOS-XE in SDA is totally different from NX-OS in ACI mode

Re: SXP with multiple border nodes recommended design

Andrii Oliinyk — Fri, 08 Sep 2023 19:13:39 GMT

ultimately, it needs time for the product to reach a mature :0D

Re: SXP with multiple border nodes recommended design

jedolphi — Mon, 11 Sep 2023 00:19:14 GMT

-Some customers move SXP to the Fusion device and propagate data plane SGTs between Fusion and SD-Access Border Nodes. I realize it's not always possible, but FYI in case helps in your scenario.

-Correct, at this time we do not allow changes under router lisp, an incorrect modification could take down the SD-Access Fabric so it's considered generally safer to block router lisp.

-If you inject Border Node loopbacks into BGP then you'll need per-VRF IBGP between the Border Nodes to propagate your manual injections when BN-Fusion BGP is down. Note that this will counterfeit Pub/Sub Dynamic Default Border behavior unless you manually block default route between BN IBGP sessions. Because you chose Anywhere Border, BGP is imported to LISP, you may not want to import IBGP routes into LISP when L3HO is down, if you want entirely optimal forwarding paths (it can be a personal preference, or a design restriction, it's not a technical restriction). Perhaps filter per-VRF IBGP to the minimum necessary routes and test it thoroughly in your specific environment.

-You're right, it's reasonable to have an option to inject manually created loopbacks into LISP, please open a feature request by clicking "Make a Wish" in the DNA Center UI, under the question mark menu in the top right corner.

Regards, Jerome