SDA Multiple Sites with IP Transit

[packet2020]

Hi,

I have an SDA multisite deployment that currently consists of two large fabric sites. Each fabric site has their own local internet breakout so IP transit has been configured from the fabric borders at each site to a local fusion switch which then connects to a firewall.  

We are now looking to add a third site and deploying SDA transit so that all three sites can communicate with each other with segmenation policy, and so that the third site can use the internet connection at either fabric site 1 or 2. When adding all three sites to the SDA transit, we noticed that fabric sites 1 and 2 advertise each others IP pools via their local IP transits in addition to their local IP pools and the IP pools for site 3. This is the same as what is detailed in BRKENS-2816 (Cisco SD-Access Transit: Advanced Design Principles) page 34.

This configuratuon causes issues with upstream routing as hosts that sit outside of the fabirc at site 1, are now trying to reach hosts that sit within the fabric at site 2 via the site 1 fabric, however the hosts that sit within the fabric at site 2 respond via their local IP transit and local firewall which causes traffic to be dropped due to out of state inspection.

I need to apply some filtering so that the fabric site 1 and 2 borders only advertise their local IP pools in addition to the IP pools for site 3. I noticed that there their is nothing in the IP transit BGP advertisements that clearly identify a local vs foreign prefix so my only option is to apply an IP prefix list on the upstream fusion switches to apply the nessessary filtering using IP address.

Question - Is what I'm experiencing a known consideration when enabling IP transit at multiple sites? Is there a better way to filter the requried prefixes or do I have to rely on standard IP prefix lists to do this?

Hope this makes sense.

[andy!doesnt!like!uucp]

"hosts that sit outside of the fabirc at site 1, are now trying to reach hosts that sit within the fabric at site 2 via the site 1 fabric" - i understand that "outside hosts" live behind FWs relevantly to site 1&2, right? how does FWs learn from your FNs sites prefixes? how does FW propagate it toward "hosts"?

be advised to attach diagram.

[jedolphi]

Hi @packet2020 , understood. For now you'll need to write custom prefix lists / route maps. In the US Cisco Live 2024 version of BRKENS-2816 (should be on ciscolive.com in next few weeks) I showed selective AS Path prepending, in essence it's hand-made prefix lists + route maps steering certain prefixes to either Fabric Site 1 or Fabric Site 2. In future perhaps we (Cisco) need some more BGP communities or something else to distinguish between routes representing different fabric sites. Also another option, possibly not feasible for your setup, but just putting it out there, is connect only one site to IP Transit in order to force routing symmetry, if you don't want to create custom prefix lists and route maps. The single site connected to IP Transit could be site1, or site2, or a new site of just 2x BN (shown below as "Hub Fabric Site") created specifically for IP Transit to SDA Transit interconnect e.g.

Also if you can please you raise this concern with your technical sales team so they can follow it up internally and/or they can raise a feature request. Also you could "make a wish" in the Catalyst Center UI. Thanks, Jerome

[packet2020]

Hi @jedolphi 

Thanks for the reply. I was in the middle or reworking the wording in my initial post along with a diagram as I didnt think that I explained my scenario very well, but yes that is exactly it. With the specifics of my scenario aside, the presence of multiple fabric sites with IP transit in addition to SDA transit, results in the IP transits to advertise routes for all fabric sites which in some instances can result in asymmetic routing. Ideally routes that originate from different fabric sites will be tagged with a different community value as this will then allow simplier/more dynamic route control. I will raise this with our technical sales team.

Also another question if I may. This existing network for this deployment has a large number of routers connected to the access layer with static routes towards these routers configured at either the distribution or core layers that are then redistributed into the IGP. As the SDA fabric edge acts as a stub network (as in we cannot route to external destinations behind the faric edge), we will either need to relocate these routers and connect to the fabric borders, or explore the concept of the border + edge which I've seen mentioned a few times in this forum. Is that correct? Are there any other ways to approach this?

[jedolphi]

Hi! You're right, EN (Edge Node) cannot route peer to an attached endpoint. Yes, relocating the routers is certainly an option. Another option, not always feasible, is to leave router in place and GRE tunnel from router IP interface in the SDA overlay subnet to something outside SDA, then over GRE tunnel route peer as required. Yet another option is to put the routers into the SDA underlay, if the business/security finds that acceptable. Another option is to put routers into L2VN with GWOTF (gateway outside the fabric) and then route peer from the router to GWOTF.

You're right again, another option is to deploy Internal Border Node and BGP peer from the router to IBN. EN+IBN was more accessible in a LISP/BGP routing architecture because it would "pull" only the LISP prefixes required to service data plane flows (there is downsides to this model through e.g. relative slowness because lookup must happen before packet forwarding can happen, static default routes, etc.). Pub/Sub is a different system because all BN will have received and stored all LISP reachability data before data plane packets arrive - you may have noted in BRKENS-2816 that all CP registrations are published from CP to all BN in a given Pub/Sub Fabric Site. This means if you configure dedicated IBN or EN+IBN then the associated switching platform will need resources to hold all LISP publications. So, for example, C9300 EN+IBN will not work if you have 30K endpoints in a Fabric Site (9300 max 16K as per Catalyst Center data sheet). In a Pub/Sub SDA fabric please make sure the IBN (or EN+IBN) has sufficient resources to handle the publication of all CP entries. In medium to long term I think there can be some optimization made in IOS XE to deprioritize publications on resource-constrainted switches, or summarize them, or something of that nature, but this again is a feature request. Please do brief your technical sales team on your requirements and they in turn can propose alternatives and/or raise SDA feature requests.

Best regards, Jerome