Underlay Design for SD-Access Distributed Campus

[packet2020]

Hi All,

I'm currently working on another SDA design that will comprise of roughly 25 fabric sites with a total of 1k fabric edge nodes. All sites are connected togther using a Metro-E service in a hub and spoke topology with the main campus site being the hub. The main campus site will provide IP Transit (external connectivity) for all fabric sites with all fabric sites connecting togther using SDA Transit.

I'm currently looking at the following options to deploy the underlay:

1) The underlay for each fabric site is deployed using LAN automation (using IS-IS as the IGP). Each fabric site then connects to the Metro-E service using eBGP with each site allocated its own private ASN. The LAN automation pool at each fabric site is summarised and advertised into BGP (each fabric site operates its own independant IS-IS area).

2) The underlay for each fabric site is deployed using LAN automation (using IS-IS as the IGP) but with IS-IS extended over the Metro-E service creating one large/flat IS-IS area with L2 adjacenices between all border/intermediate/edge nodes and sites. This is referenced in the SDA for Distributed Campus deployment guide below, however it doesnt go into detail along with recommendations and best practises.

https://www.cisco.com/c/en/us/td/docs/solutions/CVD/Campus/SD-Access-Distributed-Campus-Deployment-Guide-2019JUL.html

"Within each site, the IS-IS routing protocol is used to advertise underlay routes and to provide IP reachability between Loopback 0 addresses. Through the Metro-E circuit, the Enterprise Edge routers and switches (fabric border nodes) at each site are IS-IS adjacent with each other so that there is full, end-to-end IP reachability between sites across the private circuit"

Has anyone deployed SDA in a large distributed campus before that can share recommendations and best practises? A sinlgle IS-IS area would be the simplest to deploy, but I'm concerned that this will bloat the routing table with thousdands of p2p and loopback 0 IP addresses shared across all fabric sites, especially as more edge nodes and sites are deployed. BGP will allow us to summarise the underlay network for each fabric site while also providing more flexibility with filtering and general traffic engineering, but its an additional protocol to deploy and troubleshoot for all sites.

Thanks

[M02@rt37]

Hello @packet2020 

In large-scale SDA deployments, the design of the underlay network is crucial. As concerned yourdeployment scenario, involving roughly 25 fabric sites and 1,000 fabric edge nodes connected through a Metro-E service, presents a significant challenge in balancing simplicity against the demands of a large and growing network...

Option 1—which involves deploying independent IS-IS areas for each fabric site and using eBGP to connect these sites via the Metro-E service—offers a robust solution tailored to large-scale networks. This approach keeps the routing tables at a manageable size by summarizing routes at each site and only advertising those summarized routes via BGP. This not only prevents the network from being flooded with unnecessary route information but also provides greater control over routing policies, which is essential for effective traffic engineering and filtering. However, the trade-off is added complexity; managing both IS-IS and BGP requires a deeper understanding of each protocol and meticulous configuration, particularly as the network scales. Nevertheless, the benefits of scalability and control outweigh the complexity, making this approach well-suited for a distributed campus with significant growth potential.

Option 2 proposes extending IS-IS across all sites in a single, flat area. While this simplifies deployment by using only one routing protocol across the entire network, it introduces significant risks in terms of scalability and stability. A single IS-IS area can lead to a bloated routing table as every node in the network must process and store the routes for all other nodes. As the network grows, this can strain device resources, potentially slowing down convergence times and increasing the risk of network instability. Furthermore, any issue within the IS-IS area could have a wide-reaching impact, complicating troubleshooting and recovery efforts. Although this approach might be easier to deploy initially, its limitations become more pronounced as the network expands.

For a large and complex SDA deployment like yours, I strongly recommend Option 1. This approach, though more complex, provides the necessary scalability and control to manage a large, distributed campus effectively. The ability to summarize routes and apply detailed routing policies through BGP will be crucial as your network grows, ensuring that it remains manageable and resilient. While the additional complexity of deploying BGP requires careful planning and expertise, it is a worthwhile investment for the long-term stability and scalability of your SDA deployment.