SD-Access Transit and SD-WAN transit network requirements

[alessandro.s]

Hi All,

we recently moved our infrastructure to SD-Access interconnecting three sites via a pre-existing MPLS network using IP-Based Transit. Now we are planning to interconnect new sites and we are evaluating to use SD-Access transit or SD-WAN Transit instead but, practically speaking, what are the physical network requirements to use SD-Access and SD-WAN as transit?

and also Can we use SD-WAN Access transit with 3rd party SD-WAN solutions or we can rely, if the requirements are satisfied, in SD-Access transit?

King Regards,

[Flavio Miranda]

Hi @alessandro.s 

 What is it the definition of "Transit" here? 

[alessandro.s]

Hi Flavio,

thanks for your reply, when i write about transit what i mean is transit type used when interconnecting fabric sites between them. I know, correct me if i'm wrong, that we can use three type of transit in SDA:

1. IP-Based transit;
2. SD-Access transit;
3. SD-WAN transit.

and i understand that with IP-Based transit the traffic between the sites is treated as "classic" ip traffic but what i want to understand is what are the requirements to use SD-Access transit or SD-WAN transit netween the sites. I know  for example, we need 9100 byte MTU for SD-Access transit to mantain VXLAN headers, what else is necessary? to have from ISP side to use transit types different than ip-based?

Hope i was clear

[andy!doesnt!like!uucp]

i'll be laconic : dont use SDA-transit over WAN :0)

also, with 3rd party SD-WAN (mean. non-Cisco SD-WAN) u'll likely fallback to IP-transit 

[alessandro.s]

Hi Andy,

thanks for your kind reply! so do you think, in general, si better to not use SD-Access transit? and why, if i can ask. about my first question, as far as you know, any requirements other 9100 byte mtu to use SD-Access transit?

[andy!doesnt!like!uucp]

apart of jumbo-frames enablement just notice >1Gbps recommendation from Cisco for the SDA-transit. 

u also have to keep in mind how LISP control plane works to be quite safe to use it for inter-site communications especially in large deployments.

but if u have to interconnect several sites with use of dark fiber or similar u can try SDA-transit.

also look here for high-level thoughts about LISP Should We Use LISP? « ipSpace.net blog

[jedolphi]

While 9100 is certain to work, SDA Transit does not strictly require a 9100 transport MTU. SDA Transit needs to be able to accommodate the ingress packet size + 50B (VXLAN encap). So hypothetically speaking if all ingress packets were max 1000B in size then the VXLAN encap between Border Nodes connected to SDA Transit would require at least 1050B. Today we can template Adjust TCP MSS to Fabric Edge Node SVIs to tune down TCP Frame sizes, and in 2.3.7.x Adjust TCP MSS has been added to the Anycast Gateway UI.

In other words, the MTU required for SDA Transit is not carved in stone, it depends on the applications in use and how Adjust TCP MSS has been deployed. And there is no min bandwidth requirement for SDA Transit, FYI.

[andy!doesnt!like!uucp]

Hi Jerom

few citing from Software-Defined Access for Distributed Campus Deployment Guide - Cisco :

"The key consideration for the Distributed Campus design using SD-Access transit is that the network between fabric sites and to Cisco DNA Center should be created with campus-like connectivity. The connections should be high-bandwidth (Ethernet full port speed with no sub-rate services), low latency (less than 10ms as a general guideline), and should accommodate the MTU setting used for SD-Access in the campus network (typically 9100 bytes).  The physical connectivity can be direct fiber connections, leased dark fiber, Ethernet over wavelengths on a WDM system, or metro Ethernet systems (VPLS, etc.) supporting similar bandwidth, port rate, delay, and MTU connectivity capabilities." 

Also adjusting TCP MSS doesnt resolve UDP jumbo frames treatment (one necessary will meet it with 802.1X EAP-TLS) thus raising architectural topic to fragment it before it enters SDA transit.

[jedolphi]

Hey Andy,

*That document is over 4 years old, some of the assertions within it are no longer strict. Customers can and do run SDA Transit over lower speed WAN circuits with MTU substantially less than 9100B.

*RADIUS / EAP-TLS for Fabric Nodes runs in the underlays, thus is not subject to VXLAN encapsulation, thus standard WAN design rules apply.

*Large UDP is something we should be wary of, but there's no such thing as 9100 UDP, so what is the maximum UDP size in your network? Most will find it's <1500B.

*IOS XE 17.7+ will send an ICMP T3C4 back towards an endpoint originating a packet, if the packet + VXLAN encap exceeds the encapsulating platform's MTU. This in turn mandates a longer discussion on holistic e2e MTU designs.

*I have a plan to assemble some updated explanatory content, give me a few months.

TLDR; there's no exactly correct answer, but rather a set of rules that if followed should give us a successful outcome

[andy!doesnt!like!uucp]

Hi Jerom
thanks for update. just few notices from my side on MTU topic (basically it's not specific to VXLAN :0):
1) let's assume one overrides automated AAA configuration (which i honestly can only assume how looks like per lack of SDA handson practice :0) with vrf-specific dot1x configuration via network templates (f.e. INFRA_VN). since that point dot1x AAA will happen in overlay, right? & therefore will become subject to VXLAN. is this scenario realistic enough?
2) i saw DNS replies being 4KB+ large because server settings allowing jumboframes (&DFbit==0 luckily :0). u right w/o customization we'll have 1.5KB MTU on server side but relying on this is not quite reliable w/o , citing u, holistic e2e MTU designs.

but while it's still manageable, there is still concerns about LISP as control plane in large IP-environments where i tend to agree with Ivan P. argues. 

[jedolphi]

Hi Andy, 1. all RADIUS happens in SDA underlay, there's no way to move it to an overlay. 2. any application that requires a minimum 4K MTU will not work well with a < 4K MTU WAN, regardless of SDA Transit or no SDA Transit. I assume this DNS implementation would therefore include PMTUD capabilities (DF bit set), and as such the Edge Node ICMP T3C4 I mentioned above should take care of PMTUD. 3. Debating Ivan's article is a rabbit hole I'd rather not go down in this discussion thread, we can open another thread to debate specific points if you feel compelled . 4. We have a customer deploying an international SDA Transit now across hundreds of sites around the world now. I will admit that achieving this does require a more advanced level knowledge that's not generally available - I'd like to unpack it in a Cisco Live presentation next year if the CL session owners will allow it. Cheers, Jerome

[andy!doesnt!like!uucp]

Hi Jerom
just one point about RADIUS in GRT. i guess u mean configuration deployed by DNAC to  switch (LISP VXLAN Fabric Configuration Guide, Cisco IOS XE Cupertino 17.9.x (Catalyst 9000 Series Switches)) - ip rad so Lo0 in GRT etc. But what if netadmin decides to change automated configuration with provisioning via network templates to move RADIUS into INFRA_VN f.e.?

[jedolphi]

Yo Andy, Edge Node (EN) Interfaces in INFRA_VN are Anycast addresses, meaning they are not unique, they exist on all Edge Nodes. Anycast addresses cannot be used for source or destination of packets because there's no return route to a specific device, there's only a return route to all devices, the only exception to this rule is IP Helper with uses option 82 to steer DHCP responses back to the correct EN. Thus RADIUS must be sourced from a unique IP address on each EN, which in turn means it must be outside of INFRA_VN. Additionally we would very strongly recommend not deviating from the DNA Center automated RADIUS configurations without first having a robust conversation about the support and reliability implications of doing so. Regards, Jerome

[andy!doesnt!like!uucp]

hi Jerom
definitely i meant something different like int lo4099 under vrf. which is matter of the arbitrary architectural approach.
to avoid it to happen recommendation to not customize automated configuration must be spread world wide :0)

[jedolphi]

I take your point Andy, if we try we can find a way to break it, and conversely if we design well we can probably find a way to make it work. I'll try to create some new SDA Transit advanced design collateral over next few months. In the mean time the sales team and CX are available for a deep dive conversations.