Re: SD-Access Transit vs Ip Transit

tsmarcyes · ‎01-19-2021

I believe I understand the basic differences between using SD-Access transit and using ip transit between different fabric sites. One of the big benefits of SD-Access transit is the preservation of SGT tagging between sites. This is where Im a little hung up on though. Why cant ip transit do the same though. I understand for ip transit that the BN decapsulates the packet with vxlan header, hence the SGT tag is lost. My question is why does the BN do this? The vxlan header is encapsulated in an outside IP/UDP packet. Why doesnt the BN simply maintain the encapsulated vxlan packet and forward it to the destination BN based on the routing table? It would seem the L3 devices inbetween the BNs simply route based on the outside IP header, not the vxlan header, right? So, the BN at site1 would keep the packet encapsulated, put the destination IP in the packet for the destination Site2 BN), and route it to its next hop based on routing table. Intermediate L3 devices get the packet, and again route it to their next hop toward the destination (Site 2 BN). They dont care about the inner vxlan header as they are simply routing based on the outer ip header. Once it gets to the Site 2 BN, the BN decapsulates the packet, see the vxlan header inside, sees the SGTs, and routes accordingly within the fabic. What am I missing?

georgehewittuk1 · ‎01-19-2021

Sorry if I misunderstand you but the major reason and issue in using sd access transit for some designs is having a larger enough MTU (to preserve tags/header) and correct equipment to facilitate whether be the existing infra a business has or wan mtu issues they may have.

SXP can be used with ip transit design for preserving Sgt tagging.

tsmarcyes · ‎01-19-2021

But if you do SD access, then you will be adding the Vxlan header anyways.

georgehewittuk1 · ‎01-20-2021

In IP transit you are not using VXLAN.

SD-Access transit will build VXLAN tunnels between the transit control plane nodes at each fabric site. You must meet certain requirements for latency and importantly MTU because of the increased sizing which is not always possible with some designs.)

Jonathan Cuthbert · ‎01-20-2021

@georgehewittuk1 wrote:

In IP transit you are not using VXLAN.

SD-Access transit will build VXLAN tunnels between the transit control plane nodes at each fabric site. You must meet certain requirements for latency and importantly MTU because of the increased sizing which is not always possible with some designs.)

A small correction here for clarity. We actually build a tunnel from Loopback 0 to Loopback 0 between border nodes in different fabric sites across the SD-Access transit.

The transit control plane node is not part of the data plane forwarding process. It is used just for control plane communications (signalling). In fact, in an ideal network, your transit control plane node needs to be out of the data forwarding path. Details on this are covered in this section of the SD-Access CVD (Key Considerations for SD-Access Transits). Other design methodologies related to both site-local and SD-Access Transit Control Plane Nodes can be found here (off-path control plane nodes).

tsmarcyes · ‎01-20-2021

Thanks everyone for the clarity and the actual details have helped a lot. However, I dont think we are getting to the heart of my question.

It seems one of the main goals for the SD-access method was to allow the SGT to be preserved across the transit, correct? But my question is why for IP transit, not SDA transit - why couldn't the BN look at its routing table for the destination route for the endpoint (BN in site2 would have exported summarized prefix into the transit network), see the next hop and route the packet but also KEEP the encapsulation with the vxlan(and SGT). The IP transit would then simply route the packet towards BN2 like any other normal IP packet. BN in site2 would then decapsulate the packet, see the vxlan and route accordingly within the fabric in site2. However, in reading these other posts and if my understanding is correct, theoretically this is what SD-access IS actually doing as far as encapsulating the packet and sending it to BN2. . However sd-access is adding an additional control plane functionality with the transport CP to facilitate the mapping between fabric sites and not relying solely on the routing table for the endpoint routes/ip addresses.

This still leaves the question on why couldnt the BN simply rely on the routing table with the summarized endpoint routes/prefixes (being advertised from other sites) and then simply encapsulate the packet and send the encapsulated packet (with vxlan and SGT) to the next hop and eventually to the next BN.

However, I think I know the reason why and its because the BN cant simply ALWAYS encapsulate every packet going into the transit including the vxlan tag (and SGT), because what if the destination isnt another BN/endpoint that can decapsulate the vxlan packet. What if instead the destination is a server or firewall in the transit network? If the BN kept the encapsulation ALWAYS, then when a server or firewall received the packet, they would not understand the vxlan tag and hence most likely drop it.

Is all my understanding and assumptions correct?

Jonathan Cuthbert · ‎01-21-2021

Is all my understanding and assumptions correct?

Basically, yes.

However, I think I know the reason why and its because the BN cant simply ALWAYS encapsulate every packet going into the transit including the vxlan tag (and SGT), because what if the destination isnt another BN/endpoint that can decapsulate the vxlan packet. What if instead the destination is a server or firewall in the transit network? If the BN kept the encapsulation ALWAYS, then when a server or firewall received the packet, they would not understand the vxlan tag and hence most likely drop it.

In another post, I mentioned that across the SD-Access transit, we build a tunnel between the Loopback 0 interface of border nodes in the sites. This same tunneling behavior happens in the fabric site itself. We build a tunnel between Loopback 0 of the edge node and the border node to send the data across the fabric site.

If you want to conceptualize it differently, consider the configuration for a GRE tunnel.

tunnel source 2.2.2.2
tunnel destination 1.1.1.1

Except in the fabric site from edge node to border node we have:

tunnel source Loopback 0 (Edge Node)
tunnel destination Loopback 0 (Border Node)

and across an SD-Access transit:

tunnel source Loopback 0 (Border Node Site X)
tunnel destination Loopback 0 (Border Node Site Y)

Why am I making this correlation? The tunnel destination is a stop point for the encapsulation. At the destination, we de-encapsulate. In VXLAN with BGP-EVPN, we use the term VTEP (Virtual Tunnel End Point).

At the border node, the packet is de-encapsulated and then forward. How it is forwarded in the data plane is based on the transit type I described in another post. In IP-Based transits, the border node forwards the packet using traditional IP.

Could another method be used? Yes, but that is why we have other transit types like SD-Access Transits which tell the border node to query the transit control plane node and then forward with fabric VXLAN data plane encapsulation rather than traditional IP.

Why do we use traditional IP? Consider the sheer amount of variations in networks you have in your own deployments and just networks globally. From the perspective of the fabric site, we use encapsulation to get from edge node to border node. At the border node, there are so many variations on what is north bound. It might be a fusion device, it might be a CE router, it might be an ISP router, it may be something that is not even managed by the administrator of the border node. It could be your own network or someone else's network. The varieties are endless. The similarity between all these networks is that, all things being equal, they use IP. So by forwarding the original IP packet, de-encapsulated, we provide a way to address all of these varieties and allows you to have flexibility in your network designs and what your border node is connected to. Hopefully this all makes sense.

This still leaves the question on why couldnt the BN simply rely on the routing table with the summarized endpoint routes/prefixes (being advertised from other sites) and then simply encapsulate the packet and send the encapsulated packet (with vxlan and SGT) to the next hop and eventually to the next BN.

In short, that is not how the IP-based handoffs are designed as I described above.

Jonathan Cuthbert · ‎01-20-2021

@tsmarcyes wrote:

The vxlan header is encapsulated in an outside IP/UDP packet. Why doesnt the BN simply maintain the encapsulated vxlan packet and forward it to the destination BN based on the routing table? It would seem the L3 devices inbetween the BNs simply route based on the outside IP header, not the vxlan header, right? So, the BN at site1 would keep the packet encapsulated, put the destination IP in the packet for the destination Site2 BN), and route it to its next hop based on routing table. Intermediate L3 devices get the packet, and again route it to their next hop toward the destination (Site 2 BN). They dont care about the inner vxlan header as they are simply routing based on the outer ip header. Once it gets to the Site 2 BN, the BN decapsulates the packet, see the vxlan header inside, sees the SGTs, and routes accordingly within the fabic. What am I missing?

You have described the behavior of what happens across an SD-Access transit.

We have three different types of transit.

IP-Based Transit
SD-Access Transit
SD-WAN Transit

At first glance, it would seem that these transit indicate the physical network northbound of the border node. This is essentially true.

Another way of looking at it is that these transit types describe the behavior the border node uses when sending traffic outside of the fabric (received from devices in the fabric).

In IP-Based transits, we use native IP. This means we remove the encapsulation added in the fabric and forward the original packet.
In SD-Access transits, we use Fabric VXLAN. This means we encapsulate the original packet in VXLAN and forward it.
In SD-WAN transits, we use SD-WAN encapsulation. This means we encapsulate the original packet in SD-WAN and forward it.