Questions about ACI Cloud/anywhere and L3Out in v4.0

[m1xed0s]

Regarding the ACI Anywhere, especailly ACI in AWS/Azure, is there still a higher MTU requirement on the ISN between AWS/Azure and On-prem? If so, what is the required MTU?

With ACI v4.0, the L3Out can do host specific route advertisement. So is there still a need to use GOLF anymore?

[Remi-Astruc · Accepted solution]

Hi @m1xed0s ,

Yes. As long as ACI Anywhere still uses VXLAN encapsulation (and optionally IPSec or CloudSec on top of it), the requirement of "tenant traffic MTU + 100B" is still required.

If using Internet as the ISN, you need to set the servers MTU to <1400B.

An other option is to rely on the IPSec/CSR devices to fragment (because ACI does not).

If using a Direct Connect solution, it may support MTU of 9100B.

Right, the complexity of GOLF is not needed anymore in 4.x (unless very specific requirement).

[m1xed0s]

Thanks.

So just use ACI in AWS as the example and the transport between on-prem and AWS is the Internet with IPSec tunnel:

1. All the servers on-prem and in AWS would have to be tweaked to use 1400 MTU in order to communicate with each other (including migration)? That sounds like a lot of work as a "workaround"...Plus this potentially also affects the on-prem/AWS inter-server and client-server communication, right? 

2. If the servers can not be modified to use 1400 MTU OR customer is not willing to do so, the IPSec tunnel termination points (assuming ASR on-prem and CSR in AWS) would need to do fragmentation, right? If so, what would be the performance impact?

Lastly, what would be the specific use case/requirement for using GOLF instead of the L3Out with host specific advertisement?

[Remi-Astruc]

Hi @m1xed0s ,

1. Right.

2. Right, but you may have other mechanisms available to avoid fragmentation (Path MTU discovery, TCP adjust MSS, ...). But please note that this constraint is not specific to ACI-Anywhere but to any tunneling technology over internet!

Very specific use case for GOLF instead of v4.x Host Advertisement could be scaling up to thousands of VRFs, but you'll better use Host Advertisement in 99% of cases.

[m1xed0s]

Thanks, even with MTU Path discovery or similar feature, that would be a feature on the VPN termination points, not ACI function, right?

[Remi-Astruc]

Hi,

Right

[sdn_worker]

Q1: So without Golf you can connect your OnPrem IPsec Termination Endpoint to a Leaf Switch connected to your specific Customer Tenant/VRF/L3out? Right?

Because in every Design Document i saw the IPsec Router is connected to the ISN/IPN Network!

Q2: The OnPrem IPsec Router configuration is not automated by MSO? Right?

[m1xed0s]

The IPSec termination point would connect to spine.

https://community.cisco.com/t5/application-centric/transport-network-between-aci-on-prem-and-cloud-aci-in-aws/m-p/4050084

there is automation of the IPSec device but I think there would be template provided.

[sdn_worker]

But the IPSec Termination Endpoint doesn‘t need to Support Golf? Right? So you can also use an ISR 4K for example?!

[Huyen Duong]

Hi,

Correct, the IPSec device just does the underlay, it is not aware of GOLF and do not need GOLF.

The traffic flow from onprem to Cloud (AWS in this case) will be:

Endpoint onprem -> Leaf -> Spine -> ISN -> IPSec device --> DX/Internet -> CSR (managed by cAPIC) -> TGW -> Instances

[Huyen Duong]

Hi

We automate the IPSec configuration on CSR at Cloud side. We also export configuration template based on IOS-XE syntax, you then use that configure template to configure your IPSec termination device.

Your IPSec termination device can be anything, as long as it support IPSec (v1 or v2) and OSPF.

Cheers

[Huyen Duong]

Hi, 

We support a feature to configure TCP MSS adjust on CSR routers. Depending on your application nature, you find the right value, normally some value around 1300 should be ok. Keep in mind that it only works for TCP based application.

[talale]

---

@Remi-Astruc wrote:

Hi @m1xed0s ,

Yes. As long as ACI Anywhere still uses VXLAN encapsulation (and optionally IPSec or CloudSec on top of it), the requirement of "tenant traffic MTU + 100B" is still required.

If using Internet as the ISN, you need to set the servers MTU to <1400B.

An other option is to rely on the IPSec/CSR devices to fragment (because ACI does not).

If using a Direct Connect solution, it may support MTU of 9100B.

Right, the complexity of GOLF is not needed anymore in 4.x (unless very specific requirement).

---

Thanks for sharing such helpful information. Because I'm a beginner and I didn't know much about this problem but when I saw your post in this community, I fell in love with the solution of your problem. So I got the solution to my problem and I am very thankful to you. Thanks for sharing such helpful information in this community.