ACI Multi-Site

[AdrianT]

Hello,

I have a question for ACI Multi Site deploymend on Brown field scenario.

VLAN 4 and a VRF with a dynamic routing protocol (ex: OSPF) needs to be used in order to make the connection between Spines and ISN. Can I reuse the same VRF and the same area of OSPF between the two ISNs - for site to site connectivity, or do I need something else?

To be honest I do not see any issue in this, but do please let me know.

Thank you.

[AshSe]

Hello @AdrianT 

In an ACI Multi-Site deployment with a brownfield scenario, reusing the same VRF and OSPF area between the two ISNs for site-to-site connectivity is valid and commonly done, as long as proper attention is given to the VRF extension, routing policies, and the inter-site connection setup. If you are comfortable with the current design and it meets your requirements, there should be no issue.

Hope This Helps!!!

AshSe

Community Etiquette: 

1. Insert photos/images inline - don't attach.

2. Always mark helpful and correct answers, it helps others find what they need.

3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[AdrianT]

Hello,

Thank you for the provided answer. Can you emphasize a bit more about the proper attention to the VRF extension, routing policies and inter-site conenction?

Looking forward for your feedback.

[AshSe · Accepted solution]

Hello @AdrianT 

Let me give you some hint points about VRF extension, Routing Policies and ISN connection as below:

1. VRF Extension and Context Sharing

• VRF as the Foundation: Ensure that the VRF is consistently defined and applied across all relevant components.

• Contract Enforcement Across Sites: contracts associated with that VRF must be consistently enforced. 

• VRF Route Target (RT) and Route Distinguisher (RD): These are critical for proper VRF extension.

  • Route Distinguisher (RD): The RD must be the same on both ISN routers.
  • Route Target (RT): In your scenario, you're not creating separate VRFs for the ISN connection, so you don't need to worry about RTs for importing/exporting between VRFs.

• ACI Multi-Site Configuration: Make sure that the VRF is properly defined and associated with the appropriate tenants and application profiles in each site.

2. Routing Policies and Route Control

• Route Leaking/Redistribution (If Needed): In your scenario (reusing the same VRF and OSPF area), you shouldn't need route leaking or redistribution between VRFs for the ISN connectivity itself.

• OSPF Route Summarization: Summarize routes appropriately at the edges of the ISN.

• Filtering and Route Maps: Use route maps and access lists to filter routes and control which routes are advertised and accepted.

• BGP Integration (If Applicable): If you're using BGP within the ACI fabric (e.g., for external connectivity), you'll need to carefully consider how BGP routes are exchanged with the ISN. You might need to redistribute BGP routes into OSPF (or vice versa) and apply appropriate filtering and route maps to control the flow of routes.

• Policy Enforcement: Ensure that your routing policies align with your overall security and network segmentation goals.

3. Inter-Site Connectivity (ISN Deep Dive)

• Bandwidth and Latency: Ensure that it has sufficient bandwidth to handle the traffic between the sites and that the latency is acceptable for your applications.

• Redundancy and Resilience: Use multiple links between the sites and implement appropriate routing protocols (e.g., OSPF) to ensure that traffic can failover to a backup path in the event of a link failure.

• MTU Consistency (Critical): Ensure that the MTU is consistent across the entire path between the ACI sites, including the ISN links, the interfaces connecting the Spines to the ISN, and any intermediate devices. A jumbo MTU (e.g., 9000 bytes) is often recommended if the underlying transport supports it.

• Security: Secure the ISN with appropriate security measures, such as firewalls or access control lists (ACLs). This is especially important if the ISN traverses a public network.

• Monitoring and Troubleshooting: Implement comprehensive monitoring and troubleshooting tools to quickly identify and resolve any connectivity issues. Use tools such as ping, traceroute, and packet capture to diagnose problems.

• ISN Device Configuration: Ensure that the devices are properly configured with the correct VLANs, IP addresses, routing protocols, and security policies.

• OSPF Adjacency: Verify that OSPF adjacencies are established between the ISN routers and the Spines in each site.

Hope This Helps!!!

AshSe

Community Etiquette: 

1. Insert photos/images inline - don't attach.

2. Always mark helpful and correct answers, it helps others find what they need.

3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[AdrianT]

Thank you for the provided info.

I have one more. Keeping in mind the setup is a brownfield scenario that connects the two ACI Fabrics via VPN, do you think there will be any outage when we also configure Spines MP-BGP-EVPN policies in order to conenct ACI Spines to ISN, or when we add the ACI sites in NDO?

[AshSe]

Hello @AdrianT 

Keeping in mind the setup is a brownfield scenario that connects the two ACI Fabrics via VPN, do you think there will be any outage when we also configure Spines MP-BGP-EVPN policies in order to conenct ACI Spines to ISN, or .......

---

• MP-BGP-EVPN Configuration: This is the higher-risk operation and requires careful planning, testing, and a rollback plan. Schedule a maintenance window and implement the changes in a staged manner.

---

, or when we add the ACI sites in NDO?

---

• Adding Sites to NDO: This is generally a lower-risk operation, but still requires careful planning and validation. Back up your ACI fabric configurations and add the sites to NDO one at a time.

Hope This Helps!

AshSe

Community Etiquette: 

1. Insert photos/images inline - don't attach.

2. Always mark helpful and correct answers, it helps others find what they need.

3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[AdrianT]

@AshSe thank you for the provided answer.

[AdrianT]

@AshSe one last question. If the MTU between ISN to ISN cannot be changed from 1500 to at least 9000 due to the fact that the ISN to ISN connection is done via SVI only, will I encounter any issue along the way?

According to Cisco documentation that should not be the case " However, it is not necessarily required to support such a large MTU on the ISN routers for intersite communication, as the required minimum value mostly depends on the MTU of the traffic generated by the endpoints connected to the ACI fabric. For more information on this, please refer to the “Intersite Network (ISN) deployment considerations” section of the ACI Multi-Site paper:"

The above is taken from the following guide:

https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-multi-site-deployment-guide-for-aci-fabrics.html

If you foresee any issue, if I change the TCP MSS option in the ACi System Settings, will that solve it? And if yes to what value (keeping in mind that Spine to ISN MTU is 9150, but ISN to ISN MTU is only 1500 and even if I change to SVI, it will not apply since I cannot change on physical interface also)?

[AshSe]

Hello @AdrianT 

Here is a summary of your latest query:

• Spine to ISN MTU: 9150 (Jumbo Frames)
• ISN to ISN MTU: 1500 (Standard Ethernet) due to SVI limitation and inability to change physical interface MTU.
• ISN to ISN Connection: Via SVI (Switched Virtual Interface).

Your Concern: Potential fragmentation and performance issues due to MTU mismatch.

Potential Solution: TCP MSS Clamping

---

According to Cisco documentation that should not be the case " However, it is not necessarily required to support such a large MTU on the ISN routers for intersite communication, as the required minimum value mostly depends on the MTU of the traffic generated by the endpoints connected to the ACI fabric. For more information on this, please refer to the “Intersite Network (ISN) deployment considerations” section of the ACI Multi-Site paper:"

The above is taken from the following guide:

https://www.cisco.com/c/en/us/td/docs/dcn/whitepapers/cisco-multi-site-deployment-guide-for-aci-fabrics.html

---

You're correct to cite the Cisco documentation stating that a large MTU on the ISN routers isn't necessarily required. The key phrase is "mostly depends on the MTU of the traffic generated by the endpoints connected to the ACI fabric."

Please look at below important Caveat about MTU:

• Endpoint Traffic: If the majority of your endpoint traffic within the ACI fabric consists of small packets (less than 1500 bytes), then the MTU mismatch might not be a significant issue. The packets will be able to traverse the ISN without fragmentation.
• Large Packets: If your endpoints generate large packets (e.g., for file transfers, database backups, or applications that use large MTUs), then fragmentation will occur in the ISN, which can lead to performance degradation.

Suggestion:

• Determine the typical packet size of the traffic that will be traversing the ISN.
• Use packet capture tools (e.g., Wireshark) to analyze the traffic patterns within your ACI fabric.
• If you see a significant amount of traffic with packet sizes close to or exceeding 1500 bytes, then you need to address the MTU mismatch.

Potential Solution: TCP MSS Clamping

TCP MSS (Maximum Segment Size) clamping is a technique that allows you to reduce the MSS value advertised by TCP endpoints. This effectively limits the size of the TCP segments that are sent, preventing fragmentation.

• How it Works: TCP endpoints negotiate the MSS value during the TCP handshake. The MSS value indicates the maximum amount of data that the endpoint is willing to receive in a single TCP segment. By clamping the MSS value, you can ensure that the TCP segments are small enough to traverse the ISN without fragmentation.

• Where to Configure: You can configure TCP MSS clamping on the ACI Spines.

Important Caveat about TCP MSS clamping configuration:

If you are using latest version of APIC, say APIC v5.2(5c), you can't directly configure TCP MSS clamping as you might in newer versions. Instead, you need to use QoS policies and L3Out MTU settings to indirectly influence packet sizes. This approach is more complex and requires careful planning and testing. Ensure that you consult the official APIC documentation and thoroughly test your configuration.

HTH & Stay Curious!

AshSe

Community Etiquette: 

1. Insert photos/images inline - don't attach.

2. Always mark helpful and correct answers, it helps others find what they need.

3. For a prompt reply, kindly tag @name. An email will be automatically sent to the member.

[AdrianT]

 I can tell you the following. ACI has 6.0(3) version installed (that means I can mark the global option in the ACI) so I can use TCP MSS in System Settings, and I know there is database replication (between sites), not database backup. With that said, at the moment replication goes out ACI via L3Out (which also has MTU 1500 at the moment), and ACI connects to FW that has VPN between sites. In this situation is there any use of the setting TCP MSS in ACI? If yes, what should I set (either 1460 or 9110 in ACI System Settings?).