connecting two ACI fabric back-to-back

[e-chuah]

Hi,

My customer has an ACI fabric running 4.2.7. The leaf and spine is reaching end of life support.
We are building a new separate ACI fabric running 6.0.

The plan is to connect two leaf switches from existing fabric to another two leaf switches in new ACI fabric via direct back to back connections and using VPC.

This is to extend the vlans between the existing and new ACI fabric.
We will then progressively migrate the workload via vMotion from existing ACI fabric to new ACI fabric.
SVI in new ACI fabric will be shutdown during initial phase. VMs migrated to new ACI fabric will use the SVI in existing ACI fabric during the co-existence phase. L3OUT will also exit via existing ACI fabric during the co-existence.

After all VMs are migrated from existing ACI fabric to new ACI fabric, we will shutdown the SVI in existing ACI fabric and unshut the SVI in new ACI fabric.

My understanding is that this should work as long as we have one EPG in one BD mapping and not multiple EPGs in one BD.

Any comments, suggestions greatly appreciated.

Thanks !

Eng Wee

[AshSe]

Hello @e-chuah , please confirm the below diagram and answer the queries for better suggestion:

1. Are you going for LAN (L2) or WAN (L3) connections between the two DCs and why?
2. You mentioned about SVI, does it mean you will be assigning 4 separate IPs from the same subnet on four separate interfaces of the four separate leaf switches?  Please share the configuration for a better understanding.
3. Finally, you will use Migration Wizard available in the vSphere client to migrate VMs to new DC, is it right?

[e-chuah]

1. Are you going for LAN (L2) or WAN (L3) connections between the two DCs and why?
   only L2 connections to extend the vlans to facilitate VM migration to the new ACI fabric. The existing ACI fabric running in 4.2 will eventually be decommissioned after all workloads are migrated.
2. You mentioned about SVI, does it mean you will be assigning 4 separate IPs from the same subnet on four separate interfaces of the four separate leaf switches?  Please share the configuration for a better understanding.
   No. At one time there will only be one active pervasive gateway configured in each fabric. So during initial co-existence phase, pervasive gateway will be active at the existing fabric (running 4.2.7). VMs that are migrated to the new fabric will use the pervasive gateway in existing fabric. After all VMs in a vlan are migrated, we will shutdown the pervasive gateway at existing fabric, enable the pervasive gateway at new ACI fabric.
3. Finally, you will use Migration Wizard available in the vSphere client to migrate VMs to new DC, is it right?
   This will be done by the server team. Standard vmare vmotion.
   
   Thanks 
   Eng Wee

[AshSe]

Hey Eng Wee aka @e-chuah  from below clarification:

No. At one time there will only be one active pervasive gateway configured in each fabric. So during initial co-existence phase, pervasive gateway will be active at the existing fabric (running 4.2.7). VMs that are migrated to the new fabric will use the pervasive gateway in existing fabric. After all VMs in a vlan are migrated, we will shutdown the pervasive gateway at existing fabric, enable the pervasive gateway at new ACI fabric.

---

Pervasive gateway refers to a distributed Layer 3 gateway functionality that is implemented across the entire ACI fabric. 

For a better understanding of this implementation; we understand your logical setup as below, please confirm before we discuss the solution:

Steps of moving VMs:

Step 1) Move the VMs to New fabric (v6.0), use the pervasive gateway (10.1.1.254/24) in existing fabric (v4.2.7)

Step 2) Shut down pervasive gateway (10.1.1.254/24) in existing fabric (v4.2.7)

Step 3) Enable the pervasive gateway at new ACI fabric

[e-chuah]

that the plan. At any one time, only one pervasive gateway will be active.

[AshSe]

Hey @e-chuah 

confirm your Back-to-Back (BtB) connectivity:

[e-chuah]

Yes, that is the plan.

[AshSe]

@e-chuah Your plan to migrate workloads from an existing ACI fabric running 4.2.7 to a new ACI fabric running 6.0 using vPC connections and VLAN extension is generally sound. Here are some detailed comments and suggestions to ensure a smooth transition:

Key Considerations:

1. Compatibility and Interoperability:

   1. Ensure that the hardware and software versions of the leaf switches in both fabrics are compatible with each other for vPC and VLAN extension.
   2. Verify that the vPC configuration between the two fabrics is supported and tested in your environment.

2. VLAN and EPG Configuration:

   1. Ensure that the VLANs you plan to extend are consistently configured across both fabrics.
   2. As you mentioned, having one EPG per BD (Bridge Domain) is a good practice to avoid complexity and potential issues with overlapping policies.

3. SVI Management:

   1. During the coexistence phase, ensure that the SVIs (Switched Virtual Interfaces) in the new ACI fabric are indeed shut down to avoid any IP address conflicts or routing issues.
   2. Plan the cutover of SVIs carefully, ensuring minimal downtime and disruption. This might involve scheduling a maintenance window.

4. L3Out Configuration:

   1. Ensure that the L3Out configuration in the existing ACI fabric is robust and can handle the traffic load during the coexistence phase.
   2. Plan the migration of L3Out to the new ACI fabric carefully, ensuring that routing and external connectivity are maintained.

5. Testing and Validation:

   1. Before starting the migration, thoroughly test the vPC connections and VLAN extensions between the two fabrics.
   2. Validate that the VMs can communicate across the fabrics as expected and that there are no issues with traffic flow or performance.

6. Monitoring and Troubleshooting:

   1. Monitor the network closely during the migration process for any signs of issues or anomalies.
   2. Have a rollback plan in place in case something goes wrong during the migration.

Detailed Steps:

1. Preparation:

   1. Document the current network configuration, including VLANs, EPGs, BDs, and L3Outs.
   2. Plan the vPC configuration between the two fabrics, ensuring that the port channels are correctly configured.

2. Configuration:

   1. Configure the vPC connections between the leaf switches in the existing and new ACI fabrics.
   2. Extend the necessary VLANs across the vPC connections.
   3. Ensure that the EPGs in the new fabric are correctly mapped to the extended VLANs.

3. Migration:

   1. Migrate VMs using vMotion from the existing ACI fabric to the new ACI fabric.
   2. Ensure that the VMs in the new fabric can still communicate with the SVIs in the existing fabric.

4. Cutover:

   1. Once all VMs are migrated, plan the cutover of SVIs from the existing fabric to the new fabric.
   2. Shut down the SVIs in the existing fabric and bring up the SVIs in the new fabric.
   3. Migrate the L3Out configuration to the new fabric.

5. Post-Migration:

   1. Monitor the network for any issues and validate that all services are functioning as expected.
   2. Decommission the old ACI fabric once you are confident that the new fabric is stable and fully operational.

By following these steps and considerations, you should be able to achieve a smooth migration with minimal disruption to your network services.

PS: Please do make sure that the high level and low level physical and logical connectivity diagram are covered in your change plan. Last but not the least; please get your change plan peer-reviewed. All the very best and do your best!!!

HTH

AshSe

Please rate this post if it was helpful; your feedback is appreciated!

[e-chuah]

Hi AshSe,

Thanks for the comments.
At the back of my mind, these are some questions i have:

(1) when i connect the leaf from one fabric to another leaf in another fabric? Will the leaf detect that it is a leaf to leaf connection and disable the connection?
My thoughts is that this should be ok because each leaf register with its respective APIC. So should not be an issue.

(2) If both fabrics that i connect back to back uses fabric-id=1, will this cause an issue? Do i have to change one of the fabric to a different fabric-id?

[AshSe]

Dear @e-chuah , excellent you have raised genuine concerns. Pfb, my responses for the same:

---

(1) when i connect the leaf from one fabric to another leaf in another fabric? Will the leaf detect that it is a leaf to leaf connection and disable the connection?
My thoughts is that this should be ok because each leaf register with its respective APIC. So should not be an issue.

---

AshSe >> When connecting two separate ACI fabrics, you are essentially creating an inter-fabric connection, which is a different scenario. Thus no blocking between the leaf switches belonging to different ACI fabric.

---

(2) If both fabrics that i connect back to back uses fabric-id=1, will this cause an issue? Do i have to change one of the fabric to a different fabric-id?

---

AshSe >> Yes, having both ACI fabrics with the same fabric-id can cause issues when connecting them back-to-back. The fabric-id is a unique identifier for each ACI fabric, and it is used to distinguish between different fabrics in multi-fabric deployments. If both fabrics have the same fabric-id, it can lead to conflicts and misinterpretation of control plane information, potentially causing network instability and operational issues.

Hope This Helps!

[AshSe]

Hey @e-chuah Kindly note that upgrading Cisco ACI from version 4.2.7 directly to 6.0 is not typically supported due to the significant changes and potential compatibility issues between major versions. Cisco usually recommends a staged upgrade process, moving through intermediate versions to ensure stability and compatibility. Please check the below links for better guidance:

Cisco APIC Installation and ACI Upgrade and Downgrade Guide

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/all/apic-installation-aci-upgrade-downgrade/Cisco-APIC-Installation-ACI-Upgrade-Downgrade-Guide/m-aci-firmware-upgrade-overview.html#id_48185

Cisco Application Policy Infrastructure Controller Release Notes, Release 4.2(7)

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/4x/release-notes/cisco-apic-release-notes-427.html

Cisco Application Policy Infrastructure Controller Release Notes, Release 6.0(3)

https://www.cisco.com/c/en/us/td/docs/dcn/aci/apic/6x/release-notes/cisco-apic-release-notes-603.html

HTH

AshSe

Please rate this post if it was helpful; your feedback is appreciated!

[sk37]

Do you have Gen-1 or Gen-2 spine and leaf switches?