How to configure stretch fabric between two sites

[eahmed007]

Dear All ,

Please give your inputs on below implementation plan :

Stretch Fabric :

1.We have planned to configure Stretch Fabric using Single Fabric Cluster to make DC and NDC Active/Active using DCI technologies .Please suggest us which DCI technologies(vPC using back to back Dark Fiber , VXLAN using NX-OS in nexus 9K or OTV using ASR router ) we should choose .

On the other hand , We might not use Dark Fiber as they do not have 40KM QSFP module in leaf switch for back to back Dark fiber connectivity and VXLAN is also not possible as they do not have Nexus 9K in NX-OS mode . So only option left which is OTV using ASR router . If we choose OTV , how the traffic flow would be between DC to NDC as we need to configure Single Cluster using 5 Nodes APIC controller where 3 would be in DC and 2 would be in NDC .In addtion , We have planned to configure ACI fabric using out of band management .

OOB MGMT :

1.We have planned to connect each and every switch of DC and NDC and All APIC controllers using OOB management Switch which is Catalyst 3560 Switch for initial configuration .

However , Need your inputs to connect and discover all Spine , Leaf and APIC between two sites for the first time .


In addition ,Please assist us what would be the impact if configure APIC controller using 5 node Clusters in each data center separately . I mean , firstly we will patch all 2 Spines, 8 Leafs and 3 APIC controller of DC and configure initial configuration and later we will patch all 2 Spines, 8 Leafs and 3 APIC controller of NDC which is different location .

Looking forward to your valuable comments and suggestion .

With regards

Erfan

Is there any complexity if we patch cable and configure separately in DC and NDC or do we need to patch each and every components of DC and NDC and configure it one shot to make single Cluster using 5 Nodes APIC .

[eahmed007]

Dear All , 

Please help me on this case .What is the process to configure Single cluster using 5 Node APIC controller and what would be physical connectivity to make single cluster between two sites.

It would be highly appreciate if you reply using physical connectivity between two sites .

I am eagerly waiting for your reply .

With regards

Erfan

[RedNectar]

Hi Erfan,

Have you explored to see if the ASRs will support the ACI Multipod solution? I had a quick look at the ACI Multipod solution, but it doesn't mention ASR, so you might need to ask that question specifically.

Regarding OOB management, again I'd recommend using inband management instead, but if you must use OOB then my answer to your other question should work across a multi-pod fabric, but of course requires the fabric to be operational to maintain connectivity.

I'd advise that you have separate OOB management networks at each site, and if OOB connectivity is required between sites when the Inter-Pod connectivity is down, you'd need to connect the OOB networks some alternative way, possibly a VPN tunnel over a public network.

I hope this helps

RedNectar
aka Chris Welsh

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

[eahmed007]

Dear All ,

 Please suggest me whether we can configure Multi-POD technology between DC and NDC .I have noticed in some documents where they said that we need 40G connectivity between Spine and IPN device (Layer 3 device ) .But client does not have 40G in ASR router .As per the attached documents , Multi-POD support 10G but Spine Switch(N9K-C9336PQ) does not support 10G connectivity with IPN devices .

Please find the attach consideration for Muti-POD scenario .

Considering the above situation , I need to configure stretch fabric between two sites .Please give your valuable inputs ,how I can configure stretch fabric as I do not have any other option to create muti-pod scenario .

Please advise me how I can configure stretch Fabric between two sites .I have seen some documents that I need to create EOMPLS circuit .

What is the configuration to create EOMPLS circuit between two sites  and discover the all components as I have 8 leaf , 2 Spine and 3 Nodes APIC in each site and 8 leaf , 2 Spine and 2 Nodes APIC in other site .

Can you tell ,what would be the procedure to make single cluster first time between two sites .

I am eagerly waiting for your reply .

with regards

Erfan

[eahmed007]

Hi Welsh ,

I need your valuable comments and waiting for your reply .

Please help me on this case .

With regards

Erfan

[RedNectar]

Erfan,

I am so angry with this site - I just spent almost an hour answering your question and then the site lost it when I posted it!

Basically I said:

1. If you use ACI Multi-site, you will have to use something (a switch) to step down the 40G from your 9336-PQ spine switches.
2. If you use stretched fabric, you will have to use something (a switch) to step down the 40G from your 9336-PQ spine switches and that will be the case whether you incorporate EoMPLS or not

So therefore your #1 problem is to step 40G down to 10G

Once you have solved #1 problem, I would advise using ACI Multi-pod, with 2 APICs in the main site and 1 APIC in the other.  Configure your remaining two APICs in standby, one in each site.

Alternatively, you could buy one more APIC for the second site and run two independent sites, in which case you could connect the leaf switches from one site to the other using a L2 connection, which if you really wanted it, could be EoMPLS - but any L2 tunnel/direct connection would do.

You might get some more ideas from BRKACI-2125 - ACI Multi-Site Architecture and Deployment (2017 Las Vegas) and BRKACI-2003 - Cisco ACI Multi-Pod Design and Deployment (2017 Las Vegas)

I hope this helps

RedNectar
aka Chris Welsh

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

[eahmed007]

Hi Welsh ,

Sorry for late reply .

I got your your  points to make multi-pod or stretch Fabric .

Please give me some inputs whether I can make multi-pod or stretch Fabric connecting leaf switch N9K-C9396PX with ASR 1002 Router with 10G connectivity instead of 9336-PQ spine switches with ASR 1002 Router  as I do not have 40G connectivity in my ASR Router to connect Spine for Multi-pod or stretch Fabric .

My intention is to connect Two ASR router with L3 connectivity and configure EOMPLS circuit between two sites .So I am trying to connect leaf of each switch would be connected with each ASR router with 10G connectivity in each site.

Please suggests whether above topology would be possible or not .

Your valuable inputs is highly appreciated .

[Nik Noltenius]

Hello Erfan,

is there a special requirement why you want to use a stretched fabric design rather than Multi-Pod? Multi-Pod is very well documented in the white paper (https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-737855.html) and other configuration guides. It will give you the active / active behavior you want and you can connect your DC sites via routing. An Inter-Pod Network is required but the ASR 1k and 9k should be able to provide all functionality needed (PIM-Bidir, DHCP relay, MTU of 9150 and OSPF).

For the discovery, that works over the IPN as well. Basically you install and cable all your components where they should be and start discovery in your main data center. Once a spine connected to the IPN is discovered it will discover a spine in the remote DC and from there go down to leaf switches and APICs. Doesn't matter if you use 3 or 5 APICs. Also it happens via the infrastructure IP network and is not affected by how you manage your switches.

I'm not sure if I got your questions all right but I answered to what I understood. Feel free to clarify if I missed something.

Regards,

Nik