OOB and inb for APIC vCenter communication

[AriSadewo]

Hi we are in the middle of creating new ESX cluster with new vCenter with ACI 4.2 our goal is to create new VMM domain with new vCenter to automate pushing the port-group to each VDS by ACI.

Here is the current topology 

From above topology, we will create new vCenter in the same subnet with existing one. APIC will communicate via OOB and going thru ASA. As of now, only OOB IP address configured for APIC.

Is it possible for new vCenter to be in user tenant inside the fabric (same EPG as ESX cluster) and communicate with APIC via inb, something like this 

The questions are

• if we enable inb mgmt and try to create new VMM domain via inb, will both existing VMM domain (using OOB ) and new one (using inb) works ? any downtime expected?
•  Do we need to change connectivity preference from OOB to inb?
•  Based on our environment, what topology would work best for us ? to put vcenter inside or outside the fabric 

Thanks very much, any response are appreciated.

[joezersk]

Hello.  Sorry it took so long for someone to answer this.  I will aim to give you two things.  One, what is possible.  Two, my personal opinion as to which I prefer. 

Looking over your topology (thanks for posting such beautiful drawings to bring clarity), either approach is possible with a little bit of planning.  The thing about APIC is that it is a host like any other host.  For any networks on a host, there can be only one preferred default gateway.  APIC is no different.  The APIC forwarding logic works like this for OOB and Inband:

• Packets that come in an interface, go out that same interface
• Packets sourced from the APIC, destined to a directly connected network, go out the directly connected interface
• Packets sourced from the APIC, destined to a remote network, prefer In-band, followed by Out-of-band
• You can change the 3rd bullet in APIC, but inband preferred is the default

So, with that understanding, if you wanted to have APIC talk to vCenters / ESXi hosts inside and outside of the fabric, you would set up inband management, and also an L3Out in tenant mgmt to reach the networks outside of ACI.  If you did not set up an L3out via inband, your APIC would happily talk to anything inside the fabric via inband and anything on the same subnet as OOB, but nothing else.  At the current time, there is no way to add routes to APICs internal routing table without root (which is not available to customers).

I do have a question, out of curiosity.  Why do you need a new vCenter?  the VMM integration is just a simple API-to-API comms channel, where APIC tells vCenter what it wants and vCenter does the work using native VMware processes.  This is to say that any DVS created via this integration is not an "ACI DVS" but rather a standard VMware DVS built to the specifications that APIC asks for.  It is perfectly fine (and supported by VMware) to have many DVS on the same ESXi host.  Some can be via ACI and some can be 'traditional'.  As long as you have enough VMnics in the host, you are fine. 

The opinion part is the short part ;)  Not just with ACI, but with anything in the DC, from any vendor, I prefer OOB always.  This is because I never want to find myself in a situation where I am cutting off the branch of the tree I am sitting in.  Better to have a separate and dedicated out of band channel to access things when there are issues.  Just my opinion though.  Take it as you will. 

Hope that helps.