How APIC pushes configuration to nodes?

[suneq]

Hi there, 

I have a very basic question regarding how ACI works, I understand that APIC pushes configuration to the nodes, but I have not found any document indicating how, I mean.. does APIC pushes configuration via "to Fabric" interfaces? or OOB interfaces? or maybe both? If both can be used, which one is preferred and how to configure it?

Thank you for your advice.

[balaji.bandi]

It all depends on the deployment you choose.

there is a good presentation to understand the deployment IPN control plane

https://www.ciscolive.com/c/dam/r/ciscolive/emea/docs/2019/pdf/BRKACI-2003.pdf

[Hector Gustavo Serrano Gutierrez · Accepted solution]

Hi @suneq,

During the initial setup of the APIC controller, you choose the below parameters (among others):

• address pool for TEP addresses (default 10.0.0.0/16)
• VLAN ID for infra network
• Out-of-band management IP address and default gateway

The Out-of-band management IP address is just for the APIC at this point so you can reach its web GUI or SSH into it from your OOB network.

The VLAN ID for infra network is the infrastructure VLAN, which is used for the communication between the APIC and the directly connected Leaf Switch. The APIC uses "Fabric Ports" to connect to one or two Leaf Switches and via them, push the configurations to the rest of the Switches in the fabric once the fabric is fully discovered.

The pool for TEP addresses is the TEP Pool (Tunnel endpoint address pool). The IP addresses from this pool are assigned to all fabric elements (APIC controllers, Leaf and Spine switches) to communicate over the infrastructure VLAN. Ideally this subnet should not overlap with any other subnets in your network.

With the initial setup, you should be able to SSH into your APIC and from there, jump into any Leaf or Spine Switch.

Optionally you can:

• Configure the OOB interfaces on the Leaf and Spines Switches to SSH directly to them from your OOB network. You need to allocate a new subnet for this purpose.
• Configure In-band management to reach the APIC and Switches from an EPG or External EPG. These EPGs can connect to any Leaf Switch's front panel ports.

A good document that you can refer to is:

Cisco Application Policy Infrastructure Controller (APIC)
Setting Up a Cisco ACI Fabric:
Initial Deployment Cookbook

Hope this helps.

[RedNectar · Accepted solution]

Hi @suneq ,

Once the initial configuration for an ACI deployment is complete (ie TEP addresses have been assigned by APIC#1 to the leaf and spine switches via DHCP) each APIC establishes an SSL session with each Leaf and Spine and with each other.

These connections are via the VTEP addresses, and there is no communication between the ACI devices on the OOB interfaces unless it is specifically initiated by a user e.g. you ping the OOB IP address of a spine switch from an APIC.

So the communications between the APICs and the Leaves/Spines is via these inband fabric VTEP addresses.

The protocol used to push the configuration is called Opflex. You can read more about Opflex here.

ACI also uses ISIS and BGP to communicate routing information, and COOP to report MAC/IP address registrations.  You can read more about these in an answer I gave on this forum here and consilidated into an article here.

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[Claudia de Luna · Accepted solution]

Hi @suneq,

To answer your question. The Fabric interfaces are used.  All you need to do is discovery the fabric and that sets up the "internal" communications channels for the fabric (leafs and spines) for configuration and management via the APIC.

So the internal fabric communication is taken care of.

For us to talk to the APIC we generally configure a out of band management network and that is how humans and automation talk to the APIC controllers and switches.  There is also an in-band option if you like and recent versions of ACI allow you to configure the priority or preference on which to use.

A bit more detail...

Starting at the beginning, the controllers use LLDP to discover approved devices. The APICs run many services including DHCP.  As devices are discovered they are assigned IPs from the TEP address pool you configured when you set up the APICs.  

In the example below its 10.0.0.0/16. 

If you look in Fabric > Inventory > Fabric Membership you can see what IPs your leafs and spines got from that large TEP /16 allocation.

This is all done automatically when you bring up (discover) the fabric so the only action you need to take is to discover the leafs and spines.  As they join the fabric they get IPs that allow the APICs to "operate" the fabric switches (configure, monitor, etc.).

The APIC also run a process call the Data Management Engine that basically "runs" the fabric for you based on what you configure into the APIC.   Some additional info below on that and the discovery process.

A Data Management Engine (DME) in Cisco ACI Fabric OS provides the framework that serves read and write requests from a shared lockless datastore. The datastore is object oriented, with each object stored as chunks of data.

Cisco Application Centric Infrastructure (a bit dated but so far the best description I've found on these "internals").

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-fabric/white-paper-c11-730021.html

[suneq]

Thank you for your quick reply. It really helps.