Typically, we will just check each APIC until we find the log. But the correct method for finding out which APIC has the log is based upon the leader for the shard.
The ACI Management Information Tree (MIT), the database containing the Managed Objects (MOs) is divided into units called shards to allow for load balancing and scaling. Each managed object (MO) in the MIT is mapped to a shard number (0-31). One of the APICs, using a hash, is elected as the leader for each of the shards, which makes it responsible for providing access to the shard data and updating the shard content.
Each of the 3 APICs in the cluster has a full copy of the MIT but can only make changes to the shards it is leader for. When the leader of the shard makes changes to it, the leader sends out an update message to the other APICs so they can update their copy of the MIT. That way, only a single APIC (the leader of the shard) can make a change to a MO in the shard and the others just update their copies of the MIT to reflect the change.
When configuring service graphs, changes are first sent to the Policy Manager. The Policy Manager performs some internal calculation and creates/deletes/updates objects in its part of the MIT. It then sends a message to Script Handler which updates its part of the MIT and, in turn, it sends a message to Script Wrapper. On receiving this message, Script Wrapper updates its ConfigCache which is then checked by a constantly running task scheduled by TaskScheduler.
It's important to understand that ScriptHandler and Policy Manager are DMEs (Data Management Engines) and they have their logs in the /var/log/dme/log/ directory:
svc_ifc_policymgr.bin.log
svc_ifc_scripthandler.bin.log
while ScriptWrapper is a python process started by ScriptHandler. Calls from the APIC to the Partner's script are saved in /data/devicescript/<Partner's script name>/logs/apic.log.
The log can be found on the APIC that is the leader for the shard where the service graph MOs are located.
Can the APIC with the logs change? Yes!
Let’s say APIC#3 in the cluster is leader of the shard containing the service graph MOs and it becomes unreachable by the other 2 APICs. The other 2 APICs would be in the majority and assume leader on the shards of the unreachable APIC and the unreachable APIC would go into a minority state and its MIT would become read-only (no changes). Hence, the APIC where you would find the log with changes to the service graph would change at this point. When APIC#3 became reachable again by the other 2 APICs, the leaders for the shards would change again.
