First I would like to do policy based switching controlled by Cisco APIC between VMs inside my Vmware ESX landscape. That's it, nothing more.
Correct me if I'm wrong but I think a don't need any Leaf Switches for this right?
No, there is no Fabric with out Leaf/Spine switches. The APICs are a fabric controller that optionally can be integrated with VMware, but its not a standalone virtual networking controller like I understand you're looking for.
I read already a lot and watched a lot of videos but it's really hard to find documentation, about what's going on under the hood of APIC/ACI?
What do you want to know specifically? Under the hood we use VXLAN over an IS-IS fabric to provide multipathing and redundancy between any switch nodes. The controllers (APIC) are responsible for managing the logical configuration for the fabric, while the switch nodes resolve this logical policy into a concrete model. The APIC also gathers health analystics from all switch nodes and can apply them to logical applications (rather than physical ports or hardware only). This allows for a health score for logical applications which are a collection of endpoints (physical or virtual) that exist anywhere within the fabric or integrated solutions. If you have other specific questions, let me know.
What I didn't know is: I have VM a running on ESX a with VTEP a and I have VM b running on ESX b with VTEP b.
If I push a policy from the APIC, which allows communication from VM a to VM b, do the VTEP's already know how to find each other?
Policy is "pushed" from the APIC to fabric switches and only enforced on Leaf nodes. The APIC can push logical port groups (EPGs) to vCenter, but there's no inter-EPG communication without that traffic going up to the Leaf for policy validation and back. Only Intra-EPG communication is permitted between endpoints in the same PortGroup/EPG on the same host (much like a vSwitch). Another role of the ACI fabric is to know where every connected endpoint exists. If a VTEP doesn't know the location of a specific endpoint, it will get punted to the Spine which (which has every endpoint entry) to determine if & where that endpoint exists, and forwards it to the corresponding VTEP.
- Could VM a send traffic over VTEP a and VTEP a knows that it need to send the traffic to VTEP b because it knows that VM b is behind it?
See answer above.
- Does VTEP a itself already know that it's allowed to send traffic to VM b?
See answer above. Only Intra-EPG traffic is permitted, any inter-EPG (inter-PortGroup) traffic gets forwarded to the connecting leaf for policy validation.
Which way is the traffic going, shortest path?
Yes a VTEP is usually only ever 1-2 hops away (if you include VMM integration). VTEP (Leaf) == Spine == VTEP(Leaf)
- If ESX a and ESX b is connected to the same Nexus 55k is the traffic not touching the APIC Fabric (N9K Spine Switches)?
Traffic never flows "through" the APIC. It's just a controller. Policy enforcment and forwarding is done by the switches - mandatory. With VMM integration with the VMware vDS, the ESX hosts need to be directly connected to a Leaf. With VMM integration using the Cisco AVS and VXLAN mode, your ESX hosts can be multiple L2 hops away from the fabric. In this mode direction connections are not required, but the switching fabric is still required.
- Or is the traffic running over the APIC Fabric (N9K Spine Switches) to enforce the policy?
See above answer.
