I am deploying a Multipod datacenter site A and Site B, That one is designed for different customers so my question is if I have a IPN to extend L2 between on both side. How would I differentiate replication traffic flow for different customer due to my IPN is 500Mbps but each customer contracted their own 1Gbps link for replication traffic. Maybe I would used a l2out between both sites but it is risky because of Loop. I want to know the best practices and scalable solution.
NO! Don't ever link another L2 link between two pods in a MultiPod setup, other then the overlay tunnels, because you will create a loop.
Coming back to your question, the easiest way to differentiate traffic between tenants (supposing your customers are different tenants in ACI, or at least different application profiles - you will see in just a sec why) is to use QoS.
First you need to associate a QoS class for your customer traffic inside the fabric:
Tenant-name -> Application profile -name -> Policy -> Qos Class
Second step is to enable Cos-to-DSCP translation policy. This way packets egressing out of a POD will have the DSCP value you map for the traffic type you selected at step 1:
Infra -> Policies -> Protocol -> DSCP class-Cos translation policy
Third and final step is to configure the QoS in IPN as you wish (scheduling, priority queues, policing etc)