Hello @conf-t
In a Multi-Pod setup within a Cisco ACI environment, the issue of non-communication between VMs in different Pods but within the same EPG can be complex due to the additional layers of abstraction and potential points of failure. Here are some specific considerations and troubleshooting steps tailored for a Multi-Pod setup:
1. Multicast Configuration in Multi-Pod Setup:
- Multicast Group Consistency: Ensure that the multicast group configuration is consistent across all pods. The multicast group used for the Bridge Domain (BD) should be the same in both Pod1 and Pod2.
- PIM Configuration: Verify that Protocol Independent Multicast (PIM) is correctly configured and operational on all IPN links in the Multi-Pod setup. Ensure that PIM is enabled on all relevant interfaces and that the PIM neighbors are correctly established.
2. Inter-Pod Network (IPN) Configuration:
- IPN Device Configuration: Check the configuration of the IPN devices (routers or switches) to ensure they are correctly forwarding multicast traffic between the pods. This includes verifying PIM, IGMP, and any relevant multicast routing protocols.
- MTU Settings: Ensure that the MTU settings are consistent across all IPN links. An MTU mismatch can cause fragmentation issues, leading to packet drops.
3. ACI Fabric Configuration:
- Bridge Domain (BD) Settings: Verify that the BD is configured to support inter-pod communication. Ensure that the BD is set to flood ARP traffic and that the multicast address is correctly configured.
- EPG Configuration: Ensure that the EPG configuration is consistent across all pods and that there are no contract issues preventing communication between the VMs.
4. Spine and Leaf Switch Configuration:
- Spine Switch Multicast Configuration: Check the spine switches in both pods to ensure they are correctly joined to the multicast group. Use commands like show ip mrouteand show pim neighbor to verify multicast routing and PIM neighbor relationships.
- Leaf Switch Configuration: Ensure that the leaf switches in both pods are correctly configured to handle multicast traffic. Verify IGMP snooping settings and ensure that the leaf switches are correctly forwarding multicast traffic to the spine switches.
5. Network Policies and Contracts:
- Contracts and Filters: Ensure that there are no contracts or filters that might be blocking ARP traffic between the pods. Even though the VMs are in the same EPG, contracts can still affect traffic flow.
- Policy Enforcement: Verify that policy enforcement is consistent across the pods and that there are no discrepancies in the policy application.
6. Physical Layer and Hardware Issues:
- Link Failures: Check for any physical link failures or errors on the IPN links between the pods. This can be done by examining the interface statistics and logs.
- Hardware Issues: Ensure that there are no hardware issues with the spine switches or the IPN devices.
7. Software and Firmware:
- Firmware Versions: Ensure that all devices are running compatible and stable firmware versions. Sometimes, bugs in the firmware can cause unexpected behavior.
- Configuration Sync: Verify that the configuration is properly synchronized across the ACI fabric. Inconsistencies can lead to communication issues.
8. Advanced Troubleshooting:
- ELAM (Embedded Logic Analyzer Module): Since you are using ELAM to capture ARP traffic, ensure that you are capturing traffic on the correct interfaces and that the ELAM configuration is correct.
- Debugging Tools: Use additional debugging tools and commands to trace the ARP traffic path. Commands like show logging, show tech-support, and debug ip pim can provide more insights into where the traffic might be getting dropped.
Summary:
Given that the ARP traffic is visible on Spine Pod1 but not on Spine Pod2, it suggests a potential issue in the multicast routing or group membership on the IPN or spine switches. By systematically checking the multicast configuration, IPN device settings, ACI fabric configuration, and physical layer aspects, you should be able to identify and resolve the issue preventing communication between the VMs in different pods.
HTH
