Hosts within Same EPG Not Able to Ping Each Other in Multi-Pod Fabric

[zachartl]

Hello,

We've just encountered this situation where we have two ACI Pods connected across town via Ethernet/OSPF connection. We've a single EPG (EPG_VLAN_750) containing hosts within the same subnet, Pod-1 Host 10.75.0.136 and Pod-2 Host 10.75.0.139 cannot ping each other. Neither host has the other's ARP Entry within their ARP Tables.

The Bridge Domain is routed externally and each host is pingable from a host outside the fabric, my laptop or workstation. Each host is able to ping externally to say 8.8.8.8 or my laptop or workstation. I've checked to ensure the MTU along the path is indeed 9150 all the way through and it is. Each host is connected downstream to a Dell VLT (their version of vPC) port-channel at each site. Spanning Tree doesn't appear to be an issue.

Thank you,

Terry

[Sergiu.Daniluk · Accepted solution]

Hi @zachartl 

First, I would start verifying which direction is broken. Since ARP is resolved, it means that we have a unicast problem.

So start a tcpdump/wireshark capture on one of the end hosts and ping from the other one. See if the ICMP request is received and if reply is generated. Do the same thing but in the other direction. This way you will know which direction is broken (the one where you do not see the ICMP request received).

Once you know this, (example EP1 -> ACI POD1 (leaf1-spine1) -> ACI POD2 (spine2->leaf2) -> EP2), start the investigation from POD1. Note: if both directions are broken, well then you can start from either end.

The next step would be to use ELAM and see where is the ICMP packet dropped. https://dcappcenter.cisco.com/elam-assistant.html If you are not familiar with the tool, no worries it is very intuitive.

Once you find the problematic device where the packet is dropped, you will need to look for EP information. For this, best to check this awesome ciscolive: Mastering ACI Forwarding Behavior: A day in the life of packet - BRKACI-3545 

Stay safe,

Sergiu

[zachartl]

Thank you Sergiu,

Your response was quite helpful.