ACL_DROP on inband management

[Nik Noltenius]

Hi folks,
we've setup inband management on our ACI lab enironment. All switches and APIC can ping each other just fine. Then we have an L3Out in the management Tenant to allow external devices to reach the fabric nodes via their inband management addresses. The common default contract is provided and consumed by all internal and external EPGs.
Now, from the outside we can only reach the border leaves inband management address. All other pings are lost.
Using the ELAM assistant app we figured out that the ICMP requests reach the border leaf and are dropped there with Drop Code "ACL_DROP". Further research showed, that this can be ignored on FX switche (unfortunately we have EX) and that internal rules leading to ACL drops cannot be modified. But there must be something we can do, right? Because inband management should work...
Testing the other direction, i.e. sourcing an ICMP request from within the fabric, we can see that it is properly transported to the border leaf and that one sends it out to the external world, too. Just the reply again gets ACL_dropped.

Has anyone experienced a similar issue and might be able to help? Or do you have more detailed explanation for what ACL_Drops are and why they occur?

Thank you and kind regards,
Nik

[RedNectar]

Hi @Nik Noltenius ,

There are a couple of details that don't seem complete, specifically:

The common default contract is provided and consumed by all internal and external EPGs.

I'm assuming that the external and internal EPGs are in different VRFs (because if they were all in the same VRF, why wouldn't you just toggle the VRF Policy Enforcement to Unenforced).

if my assumption is incorrect, i.e. the external and internal EPGs are in the inb VRF, then the first thing I'd try is to toggle the VRF Policy Enforcement to Unenforced and forget what follows

So this leads to those tricky settings - firstly, that the subnets are all marked as Advertised Externally - but I'm pretty sure you've done that, and that the BDs are all linked to the appropriate L3Out. Again - not likely to be the problem, but I'd feel better if you'd tell us that these things have been checked.

But the trickiest of all are the L3 out settings. And these vary a little depending on the routing protocol you are using - which again you didn't tell us.

If you are using OSPF as your routing protocol, you may need to set the scope on the Subnet of the External EPG of the L3Out [Networking > L3Outs > your_L3Out > External EPGs > your_L3EPG >| Subnets > yourSubnet]

Make sure the following options are set:

[x] Shared Route Control Subnet

[x] External Subnets for External EPG
[x] Shared Security Import Subnet

[Nik Noltenius]

Hi @RedNectar,
thanks for your reply
 
You're absolutely right, I failed to provide all context.
So, everything is in VRF inb ans yes, there are other solutions than using a contract but all lead to the same result. Setting the VRF to unenforced is basically the first thing we do (and did in this case) for troubleshooting if there is only the slightest chance that issues might be contract related.

Our subnet is Advertised Externally and also Shared Between VRFs, although that's not required for the moment. 
We don't use any dynamic routing protocol, we only have a simple static default route towards the outside world. The firewall we connect to has a static route for our internal inband network. Subsequently our EEPG contains 0.0.0.0/0 as the only subnet and that is only set to External Subnets for the External EGP and without dynamic routing that should be fine. I've turned on the other knobs you mentioned to be sure but it didn't change the behavior we're seeing.
Anyway, I believe routing and contract must be working as we see (using ELAM) the ICMP packet coming in on the border leave. It applies policy and does not drop due to contract but with code ACL_DROP, whatever causes it to do so...

To provide full transparency I've attached our mgmt-tenants full (well... configurable) json. It's only a lab, so there are no company secrets.

Thank you and kind regards,
Nik