APIC In-band IPs unreachable outside the fabric

[Nova777]

I have in-band IPs configured for all APICs, leafs, and spines in the mgmt tenant going through an L3Out in the mgmt tenant as well. All the spines and leafs are pingable from the directly connected device (Cat 4500) including the Inb SVI that is configured on the Inb BD. Only the two APICs aren't reachable. The APIC shows configured for inband and you can ping the Inb SVI from the APIC and the other leafs and spines can ping the APICs via the inband IP. I have another fabric running the same code version and identical configuration for in-band and the APICs are pingable via Inband from outside the fabric from the same Cat 4500. Both fabrics have L3outs in the mgmt tenant using routed interfaces and static routing (0.0.0.0/0). 

IPs are:

129.1 - SVI (reachable)

129.2- APIC 1 (unreachable)

129.3- APIC 2 (unreachable)

129.6-11 Leafs & Spines (reachable)

There is no config on the 4500 that would be blocking access to just those two IPs and I've tried another subnet with the same result for the first fabric.

Any ideas?

[Sergiu.Daniluk]

Hi @Nova777 

Is the default management connectivity mode set to inband? (System > System Settings > APIC Connectivity Preferences)

Cheers,

Sergiu

[Nova777]

It is set to inband.

I just noticed L3 drops for communication to the two APIC IPs, yet the same contract allows communication for all the leafs and spines. If I set the VRF to unenforced the two APIC IPs start pinging. Not sure why the inband contract, which is configured identically to the second fabric, is dropping traffic just to and from the APICs. Attempting to ping from the APICs, while sourcing the inband interface, fails to external IPs as well. So it appears narrowed down to the inband contract or how it's getting programmed somehow to drop traffic just for the APICs.

[Robert Burns]

Connectivity preference wouldn't impact reachability to inband IPs.

Check the inband management contract. If unenforcing the vrf fixes it, it's a contract issue.

Robert

[Nova777]

Something about the contract can't be getting programmed correctly if it allows eight other devices to be reachable but not the two APICs. The contract applies to the entirety of the inband EPG to external EPG, not just a subset of endpoints in that EPG.

[Robert Burns]

Check the contract rules of the switches where the APICs are connected.

1. Get the class ID for the inband management EPG.  Grab this from the UI, or CLI

moquery -c fvAEPg | egrep "dn|fv.AEP|scope|pcTag"

2. Check the zoning rules on each leaf

show zoning-rule | grep [inb pcTag]  (take note of Rule IDs)

show system internal policy-mgr stats  | grep [Rule ID]

3. (While testing Ping between APICs) Check Denied Logging entries

show logging ip  access-list internal packet-log deny | grep [APIC_IP]

show logging ip access-list cache deny

You can also check the "security" tab in the GUI assuming you're running a later version.  This will show hit counters for the contracts

Robert

[Nova777 · Accepted solution]

To close this one out,  the issue was the 0.0.0.0/0 wasn't defined on the subnets option of the l3out and only affected the APICs, not the leafs/ spines. 

[ariannur]

hi i'm have same problem too, cannot ping inband apic from outside but the leaf can

what do you mean 0.0.0.0/0 wasn't defined on the subnets option of the l3out? i have 0.0.0.0/0 subnet for external epg in l3out

[Nova777]

Yes, the 0.0.0.0./0 on the ext epg for the mgmt tenant L3out with scope - external subnets for the external epg.