Inband management EPG as provider

[Antonio Macia]

Hi,

I've successfully configured the inband access in ACI using the mgmt tenant with the inb VRF. Consuming the L3out from the common tenant, I can remotely manage the APIC and switches, however, I cannot connect from another EPG in another tenant. The inband subnet and the consuming EPG subnet are both configured for "Shared between VRFs" and correctly leaked at each VRF after checking the routes. I've exported the inband provider contract and added it into the consumer EPG as a contract interface, but still cannot reach the inband IPs. Is this a known ACI limitation?

Regards.

[6askorobogatov]

I know it is obvious, but did you check route in  VRF mgmt:inb  for routes back to your consumers ?  

[Antonio Macia]

Hi,

Yes, the consumer routes are present in the "inb" VRF, so routing should happen normally.

[6askorobogatov]

If it's not a route,  it is contract. Is you contract subject has specific ports, by directional ?  Do you have ICMP permitted ? Can you ping ? 

The configuration should work, right now , i'm accessing APIC  inb from the workstation in another tenant ... 

[Antonio Macia]

How are your consumer and provider contracts configured? 

I think I've tried almost every possible option for the provider contract (always allowing all kind of traffic):

• "Global" and "Tenant" scope and exported to the consumer tenant. 
• Also using contracts from the common tenant as providers.

Do you reach the mgmt tenant through an external L3out or within the ACI fabric?

[6askorobogatov]

Contract should be "global" . I have  access to inb form both, L3out in number of tenants and EPGs in number of tenants. 

What about subject in the contract in the subject?  Do did you try unrestricted:   IP /both dir./ reverse port 

[Antonio Macia]

My contract is set to global and the filter allows all the traffic.

Surprisingly, if I configure the consumer EPG as the "provider" and vice-versa (the other way around), I can reach the inband EPG. For example, when the supposed consumer EPG provides ICMP access and this service is consumed by the inband EPG, then I can ping the APIC and switches, the same when the filter allows SSH (source port 22) at the consumer, then I can ssh the inband IPs. So it's like the contracts, when provided by the inband EPG, do not work for some reason.

[RedNectar]

Looks like you answered my question while I was composing it........ Do you have psychic power?

"the filter allows all the traffic."

This means the concept of Provider and Consumer go right out the window - in fact the Cosumer/Provider ONLY works for TCP. For ICMP, UDP, ESP etc etc, it makes no difference which is provider and which is consumer.

Hence your L3Out EPGs and access inside EPGs and vice versa

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[Antonio Macia]

Hi @RedNectar ,

Right now I have a simple contract that allows SSH only:

• Scope set to global.
• TCP dst 22.
• "Both directions" and "reverse port filters" enabled.

This contract is provided by the inband EPG at the "mgmt" Tenant and exported to tenant B. EPG at Tenant B consumes the contract interface. Can't SSH the APICs or switches from a VM in Tenant B. Am I missing something?

Thanks.

[RedNectar]

OK. Took me a couple of hours but I've cracked it. It will take as long again to write it up. [Edit: Family Christmas do got in the way. Nearly finished writing up, but it's 1:00am - will finish when I wake. The secret is creating an extra EPG in the mgmt tenant, gving it an IP address and making it Shared Between VRFs. All will be clear when the write up is finished]

[RedNectar · Accepted solution]

Hi @Antonio Macia ,

OK, I worked through it (making a couple of my own mistakes on the way) and got it working. I documented the complete process on my blog, but carrying on from:

a simple contract that allows SSH only:

• Scope set to global.
• TCP dst 22.
• "Both directions" and "reverse port filters" enabled.

This contract is provided by the inband EPG at the "mgmt" Tenant and exported to tenant B. EPG at Tenant B consumes the contract interface. Can't SSH the APICs or switches from a VM in Tenant B.

The bit you were missing is actually not so much what YOU were missing, but what ACI is missing, and that is the ablity to mark the inband EPG subnet as Shared Between VRFs.

The kludge I used to get around this was to create an Application Profile and EPG in the mgmt tenant, added the inband management IP subnet and checked the shared between VRFs option, and had it also provide the contract.

That allowed the route for the inband management subnet to leak into Tenant B, and it all came good.

Don't forget to check out the complete story.

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

Footnote: Regarding the inability to set the Shared between VRFs option on the Inband EPG - I tried forcing the attribute by posting this to the EPG:

 {
        "fvSubnet": {
          "attributes": {
            "annotation": "",
            "ctrl": "nd",
            "descr": "",
            "ip": "192.168.99.1/24",
            "name": "",
            "nameAlias": "",
            "preferred": "no",
            "scope": "private,shared",
            "virtual": "no"
          }
        }
      }

into the mgmt tenant - the post worked, and I could see the changed attribute in the object nrowser, but it made no difference to the behaviour.  So to fix this properly needs more than just adding the abilty to set the option.

[6askorobogatov · Accepted solution]

There is a simpler way to let EPG in another TENANT:VRF to get to inb. The issue is a route leak from inb to the client's VRF that needs an EPG subnet, not just a BD  subnet.  That is a bit different to config. in MGMT / INB epg compared to the "normal" EPG. 

So, without dealing with EPG subnet:   create global contract for ICMP in MGMP tenant apply it as  provider  to INB EPG and export to client's tenant and apply contract interface to clients VRF. Cleate . contract for ICMP in client's tenant, apply as prover to client's EPG and export it to  MGMT and apply contact interface to INB EPG. That will take care or route leak both directions.

Also, i it is nothing wrong, i believe, to allow ping between client's IPs and inb IPs. Could be  helpful  in future. 

Now you can add your SSH contact as client's EPG consumer and INB EPG provider. 

If you really want to make SSH one way only,  in subject un-check "reversed port" and  "both direction" and make 2 rules in the subject:

Consumer-provider source  TCP (default) 1024-65535 to TCP 22

Provider Consumer  TCP (stateful, Established ) TCP/22 to TCP/1025-65535

[Antonio Macia]

Thanks a lot @RedNectar your solution worked like a charm!

Very appreciated.