route leaking with subnets on BD

[bcn-jbrooks]

Hi all,

My ACI journey continues, and I invariably end up back here looking for help.  

I am currently working through a design for a "shared services" tenant in our environment.  

What I'd like to do is:

-in Shared Services tenant, have one BD with one subnet

-in the ANP, create multiple EPGs that use the subnet/BD

-specify contract/filters for each EPG

The outcome I want is to have several services on the same subnet, but access to those services constrained by policy.  So for example, I want to make DNS, NTP and SMTP available to all tenants, and have them all on the same subnet, but have the policy allow DNS queries to ONLY the DNS server, NTP queries to ONLY the NTP servers, and SMTP sessions ONLY to the SMTP servers.

I can make this work the "traditional" way (where subnets are defined on the EPG), but this method doesn't quite provide my desired outcome.

is there a way to do this?

I found some docs on the interwebs that say it can be done, but that the contracts/filters must be specified and exported/imported in both directions.  TCAM concerns aside, would this work?  I can't figure out how to make it work.  

For experimentation, I have a very simple set up:

Tenant 1 (provider)

  -BD1

  --subnet1 (192.168.0.0/24)

  -EPG1

  ---filter1: allow IP/TCP/Port 80 (stateful) -> i.e.: from ANY port to Port 80

  ---Contract (with filter1 as subject) exported to Tenant2

Tenant 2 (consumer)

  -BD2

  --subnet2 (10.0.0.0/24)

  -EPG2

  ---filter2: allow IP IP/TCP/ANY (stateful) -> i.e.: from TCP 80 to ANY port

  ---Contract (with filter2 as subject) exported to Tenant1

They all appear to be Formed, and no errors or alerts raised... however, routes only leak in one direction (in my case I can see a route to Tenant1 in Tenant2's VRF routing table, but I can't see Tenant2's route in Tenant1's VRF routing table).

I could just define small subnets at the EPG level (a /29 for NTP, a /29 for DNS, etc), but that seems counter intuitive for ACI.

Open to suggestion and ideas!

Thank you very much!

J

[bcn-jbrooks]

A quick follow up to say that I got the BD-to-BD route leaking working.  I just wasn't exporting/importing properly.

However, I'd still appreciate any guidance on the "right way" to do this... 

Another question that comes up too is whether or not there are limits on the number of tenants I can export a contract to.  When looking at the UI at:

Tenants -> Tenant1 -> Security Policies -> Contracts

I see a list of my contracts, and any that are exported have a column called "Exported Tenants".  

Can i export the same contract to hundreds of tenants?

Thanks!

J

[Joseph Young]

Hello,

The only 'supported' way to do this type of thing is by configuring the subnets underneath the epg's and then make sure you don't have hosts in the same subnet overlapping between epg's. The reason for this is that for inter-vrf communication between internal endpoints the destination class of the provider (dclass/pctag) is based upon the prefix. Then when the pctag lookup is complete the zoning-rules are evaluated to see if the traffic should be permitted. Because the pcTag lookup is based on prefix you run the risk of traffic receiving the wrong dclass if you have overlapping ip address stretched across epg's.

Yes, you can do this with the subnet configured under the bd if both epg's are providing and consuming but i don't know that you are going to have the contract granularity that you are desiring if you do it this way.

Joe

[bcn-jbrooks]

Thanks Joe.

So if I understand correctly then, this is not much different from traditional networking where servers that serve a common service (like DNS servers) need to be on the same subnet and in the same EPG.  is that correct?

I have managed to get this all working BD subnet to BD subnet with multiple EPGs on the same subnet.

In my case, I have one subnet on a BD in my "provider" tenant that has two EPGs on top of it; one for DNS and one for NTP.

I can provide/consume contracts in "provider" tenant and "consumer" tenant and selectively allow services at the EPG level.  In other words, the DNS EPG only allows systems in the "consumer" tenant to query DNS, and likewise, the NTP EPG only allows NTP queries from the "consumer" tenant.  Even more, DNS servers can not reach NTP servers and vice versa because there is no policy allowing that.  This is exactly the behaviour I wanted.

But now, with your comments, I'm not sure if i want to do this if "subnet on the EPG" is the only supported method... is that really true?  Do I need a DNS subnet and an NTP subnet?  That seems counter-ACI in my brain.

Does the key risk you describe only happen when there's risk of overlapping IP space?  In our case, we have a "public" IP space (a subnet in each tenant that is Shared and Advertised), and then a collection of "private" networks where tenants can do whatever they like.  We will only ever connect a tenant's "public" network to shared services.  Their "private" networks are always contained and constrained within the tenant and will never be allowed out through the fabric.

Eager to get more feedback!

cheers,

Joel

[Joseph Young]

Take a look at the section titled "Shared Services Contracts Usage" from this link:

http://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/aci-fundamentals/b_ACI-Fundamentals/b_ACI-Fundamentals_chapter_010100.html#concept_FD8CA8BD29EA4F70A0F96D18AC1DBD87

That is the most clear document I've found about what is supported v. what isn't for shared services.

When I say overlapping I am referring to epg's in the same vrf. If epg A and epg B both have endpoints in the same subnet but you want to allow them to provide/consume different shared services then you are going to run into issues with traffic being mis-classified since dclass lookup is done on prefix rather than endpoint. In order to get around this type of issue every shared vrf would have to have every other shared vrf's endpoint entries and vice versa.

[bcn-jbrooks]

Thanks Joe.  All very interesting!

i (sort of) understand the risk, but it just seems so counter to ACI policy model as far as I've learned so far.  It sounds like I basically need a subnet per service which I can do easily enough with any L3 network gear - as opposed to just grouping systems into EPGs (on the same BD+subnet) and applying different policy (through contracts+filters) which seems much more like what I thought ACI was offering.

Anyway, back to the whiteboard I go then.

thanks for your help!

[Jayesh Singh]

Hi Guys,

I know this is an old post, but it is possible people would land up here looking for a solution.

I have covered route leaking design in detail in one of my articles recently that may help,

Cisco ACI Inter VRF/Tenant Route Leaking Design – Simplified!

Regards,

Jayesh