EPG Subnet vs Bridge Domain - Errors after upgrading to 5.2(3g)

[bl80]

Primary question - What is the difference between the subnet defined under either the uEPG or the AP EPG vs the subnet defined on the Bridge Domain associated with that uEPG/EPG?

Secondary question - Do you need (is it common/best practice) to only set the subnet IP in the bridge domain and refrain from setting it in the subnet for a specific EPG/uEPG?

After upgrading from 4.2(5l) to 5.2(3g) we are unable to create any EPGs in our entire fabric.  We are getting duplicate Subnet errors and cannot create in any VRF.

Our standard deployment has been leveraging microsegementation --

Normal process:

Create a Bridge Domain with subnet defined, example 10.205.16.0/21.

Create Application EPG tied to that BD, set VM Domain and allow for microsegmentation.

No subnet defined under the AP EPG

Create numerous uSEG EPGs under that AP EPG

Set same VM Domain for each uEPG and set attribute under each to define the different server groups.

(do this based on server name contains, for example "-SQL-" so any server with -SQL- in the name will be auto-placed into that microsegmented uEPG.

Then we have been setting the same 10.205.16.0/21 subnet under each uEPG.  

We have set the No Default SVI Gateway on each.

The above has worked out fine, once we got other things sorted, but this has been our standard setup for a while.

When we upgraded today to the new version we cannot create any more EPGs -- get error that the subnet (10.205.16.0/21) is duplicate and used in various EPGs.  I tried to delete a subnet from an EPG and now cannot re-add it.  

The uEPG without the subnet still looks to work fine.  Servers in that group are still showing correct and no impact seen.

I am wondering if our entire SOP has been incorrect and this subnet should only be defined in the BD but dont want to cause huge disruption if I start to remove it.  

Found the exact same issue on other App EPGs that are not microsegmented.  I cannot create a new EPG under those VRFs at all until all the App EPGs have their subnets removed and ONLY the BD contains the /24 for each application EPG.  

Appreciate any insight and can provide more specifics if needed. 

[Robert Burns]

Yikes.  Lots to unravel here.  But yes, your SOP could be improved. 

EPG vs. BD Subnets.  Always create subnets under the BD unless you're doing shared services (ie. Inter-VRF leaking).  This is well explained on a bunch of posts if you need details. If you're not doing routing on the fabric, you likely don't need to define a subnet.   

Why are you defining subnets if you're setting the No Default GW SVI flag?  This is a specific use case for this.  Either you want an L2 BD (where a FW or other device is acting as your GW), or you want the fabric to do your routing.    This flag is specially for setting up an Anycast service on the fabric.   Ref: https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/kb/b_Cisco_APIC_and_Anycast_Services.html

Are you using the fabric as L2 only (no unicast routing) and if so, which device is handling L3?

Robert

[RedNectar]

HI @bl80 ,

Too slow - @Robert Burns has pretty much answered - but I had the following 90% prepared so may as well add it now

---

Primary question - What is the difference between the subnet defined under either the uEPG or the AP EPG vs the subnet defined on the Bridge Domain associated with that uEPG/EPG?

A. If an EPG/µSegEPG is the Provider of a contract shared between VRFs, then you must define the subnet under the EPG/µSegEPG so that it is allocated a globally unique pcTag when the contract is applied.  Otherwise, there is no advantage to defining the subnet under the EPG/µSegEPG - in fact, there are some disadvantages (lack of flexibility)

Secondary question - Do you need (is it common/best practice) to only set the subnet IP in the bridge domain and refrain from setting it in the subnet for a specific EPG/uEPG?

A. Absolutely. Except in the provider-of-a-shared contract case mentioned above

After upgrading from 4.2(5l) to 5.2(3g) we are unable to create any EPGs in our entire fabric.  We are getting duplicate Subnet errors and cannot create in any VRF.

A. WOW - yep. I just verified that! BUT ONLY if the subnet exists on more than one EPG.  One BD+one EPG/µSegEPG seems to work OK.

Our standard deployment has been leveraging microsegementation --

Normal process:

Create a Bridge Domain with subnet defined, example 10.205.16.0/21.

Create Application EPG tied to that BD, set VM Domain and allow for microsegmentation.

No subnet defined under the AP EPG

Create numerous uSEG EPGs under that AP EPG

Set same VM Domain for each uEPG and set attribute under each to define the different server groups.

(do this based on server name contains, for example "-SQL-" so any server with -SQL- in the name will be auto-placed into that microsegmented uEPG.

Then we have been setting the same 10.205.16.0/21 subnet under each uEPG.  

We have set the No Default SVI Gateway on each.

Note: Only need to do the No Default SVI Gateway if it is a /32 subnet (that you defining under the EPG because you are trying to advertise to an external VRF)

The above has worked out fine, once we got other things sorted, but this has been our standard setup for a while.

When we upgraded today to the new version we cannot create any more EPGs -- get error that the subnet (10.205.16.0/21) is duplicate and used in various EPGs.  I tried to delete a subnet from an EPG and now cannot re-add it.  

Comment: You don't need to. Be happy without a subnet. UNLESS you need to leak it to another VRF because the EPG is providing a contract.

The uEPG without the subnet still looks to work fine.  Servers in that group are still showing correct and no impact seen.

I am wondering if our entire SOP has been incorrect and this subnet should only be defined in the BD but dont want to cause huge disruption if I start to remove it.  

Don't fret. You can remove the Subnets form the µSegEPGs without any loss

Found the exact same issue on other App EPGs that are not microsegmented.  I cannot create a new EPG under those VRFs at all until all the App EPGs have their subnets removed and ONLY the BD contains the /24 for each application EPG.  

Appreciate any insight and can provide more specifics if needed. 

---

Seems as if all your µSegEPGs are using the same default gateway, so probably best practice (if to no other reason than to save confusion) to just have the subnet on the BD.

[bl80]

Thank you both for the quick replies..  

@RedNectar   -  regarding your statement : 

A. If an EPG/µSegEPG is the Provider of a contract shared between VRFs, then you must define the subnet under the EPG/µSegEPG so that it is allocated a globally unique pcTag when the contract is applied.  Otherwise, there is no advantage to defining the subnet under the EPG/µSegEPG - in fact, there are some disadvantages (lack of flexibility)

-- thank you for this!!  I have never heard it explained so simply before.

But admittedly, this is very concerning going forward in how we have had our system admins deploying servers with zero fabric changes required once the base uEPG (as detailed below) was in put in place.  We even leverage the Palo Alto ACI plug-in to dynamically populate FW object groups so when a server is rolled out in a uEPG its automatically added to the appropriate policies in every firewall in the company...

Specific example for our "Front End" servers in our standard deployment

FE-VRF

-FE-SYSTEMS-BD (10.205.24.1/21 - advertise externally, shared between vrfs)  -- NOTE -- edit -- was incorrect /24.  All subnets are /21 in our server BDs.

--FE-SYSTEMS-APP-PROFILE

---FE-SYSTEMS-APP-EPGs (FE-SYSTEMS-BD, VMWare VMM Domain, Micro segmentation Allowed)

----FE-ServerGroup-1-uEPG 

----FE-ServerGroup-2-uEPG

----FE-ServerGroup-3-uEPG

----FE-ServerGroup-4-uEPG

----"" "" -- 20+ different uEPG Server Groups under the APP EPG

Most servers communicate between the same FE VRF using a simple "Allow-ANY" contract set as Provider and Consumer under vzANY for each of the VRFs.  But perhaps 4 or 5 uEPGs are providers for Server Groups in other VRFs.

Hundreds of different servers in each of the 3 Vrfs defined that contain primary Server Groups.  Each their own /21.  20+ uEPGs in each VRF as well as other VRF for special case systems.

The above uEPGs would each have the same /21 subnet defined under each set with Advertise Externally and with No Default SVI Gateway.  I cant recall exactly when we started using that option but we had numerous issues in our muti-tenant deployment and once we set to No Default SVI everything started working.  We had set /32 IPs for each server in each uEPG at one time and that also worked but due to the nature of constant server deployments we want to make this as dynamic as possible.  

The above has been working.  Sys Admins can deploy servers at will to any uEPG and all communication has been stable.

Obviously seems 5.2x ACI is more strict (or smarter to ensure specific configurations are not allowed).  

I have just created a couple test VRFs and populated with the above settings and the only working solution I can get is to set everything exactly as you noted.  VRF1 <-> VRF1 works fine (as expected) without eEPG subnet defined.  VRF1 <-> VRF2 only works if there is a subnet defined (/32 or anything outside the same scope of the BD).  

Frustrating that this is how we are going to have to move forward with more manual intervention.  Might be a good time to learn how to leverage the REST API .  We have been so hands off for sever deployments this will be quite a change...

Again - truly appreciate the quick help.  

[RedNectar]

Hi @bl80 ,

Sounds like you had a pretty neat system there, and if there are some /32 IP amongst the µSegEPGs then that would explain why you needed the [x] No Default SVI Gateway option set - and so easier to set for all when automating.

Now, those "4 or 5 uEPGs are providers for Server Groups in other VRFs" will still need the subnet defined under the µSegEPG. What you might have to do is get down to the /32 IP of each server (you can have multiple subnets) and add each /32 under the µSegEPG.  Clearly not as easy as adding the /21 which is what you were doing.  In fact, now that you have explained it this way, it sounds as this is a legitimate use case for using /21 on every µSegEPG - so it might be worth raising a TAC case about this to see if you can get a valid reason WHY this restriction has been added (and IF you find out - please report back)

Mind you, I don't want to discourage you from exploring the world of ACI APIs either.

One last thing though, (and this is one of my pet hates) you mentioned:

using a simple "Allow-ANY" contract set as Provider and Consumer under vzANY for each of the VRFs.

I've NEVER understood why anyone thought this was a good idea.  It just doesn't make sense to me why you stand in one room with two doors. Door A leads directly to Room2, which is where you want to go. But instead of using Door A, you go through Door B, which leads to Room 3, where you then take another door to Room 4, which has a door that leads back to Room 2 - so you finally get to where you want to go - the long way round.

But, I'll concede that it is a popular design.  I hope someone can explain to me why it is a preferred option over just checking the Unenforced option. Both are implemented in hardware using exactly the same filters if I believe correctly.

Mind you, there are more flexible ways of allowing a bunch of servers to communicate with each other that the two mentioned (Preferred Group, Application Profile scoped Contracts) above.

[bl80]

@RedNectar -- I'll be looking to see what info TAC might be able to provide.  Luck with them has been wishy washy at best lately unfortunately.  

Regarding the vzANY contract for provider/consumer--- this is done only as we use Service Graphs on most all of our contracts to send traffic thru a Firewall for the filtering..  

Appreciate the help !!!

[RedNectar]

Hi @bl80 ,

Yep - putting a service graph in the middle of a vzAny contract makes sense - in which case it s case of "Sombody changed the default contract" - although I'll admit you didn't actually say default contract...

So my bad for assuming...

And for not turning on the spell checker (oops. Too bad)

Have a wonderful rest-of-your-day!