How can I Move a Bridge Domain to another vrf, with minimal impact

[netwerkbeheer]

In our production tenant we have two vrfs;

vrf-1 with multiple related bridge domains associated to L3out-1, which handles intertenant, and WAN traffic..

vrf-2 with one (larger subnet - Advertised Externally) bridge domain; BD-2, associated to L3out-2, which handles traffic between the two vrfs

(A very inefficiënt cumbersome way to get traffic flows through the firewall, which demanded changing server IP addresses.. 🤮)

We now have a PBR/ ServiceGraph configured, which removes need for changing server IP-addresses, potentially gets get rid of this complexity with the unnecessary extra L3out, and enables us to use consistent policies throughout the entire tenant..

Especially if we can get the one large bridge domain into the same vrf.. Then we can be consistent in contract scope as well.. 

So..

Is there a way to associate the Bridge Domain BD-2 on Vrf-2, to Vrf-1, without much impact?

Can I associate a second L3out; L3out-1, to BD-2, (currently associated to L3out-2), without screwing up my routing? 

My hope is that ACI has the intelligence to recognize that the BD/ Subnet is associated to only one of the two Vrfs, and therefore Advertises the route correctly, in the current situation, AND after I change the associated VRF to Vrf-1..

Without causing an outage due to routing loops..

Please assist

Best regards,

Rick Roersma

[AshSe]

Hello @netwerkbeheer would it be possible for you to draw the logical diagram of your above mentioned setup and share for a better understanding.

[netwerkbeheer]

Hi

I hope this diagram helps.

https://rickroersma.stackstorage.com/s/puSlR8d3natJq5Px 

Less complexity;

Currently, one part of EPG to EPG traffic within this tenant, [all BD's in VRF-1 <---> BD-144 in VRF-2] uses only firewall rules, because of the L3out..

Everything within VRF-1 needs contracts with Service Graph for the same FW treatment.

Unfortunately, they placed all EPG's from both VRFs in one big, (horrible...)  application profile, so it's not clear what is on one VRF,  what is on the other VRF..
Hence, the need for consistency, which prompted this plan of mine..

(Second stage of my project is creating proper Application-Based Application Profiles, and rearranging al servers into these profiles) 

In the desired situation, Traffic can still be looped through FW using Service Graph/ Policy Based redirect contracts between the EPG's attached to the different Bridge Domains..

The only thing is that it's our production environment and I prefer to minimize any disruptions.

So if the BD-144 can temporarily associate to both L3Outs, without causing problems, and making the shift from VRF2 to VRF1 (almost) seamless, I will be a very happy camper indeed.

Best regards,

Rick

[AshSe]

Hey Rick,

Check the below diagram based on your's, and please answer few questions for better clarity:

Current Setup:

Desired Setup:

Questions:

1. For both (current & desired) do you have a single tenant?
2. What for you setup L3Out_OSPF (Internal) for communication between EPs in between two different VRFs in Cisco ACI fabric?
3. Didn't you think of configuring Inter-Tenant, Inter-VRF communication in current setup?

Feel free to specify Tenant, VRF (dummy) names.

Happy configuration!!

[netwerkbeheer]

Hi,

Thanks for the input so far.

1. It is a single tenant, with all EPG's in both VRF's currently having intertenant / WAN connectivity.
   
   There are some factors within this tenant that cause me to want this move;
   
   Currently we have on large application profile with about a hundred EPGs, some even containing servers for different applications. Not very secure, so I want to fix this.
   If I want to have any hope of being able to organise everything into proper application based application-profiles, I can't have one EPGs endpoint within an application profile, being on another vrf, IP address wise, compared to endpoints of other EPG's in that Application Profile..
   Because; What contract scope should I use in such a scenario??????? Application Profile? VRF?
   To prevent this, and to be able to be consistent applying contracts within this tenant, this move is necessary.
   
   To prevent that scenario and any other confusing inconsistencies, I want to move BD_144 to the VRF-1.
   
   But with as little impact as possible, so I was hoping to be able to "gradually" move BD_144 to the same config as the other BDs, never cutting external connectivity to/from BD_144, as would happen if I started by deleting L3out_2, leaving no attached L3out whatsoever..
   
   - STEP 1: Add the L3Out_1 (I hope without causing routing loops or problems)
   - STEP 2: Switch VRF
   - STEP 3: Only then remove L3Out_2
   
   These 3 steps is really what it all boils down to.
   I would expect it not to be a problem, because when I switched a test BD/subnet from vrf_2 to vrf_1, but without switching out the L3out (Internal), the subnet could only be found on vrf_1, so the attached L3out only seems active for the BD/subnet when it corresponds to the attached vrf, which seems logical..
   But I want to be sure that I'm not missing some weird, or unexpected behaviour occuring when the second L3out; L3out_1 (External) gets attached.
   
   
2. The L3out (Internal) seems like basic OSPF config
   -  area 0.0.0.1 (L3Out External is area 0)
   - Send redistributed LSAs into NSSA area and Originate summary LSA are enabled.|
   - Regular area
   - Cost 1
   - Under the external EPG, there is only 0.0.0.0 with External Subnets for the External EPG
   
   - The BD_144 has Advertised Externally set on its subnet
   
   
3. Maybe i don't understand this third question correctly/ completely, but inter-tenant and inter-VRF communication is already possible for all EPG's in this tenant.

I could leave it like this, but as mentioned in my answer to question 1, that would leave me and colleagues with a very confusing situation, causing all kinds of hard to figure out security configuration. All because of an odd-ball set of EPG's living on another vrf.

Best regards,

Rick

[AshSe]

Hey @netwerkbeheer 

with regards to the below:

1. It is a single tenant, with all EPG's in both VRF's currently having intertenant / WAN connectivity.

---

1. Is it a Multi-Site topology?
2. On one side you are saying single tenant and then in the same sentence you are using "intertenant"

Apologies, I am getting confused with the usage of L3Out inside a single tenant to allow communication between two VRFs.

[AshSe]

Hey Rick, Based on your inputs, pfb my understanding of your logical diagram (Please validate/correct):

I have couple of questions to you, before penning down the solution:

1. Why are you using L3Out-2 for firewall rules implementation? Is it because you are using external hardware firewall?
2. Why do you want BD-144 to temporarily associate to both L3Outs? Is it to make sure that it can get connected to Internet, and WAN as well?
3. Have you considered changing the scope of BD-144's subnet to both: "Advertised Externally" and "Shared between VRFs"?

Please confirm:

• Subnet scope of all BDs viz. 10,20,30,40, and 50 is already changed to "Advertise Externally".

Feel free if you would like to share some more details for better understanding and solution sharing.

All the best and Do your best!!

[AshSe]

Hey @netwerkbeheer , please check my diagram, and give reply to my queries.

Btw, please help me to understand the logic of using L3Out-2 between VRF-1 and VRF-2.

Have a good one!

[netwerkbeheer]

Hi,

It was an ugly and cumbersome way thought up to get some insights into traffic flows within this open, any-any-allow, "spray-and-pray" tenant.

• Re-IP-address servers,
• Place them in a new EPG in VRF2, behind the L3out/external Firewall with an any-any allow policy
• See traffic flows to and from the EPG
• Document, and make Firewall policies/ contracts

The previous head of IT probably still had the fully managed service graph seared into his conscience from when we were the first in holland to implement ACI, that's why he never allowed implementation of ServiceGraph/ PBR, chosing this cumbersome option instead.

I do understand his disliking for that firewall device-package "interface" with the neverending expanding menu's, but this L3out+VRF_2 idea never really took off, because who likes re-IP-addressing his perfectly working server setup in production? no one..

Best regards,

Rick