ACI Microsegmentation Requirement - Physical Domain

[zachartl]

Hello,

I've received a microsegmentation requirement. It appears we're to take a base EPG and create distinct groups of hosts. The hosts are in the same IP subnet and same BD. I intend to create three micro-EPGs and utilize the IP Address attributes, for each micro-EPG in order to create those groups. Then, leverage Contracts for the L4 destination ports required for each group/micro-EPG. 

Can I do this without affecting the other EPG/BD = VLAN traffic already configured within the vPC I will need to select as part of the micro-EPG configurations? The Physical Domain portion of the configuration.

I haven't done this before and want to ensure I won't affect those other EPG's traffic when implementing this configuration.

Thank you,

Terry  

[Robert Burns]

Terry,

Yes if configured correctly you can selectively apply to uSeg matching to specific hosts only.  A couple of points.  First, make sure you understand that once a host matches the attribute (IP) its re-classified (moved) from the base EPG to the uSeg EPG.  This means any contracts or communication allowed by the base EPG no longer applies to the uSeg EPG endpoints.  Just be sure that your uSeg endpoints no longer need to communicate with the base EPG endpoints because this wont happen unless you add a contract specifically.    Second advice is as you define each IP attribute on the uSeg EPG, monitor the "Operations" tab and Client Endpoint table to ensure only the desired endpoint has been moved to the uSeg EPG.  If unexpected shows up you can revert and re-examine your config.

Robert

[zachartl]

Hi Robert,

Very good information, Thank you. 

One of the requirements are that all 3 micro-EPGs are able to communicate freely with each other. The other requirement is the 3 EPGs be assigned specific destination L4 Ports. Would the first requirement entail east-west "permit any any" contracts between micro-EPGs? And would the next requirement entail north-south contracts between the Base EPG and the micro-EPGs to make this work?

Thank you Again Robert.

Best,

Terry

[Sergiu.Daniluk]

HI @zachartl 

I just want to add something on top of @Robert Burns's great response.

You mentioned "The hosts are in the same IP subnet and same BD" and "I intend to (...) utilize the IP Address attributes".

If you plan to enforce policies (contracts) between EPs which are part of the same single BD/subnet, then IP attribute will not help you. It's the MAC attribute which you **MUST** add the uSeg EPG.

Why is that, you might ask? Explanation is simple:

When a packet is routed by a switch, the forwarding lookup is based on the IP address. When a packet is switched by a switch, the forwarding lookup is based on the MAC address even when the packet has an IP header. Similarly, when a packet is routed by a switch, the contract lookup is based on the IP address. When a packet is switched by a switch, the contract lookup is based on the MAC address even when the packet has an IP header. This behavior affects the policy enforcement inside a uSeg EPG

If you add only the IP attribute inside the uSeg EPG, then only the IP address will be moved to uSeg EPG, while the MAC address will be preserved in the Base EPG. In other hand, the contract applied to uEPG, will apply only to the routed traffic.

If you add both IP and MAC attributes, then both routed and switched traffic will be policy enforced based on contracts applied on uEPG.

Stay safe,

Sergiu

[zachartl]

Hi Sergiu,

This is Most Helpful too, Thank you. I've been going over the Cisco ACI Contract Guide, dated June 23, 2021. Within the Microsegmentation Section, I was reading that Intra-EPG isolation involves the use of PVLANs and that if there's an intermediate switch, then a PVLAN configuration is required there as well. Again, I'm going to need to use the Physical Domain as part of the configuration. Would the reason for the intermediate switch PVLAN configuration be for preserving the integrity of the entire PVLAN configuration? Would it have to do with the VLAN Tag re-write over the vPC (Trunk)? The Promiscuous versus Isolated PVLAN Trunk?

Thank you Sergiu,

Terry

[Sergiu.Daniluk]

Hello again,

IntraEPG isolation is a feature which prevents the EPs inside a EPG/uEPG to not talk to each other. If you do not need this feature, it's not mandatory to use it for uEPGs to work.

On the other hand, if you need this feature, then you must be aware of the following:

1. if you have VMM domain associated to the EPG/uESG, then the port-group created on the VMware DVS or HyperV vSwitch will use a PVLAN (primary vlan, isolated vlan) pair.

1.1. If there are no intermediary switches between Host and ACI Leaf, then you don't have to worry about anything;

1.2. If you have an intermediary switches (something like a blade switch or a fabric interconnect), then you have to make sure that PVLANs associated dynamically or statically to the port-group are also configured as private vlans on the intermediary switch. This is simply required becuase the VM will send switches to GW (which usually is on the ACI) over the isolated and the response will be returned in primary vlan. If pvlan is not configured on the intermediary switch then traffic can either be dropped or unknown unicast flooded - depending on the default forwarding on it.

2. if the servers part of the physical domains are directly connected to the leaf switches, then you do not have to worry about anything. ProxyARP will ensure the correct policy enforcement.

2.1 If there is an intermediary switch for the physical domain servers (for example virtual switches on AIX servers), then you must ensure that traffic is enforced to be sent to uplink towards ACI leaf, instead of local switching. There are different features which can help you achieve this:

- PVLAN (if is supported on the intermediary switch)

- Reflective relay or VEPA (802.1qbg)

Cheers,

Sergiu

[zachartl]

Hi Sergiu,

We've Two sites. One is setup as follows;

ESX -------- UCS Fabric Interconnect ---vPC----NXOS 9K---vPC----NXOS 7K----vPC----ACI Border-Leaf N9K

Second Site;

ESX-------- UCS Fabric Interconnect-----vPC----ACI 9K ToR------ACI Spines

We have no VMM Domains. We use Physical Domains for EPG provisioning. I was thinking and maybe I've had a misconception, we had to use micro-EPGs if we wanted to separate end-hosts in this environment? Would there be another method to achieve intra-epg isolation for end hosts within the same BD and IP Subnet in this environment?

I'm under the impression that PVLAN Configurations would be required on ALL of the intermediary switches above?

Thank you for having taken the time with this for us today Sergiu.

Best Regards,

Terry

[Sergiu.Daniluk · Accepted solution]

Indeed, you should have PVLAN on all intermediary switches.

On the other hand, building on top of Chris' (@RedNectar's) idea, you can do something like this as an alternative to PVLANs:

+ one Bridge Domain (one subnet)

+ three EPGs (Web_EPG, App_EPG, DB_EPG) associated to the same BD

+ each EPG having static ports to different vlans (Web - vlan 10, App, - vlan 20, DB - vlan 30)

+ in ESXi, each tier of VMs will be configured with a dedicated port-group, with the respective Vlans associated to them (web-pg - vlan10, app-pg - vlan20, db-pg - vlan30)

+ on the intermediary switches or FIs, you only configure the vlans as normal vlans.

+ between EPGs, you have the contracts with the appropriate filters.

Sergiu

[zachartl]

Hey Sergiu,

This could work. Wow, okay. As you probably guessed, We're a one-to-one (BD - EPG) network-centric environment. I will get with our brethren on the Server Team to see if this is something they'd be interested in as an Option.

BIG THANK YOU to You, Chris and Robert. You Guys are Great!

Best,

Terry

[RedNectar · Accepted solution]

Hi @zachartl ,

Can I make a constructive suggestion?

Forget microsegmentation. Microsegmentation reminds me of a quote of Charles Dickens.

Annual income twenty pounds, annual expenditure nineteen nineteen and six , result happiness.
Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery

Just replace the first line with EPGs and the second with uSEG

To me it appears that you are trying to achieve something "utilize the IP Address attributes" that can be much more easily done WITHOUT using microsegmentated EPGs.

Here's my suggestion

1. Turn on Preferred Groups in the VRF
2. Create 3 new EPGs, put them into the preferred group
3. Create the new contracts as you were going to create anyway

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

[zachartl]

Hey Chris,

This is a very good idea and very helpful too. Needless to say, I don't have any practical experience with ACI microsegmentation, yet. As for PVLANs, In the past, I've set those up locally, right next to the SVI on the switch and have not had a requirement to trunk them to another device. We're still very much in an ACI Network Centric prose. In fact we utilize Preferred Groups (Single VRF) and then remove the EPG from the Preferred Groups if it requires isolation and then apply a Contract. Sometimes to an External EPG (L3-Out) if necessary. I'm starting to conclude that we may have to forgo microsegmentation and simply remove this EPG from the Preferred Groups and apply a single contract provisioning the L4 ports required. Please know, I'm open to all and any ideas though.

I want to Thank You, Sergiu and Robert for weighing in on this one. 

Best Regards,

Terry

[RedNectar]

Hi Terry,

"remove this EPG from the Preferred Groups and apply a single contract provisioning the L4 ports required." sounds like a good idea - but I wouldn't put it past @Robert Burns or @Sergiu.Daniluk to come up with a better idea!

[RedNectar]

Postscript

You have to remember WHY ACI has microsegmented EPGs.

First, Cisco had EPGs, which allowed users to control traffic based on EPGs rather than purely IP addressing (or uggh- MAC addressing)

Then VMware said - "Hey - we can apply policy between micro-segmented groups" 

So Cisco had a backroom chat and realised that if they allowed one layer of nested EPGs, they could do the same thing, even if it added a layer of complexity.

Next, thing you know, Cisco announces "We can do microsegmentation too! And better.  Here - look you can use microsegmentations to do ... "

(scratch, scratch, think, think) 

"Oh yeah - to do something really useful like ..."

(scratch, scratch, think, think - 'com'on guys, there MUST be something useful we can use this feature for?')

"Like isolating a server that been compromised."

'Quick team - can we put some fancy marketing like "reduce attack surface" and "regulatory compliance" around this feature and sell it?'

And there you have it. The story (possibly somewhat embellished with a LOT of poetic licence and old-man cynicism) of how microsegmentation came about in ACI.  I'm sure it varies slightly from the official Cisco line.

But seriously now

If planned properly, microsegmentation can be implemented to achieve granular control over servers, and its best feature is that you can potentially change attributes of an endpoint (typically a VM) and have it dynamically move to a different microsegmented EPG (just like you can move an normal VM into a different EPG by assigning it to a different Port Group).  But microsegmentation gives more attributes that you could possibly re-assign several servers in one hit, such as OS or custome attribute - assuming the custom attribute has already been set.

[zachartl]

Hi Chris,

A very colorful perspective, it put a smile on my face with my coffee this morning. 

If we had to, I wouldn't have a problem with the trunking of the PVLANs through the intermediate switches. But I don't have administrative access to the FIs and I've offered to help with those in the past but the receptions have been tepid. It's not me but where I set and where the Team that administers Servers set. Me Network - ooog! Don't get me wrong they're good well intentioned people and we have a pretty good working relationship. We've endeavored onto converged networking for years now but not so much on the HR side if you know what I mean. I will tell them about the PVLAN requirement if we choose this path but I can visualize the reticence on their faces already. We've begun NSX deployment and we intend to use ACI as the underlay and I Know, I've already asked.

Thanks again weighing in on this for us.

Have a Great Sunday and the rest of the week too.

Best,

Terry