ACI l3out transit routing (eBGP) and contracts

Johannes Luther · ‎12-13-2021

Hi ACI professionals,

refer to the following figure:

So initially I had two L3outs in my GLOBAL VRF:

- Core (l3o Core)

- DMZ#1 (l3o DMZ#1)

In each L3O I have an external EPG with 0.0.0.0/0 for contract handling ("External Subnets for External EPG").

I have one contract between the Core L3o EEPG (provider) and the DMZ#1 L3o EEPG (consumer)

=> Everything works like a charm

So now I add the additional l3out in my GLOBAL VRF

- DMZ#2 (l3o DMZ#2)

To provide data plane transit between the other L3outs and the new one, I assumed I need additional contracts and assign them to the EEPG of DMZ#2. However everything works without adding a contract to the EEPG in the DMZ#2 L3o.

If I remove the existing contract between Core and DMZ#1 everything breaks down - nothing works at all.

I'm still clueless why transit traffic from and to DMZ#2 is working without a contract or any contract interfaces.

By the way: The external EPGs are not in the preferred group.

Any hints?

Sergiu.Daniluk · ‎12-13-2021

Hi @Johannes Luther

Having the ExtEPG with 0/0 prefix in multiple L3Outs is NOT supported and most likely this is causing this issue.

More precisely is because the 0/0 has a VRF-wide scope and not L3Out scope.

In your case, you most likely have DMZ1 and DMZ2 L3Outs on the same BL.

Basically there are two things which are performed once transit traffic arrives on the BL:

1. checks what is the pcTAG of the destination IP. You can check that manually as well using the command:

vsh -c "show system internal policy-mgr prefix"

You will see the VRF Name being mentioned, but no indication of L3Out. If your destination matches on 0/0 then from policy enforcement perspective a contract already exists

2. if policy enforcement is a pass, then check the routing table, where there is a route to L3Out DMZ2.

3. voilà

Cheers,

Sergiu

Johannes Luther · ‎12-14-2021

Hi @Sergiu.Daniluk

thank you for your reply

@Sergiu.Daniluk wrote:
Having the ExtEPG with 0/0 prefix in multiple L3Outs is NOT supported and most likely this is causing this issue.
More precisely is because the 0/0 has a VRF-wide scope and not L3Out scope.

I cannot find any documents, which states that multiple l3o with 0.0.0.0/0 as “External Subnets for the External EPG” are not supported.

Do you have more information?

However, I found the following hints:

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/guide-c07-743150.html

>> If there is a 0.0.0.0/0 with “External Subnets for the External EPG” scope somewhere in the same VRF, that L3Out EPG with 0.0.0.0/0 will be the fallback for all traffic in that VRF, from a contract perspective.

>> When the same subnet is configured with “External Subnets for the External EPG” scope in multiple L3Out EPGs in the same VRF, the configuration will be rejected. However, 0.0.0.0/0 is an exception. This does not mean 0.0.0.0/0 with “External Subnets for the External EPG” scope should be configured in multiple L3Out EPGs. It is strongly recommended NOT to do that to avoid traffic being allowed unexpectedly.

>> The reason 0.0.0.0/0 is an exception is because 0.0.0.0/0 with “External Subnets for the External EPG” does not use a pcTag for each L3Out EPG; instead, it always uses the reserved pcTag

Since I don't want to enforce any data plane traffic filtering over the l3outs, this should not be a problem.

Sergiu.Daniluk · ‎12-14-2021

Ok, I admit. I was too harsh in my previous post. It is supported to have 0.0.0.0/0 as "External Subnets for External EPG", but is strongly recommended to not configure it. And why is not recommended? Because there are so many limitations with multiple 0/0, and so so many situations which multiple 0/0 can break the expected behavior. and if you read between the lines, the paragraph from the whitepaper actually means - if you encounter problems, the recommended solution would be to remove the duplicate "0.0.0.0/0".

Based on my experience, I can only recommend this: do not configure multiple 0/0. If you really cannot summarize the external prefixes from an ExtEPG, simply use 0.0.0.0/1 and 128.0.0.0/1.

In the end, what you need to remember is that Ext Subnet for Ext EPG is used for policy enforcement, so the more specific, the better.

"""Since I don't want to enforce any data plane traffic filtering over the l3outs, this should not be a problem.""" - about this, since your VRF is enforced, even if it works now without a contract, I would suggest you still configure one, otherwise if you will move the DMZ2 L3out to a new Border Leaf where only DMZ2 is configured, the communication will not work anymore.

Stay safe,

Sergiu