Cisco ACI Route Leak between Multiple VRF

[sutha_entc]

Hi All, 

checking on the ACI route leaking between VRF. I have 3 VRF and I need to leak the route between all 3 VRF. as per the document we need to configure the subnet under the EPG and enable "shared between VRF" option for provider VRF. we need to enable the "shared between VRF" option under BD subnet in consumer VRF. Finally we need to have proper contract. in my case, all my VRF are provider and consumer because  I need to leak route in each and every VRF. So I applied the provider and consumer contract in all VRF but I did not configure the EPG subnet in any of the VRF. I enabled the "shared between VRF" option under BD subnet for all VRF. I can see the routes are leaking between VRF and have proper zoning rules created. I believe all the VRF are leaking routes without EPG subnet configuration is because all the VRF are consumer as well. but I just want to confirm this is valid design and won't create any problem in future. can someone help. here is simple diagram for your reference.

[M02@rt37 · Accepted solution]

Hello @sutha_entc 

I think about 2 things, contracts and zoning.

Contracts define the communication policies between EPGs. If you have applied contracts between your EPGs in different VRFs, it should facilitate route leaking. In your case, since all VRFs are both providers and consumers of contracts, it suggests a mutual sharing of routes between them.

Zoning rules play a crucial role in controlling traffic between different EPGs. Ensure that your zoning configuration aligns with the desired communication patterns between EPGs in different VRFs.

So, from my point of view it seems to be a valid design, yes.

[sutha_entc]

M02@rt37 thanks for your reply. yes sure I will take care of the desired zoning rules. 

[RedNectar · Accepted solution]

Hi @sutha_entc ,

Firstly, thanks for writing your question so clearly. It does make the job of answering much easier.

---

checking on the ACI route leaking between VRF. I have 3 VRF and I need to leak the route between all 3 VRF. as per the document we need to configure the subnet under the EPG and enable "shared between VRF" option for provider VRF.

Correct.

we need to enable the "shared between VRF" option under BD subnet in consumer VRF.

Actually, for the consumer, it doesn't matter if the "shared between VRF" option is configured under BD subnet IF it is already configured under the EPG (as in your case)

Finally we need to have proper contract. in my case, all my VRF are provider and consumer because  I need to leak route in each and every VRF. So I applied the provider and consumer contract in all VRF but I did not configure the EPG subnet in any of the VRF. I enabled the "shared between VRF" option under BD subnet for all VRF.

Indeed, since each each subnet is a consumer, the routes will be leaked.  But you are missing part of the whole picture (I'll discuss later)

I can see the routes are leaking between VRF and have proper zoning rules created. I believe all the VRF are leaking routes without EPG subnet configuration is because all the VRF are consumer as well.

As I said above - FAD (Functioning As Designed)

but I just want to confirm this is valid design and won't create any problem in future. can someone help. here is simple diagram for your reference.

---

Let's Do It

Step 1 - build it. My tenant is Chris18

OK. I built your scenario, but as yet I have not applied the contract. Forgive me, this is going to be long-winded because I'm writing as I explore!

I want you to observe the pcTag values of the EPGs BEFORE I do the contract connections.  In particular, note they are all > 16385

admin@apic1:~> icurl -ks https://localhost/api/node/mo/uni/tn-Chris18/ap-AP_ABC.json?query-target=children | jq ".imdata[].fvAEPg.attributes | {EPG: .name, pcTag: .pcTag}"
{
  "EPG": "EPG_A",
  "pcTag": "32771"
}
{
  "EPG": "EPG_B",
  "pcTag": "16386"
}
{
  "EPG": "EPG_C",
  "pcTag": "32770"
}

And the route tables. I have the endpoints for EPG_A (subnet 10.217.11.0/24) and EPG_C (10.217.12.0/24) on leaf 2201 and the endpoint for EPG_B (10.218.11.0/24) on leaf 2202.

apic1# fabric 2201-2202 show ip route vrf Chris18:VRF_A
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
<snip>

10.217.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:14:23, static, tag 4294967294
10.217.11.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.11.1, vlan20, [0/0], 00:14:23, local, local

----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------
No IP Route Table for VRF "Chris18:VRF_A"

apic1# fabric 2201-2202 show ip route vrf Chris18:VRF_B
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
No IP Route Table for VRF "Chris18:VRF_B"

----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------
<snip>

10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:14:13, static, tag 4294967294
10.217.12.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.12.1, vlan14, [0/0], 00:14:13, local, local

apic1# fabric 2201-2202 show ip route vrf Chris18:VRF_C
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
<snip>

10.218.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:14:30, static, tag 4294967294
10.218.11.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.218.11.1, vlan2, [0/0], 00:14:30, local, local

----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------
No IP Route Table for VRF "Chris18:VRF_C"

Step 2 - Add contract in ONE direction

Now let's see what happens when I add the contract (which allows all IP traffic) between EPG_A and EPG_B, with EPG_A providing the contract, and EPG_B consuming it.

admin@apic1:~> icurl -ks https://localhost/api/node/mo/uni/tn-Chris18/ap-AP_ABC.json?query-target=children | jq ".imdata[].fvAEPg.attributes | {EPG: .name, pcTag: .pcTag}"
{
  "EPG": "EPG_A",
  "pcTag": "5479"
}
{
  "EPG": "EPG_B",
  "pcTag": "16386"
}
{
  "EPG": "EPG_C",
  "pcTag": "32770"
}

Note that EPG_A, the provider EPG has a new pcTag - and it is < 16386. This makes it a globally unique pcTag.

But the routing table is not quite right - leaf 2202 (where EPG_B lives) can't see the route to EPG_A (10.217.11.0/24)

apic1# fabric 2202 show ip route vrf Chris18:VRF_B
----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------
<snip>

10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 02:01:38, static, tag 4294967294
10.217.12.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.12.1, vlan14, [0/0], 02:01:38, local, local

But leaf 2201 (where EPG_A lives) CAN see the route to EPG_B

apic1# fabric 2201 show ip route vrf Chris18:VRF_A
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
<snip>

10.217.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 02:13:58, static, tag 4294967294
10.217.11.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.11.1, vlan20, [0/0], 02:13:58, local, local
10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:12:24, static, tag 4294967294, rwVnid: vxlan-3080193

---

Step 3. The crux of your post - add the contract in the reverse directioin

So, to explore ...

"I just want to confirm this is valid design and won't create any problem in future"

... I'll add the same contract in the reverse direction, and see what changes.

admin@apic1:~> icurl -ks https://localhost/api/node/mo/uni/tn-Chris18/ap-AP_ABC.json?query-target=children | jq ".imdata[].fvAEPg.attributes | {EPG: .name, pcTag: .pcTag}"
{
  "EPG": "EPG_A",
  "pcTag": "5479"
}
{
  "EPG": "EPG_B",
  "pcTag": "27"
}
{
  "EPG": "EPG_C",
  "pcTag": "32770"
}

OK. Now EPG_B has a pcTag < 16386.  Time to check if EPG_B can see the route to EPG_A (10.217.11.0/24)

apic1# fabric 2202 show ip route vrf Chris18:VRF_B
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
<snip>

10.217.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:00:19, static, tag 4294967294, rwVnid: vxlan-2818049
10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 03:00:41, static, tag 4294967294
10.217.12.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.12.1, vlan14, [0/0], 03:00:41, local, local

Great. So it LOOKS like its working.  Time to test - first let's see if 10.217.11.10 can ping 10.217.12.10 (and ssh)

Well, that looks good! Let's try the reverse. Let's see if 10.217.12.10 can ping and ssh to10.217.11.10 

OK. Looks like you've made it work without following the rules!  The $64,000 question now, is

"Could it create a problem in the future?"

Bottom Line

No, it probably WON'T be a problem in the future, but you are relying on something non-standard, and if one of those CONSUMER EPGs stops being a PROVIDER as well as a consumer, it will loose routes!

A Better Approach - Step 1 repeat

Le's go back to Step 1, and rebuild the scenario with the IP subnets applied to the EPGs rather than the BDs (i.e. remove the IPs from the BDs).  The EPGs will get new pcTags

admin@apic1:~> icurl -ks https://localhost/api/node/mo/uni/tn-Chris18/ap-AP_ABC.json?query-target=children | jq ".imdata[].fvAEPg.attributes | {EPG: .name, pcTag: .pcTag}"
{
  "EPG": "EPG_A",
  "pcTag": "32772"
}
{
  "EPG": "EPG_B",
  "pcTag": "49153"
}
{
  "EPG": "EPG_C",
  "pcTag": "32770"
}

And the routes will be as before at Step 1 (I won't bother with VRF_C)

apic1# fabric 2201 show ip route vrf Chris18:VRF_A
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
<snip>

10.217.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:14:23, static, tag 4294967294
10.217.11.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.11.1, vlan20, [0/0], 00:14:23, local, local

apic1# fabric 2202 show ip route vrf Chris18:VRF_B
----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------
<snip>

10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:14:13, static, tag 4294967294
10.217.12.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.12.1, vlan14, [0/0], 00:14:13, local, local

Step 2 repeat

But at Step 2, when the contracts are applied in ONE direction, things are different

apic1# fabric 2201 show ip route vrf Chris18:VRF_A
----------------------------------------------------------------
 Node 2201 (Leaf2201)
----------------------------------------------------------------
<snip>

10.217.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 02:19:40, static, tag 4294967294
10.217.11.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.11.1, vlan20, [0/0], 02:19:40, local, local
10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:01:55, static, tag 4294967294, rwVnid: vxlan-3080193

apic1# fabric 2202 show ip route vrf Chris18:VRF_B
----------------------------------------------------------------
 Node 2202 (Leaf2202)
----------------------------------------------------------------
<snip>

10.217.11.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 00:00:19, static, tag 4294967294, rwVnid: vxlan-2818049
10.217.12.0/24, ubest/mbest: 1/0, attached, direct, pervasive
    *via 10.2.128.65%overlay-1, [1/0], 03:00:41, static, tag 4294967294
10.217.12.1/32, ubest/mbest: 1/0, attached, pervasive
    *via 10.217.12.1, vlan14, [0/0], 03:00:41, local, local

Observe how this is different to Step 2 above - achieved by putting the subnets on the EPG rather than the BDs.  BTW, you can have theme in BOTH paces if you want, but I only applied subnets to the EPGs in this second round!

---

The other thing to consider is that I used a STATELESS contract - my RouteLeak_Ct was to permit all IP traffic. If you used a contract that permitted only TCP, then at this stage with the contract applied only in one direction, only consumers would be able to initiate a connection to the providers.  But since you are applying in both directions, then it's a mute point.

---

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[sutha_entc]

@RedNectar thank you for the details explanation and I understand the different. but this is only works for 2 VRF. when we introduce the Third VRF, at least one of the VRF become both consumer and Provider. correct me if I am wrong.

for example. assuming I m configuring the subnet under EPG for provider.

VRF A (provider ) - VRF B (Consumer)  ---->  route leaks between VRF A and VRF B

VRF A (Provider) - VRF C (Consumer) -----> route leaks between VRF A and VRF C

now we need to leak the routes between VRF B and VRF C. lets say VRF B is provider

VRF B (provider ) - VRF C (consumer) -- routes leaks between VRF B & VRF C. 

now our VRF B both provider and consumer but  is it provider for VRF C and consumer for VRF A. but If we use the same contract for all the route leaking, technically it is both provider and consumer for both VRF A & C.

[RedNectar]

Hi @sutha_entc ,

I was getting pretty tired when I wrapped that post up last night - most of it was just documenting my exploration.  I guess the point I was making was that if the subnets are defined on the EPG rather than the BD, then the route leaking occurs before contracts are applied in both directions,

To be clear:

• Go ahead and provide and consume your contract on each EPG
• In your scenario, it won't matter if you define the subnets on the EPG or the BD. Both will work equally well
  • And, as you first observed, if
    1. the subnets are defined on the BD and marked as Shared between VRFs
    2. the contracts are applied on both directions,
  • then it is NOT necessary to define a subnet on the EPG

[sutha_entc]

@RedNectar thanks for your reply. I just wanted to confirm whatever I m doing is not creating problem in future. anyway I got your point now.