Accepted Solutions
Hi @interfacedy ,
Here's a picture:
Because some of the Endpoints have duplicated IP addresses, I'll refer to them As EP1, EP2 etc
Firstly, note EP1 and EP2. They are mapped to EPG1 (which is largely irrelevant in this case) which is linked to Bridge Domain BD1
- Note BD1 is not linked to ANY VRF - it is L2 Only, so the IP addresses of EP1 and EP2 are totally irrelevant to ACI. ACI will not even register those IP addresses. As it turns out, EP1 and EP2 are on the same subnet, so will be able to communicate. If they were on different subnets, they would need a router to communicate, just like normal L2 networks
Next, look at EP3 and EP4. They are both mapped to EPG2 which is linked to Bridge Domain BD2 which has been assigned two IP addresses which serve as the default gateway IP addresses for EP3 and EP4 respectively.
- Since EP3 and EP4 are both mapped to EPG2 then they are able to communicate freely in an ACI environment - without any contract.
- Since Bridge Domain BD2 is linked to VRF VRF1, the routes for 20.20.20.0/24 and 22.22.22.0/24 will be exist on any leaf switch that either EP3 or EP4 is attached to, within VRF VRF1.
Now take a look at look at EP5 and EP6. They are each mapped to different EPGs, and even though both EPGs (EPG3 and EPG4) are linked to Bridge Domain BD3 and both endpoints share the same default gateway, they will NOT be able to communicate in ACI without a contract.
- This is a key feature of ACI - achieving this kind of control over two endpoints in the same subnet is impossible to achieve in a standard L3 routed network.
- Since Bridge Domain BD3 is linked to VRF VRF1, the route for 10.10.10.0/24 will be exist on any leaf switch that either EP5 or EP6 is attached to, within VRF VRF1.
- Note that EP5 and EP6 have the same IP addresses as EP1 and EP2. This is of no consequence in ACI because BD1 is L2 only.
The last two endpoints, EP7 and EP8 They are mapped to the same EPG - EPG5, which in turn is linked to Bridge Domain BD4 but BD4 has NO IP addresses. Instead, the IP addresses that serve as the default gateway IPs have been assigned to EPG5 instead.
- Since EP7 and EP8 are both mapped to EPG5 then they are able to communicate freely in an ACI environment - without any contract.
- Since Bridge Domain BD4 is linked to VRF VRF2, the routes for 20.20.20.0/24 and 30.30.30.0/24 will be exist on any leaf switch that either EP7 or EP8 is attached to, within VRF VRF2.
- Note that EP7 has the same IP addresses as EP3 and the IP address 20.20.20.1/24 has been assigned to both BD2 and EPG5. This is of no consequence in ACI because BD2 and BD4 are linked to different VRFs
- For the record, this case would be exactly the same is the gateway IP addresses had been assigned as:
- 20.20.20.1/24 and 30.30.30.1/24 assigned to BD4
- 20.20.20.1/24 assigned to BD4 and 30.30.30.1/24 assigned to EPG5
- 20.20.20.1/24 assigned to EPG5 and 30.30.30.1/24 assigned to BD4
Finally, look at the Application Profiles.
Application Profiles serve virtually NO function in ACI except:
- They provide separation for EPG names - I COULD have named EPG5 as EPG1 or EPG2 or EPG3 or EPG4 if I have have wanted to, because EPG5 is in a different Application Profile
- They can be useful for restricting the scope of a contract.
So to wrap up your questions:
Hi In ACI, Bridge domain(BD) must be linked to VRF
No. A BD does NOT have to be linked to a VRF in the SPECIAL case of it being a L2 ONLY Bridge Domain
and must have at least one subnet with it.
Usually true, but again, not true in the SPECIAL case of it being a L2 ONLY Bridge Domain, AND not true if a subnet has been assigned to an EPG that is linked to that BD - although in this case the BD essentially "inherits" the subnet form the EPG anyway.
so can we say in ACI, Bridge domain(BD) must be linked to VRF,
...except in the SPECIAL case of it being a L2 ONLY Bridge Domain
and BD could have more that two subnet with it?
Correct. A BD can have many subnets
Hi @interfacedy ,
Again - let's bring the picture in
In your diagram above, the gateways for PE2 and PE3 are 20.20.20.1 and 22.22.22.1 respectively. so when PE2 ping PE3, do you think the traffic will go through VRF1? If not, how the both see each other?
OK - Firstly, let me rewrite your question the way I THINK you meant it
the gateways for EP3 and EP4 are 20.20.20.1 and 22.22.22.1 respectively. so when EP3 ping EP4, do you think the traffic will go through VRF1? If not, how the both see each other?
And the answer is YES - the trick with ACI is to understand that those IP addresses (20.20.20.1 and 22.22.22.1) will exist on each switch within VRF1 of every leaf that has an endpoint attached to EPG2.
So when EP3 pings EP4, it will send an IP ICMP packet to the MAC address of 20.20.20.1 (i.e its default gw). The leaf that EP3 is attached to will ROUTE the packet to the 22.22.22.0/24 subnet, then that same leaf will look to see if it knows the MAC address of EP4 - if so it will:
- If EP4 is on the SAME leaf as EP3...
- send the packet/frame to the MAC of EP4 via the port that EP4 is attached
- IF EP4 is on ANOTHER LEAF
- encapsulate the IP packet in VXLAN encap and send it to that other leaf's VTEP address
- when the other leaf gets the VXLAN encapsulated packet, it will add EP4's MAC address and send the frame to EP4 via the port that EP4 is attached
Hi @interfacedy ,
Here's a picture:
Because some of the Endpoints have duplicated IP addresses, I'll refer to them As EP1, EP2 etc
Firstly, note EP1 and EP2. They are mapped to EPG1 (which is largely irrelevant in this case) which is linked to Bridge Domain BD1
- Note BD1 is not linked to ANY VRF - it is L2 Only, so the IP addresses of EP1 and EP2 are totally irrelevant to ACI. ACI will not even register those IP addresses. As it turns out, EP1 and EP2 are on the same subnet, so will be able to communicate. If they were on different subnets, they would need a router to communicate, just like normal L2 networks
Next, look at EP3 and EP4. They are both mapped to EPG2 which is linked to Bridge Domain BD2 which has been assigned two IP addresses which serve as the default gateway IP addresses for EP3 and EP4 respectively.
- Since EP3 and EP4 are both mapped to EPG2 then they are able to communicate freely in an ACI environment - without any contract.
- Since Bridge Domain BD2 is linked to VRF VRF1, the routes for 20.20.20.0/24 and 22.22.22.0/24 will be exist on any leaf switch that either EP3 or EP4 is attached to, within VRF VRF1.
Now take a look at look at EP5 and EP6. They are each mapped to different EPGs, and even though both EPGs (EPG3 and EPG4) are linked to Bridge Domain BD3 and both endpoints share the same default gateway, they will NOT be able to communicate in ACI without a contract.
- This is a key feature of ACI - achieving this kind of control over two endpoints in the same subnet is impossible to achieve in a standard L3 routed network.
- Since Bridge Domain BD3 is linked to VRF VRF1, the route for 10.10.10.0/24 will be exist on any leaf switch that either EP5 or EP6 is attached to, within VRF VRF1.
- Note that EP5 and EP6 have the same IP addresses as EP1 and EP2. This is of no consequence in ACI because BD1 is L2 only.
The last two endpoints, EP7 and EP8 They are mapped to the same EPG - EPG5, which in turn is linked to Bridge Domain BD4 but BD4 has NO IP addresses. Instead, the IP addresses that serve as the default gateway IPs have been assigned to EPG5 instead.
- Since EP7 and EP8 are both mapped to EPG5 then they are able to communicate freely in an ACI environment - without any contract.
- Since Bridge Domain BD4 is linked to VRF VRF2, the routes for 20.20.20.0/24 and 30.30.30.0/24 will be exist on any leaf switch that either EP7 or EP8 is attached to, within VRF VRF2.
- Note that EP7 has the same IP addresses as EP3 and the IP address 20.20.20.1/24 has been assigned to both BD2 and EPG5. This is of no consequence in ACI because BD2 and BD4 are linked to different VRFs
- For the record, this case would be exactly the same is the gateway IP addresses had been assigned as:
- 20.20.20.1/24 and 30.30.30.1/24 assigned to BD4
- 20.20.20.1/24 assigned to BD4 and 30.30.30.1/24 assigned to EPG5
- 20.20.20.1/24 assigned to EPG5 and 30.30.30.1/24 assigned to BD4
Finally, look at the Application Profiles.
Application Profiles serve virtually NO function in ACI except:
- They provide separation for EPG names - I COULD have named EPG5 as EPG1 or EPG2 or EPG3 or EPG4 if I have have wanted to, because EPG5 is in a different Application Profile
- They can be useful for restricting the scope of a contract.
So to wrap up your questions:
Hi In ACI, Bridge domain(BD) must be linked to VRF
No. A BD does NOT have to be linked to a VRF in the SPECIAL case of it being a L2 ONLY Bridge Domain
and must have at least one subnet with it.
Usually true, but again, not true in the SPECIAL case of it being a L2 ONLY Bridge Domain, AND not true if a subnet has been assigned to an EPG that is linked to that BD - although in this case the BD essentially "inherits" the subnet form the EPG anyway.
so can we say in ACI, Bridge domain(BD) must be linked to VRF,
...except in the SPECIAL case of it being a L2 ONLY Bridge Domain
and BD could have more that two subnet with it?
Correct. A BD can have many subnets
Hi @interfacedy ,
Let me repeat my diagram for ease of reference:
Consider the following config on a Catalyst switch (I hope you are familiar with Cisco style config)
interface vlan100
ip address 20.20.20.1 255.255.255.0
ip address 22.22.22.1 255.255.255.0 secondary
Here a SVI (Switched Virtual Interface) has been allocated two IP addresses
In ACI, Bridge domains are allocated IP addresses rather than the switch - BUT the IP addresses ARE PUSHED to the switches when (and ONLY when) needed.
So in ACI, the running config for BD2 in my diagram above would look like:
interface bridge-domain BD2
ip address 20.20.20.1/24 secondary
ip address 22.22.22.1/24 secondary
OR (if the [x] Make Primary option was chosen for the 20.20.20.1/24 address)
interface bridge-domain BD2
ip address 20.20.20.1/24
ip address 22.22.22.1/24 secondary
Remember, these addresses are configured on the APIC, and pushed TO EVERY LEAF that has an endpoint in EPG2 (from my diagram) - so multiple leaves may end up with the SAME IP addresses - but this is not a problem in ACI, it is called a Common Pervasive Gateway Address.
Does this answer your question?
Hi @interfacedy ,
Again - let's bring the picture in
In your diagram above, the gateways for PE2 and PE3 are 20.20.20.1 and 22.22.22.1 respectively. so when PE2 ping PE3, do you think the traffic will go through VRF1? If not, how the both see each other?
OK - Firstly, let me rewrite your question the way I THINK you meant it
the gateways for EP3 and EP4 are 20.20.20.1 and 22.22.22.1 respectively. so when EP3 ping EP4, do you think the traffic will go through VRF1? If not, how the both see each other?
And the answer is YES - the trick with ACI is to understand that those IP addresses (20.20.20.1 and 22.22.22.1) will exist on each switch within VRF1 of every leaf that has an endpoint attached to EPG2.
So when EP3 pings EP4, it will send an IP ICMP packet to the MAC address of 20.20.20.1 (i.e its default gw). The leaf that EP3 is attached to will ROUTE the packet to the 22.22.22.0/24 subnet, then that same leaf will look to see if it knows the MAC address of EP4 - if so it will:
- If EP4 is on the SAME leaf as EP3...
- send the packet/frame to the MAC of EP4 via the port that EP4 is attached
- IF EP4 is on ANOTHER LEAF
- encapsulate the IP packet in VXLAN encap and send it to that other leaf's VTEP address
- when the other leaf gets the VXLAN encapsulated packet, it will add EP4's MAC address and send the frame to EP4 via the port that EP4 is attached
Hi @interfacedy ,
during the process, what is function of VRF1?
I used two VRFs - VRF1 and VRF2 just to show that it is possible to use more than one VRF, just like a regular (modern) router. But just like a router, endpoints connected to one VRF can't communicate with endpoints in another VRF unless you set up route leaking between the VRFs - in my example this would not work anyway because (to show the isolation of the VRFs) I have used subnet 20.20.20.0/24 in BOTH VRFs.
So the function of VRF1 is to allow routing between the 20.20.20.0/24 and 22.22.22.0/24 subnets for EP3 and EP4
I guess its function is related with routing to outside.
No - routing to outside (in ACI) requires a L3Out - that's a whole new topic
but each VRF has its own ip address,
Not really - there are NO IP addresses assigned to VRFs. VRFs are logical functions that allow the routing of packets between subnets
so what ip address the VRF1 could be?
As I said, IP addresses are NOT assigned to VRFs. ROUTES are found in VRFs, and VRFs live on leaves. The APIC pushes those routes to a leaf whenever that leaf is assigned an endpoint for an EPG that is linked to the BD that is linked to the VRF
So if so EP3 was attached to Leaf2201, I'd expect the following output from a show ip route command:
apic1# fabric 2201 show ip route vrf Tenant1:VRF1
IP Route Table for VRF "Tenant1:VRF1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF
20.20.20.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.2.24.64%overlay-1, [1/0], 3d10h, static, tag 4294967294
20.20.20.1/32, ubest/mbest: 1/0, attached, pervasive
*via 20.20.20.1, vlan15, [0/0], 3d10h, local, local
22.22.22.0/24, ubest/mbest: 1/0, attached, direct, pervasive
*via 10.2.24.64%overlay-1, [1/0], 3d10h, static
22.22.22.1/32, ubest/mbest: 1/0, attached, pervasive
*via 22.22.22.1, vlan13, [0/0], 3d10h, local, local
or what subnet address the VRF1 could be?
See above
The reason that I ask the question is because there are two BD under VRF1. The two BD have its own ip address
There is only ONE BD under VRF1, but it does have two IP addresses
