Communication among Same IP subnets in different EPGs in one BD

Thushan Pramod · ‎03-15-2017

Hi All,

Since we can configure same IP subnet in different EPGs in same BD. What is the path if the end points of those EPGs communicate where the default Gw is defined in external firewall?

dpita · ‎03-20-2017

If the gateway is outside, there is no need for a subnet under the BD, much less under the EPG

from a forwarding perspective, same BD with two different EPGs using the same subnet should be ok so long as they are different VLANs to the ACI side. at this point, traffic from subnet-A EPG-A will need pass through a contract to subnet-A EPG-B. if you need to communicate to the gw on an external device, that gets complicated.

would probably be best to use three VLANs, one for the external bridge network, the real vlan your FW is using, a VLAN for EPG-A and a VLAN for EPG-B and contracts between all of them. of course, all of them tied to the same BD

hope that helps

Thushan Pramod · ‎03-21-2017

Hi Dpita,

Thanks for the reply. Do we really need to worry about the vlan concept in ACI? for the forwarding does ACI check and worry about the vlan configuartion?

Here you are proposing three vlans as below right. If iam wrong pls comment.

Vlan10 - EPG-A (IP Subnet 1)

Vlan20 - EPG-B (IP Subnet 1)

here does external bridge network means external EPG which connects FW?

Maurlai · ‎07-25-2018

I have same issue but without external bridge.

I have this scenario:

Vlan10 - EPG-A (IP Subnet 1)

Vlan20 - EPG-B (IP Subnet 1)

Issue is IP 10.0.0.2/24 in EPG-A doesn't ping IP 10.0.0.3/24 in the EPG-B.

EPG-A and EPG-B are linked with a permitted contract.

If I change Network ID on one of IP, all works!

Anyone can explain that behavior?

Both EPGs are in the same Bridge Domain and it have only one subnet.

Chris010 · ‎07-27-2018

Hi Maurlai,

Why do you need to declare subnet in yours EPGs ? Do you want to share your service with other tenant, VRF ?

IP Subnet EPG specify which part of the subnet you wish to advertise. (Shared between VRF).

Maurlai · ‎08-09-2018

I don't declare Subnet in my EPG.

Subnet is declared only in the Bridge Domain.

I want to understand why 2 different EPG (containing 2 servers in same subnet IP) linked with a contract, they don't reach together.

Example:

Vlan10 - EPG-A (IP 10.0.0.2/28)

Vlan20 - EPG-B (IP 10.0.0.3/28)

EPG-A with contract to EPG-B and reverse.

micgarc2 · ‎08-09-2018

What TEP address space are you using? Typically that subnet range is used for the TEP pool. If you didn't change the default 10.0.0.0/16. If you do show controller :

a-apic1# show controller
Fabric Name : calo-a
Operational Size : 3
Cluster Size : 3
Time Difference : 319
Fabric Security Mode : permissive

ID Pod Address In-Band IPv4 In-Band IPv6 OOB IPv4 OOB IPv6
Version Flags Serial Number Health
---- ---- --------------- --------------- ------------------------- --------------- ---------------------------
--- ------------------ ----- ---------------- ------------------
1* 1 10.0.0.1 14.2.104.228 fc00::1 10.122.141.98 fe80::5a97:bdff:fe5:dd5a
3.2(2l) crva- FCH1929V153 fully-fit
2 1 10.0.0.2 14.2.104.229 fc00::1 10.122.141.99 fe80::a2e0:afff:fe33:945a
3.2(2l) crva- FCH2045V1X2 fully-fit
3 2 10.0.0.3 14.2.104.230 fc00::1 10.122.141.100 fe80::fac2:88ff:fe1b:bf88
3.2(2l) crva- FCH1824V2VR fully-fit
4~ 10.0.0.4

you can see that my APICs are using 10.0.0.1,.2,.3,.4. You may run into issues if you try to use those IPs in your fabric as they are suppose to be used for the infrastructure.

Maurlai · ‎08-09-2018

It was a subnet example only.

The concept is that I can't contract two IP virtual machines in this way:

Vlan10 - EPG-A (IP 1.2.3.3/28)

Vlan20 - EPG-B (IP 1.2.3.4/28)

Contract is ok.

If you can, do the same config (VMM Domain) in your lab.

micgarc2 · ‎08-10-2018

Is this a typo ?

Vlan20 - EPG-B (IP 10.2.3.4/28)

From previous posts you said these were all in the same BD/subnet.

If VLAN20 VM was suppose to be 1.2.3.4 and not 10.2.3.4 that is a very straight forward setup. I would verify on your VMs if ARP is getting resolved to the other EP. The GW is not involved here since both hosts are in the same subnet. If that all looks good check rules on the leaf to make sure traffic is not being dropped.

leaf# show logging ip access-list internal packet-log deny | grep 1.2.3.3 | grep 1.2.3.4

Maurlai · ‎08-10-2018

It was a mistake. Post corrected!

Maurlai · ‎08-10-2018

This is the output:

Leaf-203-Fiber-Siusi# show logging ip access-list internal packet-log deny | grep 10.1.91.101| grep 10.1.91.150
[ Fri Aug 10 14:05:58 2018 18989 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98
[ Fri Aug 10 14:05:57 2018 18919 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98
[ Fri Aug 10 14:05:56 2018 18897 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98
[ Fri Aug 10 14:05:55 2018 18853 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98
[ Fri Aug 10 14:05:54 2018 18880 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98
[ Fri Aug 10 14:05:53 2018 18834 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98
[ Fri Aug 10 14:05:52 2018 18814 usecs]: CName: MB_Tenant:Multisoc(VXLAN: 2719744), VlanType: FD_VLAN, Vlan-Id: 10, SMac: 0x000c29597f16, DMac:0x0050568169b3, SIP: 10.1.91.150, DIP: 10.1.91.101, SPort: 0, DPort: 0, Src Intf: port-channel5, Proto: 1, PktLen: 98

micgarc2 · ‎08-10-2018

Looks like its getting dropped. Are these in the same BD?

I would verify one EPG is the consumer (initiator of traffic) and one is the provider (receiver of traffic) as well as the filter allowing the type of traffic you are testing.

micgarc2 · ‎08-10-2018

Vlan10 - EPG-A (IP 1.2.3.3/28)

Vlan20 - EPG-B (IP 10.2.3.4/28)

If those are actually the IPs you are using and you still want these EPGs the same BD, you can define two SVIs under the bridge domain, one for the 1.2.3.0/28 subnet and one for 10.2.3.0/28 subnet

RedNectar · ‎08-10-2018

Wow - a lot of confusion here.

It seems we have two mixed threads, or a hijacked thread

Thread #1 as asked by Thursand

@Thushan Pramod wrote:

Hi All,

Since we can configure same IP subnet in different EPGs in same BD. What is the path if the end points of those EPGs communicate where the default Gw is defined in external firewall?

Thread #2 as asked by Maurlai (or what I believe was meant to be asked)

I have two hosts on the same subnet, same BD but in different EPGs with a permit all contract between them. Why can't I get them to communicate?

Now let's deal with Thread #1

Daniel's (dpita) first answer was spot on - have three EPGs, one of them being for the external FW. Thursan, the fact that he mentioned VLANs seems to have confused you. You just need three EPGs. Oh of course each EPG will need to be allocated a VLAN from you VLAN pool that is linked to the domain (Physical, L2External or VMM) that links to the AAEP that contains the ports where your hosts are, so that's why he referred to VLANs. You are correct in that the VLAN concept is different in ACI. VLAN tags are used to identify which EPG a packet was sourced from - so yes ACI really DOES check the VLAN tag.
However, Daniel forgot to mention that in this scenario, you will probably want to enable ARP flooding for the Bridge Domain, or assign an IP address to the BD in the same subnet to enable ARP Gleaning.

And now let's deal with Thread #2

Maurlai, I think your problem can also be solved by either enabling ARP flooding on the BD or assigning an IP address to the BD to enable ARP Gleaning. However, you could also make the EPGs members of the Preferred Group is you want to allow all traffic between the EPGs.
Another problem could be your contract. You say that "EPG-A and EPG-B are linked with a permitted contract" - but you may not have implemented that contract properly - it worries me that you see pings denied between the end points as shown in your logging output.
- To test if it is your contract, make one EPG consume the default contract in the common Tenant and the other EPG consume it. If they STILL can't ping each other, and you HAVE enabled either ARP flooding or ARP gleaning, then get back to us!

And finally, just a note about the IP addresses in the TEP space. The IP addresses used in the TEP space has nothing to do with either of these - don't let that thought distract you.

I am in the process of writing a blog post explaining how ARP Gleaning works, so if you want to know more, look out for it.

I hope this helps

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

RedNectar aka Chris Welsh.
Forum Tips: 1. Paste images inline - don't attach. 2. Always mark helpful and correct answers, it helps others find what they need.

RedNectar · ‎08-10-2018