Endpoints in same epg (same tenant/ VRF) have issue in communication

[Daps]

My network topology is simple to understand.

ACI network configured as L2 OUT. Mapping of VLAN to BD is BD=EPG=VLAN.

Gateways of all the vlans are on firewall.

Servers are connected to cisco 3560 switches and switches connected to LEAF switches. 

I am facing the issue like some of the End points in same EPG not communicated to each other. Tenants and VRF are same. BD is common for both the EPGs, with same subnet.

Diagram is attached with the post. Kindly share your suggestions.

Checked: STP (legacy l2 domain connected to LEAF), VLAN Pruning, EPG deployment, No firewall in between. MAC learning of laptop and server on both the L2 switches.

[RedNectar]

Hi @Daps ,

Your picture doesn't match your explanation: The key point being that you have omitted the firewall.

You also have two possibly contradicting statements:

1. "ACI network configured as L2 OUT."
2. "Mapping of VLAN to BD is BD=EPG=VLAN"

L2Outs (or External Bridged Networks) don't have EPGs - although they do have a "Network" configuration which is often referred to as a "L2EPG".

My advice is to never use a L2Out when a regular EPG can do the job, and a regular EPG can do the job 100% of the time. So my advice is to never use L2Outs.

Now, the firewall placement COULD be the problem.

If the firewall is connected to the ACI leaf there is definately a potential problem, but it's going to take me awhile to explain it, so I won't go to that trouble until you can answer ther queston:

Is the firewall connected to the ACI leaf?

[Daps]

Today, what I did was , created 1 EPG and add both the server and laptop to that EPG. No firewall, no l2out. 1 server , 1 laptop, 2- L2 switches and 2 leafs.

MAC address of the laptop and server have been Learned to ACI leafs and both the L2 switches. However, not able to take remote desktop, no ping response.

[Vetri C]

Hello,

Could you confirm the BD which is associated with mentioned EPG group configured as L2 BD or L3 BD.

If it's L2 BD,l3 BD, what is option you chose " Private to VRF or Advertisely Externally or other one.

You mentioned Mac address is learning for both laptops. if then you can try to ping the device from L3 Device (who has subnet/default gateway for the subnet).

Also can you share login to leaf's and run the below command to check the connectivity.

Show endpoint ip / interface (interface ID)

[Daps]

Gateway is reachable from both the machines. BD /VRF is common for both the machines. 

[Daps]

Vetri,

Please find the show endpoints 

[RedNectar]

@Daps ,

I've been out of Internet access for a couple of days, but had prepared this on a plane trip a few days ago and forgot to post it:

Try this:

Tenants > YourTenant > Networking > Bridge Domains > YourBridgeDomain >| Policy | General  and check the Limit IP Learning to Subnet checkbox. You will get a warning when you check the box.

Acknowledge the warning and click Submit then Submit Changes

If this fixes it and you want to know why, please ask and I'll give you the explanation.

If it does NOT fix it, I am out of ideas!

[RedNectar]

Hi @Daps,

Sorry for two replies on the one day - it's just that I wrote the other one a few days ago

Looks like there are two TS scenarios being discussed here.  One where there is a firewall and one without a firewall. Makes it very hard to TS via messages.

So: In response to

Today, what I did was , created 1 EPG and add both the server and laptop to that EPG. No firewall, no l2out. 1 server , 1 laptop, 2- L2 switches and 2 leafs.

The way ACI handles this depends on your bridge domain settings:

1. If you have an IP address assigned to the DB (or EPG) that is in the same subnet, then ACI should try and glean the IP address of the second PC and they shoudl communicate (google ACI+Arp+gleaning - first hit should be a good one)
2. If you DO NOT have an IP address assigned to the DB (or EPG) that is in the same subnet, then you will need to enable ARP Flooding on the BD

As I said before, if the firwall is connected to ACI as well, life can get more difficult, but is generally solved by chceking the Limit IP Learning to Subnet checkbox

[chathurangaj@kbsl.lk]

Im Just saying,

Have you created contract between two EPGs properly to communicate ?

[Daps]

It's same epg and no contracts required

[Daps]

Hi, 

Let me clear all of you guys, I have configured new BD and  EPG  to bypass all the things, no subnet configuration, no firewall, no l2 out, no l3 out.  Now, it's simple MAC Learning from host to L2 switch to Leaf switches. Please find the BD and EPG configuration. Still, Both the end points are neither pinging nor take remote desktop. Server and laptop OS firewalls off. 

i.e.  "70-BD" = "70-EPG" = "VLAN ID - 70"

moreover, for clear my doubts, I have connected laptops on both the RACK switches and configured the port same as server uplink ports to switch. As L2 switch, both the server and Laptop communicating to their respective rack end points. When ACI comes into picture, neither pinging nor ssh or remote desktop. Apart from this, I have not created the policies contracts in Intra epg communication.   

[RedNectar]

@Daps ,

I have to say you have stumpted me. Everything you have seems OK. Although I notice that the VRF you have used is the common/default VRF. So I assume the BD and the EPGs you have configured are also in the common tenant.  If not, it might be worth trying creating a VRF in the tenant you are using and link the BD to that.

The other thing I'd try is allocating an IP address to your BD and then testing to see if you can use iping from each leaf to see if they can ping each endpoint.  Having said that, I'm not sure where to go next.

[Daps]

Hi  Chris,

tried but not work. Will you please share how can I capture traffic using ELAM? command please

[Alejandro AP .:|:.:|:.]

Hi @Daps ,

This is a really interesting case , when I face one of this issue I usually check the basics for all communications , is the ARP getting complete in both ways ? Meaning both endpoints know the Mac address of the other host ? As you said I dont think this is Firewall related unless the Firewall has the proxy ARP enable and the ARP on both EP are pointing to the FW , but otherwise communication should be straight forward between these 2 EPs.No contract require and not even for a L3 config on the BD.

If ARP is not complete between EPs I would focus on why BUM traffic is not working properly in ACI, if it is then I would run the ELAM as you mentioned.

The ELAM command have some variables depending on the Model of Leaf you have , if it is an EX-FX it would be something like this :(Using this as a reference since it is the most common deployed as far as my expertise)

1)vsh_lc

2)debug platform internal tah elam asic 0  >>> Here you are specifying the ASIC where you will run the capture

3)trigger reset  >>> as a best practice to clear out any previous trigger

4)trigger init in-select 6 out-select 1 >>> Here I usually run 6 for in and 1 for incoming traffic on the Front Panel Ports , no the Fabric ports looking only in the outer header (wont give much detail since we can spend  lot of time explaining this in deep for the different variables)

5)set outer ipv4 src_ip x.x.x.x dst_ip e.e.e.e <<< here you are specifying the packet you want to match I just did it base on IP.

6)start  <<< After you set all the variables you start the capture and start checking if it hits the match criteria with :

7)status << You should see something like this :

ELAM STATUS
===========
Asic 0 Slice 0 Status Armed
Asic 0 Slice 1 Status Armed

While it is in Armed , is means there is not a packet that has match the criteria you set, if it turns into this :

ELAM STATUS
===========
Asic 0 Slice 0 Status Triggered
Asic 0 Slice 1 Status Armed

It means we got a packet match and we have the info to be review.

To check the info you need to run the command "report detail" it will provide the hex values of variable and decision.

If you are at this point let me know so we can check the results of the ELAM .

A reference of checking this and good document is here as well :

https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/213346-ex-hardware-aci-packet-forwarding-deep.html

Alejandro Avila Picado .:|:.:|:.

[Daps]

thanks. Issue resolved