Endpoint Learning

[Suprit Chinchodikar]

Hello,

Let's say I have learned a EP from the L3 out. So, the source mac/ip will be learned as local endpoint in the endpoint table of the leaf? Also, if it is learned as local endpoint that info is shared to the spines to update the COOP DB?

Thanks,
Suprit

[RedNectar · Accepted solution]

Hi @Suprit Chinchodikar ,

ACI doesn't learn IP EPs from L3Outs. L3Outs are based on traditional routing tables + ARP Cache.

So therefore, any endpoint (IP+MAC in the ARP cache) seen on a L3Out is seen only on the Leaf/VPC Pair that it is attached to, and only the MAC address is reported to the COOP database - and even then, it won't show in the show endpoint command for the leaf.

HOWEVER, the Cisco GUI does do a little trick when viewing Fabric > Inventory >> Topology >| Global-EndPoints  to make those ARP entries appear in the MAC Endpoints and IP End-Points tables - even though they are not in the COOP database.

Here's my lab

Note particularly the IP address of the remote router - which is the only directly connected endpoint on this L3 Out (as would typically be the case). Observe that it is 10.118.1.1. Also note that the interface on Leaf1201 to which the router connects is Eth1/10

Here is the output of a  show endpoint command on Leaf 1201 for this VRF

apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
<snip>
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
Tenant18:Production_VRF                                  10.118.0.201 L                         lo3
20                                        vlan-1181    a036.9f56.20fc L                     eth1/28
Tenant18:Production_VRF                   vlan-1181      10.118.11.10 L                     eth1/28
23                                        vlan-1183    0050.569b.5766 LV                        po5
Tenant18:Production_VRF                   vlan-1183     10.118.11.200 LV                        po5
22                                        vlan-1184    0050.569b.0283 LV                        po5
Tenant18:Production_VRF                   vlan-1184     10.118.12.200 LV                        po5
24/Tenant18:Production_VRF           vxlan-14843891    380e.4d48.8df5 L                     eth1/10

Note that the only address shown agains interface Eth1/10 is the MAC address of the router - NO IP address is shown.

Here is the output of a  show ip arp command on Leaf 1201 for this VRF - which shows the MAC to IP binding.

apic1# fabric 1201 show ip arp vrf Tenant18:Production_VRF
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
<snip>
IP ARP Table for context Tenant18:Production_VRF
Total number of entries: 1
Address         Age       MAC Address     Interface
10.118.1.1      00:00:05  380e.4d48.8df5  vlan24

But if I look at Fabric > Inventory >> Topology >| Global-EndPoints I see the 10.118.1.1 endpoint!

Which of course makes it LOOK like the IP endpoint IS in the global proxy table.

To prove otherwise is going to involve digging into the actual proxy table itself, which is not a common thing to do, and requires some system level commands on the a spine.

The commands to check the entire COOP database on a spine all begin with  show coop internal info. And the ones we want are show coop internal info ip-dband show coop internal info repo ep  

When I look at Spine1101, I see that the IP endpoint 10.118.1.1 does NOT appear in the COOP database!

apic1# fabric 1101 show coop internal info ip-db | egrep "IP address" | egrep 10.118

IP address : 10.118.11.1
IP address : 10.118.11.10
IP address : 10.118.11.200
IP address : 10.118.1.201
IP address : 10.118.12.1
IP address : 10.118.0.201
IP address : 10.118.12.200
IP address : 10.118.12.10

Note that 10.118.1.1 endpoint does NOT appear.

But, if we look at L2 EP info in the COOP database: (Remember the MAC address of 10.118.1.1 is 38:0E:4D:48:8D:F5)

apic1# fabric 1101 show coop internal info repo ep | egrep "EP mac"
EP mac : A0:36:9F:61:8E:EB
EP mac : A0:36:9F:56:20:FF
EP mac : 00:50:56:9B:57:66
EP mac : 38:0E:4D:48:8D:F5
EP mac : 34:ED:1B:8B:5A:2B
EP mac : 34:ED:1B:8B:5A:2B
EP mac : 00:50:56:9B:02:83
EP mac : 00:50:56:9B:1F:E6
EP mac : A0:36:9F:61:8E:E8
EP mac : 00:22:BD:F8:19:FF
EP mac : 00:50:56:9B:F0:43
EP mac : A0:36:9F:56:20:FC
EP mac : 00:50:56:9B:77:DB

we see the L2 info - which is consistent with the show endpoint command on Leaf 1201 issued earlier.

Summary

Only MAC address information learned from L3Outs is learned in the local endpoint table and reported to the COOP database.  IP information still appears in the ARP cache for the attached leaf/VPC pair, but IP information is not sent to COOP for external L3 Out endpoints.

Note that this only applies to DIRECTLY connected endpoints - endpoints such as the remote host in my diagram above (10.118.10.10) will NEVER appear in an endpoint table.

I hope this helps.

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.

---

[Suprit Chinchodikar]

@RedNectar,
Thanks for the brief explanation, this clear to me now.
Also, I was refereeing to the flags from the first command output so does V stand for VPC?

[RedNectar · Accepted solution]

Hi @Suprit Chinchodikar ,

---

Also, I was refereeing to the flags from the first command output so does V stand for VPC?

---

Correct.  Here is the expanded output.

apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
Legend:
 S - static           s - arp              L - local            O - peer-attached
 V - vpc-attached     a - local-aged       p - peer-aged        M - span
 B - bounce           H - vtep             R - peer-attached-rl D - bounce-to-proxy
 E - shared-service   m - svc-mgr
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
Tenant18:Production_VRF                                  10.118.0.201 L                         lo3
20                                        vlan-1181    a036.9f56.20fc L                     eth1/28
Tenant18:Production_VRF                   vlan-1181      10.118.11.10 L                     eth1/28
23                                        vlan-1183    0050.569b.5766 LaV                       po5
Tenant18:Production_VRF                   vlan-1183     10.118.11.200 LV                        po5
22                                        vlan-1184    0050.569b.0283 LV                        po5
Tenant18:Production_VRF                   vlan-1184     10.118.12.200 LV                        po5
24/Tenant18:Production_VRF           vxlan-14843891    380e.4d48.8df5 L                     eth1/10


+------------------------------------------------------------------------------+
                             Endpoint Summary
+------------------------------------------------------------------------------+
Total number of Local Endpoints     : 5
Total number of Remote Endpoints    : 0
Total number of Peer Endpoints      : 0
Total number of Peer Rl Endpoints   : 0
Total number of vPC Endpoints       : 2
Total number of non-vPC Endpoints   : 3
Total number of MACs                : 4
Total number of VTEPs               : 0
Total number of Local IPs           : 4
Total number of Remote IPs          : 0
Total number All EPs                : 5

One curious thing about the Flags in the output of this command is the effect that a VPC pair has on the endpoints with single attachments to the remote leaf that are configured for a different VLAN.

For instance, in my diagram above (here it is again, with the host in question highlighted) where ALL endpoints have communicated with each other...

... I would EXPECT to see (in the output of the fabric 1201 show endpoint vrf Tenant18:Production_VRF command) the endpoint 10.118.12.10 to appear. And to appear with no flags (indicating a Remote endpoint - i.e endpoint attached to anther leaf)

But it doesn't!

The explanation for this has to do with the fact that VPC leaf pairs actually share ALL endpoint information, not just the VPC endpoints.  If I kill the VPC (.i.e. Fabric > Access Policies >> Policies > Switch > Virtual Port Channel Default and delete the Explicit VPC Protection Group), the output is quite different:

apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
Legend:
 S - static           s - arp              L - local            O - peer-attached
 V - vpc-attached     a - local-aged       p - peer-aged        M - span
 B - bounce           H - vtep             R - peer-attached-rl D - bounce-to-proxy
 E - shared-service   m - svc-mgr
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
Tenant18:Production_VRF                                  10.118.0.201 L                         lo3
Tenant18:Production_VRF                                 10.118.11.200                       tunnel3
Tenant18:Production_VRF                                  10.118.12.10                       tunnel3
Tenant18:Production_VRF                                 10.118.12.200 B                     tunnel3
14/Tenant18:Production_VRF           vxlan-16351138    0050.569b.5766                       tunnel3
15                                        vlan-1181    a036.9f56.20fc L                     eth1/28
Tenant18:Production_VRF                   vlan-1181      10.118.11.10 L                     eth1/28
16/Tenant18:Production_VRF           vxlan-15794150    0050.569b.0283 B                     tunnel3
19/Tenant18:Production_VRF           vxlan-15695749    380e.4d48.8df5 L                     eth1/10

Note that 10.118.12.10 now shows up as REMOTE endpoint (annoyingly, there is no REMOTE flag - just the absence of any other flag indicated REMOTE)

And, since I broke the VPC, one of the VPC attached endpoints show up with a Bounce entry (as expected). But that's a whole discussion for another day.

---

[Added 2024.04.19 - Curious variation]

If I leave the VPC intact, and simply change the VLAN encapsulation for the single attached host to match the VLAN used on the VPC (1183 rather than 1181), I get a different result - I see 10.118.12.10 as a peer-attached host as indicated by the O flag - both for Layer 2 and Layer 3

apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF
----------------------------------------------------------------
 Node 1201 (Leaf1201)
----------------------------------------------------------------
Legend:
 S - static           s - arp              L - local            O - peer-attached
 V - vpc-attached     a - local-aged       p - peer-aged        M - span
 B - bounce           H - vtep             R - peer-attached-rl D - bounce-to-proxy
 E - shared-service   m - svc-mgr
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
Tenant18:Production_VRF                                  10.118.0.201 L                        lo3
20                                        vlan-1184    a036.9f56.20ff O                     tunnel3
Tenant18:Production_VRF                   vlan-1184      10.118.12.10 O                     tunnel3
20                                        vlan-1184    0050.569b.886a LV                       po12
Tenant18:Production_VRF                   vlan-1184     10.118.12.200 LV                       po12
23                                        vlan-1183    a036.9f56.20fc L                     eth1/28
Tenant18:Production_VRF                   vlan-1183      10.118.11.10 L                     eth1/28
23                                        vlan-1183    0050.569b.2314 LV                       po12
Tenant18:Production_VRF                   vlan-1183     10.118.11.200 LV                       po12
24/Tenant18:Production_VRF           vxlan-14843891    380e.4d48.8df5 L                     eth1/10