03-12-2024 09:51 PM
Hello,
Let's say I have learned a EP from the L3 out. So, the source mac/ip will be learned as local endpoint in the endpoint table of the leaf? Also, if it is learned as local endpoint that info is shared to the spines to update the COOP DB?
Thanks,
Suprit
Solved! Go to Solution.
03-13-2024 05:16 PM - edited 04-18-2024 08:55 PM
Hi @Suprit Chinchodikar ,
ACI doesn't learn IP EPs from L3Outs. L3Outs are based on traditional routing tables + ARP Cache.
So therefore, any endpoint (IP+MAC in the ARP cache) seen on a L3Out is seen only on the Leaf/VPC Pair that it is attached to, and only the MAC address is reported to the COOP database - and even then, it won't show in the show endpoint
command for the leaf.
HOWEVER, the Cisco GUI does do a little trick when viewing Fabric > Inventory >> Topology >| Global-EndPoints to make those ARP entries appear in the MAC Endpoints and IP End-Points tables - even though they are not in the COOP database.
Here's my lab
Note particularly the IP address of the remote router - which is the only directly connected endpoint on this L3 Out (as would typically be the case). Observe that it is 10.118.1.1
. Also note that the interface on Leaf1201 to which the router connects is Eth1/10
Here is the output of a show endpoint
command on Leaf 1201 for this VRF
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- <snip> +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 20 vlan-1181 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1181 10.118.11.10 L eth1/28 23 vlan-1183 0050.569b.5766 LV po5 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po5 22 vlan-1184 0050.569b.0283 LV po5 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po5 24/Tenant18:Production_VRF vxlan-14843891 380e.4d48.8df5 L eth1/10
Note that the only address shown agains interface Eth1/10
is the MAC address of the router - NO IP address is shown.
Here is the output of a show ip arp
command on Leaf 1201 for this VRF - which shows the MAC to IP binding.
apic1# fabric 1201 show ip arp vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- <snip> IP ARP Table for context Tenant18:Production_VRF Total number of entries: 1 Address Age MAC Address Interface 10.118.1.1 00:00:05 380e.4d48.8df5 vlan24
But if I look at Fabric > Inventory >> Topology >| Global-EndPoints I see the 10.118.1.1
endpoint!
Which of course makes it LOOK like the IP endpoint IS in the global proxy table.
To prove otherwise is going to involve digging into the actual proxy table itself, which is not a common thing to do, and requires some system level commands on the a spine.
The commands to check the entire COOP database on a spine all begin with show coop internal info
. And the ones we want are show coop internal info ip-db
and show coop internal info repo ep
When I look at Spine1101, I see that the IP endpoint 10.118.1.1
does NOT appear in the COOP database!
apic1# fabric 1101 show coop internal info ip-db | egrep "IP address" | egrep 10.118 IP address : 10.118.11.1 IP address : 10.118.11.10 IP address : 10.118.11.200 IP address : 10.118.1.201 IP address : 10.118.12.1 IP address : 10.118.0.201 IP address : 10.118.12.200 IP address : 10.118.12.10
Note that 10.118.1.1
endpoint does NOT appear.
But, if we look at L2 EP info in the COOP database: (Remember the MAC address of 10.118.1.1
is 38:0E:4D:48:8D:F5
)
apic1# fabric 1101 show coop internal info repo ep | egrep "EP mac"
EP mac : A0:36:9F:61:8E:EB
EP mac : A0:36:9F:56:20:FF
EP mac : 00:50:56:9B:57:66
EP mac : 38:0E:4D:48:8D:F5
EP mac : 34:ED:1B:8B:5A:2B
EP mac : 34:ED:1B:8B:5A:2B
EP mac : 00:50:56:9B:02:83
EP mac : 00:50:56:9B:1F:E6
EP mac : A0:36:9F:61:8E:E8
EP mac : 00:22:BD:F8:19:FF
EP mac : 00:50:56:9B:F0:43
EP mac : A0:36:9F:56:20:FC
EP mac : 00:50:56:9B:77:DB
we see the L2 info - which is consistent with the show endpoint
command on Leaf 1201 issued earlier.
Only MAC address information learned from L3Outs is learned in the local endpoint table and reported to the COOP database. IP information still appears in the ARP cache for the attached leaf/VPC pair, but IP information is not sent to COOP for external L3 Out endpoints.
Note that this only applies to DIRECTLY connected endpoints - endpoints such as the remote host in my diagram above (10.118.10.10
) will NEVER appear in an endpoint table.
I hope this helps.
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.
03-14-2024 02:38 PM - edited 04-18-2024 08:50 PM
Hi @Suprit Chinchodikar ,
Also, I was refereeing to the flags from the first command output so does V stand for VPC?
Correct. Here is the expanded output.
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: S - static s - arp L - local O - peer-attached V - vpc-attached a - local-aged p - peer-aged M - span B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy E - shared-service m - svc-mgr +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 20 vlan-1181 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1181 10.118.11.10 L eth1/28 23 vlan-1183 0050.569b.5766 LaV po5 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po5 22 vlan-1184 0050.569b.0283 LV po5 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po5 24/Tenant18:Production_VRF vxlan-14843891 380e.4d48.8df5 L eth1/10 +------------------------------------------------------------------------------+ Endpoint Summary +------------------------------------------------------------------------------+ Total number of Local Endpoints : 5 Total number of Remote Endpoints : 0 Total number of Peer Endpoints : 0 Total number of Peer Rl Endpoints : 0 Total number of vPC Endpoints : 2 Total number of non-vPC Endpoints : 3 Total number of MACs : 4 Total number of VTEPs : 0 Total number of Local IPs : 4 Total number of Remote IPs : 0 Total number All EPs : 5
One curious thing about the Flags in the output of this command is the effect that a VPC pair has on the endpoints with single attachments to the remote leaf that are configured for a different VLAN.
For instance, in my diagram above (here it is again, with the host in question highlighted) where ALL endpoints have communicated with each other...
... I would EXPECT to see (in the output of the fabric 1201 show endpoint vrf Tenant18:Production_VRF
command) the endpoint 10.118.12.10 to appear. And to appear with no flags (indicating a Remote endpoint - i.e endpoint attached to anther leaf)
But it doesn't!
The explanation for this has to do with the fact that VPC leaf pairs actually share ALL endpoint information, not just the VPC endpoints. If I kill the VPC (.i.e. Fabric > Access Policies >> Policies > Switch > Virtual Port Channel Default and delete the Explicit VPC Protection Group), the output is quite different:
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: S - static s - arp L - local O - peer-attached V - vpc-attached a - local-aged p - peer-aged M - span B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy E - shared-service m - svc-mgr +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 Tenant18:Production_VRF 10.118.11.200 tunnel3 Tenant18:Production_VRF 10.118.12.10 tunnel3 Tenant18:Production_VRF 10.118.12.200 B tunnel3 14/Tenant18:Production_VRF vxlan-16351138 0050.569b.5766 tunnel3 15 vlan-1181 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1181 10.118.11.10 L eth1/28 16/Tenant18:Production_VRF vxlan-15794150 0050.569b.0283 B tunnel3 19/Tenant18:Production_VRF vxlan-15695749 380e.4d48.8df5 L eth1/10
Note that 10.118.12.10
now shows up as REMOTE endpoint (annoyingly, there is no REMOTE flag - just the absence of any other flag indicated REMOTE)
And, since I broke the VPC, one of the VPC attached endpoints show up with a Bounce entry (as expected). But that's a whole discussion for another day.
[Added 2024.04.19 - Curious variation]
If I leave the VPC intact, and simply change the VLAN encapsulation for the single attached host to match the VLAN used on the VPC (1183 rather than 1181), I get a different result - I see 10.118.12.10 as a peer-attached host as indicated by the O flag - both for Layer 2 and Layer 3
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: S - static s - arp L - local O - peer-attached V - vpc-attached a - local-aged p - peer-aged M - span B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy E - shared-service m - svc-mgr +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 20 vlan-1184 a036.9f56.20ff O tunnel3 Tenant18:Production_VRF vlan-1184 10.118.12.10 O tunnel3 20 vlan-1184 0050.569b.886a LV po12 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po12 23 vlan-1183 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1183 10.118.11.10 L eth1/28 23 vlan-1183 0050.569b.2314 LV po12 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po12 24/Tenant18:Production_VRF vxlan-14843891 380e.4d48.8df5 L eth1/10
03-13-2024 05:16 PM - edited 04-18-2024 08:55 PM
Hi @Suprit Chinchodikar ,
ACI doesn't learn IP EPs from L3Outs. L3Outs are based on traditional routing tables + ARP Cache.
So therefore, any endpoint (IP+MAC in the ARP cache) seen on a L3Out is seen only on the Leaf/VPC Pair that it is attached to, and only the MAC address is reported to the COOP database - and even then, it won't show in the show endpoint
command for the leaf.
HOWEVER, the Cisco GUI does do a little trick when viewing Fabric > Inventory >> Topology >| Global-EndPoints to make those ARP entries appear in the MAC Endpoints and IP End-Points tables - even though they are not in the COOP database.
Here's my lab
Note particularly the IP address of the remote router - which is the only directly connected endpoint on this L3 Out (as would typically be the case). Observe that it is 10.118.1.1
. Also note that the interface on Leaf1201 to which the router connects is Eth1/10
Here is the output of a show endpoint
command on Leaf 1201 for this VRF
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- <snip> +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 20 vlan-1181 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1181 10.118.11.10 L eth1/28 23 vlan-1183 0050.569b.5766 LV po5 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po5 22 vlan-1184 0050.569b.0283 LV po5 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po5 24/Tenant18:Production_VRF vxlan-14843891 380e.4d48.8df5 L eth1/10
Note that the only address shown agains interface Eth1/10
is the MAC address of the router - NO IP address is shown.
Here is the output of a show ip arp
command on Leaf 1201 for this VRF - which shows the MAC to IP binding.
apic1# fabric 1201 show ip arp vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- <snip> IP ARP Table for context Tenant18:Production_VRF Total number of entries: 1 Address Age MAC Address Interface 10.118.1.1 00:00:05 380e.4d48.8df5 vlan24
But if I look at Fabric > Inventory >> Topology >| Global-EndPoints I see the 10.118.1.1
endpoint!
Which of course makes it LOOK like the IP endpoint IS in the global proxy table.
To prove otherwise is going to involve digging into the actual proxy table itself, which is not a common thing to do, and requires some system level commands on the a spine.
The commands to check the entire COOP database on a spine all begin with show coop internal info
. And the ones we want are show coop internal info ip-db
and show coop internal info repo ep
When I look at Spine1101, I see that the IP endpoint 10.118.1.1
does NOT appear in the COOP database!
apic1# fabric 1101 show coop internal info ip-db | egrep "IP address" | egrep 10.118 IP address : 10.118.11.1 IP address : 10.118.11.10 IP address : 10.118.11.200 IP address : 10.118.1.201 IP address : 10.118.12.1 IP address : 10.118.0.201 IP address : 10.118.12.200 IP address : 10.118.12.10
Note that 10.118.1.1
endpoint does NOT appear.
But, if we look at L2 EP info in the COOP database: (Remember the MAC address of 10.118.1.1
is 38:0E:4D:48:8D:F5
)
apic1# fabric 1101 show coop internal info repo ep | egrep "EP mac"
EP mac : A0:36:9F:61:8E:EB
EP mac : A0:36:9F:56:20:FF
EP mac : 00:50:56:9B:57:66
EP mac : 38:0E:4D:48:8D:F5
EP mac : 34:ED:1B:8B:5A:2B
EP mac : 34:ED:1B:8B:5A:2B
EP mac : 00:50:56:9B:02:83
EP mac : 00:50:56:9B:1F:E6
EP mac : A0:36:9F:61:8E:E8
EP mac : 00:22:BD:F8:19:FF
EP mac : 00:50:56:9B:F0:43
EP mac : A0:36:9F:56:20:FC
EP mac : 00:50:56:9B:77:DB
we see the L2 info - which is consistent with the show endpoint
command on Leaf 1201 issued earlier.
Only MAC address information learned from L3Outs is learned in the local endpoint table and reported to the COOP database. IP information still appears in the ARP cache for the attached leaf/VPC pair, but IP information is not sent to COOP for external L3 Out endpoints.
Note that this only applies to DIRECTLY connected endpoints - endpoints such as the remote host in my diagram above (10.118.10.10
) will NEVER appear in an endpoint table.
I hope this helps.
Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem.
03-14-2024 02:44 AM
@RedNectar,
Thanks for the brief explanation, this clear to me now.
Also, I was refereeing to the flags from the first command output so does V stand for VPC?
03-14-2024 02:38 PM - edited 04-18-2024 08:50 PM
Hi @Suprit Chinchodikar ,
Also, I was refereeing to the flags from the first command output so does V stand for VPC?
Correct. Here is the expanded output.
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: S - static s - arp L - local O - peer-attached V - vpc-attached a - local-aged p - peer-aged M - span B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy E - shared-service m - svc-mgr +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 20 vlan-1181 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1181 10.118.11.10 L eth1/28 23 vlan-1183 0050.569b.5766 LaV po5 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po5 22 vlan-1184 0050.569b.0283 LV po5 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po5 24/Tenant18:Production_VRF vxlan-14843891 380e.4d48.8df5 L eth1/10 +------------------------------------------------------------------------------+ Endpoint Summary +------------------------------------------------------------------------------+ Total number of Local Endpoints : 5 Total number of Remote Endpoints : 0 Total number of Peer Endpoints : 0 Total number of Peer Rl Endpoints : 0 Total number of vPC Endpoints : 2 Total number of non-vPC Endpoints : 3 Total number of MACs : 4 Total number of VTEPs : 0 Total number of Local IPs : 4 Total number of Remote IPs : 0 Total number All EPs : 5
One curious thing about the Flags in the output of this command is the effect that a VPC pair has on the endpoints with single attachments to the remote leaf that are configured for a different VLAN.
For instance, in my diagram above (here it is again, with the host in question highlighted) where ALL endpoints have communicated with each other...
... I would EXPECT to see (in the output of the fabric 1201 show endpoint vrf Tenant18:Production_VRF
command) the endpoint 10.118.12.10 to appear. And to appear with no flags (indicating a Remote endpoint - i.e endpoint attached to anther leaf)
But it doesn't!
The explanation for this has to do with the fact that VPC leaf pairs actually share ALL endpoint information, not just the VPC endpoints. If I kill the VPC (.i.e. Fabric > Access Policies >> Policies > Switch > Virtual Port Channel Default and delete the Explicit VPC Protection Group), the output is quite different:
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: S - static s - arp L - local O - peer-attached V - vpc-attached a - local-aged p - peer-aged M - span B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy E - shared-service m - svc-mgr +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 Tenant18:Production_VRF 10.118.11.200 tunnel3 Tenant18:Production_VRF 10.118.12.10 tunnel3 Tenant18:Production_VRF 10.118.12.200 B tunnel3 14/Tenant18:Production_VRF vxlan-16351138 0050.569b.5766 tunnel3 15 vlan-1181 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1181 10.118.11.10 L eth1/28 16/Tenant18:Production_VRF vxlan-15794150 0050.569b.0283 B tunnel3 19/Tenant18:Production_VRF vxlan-15695749 380e.4d48.8df5 L eth1/10
Note that 10.118.12.10
now shows up as REMOTE endpoint (annoyingly, there is no REMOTE flag - just the absence of any other flag indicated REMOTE)
And, since I broke the VPC, one of the VPC attached endpoints show up with a Bounce entry (as expected). But that's a whole discussion for another day.
[Added 2024.04.19 - Curious variation]
If I leave the VPC intact, and simply change the VLAN encapsulation for the single attached host to match the VLAN used on the VPC (1183 rather than 1181), I get a different result - I see 10.118.12.10 as a peer-attached host as indicated by the O flag - both for Layer 2 and Layer 3
apic1# fabric 1201 show endpoint vrf Tenant18:Production_VRF ---------------------------------------------------------------- Node 1201 (Leaf1201) ---------------------------------------------------------------- Legend: S - static s - arp L - local O - peer-attached V - vpc-attached a - local-aged p - peer-aged M - span B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy E - shared-service m - svc-mgr +-----------------------------------+---------------+-----------------+--------------+-------------+ VLAN/ Encap MAC Address MAC Info/ Interface Domain VLAN IP Address IP Info +-----------------------------------+---------------+-----------------+--------------+-------------+ Tenant18:Production_VRF 10.118.0.201 L lo3 20 vlan-1184 a036.9f56.20ff O tunnel3 Tenant18:Production_VRF vlan-1184 10.118.12.10 O tunnel3 20 vlan-1184 0050.569b.886a LV po12 Tenant18:Production_VRF vlan-1184 10.118.12.200 LV po12 23 vlan-1183 a036.9f56.20fc L eth1/28 Tenant18:Production_VRF vlan-1183 10.118.11.10 L eth1/28 23 vlan-1183 0050.569b.2314 LV po12 Tenant18:Production_VRF vlan-1183 10.118.11.200 LV po12 24/Tenant18:Production_VRF vxlan-14843891 380e.4d48.8df5 L eth1/10
Discover and save your favorite ideas. Come back to expert answers, step-by-step guides, recent topics, and more.
New here? Get started with these tips. How to use Community New member guide