Multi-pod endpoint learning

[Antonio Macia]

Hi,

The ACI Multi-pod white paper explains how remote endpoints belonging to different pods are associated to the spines Anycast VTEP address (Proxy), minimising the number of MP-BGP EVPN updates between pods when the endpoint performs local movement between leafs in the same pod. That makes complete sense.

However, further down the document, when it shows how endpoints are learnt with arp-flooding enabled and disabled (ARP-gleaning), the final picture in both cases display the local leafs endpoint table pointing to the specific remote leafs instead of the spines proxy IP.

How is this actually implemented?

Thanks.

[Sergiu.Daniluk · Accepted solution]

Hi,

There are two distinct locations we are discussing about:

1. Leaf endpoint table
2. Spine COOP database

Let's discuss about each of them:

• Leaf endpoint table

Remote endpoint is being learned when a packet comes into a Cisco ACI leaf switch from another leaf through fabric ports. Learning is done based on outer source IP which represents the source Leaf TEP. Thus the tunnel interface will point to the leaf where the endpoint is connected.

Output from POD2 leaf, about EP from pod1

Leaf111# show endpoint ip 172.16.1.1
Legend:
 s - arp              H - vtep             V - vpc-attached     p - peer-aged       
 R - peer-attached-rl B - bounce           S - static           M - span            
 D - bounce-to-proxy  O - peer-attached    a - local-aged       m - svc-mgr         
 L - local            E - shared-service  
+-----------------------------------+---------------+-----------------+--------------+-------------+
      VLAN/                           Encap           MAC Address       MAC Info/       Interface
      Domain                          VLAN            IP Address        IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
SD:SD                                                      172.16.1.1                      tunnel25

Leaf111# show int tun 25            
Tunnel25 is up
    MTU 9000 bytes, BW 0 Kbit
    Transport protocol is in VRF "overlay-1"
    Tunnel protocol/transport is ivxlan
    Tunnel source 10.1.184.65/32 (lo0)
    Tunnel destination 10.0.48.64

Leaf111# acidiag fnvread
      ID   Pod ID                 Name    Serial Number         IP Address    Role        State   LastUpdMsgId
--------------------------------------------------------------------------------------------------------------
     101        1              Leaf101      AAAAAAAAAAA      10.0.48.64/32    leaf         active   0

• Spine COOP database

In case of multi-pod setup, the spines from a pod will update the COOP databases with the remote EPs information based on MP-BGP updates generated by remote pod's spines (etep). Based on this information, the spines will generate a tunnel with the anycast IP of the remote pod spines.

Output from POD2 spine, about EP from pod1:

Spine211# show coop internal info ip-db key 2490368 172.16.1.1
IP address : 172.16.1.1
Vrf : 2490368
Flags : 0x2
EP bd vnid : 16318376 
EP mac :  00:00:00:00:00:02
Publisher Id : 192.168.1.1
Record timestamp : 01 01 1970 02:00:00 0
Publish timestamp : 01 01 1970 02:00:00 0
Seq No: 0
Remote publish timestamp: 03 29 2020 17:20:31 197393676
URIB Tunnel Info
Num tunnels : 1
        Tunnel address : 10.0.0.34
        Tunnel ref count : 1

Spine201# show ip interface | grep -B 1 "192.168.1.1/"
lo13, Interface status: protocol-up/link-up/admin-up, iod: 22, mode: etep  IP address: 192.168.1.1, IP subnet: 192.168.1.1/32  

Spine201# show ip interface | grep -B 1 "10.0.0.34"
lo8, Interface status: protocol-up/link-up/admin-up, iod: 17, mode: anycast-v4,external
  IP address: 10.0.0.34, IP subnet: 10.0.0.34/32  

In case of a EP move, let's say from Leaf101 to Leaf102, the local spines will generate a bounce entry on Leaf101. When traffic generated by EP will arrive on Leaf111 (pod2), the leaf will update the tunnel information based on source TEP in the outer header. Because the EP is still located in POD1, the spines from pod1 do not need to generate any additional MP-BGP update.

Regards,

Sergiu

[Antonio Macia]

Thank you Sergiu, that was a fantastic explanation!

[Sergiu.Daniluk · Accepted solution]

Hi,

Glad to hear that it is helpful!

I forgot to mention how the info in the COOP database for the remote EP is useful. This is used when it comes to unknown unicast or hardware proxy traffic. For example, if the leaf prom pod2 does not know about remote EP, and hardware proxy is enabled in BD, packets will be forwarded to spines (pod2), and normally, the spines will know the correct pod (spines) where to forward further the packets (let's say spines from pod1). Once the traffic reaches the spines from pod1, the local coop db, will point out to the leaf where the EP is locally connected.

Cheers,

Sergiu