MP-BGP , IS-IS, COOP

[mgual57]

Hello,

Can someone explain to me what are doing those 3 technologies ?

Thanks

[Claudia de Luna]

Hi [@mgual57@gmail.com]  

MP-BGP, when enabled by configuring an ASN and enabling your spines as route reflectors,  handles distributing the (host and external) routing table to the devices in the fabric (i.e leafs) and to external devices via L3outs.

Think of IS-IS as the IGP within the fabric that is building the routing tables.

Think of COOP (Council of Oracle Protocol) as the protocol that that helps to build the tables that identify endpoints...what helps build the "arp table" across the entire fabric, not just on a sigle device.

This is how I like to think of them...there is alot more "under the hood" but one of the nice things about ACI is that it does alot of that for you!

[mgual57]

Thanks you !

So if I understand well , IS-IS is used for routing between leaves and spines and VTEp , MP-BGP is used for routing to external devices but leaves learn Endpoint's mac and BGP send it to spines. BGP also store information about the IP address of VTEP. And finally COOP is used for mac discovery of the host?

[mgual57]

One more question : When is COOP used exactly , cause MP-BGP can learn host mac address so I dont really understand the utility of COOP

[RedNectar · Accepted solution]

Hi mgual57@gmail.com

Let me start with a visual picture.  Imagine a simple 2leaf/2spine topology with HostA attached to to Leaf1 and with HostB attached to to Leaf2

Leaf1 has a VTEP address of 10.0.1.101

Leaf2 has a VTEP address of 10.0.1.102

Spine1 has a VTEP address of 10.0.1.201

Spine2 has a VTEP address of 10.0.1.202

HostA has a MAC address of A and an IP address of 192.168.1.1 and is attached to port 1/5 on Leaf1

HostB has a MAC address of B and an IP address of 192.168.1.2 and is attached to port 1/6 on Leaf2

Enter IS-IS

The leaves and spines will exchange IS-IS routing updates with each other so that Leaf1 sees that it has two equally good paths to reach Leaf2, and Leaf2 sees that it has two equally good paths to reach Leaf1

For now, that's all we need to know about IS-IS - IS-IS is the routing protocol used by the VTEPs to learn how to reach the other VTEPs.

Now think about the hosts.

This is where COOP comes in.

When Leaf1 learns about HostA because say HostA sent an ARP request seeking the MAC address of 192.168.1.2 (which you know is HostB, but that's not relevant at the moment)

Leaf1 looks at that ARP request, and just like a normal switch, learns that MAC A is present on port 1/5.  But the leaf is a bit more clever than that, and looks INSIDE the payload of the ARP packet and learns that Host1 also has an IP address of 192.168.1.1 and records all this information in its Local Station Table

AND THEN reports this information to one of the spine switches (chosen at random) using the Council Of Oracles Protocol (COOP).  The spine switch that was chosen then relays this information to all the other spines so that every spine has a complete record of every end point in the system.

The spines record the information learned via the COOP in the Global Proxy Table, and this information is used to resolve unknown destination MAC/IP addresses when traffic is sent to the Proxy address.

Note that all of this happens without anything to do with BGP.

But to round off the COOP story, we would assume that at some stage Leaf2 will also learn HostB's MAC and IP and also inform one of the spines at random of this information using the COOP.

So COOP is used solely for the purpose of distributing endpoint information to Spine switches. As far as I know, Spine switches never use COOP to distribute end host information to leaf switches.

So where does BGP fit in?

BGP is not needed until an external router is connected.  So now imagine that Leaf2 has had a router connected and has learned some routes from that external router for a particular VRF for a particular Tenant. 

How can Leaf2 pass this information on to Leaf1 where HostA is trying to send packets to one of these external networks?  For Leaf2 to be able to pass routing information on to Leaf1 and keep that information exclusive to the same VRF, we need a routing protocol that is capable of exchanging routing information for multiple VRFs across an underlay network.

Which is exactly what MP-BGP was invented for - to carry routing information across MPLS underlay networks.  In the case of ACI, BGP is configured by choosing an Autonomous System number and nominating one of the spine switches to be a route reflector.  MP-BGP is self configuring, you don't need to do anything to make it work!

Hope this helps.

Reference: A Version of this post with illustrations and examples can be found here.

[mgual57]

Just one last question , I thought MP-BGP can be used to learn host mac address but it seems like I was wrong or I misunderstood your explaination?

[RedNectar]

Ahh - now you are getting into the realm of MP-BGP EVPN (what a mouthful) which is used between pods in a multi-pod environment and between sites in a multi-site environment, but I don't have those details in my head, I'd have to read http://www.cisco.com/c/en/us/products/collateral/switches/nexus-9000-series-switches/guide-c07-734107.html as a starting point

So for now I'll leave this - perhaps you should ask this as a new question to attract other respondents who by know are skipping this thread.

CW

[moumoham]

Indeed your understanding is correct and this can be done using BGP as well but not in ACI

[Rick1776]

This is a fantastic, clear, and super concise write up of the ACI fabric in a nutshell. Thank you for writing this.

[Maren Mahoney]

As a teacher, I appreciate the clear, concise story you told to explain these concepts. If you are not a teacher, please consider it. You will improve the lives of many.

[asif.naveed]

Dear Chris
Thanks a lot but where does Multicast role is here

[JoaoCadavez]

Just one minor detail. I'm trying to understand the use of IS-IS in this particular scenario. In my understand, IS-IS is mandatory because it runs over layer 2, in opposite to OSPF or even IGRP that run over layer 3. Am I right, or there is no reason to use IS-IS in particular? Thanks!

[RedNectar]

Hi @JoaoCadavez ,

I hope I can settle your concerns.

IS-IS is a routing protocol

Just one minor detail. I'm trying to understand the use of IS-IS in this particular scenario. 

---

Although IS-IS was originally defined to run on CLNS and does run at the Data Link Layer, it still carries routing information.  In 1990, extensions to the IS-IS protocol were defined in RFC 1195 to allow it to carry IP routing information as well as CLNS routes.

Why IS-IS is Mandatory

In my understand, IS-IS is mandatory because it runs over layer 2, in opposite to OSPF or even IGRP that run over layer 3. 

---

The fact that IS-IS runs at Layer 2 dies not make it mandatory.

To build a Leaf-Spine underlay toploogy, whether it be ACI or any other variation, an underlying routing protocol is required to maintain the mapping of the Leaf-Spine topology.  Some vendors chose BGP as the underlying protcol. But BGP lacks the super-fast convergence that can be achieved with a link-state routing protocol. When Insieme developed ACI's Leaf-Spine topology, they chose IS-IS with its IP extensions as the underlying protocol.

If you use ACI, you get IS-IS as the underlying protocol, but that it totally transparent to the ACI administrator. I have been working extensively with ACI since version 1.01 and have NEVER had to look at any of the workings of IS-IS other than to stisfy my curiosity.

There's no reason to use IS-IS - in fact you can't!

Am I right, or there is no reason to use IS-IS in particular? Thanks!

---

As far as the end-user is concerned, you will NEVER have to do any IS-IS configuration. The only routing protocol you'll need to configure is between ACI and the outside world. As of 2020, your choices are limited to using OSPF, BGP or EIGRP.  You don't even have IS-IS as a choice!

The advantage IS-IS gives ACI

ACI leaf-spine structure is almost as transprent to the end user as the backplane of say a Nexus 7000 or 9000 chasis.  You as the user will never have to worry about the underlying leaf-spine topolgy.  There are chaper SDN solutions that allow you to build your own leaf-spince topolgy, where you get to manage all that leaf-spine infrastructure and routing yourself. In fact you can even buy the same Nexus 9000 switches running in NXOS mode to do just that. But I wouldn't advise doing that when you can buy a ready made ACI solution.

[JoaoCadavez]

Thank you so much!

[moj_ccie]

That was Great 

As you Say:

IS-IS Robust Routing Protocol is IP Fabric in ACI Underlay (It's mean for Establish Great IP Routing For All Leaf & Spine in Fabric) == Transparent & Permanent 

BGP Exterior Routing Protocol is Only For L3Out, WAN Network (ISN) & Multi-Site & Multi-POD IP Routing Establishment - From Leaf should be Redistributed Route Table (Ingress BGP - Egress OSPF/EIGRP)

COOP Oracle Protocol For:

1: Update Local Station Table (Leaf Host Table) to Proxy Station Table (Spine Table) 

2: Update Proxy Station Table (Spine Table) of All Spine in Fabric

3. Proxy Spine Anycast

 Best Regards.