Cisco ACI Multisite - ISN

[Newbie..9109]

Hi,

Basically this is simple question. But, I'm still confused even I have read document cisco ACI multisite whitepaper.

Basically, ACI use MP BGP EVPN (between spines) to build underlay between sites. Hence, it is needed to configure OSPF and BGP between spine and ISN equipment (let's say router).

Should i also use BGP-EVPN between ISN? I mean between ISN, not between spine and ISN. Or I can use any other routing protocol like IGP between ISN? 

[Robert Burns]

Between ISN devices (after the first SPINE > ISN Device hop), you can use any IP routing method.  Just needs to support larger MTU to support the extended VXLAN headers.  

Robert

[Newbie..9109]

So it means between ISN, we can apply any IGP such as only OSPF between ISN? No need either BGP or even MP-BGP MPLS?

[Robert Burns]

Correct.

[Newbie..9109]

Okay Robert,

one more thing to make it clear. So devices which will run MP-BGP EVPN is only the spine with another spine as a target (another site) vice versa?

from spine to ISN devices (router) we only need to run OSPF (due to this is the one only supported) as undelay routing to be traversed by MP-BGP EVPN?

[sidshas03]

In an ACI multisite deployment, the underlay network between sites is built using MP-BGP EVPN between the spines, as you mentioned. However, the routing protocol used between the ISN (inter-site network) equipment, such as routers, is independent of the underlay protocol.

In general, you can use any routing protocol between ISN equipment that is supported by both devices. This could include OSPF, BGP, or any other IGP (Interior Gateway Protocol) such as IS-IS or EIGRP.

It's important to note that the choice of routing protocol between ISN equipment may impact the ability to support certain ACI multisite features, such as stretched Layer 2 segments or end-to-end policy enforcement. For example, if you want to extend a Layer 2 segment between sites, you may need to use a protocol that supports MAC address mobility, such as BGP EVPN.

In summary, while MP-BGP EVPN is used to build the underlay network between sites in ACI multisite, the choice of routing protocol between ISN equipment is independent and can be any protocol that is supported by both devices.

[Newbie..9109]

Hi,

Let me summarize it to make it clear.

Regardless of layer 3 or layer 2 connectivity between sites, spines still use MP-BGP EVPN to make underlay between sites?

To support layer 3 connectivity (different subnet between each sites), I only need to run either IGP or BGP between ISN. It's enough?

Yet, to support layer 2 connectivity (span one subnet across sites), I need to run BGP EVPN between ISN? It's a must? any other option to support layer 2 connectivity between ISN?

[sidshas03]

Yes, that's correct.

Regardless of layer 2 or layer 3 connectivity between sites, the spine switches use MP-BGP EVPN as the underlay protocol to build connectivity between sites.

If you want to support layer 3 connectivity between sites (different subnets between sites), you can use any routing protocol like IGP (such as OSPF or IS-IS) or BGP to exchange the endpoint information between the ISN equipment.

To support layer 2 connectivity between sites (spanning one subnet across sites), BGP EVPN between the ISN equipment is typically the best option. However, you can also use alternative protocols like Virtual Extensible LAN (VXLAN) over IPsec or VXLAN with multicast over WAN. However, these options may have some limitations, such as increased complexity and potential scalability issues.

[Newbie..9109]

Your answer is very loud and clear.

1. Do you mean MP-BGP EVPN between ISN? BGP EVPN or MP-BGP EVPN? Is it same or different?

2. If we use BGP EVPN between ISN (let's say to stretched layer 2 between sites), can we still run only OSPF between spine and ISN device (router)? Or we also need to run BGP on top of OSPF between spine and ISN device? Or maybe can only BGP between spine and ISN (no need OSPF)?

[sidshas03]

1. Yes, I apologize for any confusion. When referring to MP-BGP EVPN, I meant the use of BGP EVPN as the control plane for the multiprotocol BGP (MP-BGP) protocol. BGP EVPN is a type of MP-BGP protocol used for the control plane of Ethernet VPNs (EVPNs), which are used to provide layer 2 connectivity between sites in a data center network.

2. It is possible to run only OSPF between the spine and ISN device (router) even if BGP EVPN is used to stretch layer 2 between sites. OSPF would be used to exchange IP routing information between the spine and ISN device, while BGP EVPN would be used to exchange information about the stretched layer 2 Ethernet segments. However, it is also possible to run BGP between the spine and ISN device (router) instead of OSPF, in which case BGP would be used for both the IP routing information and the Ethernet segment information. The choice between OSPF and BGP depends on the specific requirements and design of the network.

[brlehigh · Accepted solution]

I think we're confusing "sites" with "ACI fabrics".  
Unless I'm mistaken, If the ISN is only being used for ACI multi-site connectivity we should be able to run any protocol within the ISN.  As long as there's IP reachability between the ACI fabrics across the ISN it should work (whether that's OSPF, BGP, static, etc).  We should be able to stretch layer 2 or layer 3 segments from one ACI fabric to another regardless of the protocol used within the ISN.
Now, if the ISN is being used for other traffic outside of ACI and you're trying to stretch layer 2, then yes the protocol would matter.

[Newbie..9109]

Hi,

Okay, so which one is correct? 

We can stretched either layer 2 or 3 with any routing protocol between ISN? Or as mentioned by @sidshas03? to stretch layer 2 we need BGP EVPN between ISN, and to stretch layer 3 we can apply any routing protocol?

@Robert Burns 

[sidshas03 · Accepted solution]

You are correct. My previous response was assuming a traditional data center network where the ISN is used for general purpose connectivity between sites, but if the ISN is being used solely for ACI multi-site connectivity, any protocol that provides IP reachability between the ACI fabrics should work, including OSPF, BGP, or static routing.

As long as the appropriate network policies and contracts are in place to allow the stretched segments to communicate with each other, the choice of the protocol used within the ISN would not have any impact on the ability to stretch layer 2 or layer 3 segments between ACI fabrics. Thank you for bringing this to my attention and apologies for any confusion caused.

[Newbie..9109]

Thanks a lot @sidshas03 for the confirmation.

One more thing, I just noticed that spine will do mutual redistribution IS-IS to OSPF/BGP (spine - ISN devices). 

Which prefixes will be redistributed? Only the prefix within VTEP environment (IP reachability between VTEP site 1 and site 2) or all the entire prefixes within Cisco ACI Fabric (include VLAN endpoint prefixes)?

[sidshas03]

The prefixes that will be redistributed depend on the redistribution configuration. If you configure redistribution to include all prefixes within the Cisco ACI fabric, then all the prefixes, including VLAN endpoint prefixes, will be redistributed. However, if you configure redistribution to include only the prefixes within the VTEP environment (IP reachability between VTEP site 1 and site 2), then only those prefixes will be redistributed.

It's important to carefully consider which prefixes to redistribute and how to configure redistribution to ensure efficient and effective routing within the ACI fabric and between the ACI fabric and external networks.