ISN Device connections to spines in ACI MSO deployment

[Aaron_un]

This is the config on the IPN side(my question is not about the config on spines, we assume that they are fully confged)

src: y0chub.cm/watch?v=HJJ8lznodN0

"The spine interfaces are connected to the ISN devices through point-to-point routed interfaces. However, traffic originating from the spine interfaces is always tagged with an 802.1q VLAN 4 value." this is from Cisco ACI Multi-Site Architecture White Paper

the part that really confuses me is about the placement of the IPN device(or that ISN cloud) and what the ISN network architecture really is, if it's located in the service provider, you can't have it connected back to the spine over the ospf protocol since it's in two geo disperse locations over an internet wan link, and if it's on premise how can you possibly connect the other interface to the other site(you can connect to the local spine but not to the other one)?? not unless we're using dark fiber or leased lines.. but in documentation there was this part that I can't find for some reason that says all you need for ISN to work is internet infra.

Thanks

[Robert Burns · Accepted solution]

Aaron, 

There are two means to interconnect Fabrics with Multi-site.  You can do Back-to-Back Spine connections, requires dedicated dark fiber links directly between the spines, or you can employ an InterSite Network (ISN), which most customers opt for as its far more scalable and not restrictive.

ISN devices would be hosted in your DC, and act as the aggregation point for a fabrics Inter-site communication.  In most common deployments a customer will have a pair of ISN devices per site.   Some customer opt to double-duty existing Switches/Routers to provide the ISN with simple VRF separation.  The ISN is not managed by ACI, it must be manually configured.  Between the ISN devices and the Spines we require OSPF (now also BGP) for the first hop.  Sub Int VLAN 4 was chosen to allow for predictive discovery of remote spines (more so for Multipod but was also leveraged for Msite).   Between the ISN sites, you can run any L2/L3 connectivity you want.  WE just require IP connectivity & jumbo frame support to support VXLAN headers.  Between your ISN devices at each site you can run any protocol -MPLS, Dark Fiber, Routed WAN etc - doesn't matter.  For implementations where the ISN will traverse any public/shared WAN links, we'd recommend enabling CloudSec which will encrypt all inter-site communications.

Make sense?

Robert

[Aaron_un]

Totally!

Can I even use OSPF over IPSec, as ip connectivity for between the ISN devices? instead of MPLS, Dark Fiber, Routed WAN.