Connection of access layer switches to ACI leaf nodes

[Andrew C]

Hi,

I've been thinking this through for a few days and would like to gauge public opinion on how others would approach it.

Background:

We have 2x Cat6500's for the network core. Into these connected the data center switches (more 6509's) and the access layer switches (mostly Cat 3850's and a couple of Cat 3750's).

We have replaced all the 6500's with ACI - 2x Spine, 4x Leaf. Two of the leaf nodes are used for data center connectivity and the other two as border leaf nodes for external connectivity.

Current issue:

To get the decommissioning of the Cat 6500's completed, we need to migrate the 22 access layer switches to something else. I think there are 3 options:

Option 1:

Connect each access layer switch to the border leaf nodes. This would be 22 connections to border leaf A, and 22 connections to border leaf B. Would have standard port channel on the 3850 and VPC in ACI. The gateways for the various VLANs would be moved from the Cat 6500's to Bridge Domains in ACI.

While this would work, I think it would be an expensive option as the cost per port on a Nexus 9396 is quite high.

Option 2:

Connect each access layer switch to a Nexus 5548UP and have that act as a distribution layer. The 5548 would then connect to the ACI border leaf nodes where the gateways will be. Would be standard port channel on the 3850's, VPC on the 5548, and VPC in ACI.

I think this would work - but does anyone know/think otherwise? Can we do VPC to VPC between 5548 and ACI? The 5K's don't have L3 daughter cards or licenses so can't do the gateways on the 5K's.

Option 3:

Connect each access layer switch to a Nexus 5548UP, but instead of having the 5548 connect to ACI upstream, have it connect to our ASR 1001 routers instead. The gateways would then be created as sub-interfaces and use HSRP to get active/standby between the 2 ASR's.

This would work and we are doing something similar at other (smaller) sites, but is it the best option?

My personal "favourite" is to use Option 1 (access layer direct to ACI leaf node) but maybe Option 2 (to ACI via 5548) gives more future expansion options. Would anyone care to offer their 2 cent's worth?

Many thanks,

Andrew

[Claudia de Luna · Accepted solution]

Hi Andrew,

In a "Classical Ethernet" environment I don't think anyone would suggest connecting their Access directly to their Data Center switches.  Having a Distribution layer is always recommended.  With both Option 1 and Option 2 all your Access flows even if the destination is not a data center service on ACI will have to traverse your data center fabric.  This "overuse" of the ACI fabric is something I always try to caution my clients against.  Can it do it.  Of course.  But at the end of the day ACI is just a data center network and many of the best practices we are used to still apply.  This is one.  I'd look at getting a pair of N3Ks or N9Ks that support layer 3 to provide a collapsed core/distribution where you can land your 22 Access switches and provide their L3 boundary.  Your ASRs can land here as well as the ACI Fabric.  The links to the fabric could be 4 x 10G L3 ECMP and your links up to your ASRs can also be L3.

Option 3 has a similar issue in that now all your access traffic flows that are intended for data center services bounce of your WAN or Edge router.   

So I hate to do an "it depends" on you but barring new gear for a collapsed core/distribution then I would let your traffic patterns guide your selection of the three options you have outlined.

You can do a dual sided VPC between anything that supports vpc (your N5Ks) and ACI.  

If I had to pick an option Id probably pick option 3 but I might make a different choice with a better understanding of the traffic patterns.

Either way, I'd use the N5K for the ports as you suggested and for at least a Layer 2 distribution that I could later swap out for L3.

Probably not that helpful, I know, but maybe a few things to chew on :D?

[Andrew C]

Claudia,

You say "Probably not that helpful" - are you kidding?! This is great, thank you! Certainly a lot to chew on!

Thank you for taking the time to think about and write this

Andrew

[aymen.khelifi]

super helpful