Cisco ACI Technical Question ( ACI Multi-tier Design or ACI with Fex Design)?

[S. B]

I have seen 2 scenarios for designing the DataCenter with the ACI Architecture, inside a POD.

1- Cisco ACI Multi-tier Architecture

Using Leaf Layer-2 (ToR), Leaf Layer-1 (EoR) and Spine  

https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/white-paper-c11-742214.html

2-  Fabric Extenders (FEX) in ACI Architecture

Using Fex(ToR), Leaf (EoR) and Spine

https://www.cisco.com/c/en/us/support/docs/cloud-systems-management/application-policy-infrastructure-controller-apic/200232-Configuring-vPC-from-a-Host-to-Fabric-Ex.html

1. I wanted to know the main differences between these 2 solutions?
2. Are there any comparison tables and technical documents, comparing these 2 solutions from the Networking perspective and ACI perspective
3. The advantage and disadvantage of these 2 solutions compared with each other?
4. What options/benefits do customers loose when using Fex instead of a Leaf (as the Top of rack switch)? 

Thanks

[RedNectar]

Hi @S. B,

The main difference can be summarised in two points

1. $
2. Gb/s

Design #1 Cisco ACI Multi-tier Architecture requires Nexus 9K (2nd generation) switches ($$$), but does give you aggregated 10Gb/s connectivity to your attached devices (servers) albeit via less-than-aggregate uplinks

To me, I don't see much sense in this unless you are running short of Spine Interfaces - you may as well connec the Tier-2-leaf switches directly to the Spines and make them Tier-1-Leaves.

One advantage Tier-2-leaf switches have over FEXes is that they can be attached to the ACI Tier-1 switches as VPCs. FEXes must be connected to a single ACI Switch.

Design #2 Fabric Extenders (FEX) in ACI Architecture uses Nexus 2K FEXes ($) - which if connecting say 1Gb/s attached devices gives economical connectivty, using say a Nexus 2248TP GE FEX.

There are 10Gb/s FEXes too - Nexus 2232PP 10GE and Nexus 2348UPQ 10GE which I suspect are more economical than the 9K switches.

Keep in mind, N9K-C9348GC-FXP, N9K-C9372TX, N9K-C9372TX-E, N9K-C9396TX, N9K-93108TC-EX, N9K-93108TC-FX, N9K-C93120TX, N9K-C93128TX, and N9K-C93180TC-EX do not support FEX in ACI Mode.

(See https://www.cisco.com/c/dam/en/us/td/docs/Website/datacenter/fexmatrix/fexmatrix.html for more detail)

Also keep in mind that neither design supports L3Out connections on the lower Tier. [Edit: Not true for Tier-2 Switches as I discovered later. But certainly true for FEXes]

I hope this helps

---

Don't forget to mark answers as correct if it solves your problem. This helps others find the correct answer if they search for the same problem

---

[S. B]

Dear @RedNectar 

Special Thanks

So, as a summary, the most important consideration is the L3Out, which none of themsupport in the lower layer. But this can be handled in the upper layer leaf, which can solve the problem. Yes? I mean will there be any limitations or traffic pattern, traffic flow, capacity problem, with connecting the L3 out from the upper layer leaf?

The reason of my question is that, if there will no no problem with connecting through upper layer leaf, this limitation will be less priority, yes?

have you seen any documents or inside ciscolive sessions talking about this ?

[RedNectar]

Hi @S. B ,

Special Thanks

So, as a summary, the most important consideration is the L3Out, which none of themsupport in the lower layer. But this can be handled in the upper layer leaf, which can solve the problem. Yes? I mean will there be any limitations or traffic pattern, traffic flow, capacity problem, with connecting the L3 out from the upper layer leaf?

No - you can connect L3Outs to the Tier-1 Leaf. I guess I mentioned it because I tried to connect a L3Out to a FEX just the other day.

The reason of my question is that, if there will no no problem with connecting through upper layer leaf, this limitation will be less priority, yes?

 No problem.

have you seen any documents or inside ciscolive sessions talking about this ?

---

No - not a hot topic really.  My feeling is that it is more a way of making ACI a bit more afforable for those with stretched budgets (which is why my (sponsor's) lab has just added some FEXes)

[S. B]

Dear Chris @RedNectar 

I didnt get the point.

So, you mean that the L3out cannot be connected to the Tier-1 leaf and the Tier-2 leaf?

[RedNectar]

Hi again @S. B ,

Sorry to confuse you.

L3Outs can't be connected to FEXes

L3Outs CAN be connected to both Tier-1 and Tier-2 Leaf switches.

Sorry for the confusion caused by my first answer (I've put an edit in my first answer to hopfully explain)

[RedNectar]

And one more thing (that I though of as I was climbing into bed)

I've assumed that you DO understand the difference between FEXes and Switches

With a FEX - EVERY SINGLE PACKET that arrives on a FEX is pushed to the upstream switch for processing. There is NO local switching on a FEX

With a switch of course, there is local switching, only packts that have to go to other switches are pushed up the line

[S. B]

Dear @RedNectar 

Thanks

About:

Are there any considerations for the ACI point of view?

I mean the policy,configurations, .....?

Does it make any difference from the ACI point of view: 

1. When you have a server connected to a physical port of a Leaf switch( ACI Single tier/Multi-tier Architecture) 
2. and When you have a server connected to a Fex, which then all the servers on a single FEX are connected to a physical port of the leaf switch. ( Fex connected to Leaf in ACI Architecture) 

From the port configuration of the Leaf switch , which the ACI and APIC see as the last port of the fabric connected to the server, are there any limitations between these 2 solutions?

Thanks

[RedNectar]

---

@S. B wrote:

Are there any considerations for the ACI point of view?

I mean the policy,configurations, .....?

Yes there are: See https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-737909.html#_Toc6452862

In Summary:

FEX has many limitations compared to attaching servers and network devices directly to a leaf. The main ones are the following:

●   No support for L3Out on FEX

●   No Rate limiters support on FEX

●   No Traffic Storm Control on FEX

●   No Port Security support on FEX

●   FEX should not be used to connect routers or L4–L7 devices with service-graph redirect

●   The use in conjunction with microsegmentation works, but if microsegmentation is used Quality of Service (QoS) does not work on FEX ports because all microsegmented traffic is tagged with a specific class of service. Microsegmentation and FEX is a feature that at the time of this writing has not been extensively validated.

It seems there ar no restrictions for Tier-2 Leaf switches (I'm sorry - I was mistaken about the L3Outs on Tier-2 switches)

See https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/white-paper-c11-742214.html where it states "There are no restrictions of EPG, L3Out, APIC or FEX connectivity to tier-2 leaf switches. They can be connected to tier-1 leaf switches or to tier-2 leaf switches. Per leaf scale is independent regardless of Tier-1 or Tier-2."

Wow. You can even Add FEXes to Tier-2 switches!!!!

Does it make any difference from the ACI point of view: 

1. When you have a server connected to a physical port of a Leaf switch( ACI Single tier/Multi-tier Architecture) 
2. and When you have a server connected to a Fex, which then all the servers on a single FEX are connected to a physical port of the leaf switch. ( Fex connected to Leaf in ACI Architecture) 

From the port configuration of the Leaf switch , which the ACI and APIC see as the last port of the fabric connected to the server, are there any limitations between these 2 solutions?

Thanks

---

Once your servers are connected to the FEX, you configure the server-facing FEX ports like you would server-facing Switch ports - they just show up as interface 191/1/1 rather than 101/1/1 (where 191 is the FEX ID and 101 is the switch ID). See picture.  I don't have a picture for the Teir-2 leaf solution, but Tier-2 leaves look like Tier-1 leaves as far as configuring Server Ports.

[jason-lew]

I came across this as I was searching on ACI multi-tier leaf design info. As a late contribution to the discussion on why choose one over the other, the requirement that typically drives a multi-tier leaf design or FEX-attached design is the need to economically support lots of copper-based 1 Gbps  connections. Inserting GLC-TE SFPs into Tier-1 leaf ports and doing that in very high quantities is not sensible to do. Chris does refer to this when he talks about budget.

That then leads to the two alternative solutions: Attach N2K copper FEXs to the Tier-1 leaves or attach Tier-2 leaves to the Tier-1 leaves. Aside from the limitations that Chris pointed out, other limitations to consider include the VLAN limits. According to the Cisco Application Centric Infrastructure Design Guide White Paper:

• The total scale for VRFs, Bridge Domains (BDs), endpoints, and so on is the same whether you are using FEX attached to a leaf or whether you are connecting endpoints directly to a leaf. This means that, when using FEX, the amount of hardware resources that the leaf provides is divided among more ports than just the leaf ports.
• The total number of VLANs that can be used on each FEX port is limited by the maximum number of P,V pairs that are available per leaf for host-facing ports on FEX. As of this writing, this number is ~10,000 per leaf, which means that, with 100 FEX ports, you can have a maximum of 100 VLANs configured on each FEX port.
• At the time of this writing, the maximum number of encapsulations per FEX port is 20, which means that the maximum number of EPGs per FEX port is 20.
• At the time of this writing, the maximum number of FEX per leaf is 20.

For me the VLAN limit per FEX HIF was a concern, and since it is somewhat complicated to track on a per FEX HIF basis, to make it easy for the support team, it was far simpler to conceptually limit the number of VLANs to FEXs at 20. That way, you're unlikely to go over the 20 VLANs per FEX HIF limit. With that in mind, the multi-tier leaf solution is appealing as the FEX-related VLAN scaling limits don't apply. The other advantage of the multi-tier leaf solution has over the FEX-attached solution is that you can vPC the uplink of the Tier-2 leaf into two Tier-1 leaves. From an operational support perspective that assists change control and outage impact enormously - a singly attached host will not be affected if one of the two Tier-1 leaves is down.

Regards

Jason