lamav wrote:
I have been reading this document on the Nexus 7000 and Catalyst 6500 architectures and designs in a server farm...
http://www.cisco.com/en/US/prod/collateral/switches/ps9441/ps9402/ps9512/White_Paper_C17-449427.pdf
(focus on pages 19 to the end).
I am pretty surprised by what I am reading.
I have never configured the Nexus 7000, nor have I deployed any sort of Virtual Switching, but the new design approaches described in this document seem to go completely against the grain of what Cisco has been peddling for the last 5 years or so.
When it comes to the Cisco hierarchical model, I have read many Cisco documents touting the benefits of minimizing and mitigating the L2 domain, maximizing L3 isolation, migrating the access layer to a routed one, leveraging route summarization, stub routing, ECMP, deploying services in the aggregation layer, leaving the core to do nothing but high speed L3 switching with all the route policies and filtering left to the aggregation layer...etc.
Now, in this document, there are recommendations to move the L2/L3 boundry all the way up to the core! Switching in the core? STP in the core? There is talk in the document of the need to sometimes span a VLAN across the entire data center and doing so by migrating the L2 domain to the core. Confusingly enough, it then goes on to mention the drawbacks of doing that (huge fault domains), and as a "solution," recommends creating a separate core layer for each and ever distribution block!
So, the traditional 2-layered distribution block, with L3 uplinks to a core shared across the entire data center is now a 3-layered model, with a separate core layer for each distribution block. The reasoning for this is to span the VLAN across a "zone," which is nothing more than the traditional distribution block. So why couldnt I just span the VLAN using the 2-layered distribution block, as always...and use a shared core?
This document is confusing and reads a lot more like sales and marketing than engineering....
Would love to hear some thoughts.
Hi Victor
Another one of those L2/L3 design questions eh .
I guess the first thing to be said is that designs do change when new technologies become available. And the key thing about this document is really STP and the limitations STP introduces ie. i can remember years ago a Nortel salesman telling me that they did not rely on STP at L2 and hence you had more flexibility in design.
Cisco relied on STP and so their advice was simply if you want to get rid of STP then go to L3. They recognised the limitations and together with the advent of L3 wire speed switching they pushed designs that mitigated STP ie. L3 in the core, L3 pushed to access-layer if possible. But even if you had a L2 access-layer they still recommended L3 in the core.
But in DC environments a L3 core worked well until your aggregation layer could no longer cope and then things got a bit messy. The problems with L3 in the core are -
1) what happens if you have 3 aggregation blocks L3 connected to a routed core but then you suddenly needed a vlan in one of the aggregation blocks to be available in one of the other aggregation blocks.
2) Scalablilty and capacity ie. for L2 adjacency to the servers you needed to deploy service modules into the aggregation layer but if you needed a new aggregation layer you then needed a new set of service modules and you had the extra overhead of keeping consistent rule bases etc. across service modules
Moving L2 to the core in the DC actually makes a lot of sense and gives the designer increased flexibility. And what makes this possible is VSS & virtual portchannels because these new technologies relegate STP to simply a backup failsafe ie. they create their own L2 loop free topology. Suddenly not having to rely primarily on STP to form a L2 loop free topology means you have removed one of the most limiting factors.
If you can move L2 to the core then your service modules can actually sit in the core and apply to all zones as they are called in this paper. You can now have centralised services not dispersed services per aggregation block. You also now have the ability to deploy a server into any vlan in any switch block.
In addition you can also overcome the one aggregation pair per zone. There is nothing in this document that says you can't have multiple aggregation pairs per zone, indeed the diagrams do show that, and because you are connecting via L2 to the core you can span the same vlan across multiple aggregtion pair per zone. So a zone is not a typical "distribution block" where you have a pair of aggregation layer switches connected via L3 to the core. This can be a big gain. I personally have come across this limitation in DCs where there is a L3 routed core but you need the same vlan on multiple aggregation pairs - not possible.
Where the doc is a little unclear is when they talk about multiple cores and show the diagram for that. It looks from the diagram as though the cores are independant ie not interconnected which could negate some of the benefits above. But that aside, because of VSS, virtual portchannels extending L2 back to the core now becomes a reasonable design choice and the additional flexibility it gives is most welcome.
Jon
