best practice question on # of access switches per distribution switch

[Alexander Beirlant]

In our biggest building we have 44 * 3560E-48 switches (avg 90% port utilization) connecting to a pair of 5548's (no vPC set up), on top of that we have several other buildings connecting to these 5548's (in total traffic of another 43 access switches connected to a set of 4900M's and then to the 5548's). 

The 5548's do L3 switching, and are then connected to our 7k's.

Currently these 5548's do not seem to have problems handling the actual traffic, but management is planning on adding another building (which would connect to these 5548's) servicing another 200-400 ports. We daisy chain our 3560E's in access closets where there is more than 1 switch.

My question is how many 48 port access switches should a L3 distribution switch service according to best practices.

[Shaunak]

Hi Alexander,

As per the campus design model it is recommended to have a 20:1 over-subscription between the access and the distribution layers and a 4:1 over-subscription between the distribution and the core layers.

Of course these numbers may vary in your network depending on a lot of factors like time sensitive data etc. The above mentioned ratios might be a good starting point for bandwidth and link evaluations in the network or any upgrades in the network.

Hope this helps.

Thanks,

Shaunak

[Alexander Beirlant]

Thanks for the information. What other factors should I look at? I know the internal switching maximum is an important factor, but with Layer 3 switching I am not sure what else comes into play.

[Shaunak · Accepted solution]

Alexander,

You're welcome.

Nowadays the multi-layer switches portfolio usually forwards data with the help of hardware based ASICs. If you're using layer 3 SVI's on the switch the inter VLAN routing will be handled by the backplane of the device and should not be a major concern.

However be careful about any inline features that you might implement as some of them might slow down the throughput of the device like ACLs or deep packet inspection etc. The hardware data sheet should be a good place to start and to evaluate the device's performance and feature set against the kind of services and throughput you require in your network since in some models ACLs or QOS can be offloaded to the hardware ASICs as well hence despite of implementing these features data might still be forwarded at wire speed.

You should also keep the growth plan of your network and enterprise in hindsight while planning this and try to use the hardware or technologies which can future proof your network for at least 3 plus years or so.

Hope this helps.

Thanks,

Shaunak

[Alexander Beirlant]

Shaunak,

Sorry for the late reply. We would have a number of ACL's on these devices, and QoS will be mainly respecting the existing tags. We will also not use any deep packet inspection on the distribution switches. 

Thank you again for the great information. 

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

I'm unaware of a "best practice" for numbers of downstream switches.  There many variables involved, some being the capacity of the actual devices involved and their typical loading.  Some dealing with design, and for that, there are best practices.

You mention 90% average port utilization.  That's busy.  Busy enough to be concerned about transient congestion.  Generally, once you push about 2/3 utilization, queuing may become an issue.

Shaunak, mentions the old design recommendations for access and distribution over-subscription ratios.  I've found those ratios seem to work better with old equipment, i.e. 10 Mbps users, and 100 Mbps infrastructure and servers.  Once you get into gig and 10g (or better), many hosts cannot push/pull that much traffic (except for some very busy hosts, e.g. SANs, large/busy VM servers, etc.)

[Alexander Beirlant]

95% of our end-user connections would be 100 Mbps lines due to our Cisco phones, but the regular load is nowhere near reaching that except for some of our developers and DBA's. What would you say is a a more modern design recommendation for access and distribution over-subscription ratios?

Most busy hosts are segmented to 1 physical location.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages wha2tsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Again, I'm unsure there's any really effective general rule for over subscription.  As your post mentions, developer or DBA user hosts tend to be much busier then "ordinary" user hosts.  So, what might be a fine over subscription ratio for "ordinary" users, likely wouldn't be for developers or DBAs.

You also mention Cisco phones.  VoIP's service level network requirements are much more restrictive than much other traffic.  You generally never want to over subscribe VoIP, but you can oversubscribe links that VoIP shares with other traffic if you have QoS that prioritizes VoIP.  So, depending on the traffic mix, on a link, and platform QoS features, optimal over subscription ratios would be impacted.

Today, you might see VDI traffic, which also shouldn't be oversubscribed.  Even something like email, ideal over subscription will be impacted by how an end user client interacts with an email server.  Again, over subscription ratios would be impacted by the network service needs of your traffic.  I.e. background bulk data transfer needs is quite different from VoIP needs.