Switch inter-connectivity question

[Sam Sanders]

Good day to all.  We have a small LAN with less than 10 switches total.  Including all the network printers and PCs and servers there are about 70 nodes in this building.  In the data room, there are (6) 24-port Catalyst switches.  I am trying to determine the best way to inter-connect them.  2 of the switches are c3750x and are stacked.  2 are C3560X and cannot be stacked, and the other 2 are C2960G and cannot be stacked.  Also, there is a C2960G in another building connected to one of these 6 via a 1G uplink.  I wish I had gotten more of the switches with stack capability, but at the time I was not aware of the advantages of stacking.

So... The 2 C3560X switches each have a 10-G network module installed.  One of the 2 3750X's (the stack master, call it Switch-A), in the stack has a 10G module as well.  In a perfect world, I would take the C3560X (the one that is layer-3 with routing enabled and also the default gateway), and connect all the other switches to that one, and use both the 10-gigabit uplinks and then use 1-G uplinks for the rest.  But if I use the 10-G ports, then I cannot use the other 2 1G SFP ports in the network module.  So I am limited to 2 10-G uplinks per C3560 switch.  I can't connect all the other switches to this one unless I used some of the standard (non-SFP) ports.

So here is the way I have it set up right now: there is a 1-G fiber uplink from another building and the fiber is FDDI so it is not possible to use a 10-G uplink between the buildings; they are too far apart for FDDI.  We might run some OM3 fiber later, but for now, the 1-G uplink is the best we can do.  So the 2 buildings are connected using a gigabit fiber uplink with SFP ports, and the fiber coming from the other building is connected to the C3560 that is not the default gateway.  The default gateway switch, call it switch-1, is using both the 10-G uplinks and has one 10-G port connected to the master in the 2-switch stack, (one of the C3750X's, call it Switch-A).  The other 10-G uplink goes to the other C3560X, call it Switch-2 (Switch-2 is the one with the fiber uplink from the other building).  Switch B is the stack slave and is only connected via the stack cables.  The other 2 switches are C2960G-24TC-L and one is connected to Switch-A with a 1-G copper SFP uplink, call it Switch-3, and Switch3 is connected to the other C2960G with a 1-G copper uplink.  I have attached a basic diagram.

So... my question is, without spending more money, is this the best way to connect them?  I could use more of a hub and spoke topology if I did not stick to SFP ports for all the up-linking.  It was my understanding that the SFP ports are the best ones to use for uplinks.  But due to the limitations of the network modules, I could not connect more than 2 SFP uplinks from a given switch, or 3 in some cases where only 1G uplinks were used.

If we did not mind spending a few more thousand (I would love to run OM3 between the buildings and get the speed up to 10G), so if we had another 5 to $10,000 to spend, would it be worth getting a switch full of fast SFP modules and using it to uplink all the other switches (a total of 5 counting the link from the other building and not counting Switch-B since it connects via the stack)?  Everything is working well, so it might be better to leave it alone.  Any helpful comments and/or suggestions are welcomed with a heaping helping of sincere appreciation!

Thanks!

Flux

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

What you might consider is using your dual 3750X stack as the physical core and as the L3 router.  All your other switches (except the off-site switch) would have dual port channel links, one link to each 3750X.  This provides core and L3 redundancy and can provide more inter switch bandwidth.

For you critical single home devices, such as your Internet and off-site switch connections, you can have a logical configuration "mirror" port ready on the second 3750X stack member, so if the member with the active link fails, you only need to repatch.  (This can also be use for other critical single homed servers).

On a 3560X/3750X switches you do want to use the SFP ports for your busy ports (because they have their own reserved hardware buffers).  However, if you use 10g port, as 10g, you lose half your SFP ports.  Without knowing your expected traffic flow patterns, I cannot not say whether using all the SFP ports for dual gig port-channels would be better than having some on copper edge ports and some as dual 10g fiber ports.

[Sam Sanders]

Thanks. I never understood why we lose half of the SFP ports when using 10g.  But your answer is very helpful because of the comment about the SFP ports "having hardware reserve buffers".  I always knew these ports had something more to them than a standard switch port, but did not know exactly what it was.  I really do appreciate the tip!

Thanks again!

Sam

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

Sorry for not being clear, but when I wrote you would lose half your SPF ports, that was assuming you're running in a dual stack member configuration and one stack member fails.  Basically, in such a configuration, losing one stack member (of two), you lose half of all your ports.  (NB: I'm also assuming you're using like stack member models.)

Regarding reserved hardware buffers, Cisco says:

The total available common pool for egress buffers varies from one platform to the other. They are more limited in 2960-S: 2MB for the whole system (downlink ports + uplink ports), while 3750-X has 2MB for each set of 24 downlink ports and 2MB for uplinks.

from: http://www.cisco.com/c/en/us/support/docs/switches/catalyst-3750-series-switches/116102-qanda-egress-00.html

[Sam Sanders]

I am not sure I was using correct terminology. By "we lose half of the SFP ports when using 10g" I meant... these network modules that slide into the opening on the right front of the switch, which each contain 4 SFP ports... when we use 10 gigabit SFP transceivers in the ports, we can only use up to 2 of the 4 ports. I am sure there is a good reason for this. It probably has to do with the total available common pool for egress buffers that you mentioned in your most recent post.  We have 2 switches that are stacked, but unfortunately the other switches are standalone, and I have always used the SFP ports to link these.  I wish I had purchased stackable switches for all of the switches we have purchased in the past 2 years.  But live and learn.  One thing I have thought about would be to get one of these Catalyst 3750X-12S-S switches, in which all of the ports are SFP ports, and using it as the central switch of all the other daisy-chained switches.  Or, replace the 2 Catalyst 3560X switches with stackable 3750X switches and have 4 switches in the stack.  The second option here would probably be best, because in the first option, if the Catalyst 3750X-12S-S were to fail, it would need to be replaced, but if one switch in the stack failed, we could move the links to a different switch. Yes, I think we will just get more stackable switches moving forward.  Thanks for your replies!

-Flux

[Sam Sanders]

Thanks again everyone for the helpful information. My background for almost 20 years was small, boring networks with cheap switches, no VLANs, no layer 3 switches.  The past 2 years has been much more fun as I have finally been introduced to some routing and VLANs.  It is like the first semester of calculus after taking boring algebra classes, it is starting to get interesting, but there is still so much I need to learn. My latest challenge is getting refurbished switches with different versions of software license. Fun stuff.