core switch recommendations (upgrade from 3750 stack acting as core)

[keithsauer507]

Hi all,

we currently have a stack of 3750v2's and 3750x's acting as our core.  There are also a lot of end users plugged into this stack as well since the way the home run Ethernet distance is in our building.  We do have two fibers off here, one to a third floor (due to distance) where it has its own separately managed 3750v2/x stack, and one to a roof antenna that points to another building that has its own 3750x stack.

Then for long haul the core is connected to a 2901 router where it has 3 providers (two wireline, one 4G LTE backup).  This connects us with 4 other offices within a 20 mile radius.

So the issue with the core is that I know it consists of a few 3750v2's in it, which are now getting up there in age.  Though things work fine, its that time of the year to come up with a budgetary wishlist.  We long dreamt of having a redundant core... one where you can update IOS image on it but the network stays operational. Or where if the core goes down, there is a second core that picks up the load.  Heck our firewalls work this way.  We have two ISP's, two load balancers, two firewalls... but what good is all of that redundancy if the core switch goes down?  It all means nothing really.

Whats a good successor that can be redundant?  I though the Nexus 7004 looked nice and we could split things up more with it being the "core" and everything else turning into a true "access" layer.  However this is very pricey and seems complex.  Is there other alternatives for a 160 employee organization?  I guess I'm looking at reusing as much as possible the existing 3750's but possibly putting modules in them and linking them with 10gig connections to a real redundant core switch that would assume all vlans routing and gateway IP's.  I guess each smaller 3750x stack would need a 10gig link to core 1 and a 10 gig link to core 2, and to prevent flapping or network loops, we would want to make sure there is some sort of HSRP or likewise similar setup between the cores.  Any help would be appreciated.  

[Ganesh Hariharan]

Hi,

Do you have architecture diagram to comment more on efficient way.

A better design is to have high availability with fast performance , cisco 3750 switches has come to life but are still serving good. World has entered into nexus arena for high end security with iOS level as well.

But for organisation with 160 employee and based on requirement of traffic flow and load. we can check of nexus 5k as well.

post the architecture for more discussion.

--GI

[keithsauer507]

Here is a diagram.  I hope it isn't too messy.

See looking at the current core in this image, at the least - breaking it up.

Issue is what if something happens to this current core we have today.  Either we want to update IOS, or there is another issue.  It takes everything down really.

Need to draft up something a little more redundant.  Like maybe 4500-X with VSS as the core, if you think thats a good fit.  Then everything kind of aggrigates to there.  That would also mean investing in some 10gig SFP+ (or 1gig SFP depending on the other device) modules for them, and archetecting it in a way there are redundant connections if one 4500-X goes down.

Also thought of the Nexus line but think it may be overkill.  6500 and 6900 series may be a little overkill for what we are pushing as well.

About 160 employees total, with the majority connected to that core / third floor and the other building you see in the top left of the image via Lightpointe free space optics.

EDIT:  You may have noticed switch 4 in the stack is missing.  We removed it 1 year, 45 weeks, 2 days, 20 hours ago.  Just haven't rebooted the stack since.  It needed to be removed because with port-security on, it was suggested that 9 switches in the stack was just killing it.  Found this during a failed upgrade to the 15.2 train and had to revert each switch one at a time back to 12.2(55)SE8.  Again that was, according to uptime almost 2 years ago.

The connectivity with the lightning bolts to other branch offices or the internet is brought in by either a local cable company or telco, depending on the other offices location.  The home office has both a fiber feed from a telco and cable company.  We are using routers at the branch office because in our experience it was simpler to do the QoS config for the VOIP vs. traditional network traffic, and the IP-SLA for 4G LTE failover.  Plus we have netflow support at each office to feed whatsupgold.

If you think its a good idea, I can engage a Cisco sales engineer and pretty much copy this diagram and discussion, but I'd love to hear other experts or end users options as well.  Theres so many different ways to skin this, and we must balance cost as well.  If I were a multi-building college / campus I surely would consider 6500/6900 or better yet the Nexus line.  I see applications for the big iron, but not sure if that cost proves benefit to our little organization.

[Ganesh Hariharan]

Hi, Thanks for the architecture diagram.

As i understand by seeing this diagram , stacking is done with full redundancy at core level with cisco 3750 switches.

But just give a thought that these switches are EOL by cisco.

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/eos-eol-notice-c51-730227.html

Important Point:-

1) How many users are connected with this core ??

2) Is this core doing dynamic or static routing ?

Based on the above points we can think of cisco modular switches or nexus we can opt.

and what is current support contract from cisco for these cisco 3750 switches.

Hope that Helps

-GI

Rate if that Helpful

[keithsauer507]

Ok I did a sh interface summary and the ones connected (*) I count 236 interfaces up.

16 VLAN interfaces, 15 up.

5 / 5 Port-channels up.

2.  Static routing at this time.

We renew Cisco Smartnet yearly.  I believe this stack has good smartnet coverage until Feb 2016.  We are on a calendar year budget, so assuming we decide its time to re-asses this and get budget money in place, we could have a new solution in place for Q1 next year.  The last 3 switches are the 3750X.  They were added in Q4 2013 when we switched over to VoIP phone system.  The reason they were added was because they are PoE.  All branch offices have these for their phones as well.

[Jason Dance]

Hi Keith.

As Ganesh mentioned, the 3750s are almost EOL.  A few questions:

Have you considered separating your current core and access layer onto different devices? Eg one set of multilayer switches for your core/servers and reusing the 3750s for workstations/phones until you can get budget to replace them for a new access layer (eg 3560s with the IP Base feature set)?

Do you have monitoring in place to measure the amount of bandwidth in use on your critical ports? eg links to your other access layer switches, firewalls etc, and traffic between vlans. This is so you can size your links connecting your workstations, phones, servers and other elements like internet etc.

Have you considered the Nexus 3000 series in your investigations?  We use a pair of 3048s in one of our datacenters, and they're configured with HSRP for routing resiliency and vPC to allow etherchannel uplink resiliency from each access layer switch up to each of the two 3048s.  Its a fairly capable multilayer switch, has 4 SFP+ for 10Gbps connectivity, and their price point is not that much more than you would expect to pay for a fully loaded 3560 (or 3750 at the height of their life).  They do come with NXOS which will take some getting used to (especially if you're an IOS hack like me :) ).

Here's a primer on vPC: http://www.cisco.com/c/en/us/products/collateral/switches/nexus-5000-series-switches/configuration_guide_c07-543563.html

And here's the models available in the 3000 series: http://www.cisco.com/c/en/us/products/switches/nexus-3000-series-switches/models-comparison.html#~Nexus3000

HTH.

Jason

PS: could this be a double post with this one?:https://supportforums.cisco.com/discussion/12613011/could-pair-3850-make-good-redundant-core-small-business

[keithsauer507]

Jason,

Yes we thought about splitting the current stack up, or using the existing stack but adding a new core and moving all IP's, vlan interfaces, vtp server, routing etc... to a new core (or pair of cores in a redundant configuration).

Yeah we know IOS, so NXOS (or any other vendor really) would be a learning curve, but its not like it has to go into production the day it arrives.  It can sit on a bench where we play with it for a few months learning all the bells and whistles first.

One thing with the 3750's is it does not seem to support netflow.  I think the core switch should be able to send netflow information to a collector where network performance and flow monitoring can be achieved.  We do that now but with our routers and asa firewalls, so we can collect flows for anything coming and going on our network, but not anything internally.

We do graph all ports bandwidth metrics via SNMP to CACTI and we are well within the tolerance of port usage.  

2gbps lacp link to our vm rack 3560x switch only maxed at 78.55 MBps.  

2gbps lacp link to our other vm rack 3750g switch only maxed at 30.91 MBps.

4gbps port channel to Cisco 5508 wireless lan controller and its maximum was 41.65 MBps.  

2gbps link to our third floor switch stack maxed at 35.14 MBps.

Our maximum to the other building over 1gbps lightpointe free-space optics maxed out at 21.67 MBps usage.

I don't think we are hurting for performance.  Just looking to add redundancy and stay current for future growth.

Oh if I do a count on all occurrences of "switchport voice vlan" I get 101 matches.  So that is 101 ports connected to a phone and pc (end users).  If I search for "printer" I get 18 matches, as long as our port descriptions are current.  I do a search for "server" I get 12 matches, again as long as our port descriptions are current.  If I search for our computer naming scheme prefix, I get 50 matches (some pc's may not have phone pass through).  I know there are 4 ports to a 5508 WLC and 5 ports to lightweight access points on this main stack.

Theres 1gbps uplinks to firewalls and our branch office router.  2gbps links to a switch in the vm rack, 2gbps to another switch in the vm rack.  2gbps to our third floor, and 1gbps to a roof mounted laser unit that connects us with another building about a football fields length away.

I appreciate everyones input and suggestions.  Its all good talking points to bring up with a Cisco sales engineer.  This thread has been a huge help so far.  Any further info is greatly appreciated and welcome for open discussion.

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

"One thing with the 3750's is it does not seem to support netflow."

The 3750X support the C3KX-SM-10G module, which, I recall, supports NetFlow on the module ports.

The 3650 and 3850 support Flexible NetFlow.

"We do graph all ports bandwidth metrics via SNMP to CACTI and we are well within the tolerance of port usage.  

.

.

I don't think we are hurting for performance.  Just looking to add redundancy and stay current for future growth."

I cannot say you're incorrect, but it's possible you might be incorrect, too.

SNMP stats, especially if taken over the typical 5 minute polls, don't really indicate what might be happening at the millisecond level.

Others have suggested VSS and vPC solutions, and those too can be fine solutions, but both of those really work best when there's no need for traffic between the pair.  I.e. You really want redundancy from the core (pair) to every device that connects to them.

Another approach, would be a L3 core, with everything that connects to it also L3.  For that approach, you don't need stackable, VSS or vPC devices, but, again, your attached devices need to support L3.  One advantage of using an L3 pair core (or traditional L2 core pair), you're not relying on proprietary protocols; you should even be able to mix different models or even different vendor devices.

[keithsauer507]

I did find this article, although a few years old it does summarize another opinion on various options... http://majornetwork.net/2012/02/choosing-your-core-switches/

I'm almost leaning towards two 32-port Catalyst 4500X take over core duties, and breaking the current 3750g/v2/x stack into two- the 2nd gen 3750g/v2 in one "access stack" and another stack of just the 3750x poe's.

Just want to make sure its possible to keep it redundant with VSS.  Watched a few videos on it, seems straighforward, I just would love a hands on with it.

[Jason Dance]

Ah, the one drawback to 3048 is that it supports sflow instead of netflow.  You will get statistics with the right monitoring solution, but if you are used to netflow then there is a difference in what information is gathered.

Given the stats you list, I think you'll be fine with what you are looking to do, depending on your poll periods.  I would definitely separate your core from your access layers if you have the budget to do so.

[Ganesh Hariharan]

Ok I did a sh interface summary and the ones connected (*) I count 236 interfaces up.


16 VLAN interfaces, 15 up.

5 / 5 Port-channels up.


2.  Static routing at this time.

We renew Cisco Smartnet yearly.  I believe this stack has good smartnet coverage until Feb 2016.  We are on a calendar year budget, so assuming we decide its time to re-asses this and get budget money in place, we could have a new solution in place for Q1 next year.  The last 3 switches are the 3750X.  They were added in Q4 2013 when we switched over to VoIP phone system.  The reason they were added was because they are PoE.  All branch offices have these for their phones as well.

Hi ,

We need to know exact amount of user ports , so that the design consideration can be made with 5 % growth for future. SVI calculation is for logical separation of your network.

As Jason pointed, Consider a design with collapsed core architecture having nexus switches or 4500 or 6500 MLS based on budget and requirement on technology.

-GI

[keithsauer507]

So you want to know the difference between what ports are phone-pc pass through, (should be easy enough doing a count for switchport voice vlan), and then what ports are printers or other peripherals, vs. what ports are servers, uplinks, trunks, etc?

[Ganesh Hariharan]

So you want to know the difference between what ports are phone-pc pass through, (should be easy enough doing a count for switchport voice vlan), and then what ports are printers or other peripherals, vs. what ports are servers, uplinks, trunks, etc?

Hi,

If we have tentative port usage calculation, then we can size the access layer accordingly along with HA core architecture.Like you can finalise a switch mode with 48 ports for usage considering buffer of 5 %.

So that was the main reason for query.

Hope that Helps..

-GI

[Joseph W. Doherty]

Disclaimer

The Author of this posting offers the information contained within this posting without consideration and with the reader's understanding that there's no implied or expressed suitability or fitness for any purpose. Information provided is for informational purposes only and should not be construed as rendering professional advice of any kind. Usage of this posting's information is solely at reader's own risk.

Liability Disclaimer

In no event shall Author be liable for any damages whatsoever (including, without limitation, damages for loss of use, data or profit) arising out of the use or inability to use the posting's information even if Author has been advised of the possibility of such damage.

Posting

If you have a stack, you should already have redundancy.  Do you believe there is some issue with stack redundancy?

The Cisco follow-on for the 3750 series, would be the 3650s or the 3850s. Your lease expensive option might be to build a dual stack of either of those as your core, and retain the 3750Xs for access edge devices, ideally also still stacked and with MEC between them and the new core.

What I recommend to avoid, if possible, is stacking StackWise with StackWisePlus 3750s, as the former in the stack negates most of the latter's improved Stack features.