Network Design Leaf LAN switches and Core Switches

[julito4589]

Hi,

This may be a bit of a noob question, but bear with me.

I have 10 24-port 2950 switches for LAN users. All these switches connect to a WS-C3560G-48TS-S via Cat5e. There are other devices connected to the 3560 (e.g. servers that provide resources to LAN users, plus other supporting devices). The 3560 is in turn connected to a Cisco 3825 router which performs all the VLAN routing and gateway functions.

I know that eventually I can replace the 3560 with a layer 3 switch that can do the VLAN routing so requests between VLAN don't have to go to the 3825. But I'm wondering if I'm likely to run into congestion issues by having one switch handle all the traffic from the 2950's as well as the traffic from the servers? In fact, I'm wondering if I'm having that issue now.

If my math is correct, each 2950 can potentially generated 48Gbps ([1Gb Up + 1Gb Down] x 24 ports). This is the max switching capacity in those switches. For 10 2950's, the total potential traffic is 480Gbps. The switching capacity of the 3560 is about 100Gbps. I know that considering the max capacity of the 2950's may not necessarily be practical because it's unlikely that all ports in all switches will pass full traffic. But I want to run this by others and find out how you deal with similar situations.

[Kevin Dorrell]

It is quite normal to have uplinks that have a bandwidth less than the total of the access links.  It is known as over-subscription.  As you observe, it is extremely unlikely that all the LAN users will be generating full traffic all at the same time. If there is contention for the uplink all switches will have a certain amount of buffering to smooth out the data flow.

Having said that, my experience is that the 2950 and 2960 are not the best for amount of buffer.  I do sometimes have output drops on the uplinks, but that is when I have several 1 Gbps servers on the same switch trying to do a backup all at the same time.  The TCP can handle the dropped packets, and recovers gracefully.

I would suggest the best thing to do is get an SNMP-based monitoring tool like PRTG or MRTG that will graph the traffic on each of your links.  That will give you a good idea whether you really are seeing any bottlenecks.  My guess is that your biggest bottleneck ( ... or should that be the smallest? ...) is the router-on-a-stick link, and you would be much better off routing the VLANs in the layer 3 switch.

Be aware that your 3560 is end-of-life:

http://www.cisco.com/c/en/us/products/collateral/switches/catalyst-3750-series-switches/eol_c51-696372.html

Hope this helps

Kevin Dorrell

Luxembourg

[julito4589]

Kevin,

Thank you for the tip on PRTG. I have it running on the network and monitoring this device. Hadn't thought of checking the traffic on those interfaces.

Based on a quick look through all the interfaces, it looks like the top one right now is passing (real time) about 600Mbps. All other interfaces are passing a lot less than that. The 2 day and 30 day averages looks good. So this at least gives me some relief that this switch is not getting pounded. Now I'm going to look at the interface that connects to the 3825.

As for the eol for the 3560, I'll be replacing it with a WS-3560X-48T-S, which also has some of the basic L3 features I want to use to route VLANs in this switch.

[Kevin Dorrell]

I find PRTG really useful in these cases.  I use it in my network - I have more than 3000 traffic sensors, each polled at an interval of 60 seconds.  The graphs give a great visual on the correlation between the traffic on the various interfaces.  Also, monitor the CPU, but in fact I find it is very rare for a switch to be CPU-bound unless there is something really wrong with the network.  The CPU scheduling algorithms are usually pretty good.

One tip: If you have a link that might be oversubscribed, for example your router-on-a-stick link, or your switch uplinks, monitor the link at both ends and include the discards in/out channel in the sensors.  Simply, if you see any DiscardsOut at either end, then the link is congested; if you don't, then the buffering is doing its job.  I set a warning thresholds of 1#/sec on the DiscardsOut counter so that I get an instant warning when there is congestion.  It also gives me good evidence if I am trying to persuade my boss to invest in an upgrade.

Good luck, and let us know how it works out.

Kevin Dorrell
Luxembourg

[educruz]

Hi, 

To discard whether there is a capacity issue now, you would need to check whether the current throughput between the 3560 switch trunk link and the 3825 router is stable. Preferably, check this during peak business hours.

See whether the link already presents errors and check the current and historical switch CPU usage. 
(show process cpu | ex 0.00% or show process cpu history)

Switch#show interface (trunk link to the router)
...output omitted...
     reliability 255/255, txload 1/255, rxload 1/255  <<<< Low values/255 are good. 
...output omitted...
  5 minute input rate 0 bits/sec, 0 packets/sec    <<<< Throughput
  5 minute output rate 0 bits/sec, 0 packets/sec   <<<< Throughput
     956 packets input, 193351 bytes, 0 no buffer
     Received 956 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
     0 watchdog, 0 multicast, 0 pause input
     0 input packets with dribble condition detected
     2357 packets output, 263570 bytes, 0 underruns
     0 output errors, 0 collisions, 10 interface resets
     0 babbles, 0 late collision, 0 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out

For scalability options, distribution switches such as 3700 or 3800 series or core (4000 series) would be considerable, depending on the needs of your business.

Hope this helps.

- Ed

[julito4589]

Thanks for the input.

Reliability figures are 2/255.

5 min input and output rates on that interface are at ~20Mbps and ~10Mbps, respectively. I'll be monitoring it some more, but at least for now I'm a little more reassured that it doesn't look like this link is getting maxed out.

Could you expand a little more on your comment about 3700 and 3800 series switches?

Even with the current traffic volumes, I'm considering the option of splitting the LAN users and the servers (and related supporting services) into two separate switches. In other words, the servers would stay on the 3560X and the LAN users would be on their own core switch (perhaps another 3560X or a 3700 series?). These two switches would of course be connected (with Cat6 or higher capacity medium).

There are about 200 LAN users and about 25 servers behind these switches. I'm not sure if two core switches would be overkill. Do you have an opinion on this?

[educruz · Accepted solution]

My apologies as I assumed there was no monitoring system in place.

Taking some feedback from Kevin, PRTG should show how maxed out the trunk link between the 3560 and the router is.

Based on the above result plus the switch and router CPU usage history (in which also PRTG can help with a report), I see two ways:

1) If the current switch is OK in terms of CPU and throughput on its trunk link to the router, an only 3650x switch as a replacement would suffice for inter-VLAN functions.

2) If the results show a possible capacity issue with the CPU or throughput, two 3560x switches for separate traffic would be optimal. The 3825 router has two fixed GE ports. 

My comment about 3700 and 3800 series switches came before I knew that you considered a 3560x, but the latter model should cope with the needs pretty well here.

I agree with you. No need of a fancier router unless there is an intention for the business to grow considerably.

[julito4589]

Really appreciate the input.