Hello,
The task you are faced with is never schematic with a simple solution. This will be quite a daunting task to pull off.
The topology seems to be okay at first sight - it follows the classic collapsed-core design.
The primary thing you now need is start collecting exact information about the performance of your network. The feeling of your users that the network is "slow" is a symptom that could be valid but you cannot base your troubleshooting merely on a vague feeling of users without any data to support that feeling. If there truly is a bottleneck then it is still uncertain whether it is caused by the network or by end hosts. Therefore, it is important to start collecting operation data.
Personally, I would suggest collecting the following data from each switch and router in your network:
- Utilization of each interface (ideally both packets and bytes)
- Error counters on interfaces
- CPU load
- Memory utilization
These data can be pulled from your devices using SNMP. Visualizing them is a matter of choosing a proper tool. MRTG, RRD, and Cacti are among the tools that are very popular for this purpose. Please note that this may require some experimenting and tweaking to first find out the proper SNMP OIDs of all these counters, and then configuring MRTG/RRD/Cacti to properly fetch them from devices and draw them in a graph.
There are also commands that allow you to see the current utilization of ports, CPUs etc. but if the users complain only about transient performance issues then these commands are of little use unless you are at the console of the switch right in the moment of a performance problem, as they always display the current load and do not generally store historical information:
show controllers utilization
show interfaces
show interfaces accounting
show interfaces counters
show interfaces counters errors
show interfaces summary
show processes cpu sorted
show memory free totals
After these data start being collected, you then need to look for any anomalies. For example, error counters on interfaces should not increase at all. Interface loads should not be continuously close to the maximum capacity of the interface. The CPU load on switches shall generally be low, under, say, 30-50%. Vice versa, whenever a user reports a performance issue, have a look at the collected operation data if there is any clue hiting toward a possible cause.
As I said, troubleshooting a performance problem is generally one of the harder tasks of a network administrator, and it often involves gathering lot of information before even being able to formulate a hypothesis. You are now in need of that information so as a first step, I suggest preparing a monitoring infrastructure that will provide you with up-todate and accurate network health information 24 hours a day.
Best regards,
Peter
