how to indenting cause of slow network performances that varies greatly

[nkosiedread10963]

Greetings,

I have 3 branches running 4500-x collapsed core and 2960 access switches with 1GB access ports and 20GB ether channel uplinks.

The network performances is significantly slower in the morning with speeds ranges of 0Mbps to max of 7Mbps and later on during the day performance spikes  to 35Mbps, 50-70Mbps.

My challenge is that the network is unusable daily in the morning later on speeds are great.

They seems to be no signs of over utilization/error of the access and trunk ports through show controllers utilization and show interfaces. Load on the access and trunk is around 1/255 and 9 and 13/255 on the 1GB core breakout.

Trace to the core varies significantly as well from 4ms to 8ms and hitting 27ms and 38ms.

Unfortunately due to some firewall or something  I can not trace or ping outside. The branches are connected together in a 1GB ring and I suspect all are sharing a single main internet breakout.

I run intravlan and extravlan test it varied from hundreds Mbits to max average of 400Mbits when performance will be  good. I captured the load test and internet access during slow times as well but I'm to research how to analyze them.

May be my convenience sake will appreciate a helping had with analyzing the captures if there are helpful.

A good fellow I trust hinted on checking buffer bloats and routing issues but hasn't had much lucky in executing the process well, will also appreciate if I could get help on the how to's and tools to use.

I welcome all willing to assist to identify the cause of the problem and how to fix it.

Thanks to every readers too.

[Seb Rupik]

Hi there,

It would be useful to see a topology diagram.

You mention network throughput, but have not said where the source and destination are. Likewise you latency figures, it would be good to know the source and destination.

If you could annotate your diagram with this detail it would be most helpful.

First question, do you have an network monitoring available across your network which alerts on interface thresholds? It would be be useful to see these alerts in chronological order to determine an area to focus on in your network.

cheers,

Seb.

[nkosiedread10963]

Hi Sep,

Thank you so much for your generous contribution.

Topology is for all three branches is 2960 ==> 4500-x  with 2 x10GB uplinks bundled together.

There is no monitoring tool in place, will have to implement one, which one would you recommend that have a less steeper learning cave.

Currently I'm focused on the options that can help pinpoint the cause at the soonest possible time.

Thanks

Nkosana

[Seb Rupik]

No monitoring? ouch! Any opensource product will do for now, I personally use https://www.cacti.net/  .

It would still be good to have an annotated diagram of your network, highlighting where you recorded those performance metrics.

cheers,

Seb.

[Joseph W. Doherty]

It's not 100% clear.  Is the slowness with traffic just within the local LAN, or when traffic goes in/out or on/off the local LAN?  If the latter, you don't have any stats for the "WAN" either?

Any network caching appliances with this infrastructure?

[nkosiedread10963]

Hi Joseph,

Thank you for asking.

The network is slow when traffic is going in and out of the the LAN.

Did some LAN load test and I'm getting a constant good performance.

So far my plan is to capture WAN traffic then analyze it further.

Do you think it will give us all the answers that we are looking for ?

Thanks

Nkosana

[nkosiedread10963]

Hi Joseph.

The internet is the one that is slow.

Checked local WAN breakout and there no signs of over utilization.

I'm suspecting that congestion is happening mostly on the main internet breakout of all the 3 branches and that's where I should be placing my catching appliance.

Correct me if I'm lost.

Thanks

Nkosana

[Joseph W. Doherty]

In the performance issues all seem tied to Internet, yup would be the place to start with analysis.

Since you noted performance is worst in the mornings, it makes me wonder if there is some network application pulling (or pushing) lots of data from (to) the Internet during the morning.