I asked them and the answer was what’s the point of running speed test at off-peak hours?!
It often removes concurrent congestion issues. If fact, ideally it should be done excluding all other traffic.
It helps to narrow the scope to other potential issues, like the hardware/firmware issue I mentioned in my prior reply, or misconfiguration errors.
In some ways, performance testing might be approached like BERT. Such testing should be more comprehensive than a simple Internet speed test. This not only to identify an issue, but specific tests can provide clues to likely causes.
For example, I would often start my testing using UDP at the CIR. Does all the traffic get through? Then I up the transmission rate by 1, 10 and 50%. Do I still obtain the CIR rate?
Then I get into TCP testing, but its performance is complex based on TCP "flavor", it's parameter settings and transit device buffer settings. For example, a too shallow or too deep buffer, on a transit device will reduce TCP performance. (Poorly configured shapers or [especially] policers are common culprits, so may be RED or even default interface buffer settings.)
BTW, the afore mentioned considerations may be more impactful with plentiful concurrent traffic.
To recap, especially nowadays, it's not difficult to saturate 10 Mbps, so with concurrent traffic, as you suspect, that alone could provide poor speed test results. However, because that is so common, it can easily mask other issues. So, ideally, you want to identify whether what the customer sees is truly expected performance, or there's something else. Testing during non peak usage hours is helpful to make this determination.
BTW, in the case I mentioned previously, the SP even went to my manager, asking could you get this engineer to stop saying there's an issue. My manager said, if Joe thinks there's an issue, good chance he's correct.
What I saw, was an over all drop rate about 1% more than I expected. As far as the SP was concerned, they didn't see how I could be that exact, so my calculations were in error, not the circuit.
Well, with my constant proding, when they finally found a transit interface without the latest firmware, they discovered the version they were running would drop slightly more packets than it should when its load exceeded 97%. After they updated the firmware, the overall drop rate became what I expected.
(Also, BTW, if you're wondering how I noticed the issue, it was quite simple. I was shaping to conform to CIR, but end-to-end, I was seeing drops in excess of my shaper's drops. But, if you conform to CIR, SP shouldn't be dropping packets, and so they do claimed.)
Anyway, the moral of my tale, although customers/users often have unrealistic expectations concerning network performance, they're not always mistaken. ; )
