This is very hard to answer because it varies many factors and requires more testings.
Let me put my way of diagnosis.
1. If you have any Monitoring system in Place - On ISP Port what level bandwidth it uses at that time of the complaint . (if it is more than 85% yes the user complain is valid).
2. if the Bsandwidh not as mentioned above - this very tricky here - what site or download site user was complained (since it is the far end we do not know their infrastructure) - so it is hard to say.
3. Can the user download any other download what the experience the same all sites?
4. what level of FW traffic you having here ( or is this reached peak of FW capacity )?
5. Do you have any system in place that you can test directly user download bypassing FW (what is the results)?
Last and Least test you can do is when you have off-peak business hours, can the user can test the same download and confirm is this better?
I hope these steps help you to understand the problem?